Language Selection

English French German Italian Portuguese Spanish

Debian

Syndicate content
Planet Debian - https://planet.debian.org/
Updated: 11 hours 29 min ago

Shirish Agarwal: India doesn’t need women or doctors

Monday 30th of September 2019 12:38:56 AM

This is again going to be a long one hence I want to start by sharing some positive news first. Few days back, a vlogger Dhruv Rathee made a vlog review about Jatayu nature and park open in Kerala.

Now, why is it important and good. While it is a profitable initiative, it has been made by private money at the cost of INR 100 crores. It has been built which was a degraded barren land surrounded by forest. While one could argue that even such lands should not be disturbed and one perhaps might be right about that, the research I found seem to be inconclusive. I was looking at one study sometime back in which two adjacent plots of land were taken, both degraded, barren patches of land . One land was left alone while the other had some sort of stable, with animals in it, horses, pigs, donkey etc. and they left the land pretty much besides coming at intervals to see if the animal feed was good enough of them, veteniary medical checkups etc. At the end of couple of years, they checked the micro-nutrients of the soil to see which had more mico-nutrients. It was found out that the ones which had the animals was more fertile and had slightlly more/better ecosystem than the one which was left. IIRC, they published the result in some magazine like ‘Nature’ or some such peer-reviewed publication and other scientists were apply to replicate the results with varying degrees of success. While I remember the simplified version I am sure it is far more complex than I have described. One of the best things they have shared in the review, that the land has been leased from the State Govt. for a period of 30 years after which it will given back to the State of Kerala.

FWIW, Jatayu is the name of a mythological bird taken from Ramayana. Instead of wasting 3000 crores of taxpayer’s money for one single statue and instead invested in in health, education, safe drinking water, employment generation etc. it would have enriched not just the people benefiting from it, but also made x times productivity growth as it has been proved time and again that any improvement in people’s lives not just makes them better, but also enhances countries growth as well. I have given the number X as right now India has 0 people in its Statistical Commisson as the last two full-time membes resigned couple of years back. The only somewhat factual numbers that are in India are provided by CMIE which is a private institution and obviously doesn’t have neither the funds nor the reach that a Government body can. CMIE does share some interesting facts and figures but that probably is a story for another day. For those who might want to visit Jatayu can visit Jayatu Center website for the same. The image shared above is taken from keralatourism.org website and is copyrighted to them.

No Country for Women

While I have written on this topic a few times before, each time an incident happens and I feel do we really deserve women ? Many a times when a woman (young or old) goes to the police she is asked to present evidence. Now a young woman who was persistently blackmailed, raped by a person of the ruling party, a ‘Swami’ , a nomenclature reserved for a seer who is supposed to be beyond temptation presented 45 videos of the gentleman to the police. She also leaked couple in social media so that the videos don’t disappear into thin air and she and her family doesn’t get killed as was attempted in Unnao rape case. It is only because the facts came in public that the MLA accused in the Unnao rape case got expelled from the ruling partt, In fact, even the killers in Nirbhaya Rape case , even they haven’t been hanged.

Paper Clipping of Chinmayanand Rape Case

The sad part part is that in this case, even after evidence she has been asked to produce two witnesses who would say that she has been raped. I haven’t ever heard a more bizarre story while siding with the seer who has claimed that she was extorting money from him. The list goes just goes on and on, there were 6 women journalists who claimed sexual harassment against MJ Akbar. The case is on-going in the Supreme High Court where it will be heard now after Dusshera holidays. The last hearing was done on 9th September and will start anytime after 20th October when the Supreme Court starts. And there are several more cases, like the Kathua case rape, the Muzaffarpur Shelter home case, the case goes on and on. Sadly, we don’t even have latest stats as there are no statisticians in Indian Govt. and the only report we have is the 2016 NCRB report which does show the trend that there is rising crime in India. It is party to joblessness which is rampant, and partly perhaps of our conservative mindset towards sex, sex-education .

There were two good movies made in India on the subject, one which sank in Bollywood without a trace called Khandaani Shafakhana which more or less only talked about erectile dysfunction and tried to make few jokes about it. There was Vicky Donor which talked about sperm donation which did good business a while back. The movie which touched my heart recently was though was the malyalam movie called Peranmbu starring ever-green star Mammooty. While I don’t speak Malyalam, you may get the movie on netflix or Amazon prime with english subs. While I don’t want to give the whole story of the movie, there is one scene in which Mammooty visits a woman so he can hire a male escort for his daughter and gets slapped. Many people, especially boys didn’t like that scene and said why he had to go there, but as a viewer if you see the movie from a father’s eyes he did what any sane father who loves his child will do. While at the end, they didn’t give any solution to the issue or it got censored, when you see the movie you can imagine the plight of such children’s fathers, relatives etc. It is sad when such movies which make you think aren’t even part of the national discourse then how are people to grow their consciousness, their humanity. When I hear of such incidents as above, I genuinely, does India really need women ? Shouldn’t women coming from other countries to India be given travel advisories stating that they should either have black belt in Karate or some defence techniques and carry a deadly weapon with them all times to defend themselves from us. It seems we, Indian men have no control

Mike Gabriel: Results produced while at "X2Go - The Gathering 2019" at LinuxHotel in Essen a.d.R., Germany

Sunday 29th of September 2019 05:27:49 PM

Over the past weekend I attended "X2Go - The Gathering 2019". This year's venue was LinuxHotel in Essen. It was good to come back here.

Things that I got DONE while at the Gathering X2Go related topics I worked on...
  • Three informal talks about:
    • the new/alternative X2Go Kdrive graphics backend for X2Go
    • status report of my work on the X2Go Plugin for Remmina
    • brain storming session: accessing X2Go sessions from a web browser
  • Get Ubuntu Gnome Desktops (from 18.04 or later) working in X2Go (with X2Go Kdrive backend being used)
  • Hide color manager authentication dialog on session startup of Gnome-based sessions in X2Go by nastily tweaking colord's policy kit rule set
  • Discuss various issues around nx-libs with Ulrich Sibiller and Mihai Moldovan
  • Discuss Free Software and Civil Administration with Heinz-M. Graesing
  • Discuss Free Software solutions for schools with Heinz-M. Graesing
  • Discuss a Thin Client concept developed by Kjetil Fleten at fleten.net and deepen the partnership of our companies
  • Discuss the benefits of using Weblate for translating X2Go components with Juri Grabowski, providing him with a best practice workflow (ToDo -> Juri)
  • Discuss switching X2Go Git from plain SSH + Git bare repositories to Gitolite with Juri Grabowski (ToDo -> Juri)
  • Share various new developments with Nito Martinez and Juan Zea going on at Qindel
  • File a pull request (PR) against screenshoter [1] porting it to GDK v3 and making it build and run on recent Linux systems (we might need this for the X2Go WebUI implementation)
  • Attend the yearly members' meeting of ORCA e.V.
  • Write the protocol for the yearly members' meeting of ORCA e.V.
  • I got elected as 2nd chair into the board of the ORCA e.V.
  • Review 3 partially quite longish pull-requests (PR) by Ulrich Sibiller for nx-libs (all passed through) [2-4] and one by myself [5].
Non-X2Go related topics I worked on...
  • Upload 35 MATE Desktop Environment related packages to Debian unstable
  • Upload Veyon 4.2.5 to Debian unstable
  • Upload FusionDirectory to Debian unstable (+ as-is Argonaut source-only upload)
  • Upload TigerVNC 1.9.0+dfsg-4 to unstable (sponsored upload on behalf of Joachim Falk)
Credits

The main big thanks goes to Stefan Baur who did most of the event organizing work. Well done, again, Stefan. Thanks for making these events possible on a yearly basis. Much much appreciated.

Another big thanks goes to Mirko Glotz for doing video recordings of all talks and discussions during the event.

Thanks so much to everyone who attended the meeting. We have been 15 people this year! Awesome!

light+love
Mike

References

Ben Hutchings: Linux Plumbers Conference 2019, part 3

Sunday 29th of September 2019 04:12:46 PM

Here's the last chunk of notes I took at Linux Plumbers Conference earlier this month. See part 1 and part 2 if you missed them.

Real-time track

Etherpad: https://etherpad.net/p/LPC2019_Real_Time/timeslider#4945

Core scheduling for RT

Speaker: Peter Zijlstra

Details: https://linuxplumbersconf.org/event/4/contributions/417/

LWN article: https://lwn.net/Articles/799454/

This was about restricting which tasks share a core on CPUs with SMT/hyperthreading. There is current interest in doing this as a mitigation for speculation leaks, instead of disabling SMT altogether.

SMT also makes single-thread processing speed quite unpredictable, which is bad for RT, so it would be useful to prevent scheduling any other tasks on the same core as an RT task.

Gen-Z Linux Sub-system

Speakers: Jim Hull and Betty Dall of HPE

Details: https://linuxplumbersconf.org/event/4/contributions/301/

Summary
  • New interconnect protocol developed by large consortium
  • Memory-like fabric scalable to large numbers of components
  • Multiple PHY types supported (PCIe gen4, 25/50 Gbit Ethernet PHYs) for different reach/bandwidth/latency trade-offs
  • Can support unmodified OS through "logical PCI devices" and ACPI device description

Connections are point-to-point between "components". Switch components provide fan-out.

Components can be subdivided into "resources" and also have "interfaces".

No requirement for a single root (like typical PCIe) and there can be redundant connections forming a mesh.

Fabric can span multiple logical computers (OS instances). Fabric manager assigns components and resources to them, and configures routing.

Protocol is reliable; all writes are acknowledged (by default). However it is not ordered by default.

Components have single control space (like config space?) and single data space (up to 2⁶⁴ bytes). Control space has a fixed header and then additional structures for optional and per-interface registers.

Each component has 12-bit component ID (CID) which may be combined with 16-bit subnet ID (SID) for 28-bit global component ID (GCID).

Coherence is managed by software.

Bridge from CPU to Gen-Z needs MMUs to map between local physical address space and fabric address space. Normally also has DMA engines ("data movers") that can send and receive all types of Gen-Z packets and not just read/write. These bridges are configured by the local OS instance, not the fabric manager.

Adding a Gen-Z subsystem

Needed to:

  • Enable native device drivers that know how to share resources
  • Enable user-space fabric managers and local management service

Should behave similarly to PCI and USB, so far as possible. Leave policy to user-space. Deal with the fact that most features are optional.

The Gen-Z subsystem needs to provide APIs for tracking PASIDs in IOMMU and ZMMU. Similar requirements in PCIe; should this be generic?

How can Gen-Z device memories be mapped with huge pages?

Undecided whether a generic kernel API for data movers is desirable. This would help kernel I/O drivers but not user-space I/O (like RDMA).

Interrupts work very differently from MSI. Bridge may generate interrupts for explicit interrupt packets, data mover completions, and Unsolicited Event Packets (link change, hotplug, …).

Device discovery

All nodes run local management services. On Linux these will be in user-space (LLaMaS).

(This means LLaMaS will need to be included in the initramfs if the boot device is attached through Gen-Z.)

Manager will use netlink to announce when resource has been assigned to the local node. Kernel then creates kernel device for it.

Live patching

Etherpad: https://etherpad.net/p/LPC2019_Live_Patching/timeslider#3799

Do we need a Livepatch Developers Guide?

Moderator: Joe Lawrence

Details: https://linuxplumbersconf.org/event/4/contributions/512/

Reflections on kernel development process, quality and testing

Speaker: Dmitry Vyukov

Details: https://linuxplumbersconf.org/event/4/contributions/554/

Slides: https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf

Dmitry outlined how the current kernel development processes are failing:

  • Processes are inconsistent between subsystems, and often undocumented
  • Regressions don't consistently get fixed even when they are reported
  • Test coverage is poor and there are several independent automated testing initiatives, that partially overlap
  • Important fixes don't always get backported to the stable branches that need them

It takes a long time for new developers to become productive, or for developers to contribute to unfamiliar subsystems.

(None of this was new to me, but spelling out all these issues definitely had an impact.)

He advocates more consolidation and consistency, so that:

  • Tools can work with and report on proposed/committed changes across the kernel
  • Developers see all test results for a change in one place
  • There is less duplicated work on tools, testing, reporting

There was further discussion of this at the Kernel Maintainer Summit, reported in https://lwn.net/Articles/799134/.

Michael Stapelberg: Debian Code Search: positional index, TurboPFor-compressed

Sunday 29th of September 2019 11:20:18 AM

See the Conclusion for a summary if you’re impatient :-)

Motivation

Over the last few months, I have been developing a new index format for Debian Code Search. This required a lot of careful refactoring, re-implementation, debug tool creation and debugging.

Multiple factors motivated my work on a new index format:

  1. The existing index format has a 2G size limit, into which we have bumped a few times, requiring manual intervention to keep the system running.

  2. Debugging the existing system required creating ad-hoc debugging tools, which made debugging sessions unnecessarily lengthy and painful.

  3. I wanted to check whether switching to a different integer compression format would improve performance (it does not).

  4. I wanted to check whether storing positions with the posting lists would improve performance of identifier queries (= queries which are not using any regular expression features), which make up 78.2% of all Debian Code Search queries (it does).

I figured building a new index from scratch was the easiest approach, compared to refactoring the existing index to increase the size limit (point ①).

I also figured it would be a good idea to develop the debugging tool in lock step with the index format so that I can be sure the tool works and is useful (point ②).

Integer compression: TurboPFor

As a quick refresher, search engines typically store document IDs (representing source code files, in our case) in an ordered list (“posting list”). It usually makes sense to apply at least a rudimentary level of compression: our existing system used variable integer encoding.

TurboPFor, the self-proclaimed “Fastest Integer Compression” library, combines an advanced on-disk format with a carefully tuned SIMD implementation to reach better speeds (in micro benchmarks) at less disk usage than Russ Cox’s varint implementation in github.com/google/codesearch.

If you are curious about its inner workings, check out my “TurboPFor: an analysis”.

Applied on the Debian Code Search index, TurboPFor indeed compresses integers better:

Disk space

  8.9G codesearch varint index

  5.5G TurboPFor index

Switching to TurboPFor (via cgo) for storing and reading the index results in a slight speed-up of a dcs replay benchmark, which is more pronounced the more i/o is required.

Query speed (regexp, cold page cache)

  18s codesearch varint index

  14s TurboPFor index (cgo)

Query speed (regexp, warm page cache)

  15s codesearch varint index

  14s TurboPFor index (cgo)

Overall, TurboPFor is an all-around improvement in efficiency, albeit with a high cost in implementation complexity.

Positional index: trade more disk for faster queries

This section builds on the previous section: all figures come from the TurboPFor index, which can optionally support positions.

Conceptually, we’re going from:

type docid uint32 type index map[trigram][]docid

…to:

type occurrence struct { doc docid pos uint32 // byte offset in doc } type index map[trigram][]occurrence

The resulting index consumes more disk space, but can be queried faster:

  1. We can do fewer queries: instead of reading all the posting lists for all the trigrams, we can read the posting lists for the query’s first and last trigram only.
    This is one of the tricks described in the paper “AS-Index: A Structure For String Search Using n-grams and Algebraic Signatures” (PDF), and goes a long way without incurring the complexity, computational cost and additional disk usage of calculating algebraic signatures.

  2. Verifying the delta between the last and first position matches the length of the query term significantly reduces the number of files to read (lower false positive rate).

  3. The matching phase is quicker: instead of locating the query term in the file, we only need to compare a few bytes at a known offset for equality.

  4. More data is read sequentially (from the index), which is faster.

Disk space

A positional index consumes significantly more disk space, but not so much as to pose a challenge: a Hetzner EX61-NVME dedicated server (≈ 64 €/month) provides 1 TB worth of fast NVMe flash storage.

   6.5G non-positional

  123G positional

    93G positional (posrel)

The idea behind the positional index (posrel) is to not store a (doc,pos) tuple on disk, but to store positions, accompanied by a stream of doc/pos relationship bits: 1 means this position belongs to the next document, 0 means this position belongs to the current document.

This is an easy way of saving some space without modifying the TurboPFor on-disk format: the posrel technique reduces the index size to about ¾.

With the increase in size, the Linux page cache hit ratio will be lower for the positional index, i.e. more data will need to be fetched from disk for querying the index.

As long as the disk can deliver data as fast as you can decompress posting lists, this only translates into one disk seek’s worth of additional latency. This is the case with modern NVMe disks that deliver thousands of MB/s, e.g. the Samsung 960 Pro (used in Hetzner’s aforementioned EX61-NVME server).

The values were measured by running dcs du -h /srv/dcs/shard*/full without and with the -pos argument.

Bytes read

A positional index requires fewer queries: reading only the first and last trigram’s posting lists and positions is sufficient to achieve a lower (!) false positive rate than evaluating all trigram’s posting lists in a non-positional index.

As a consequence, fewer files need to be read, resulting in fewer bytes required to read from disk overall.

As an additional bonus, in a positional index, more data is read sequentially (index), which is faster than random i/o, regardless of the underlying disk.

1.2G 19.8G 21.0G regexp queries

4.2G (index) 10.8G (files) 15.0G identifier queries

The values were measured by running iostat -d 25 just before running bench.zsh on an otherwise idle system.

Query speed

Even though the positional index is larger and requires more data to be read at query time (see above), thanks to the C TurboPFor library, the 2 queries on a positional index are roughly as fast as the n queries on a non-positional index (≈4s instead of ≈3s).

This is more than made up for by the combined i/o matching stage, which shrinks from ≈18.5s (7.1s i/o + 11.4s matching) to ≈1.3s.

3.3s (index) 7.1s (i/o) 11.4s (matching) 21.8s regexp queries

3.92s (index) ≈1.3s 5.22s identifier queries

Note that identifier query i/o was sped up not just by needing to read fewer bytes, but also by only having to verify bytes at a known offset instead of needing to locate the identifier within the file.

Conclusion

The new index format is overall slightly more efficient. This disk space efficiency allows us to introduce a positional index section for the first time.

Most Debian Code Search queries are positional queries (78.2%) and will be answered much quicker by leveraging the positions.

Bottomline, it is beneficial to use a positional index on disk over a non-positional index in RAM.

Steinar H. Gunderson: All Unicode characters

Saturday 28th of September 2019 10:45:42 AM

I made this during the Unicode 5.2.0 days; now it's time for an upgrade:

grep -viE 'LEFT-TO-RIGHT|RIGHT-TO-LEFT|SURROGATE|;Cc;' UnicodeData.txt | perl -U -e 'binmode STDOUT, ":utf8"; while (<>) { my ($u, $name, undef, undef, undef, undef, undef, undef, undef, undef, $alias) = split /;/; printf "%c\tU+%s\t%s\t%s\n", oct("0x$u"), $u, $name, $alias; } ' > /srv/storage.sesse.net/unicode.txt

Result at http://storage.sesse.net/unicode.txt. It doesn't include all the CJK ideographs, but apart from that, it should be fairly complete.

Joey Hess: turing complete version numbers

Friday 27th of September 2019 07:38:39 PM

A quick standard for when you want to embed an arbitrary program in the version number of your program.

2 increment the data pointer (to point to the next cell to the right). 3 decrement the data pointer (to point to the next cell to the left). + increment (increase by one) the byte at the data pointer. - decrement (decrease by one) the byte at the data pointer. . output the byte at the data pointer. 4 accept one byte of input, storing its value in the byte at the data pointer. 6 if the byte at the data pointer is zero, then instead of moving the instruction pointer forward to the next command, jump it forward to the command after the matching 9 command. 9 if the byte at the data pointer is nonzero, then instead of moving the instruction pointer forward to the next command, jump it back to the command after the matching 6 command.

This is simply Brainfuck with operators that are legal in (Debian) version numbers kept as-is, and some numbers replacing the rest.

Note that all other operators are ignored as usual. In particular, 1 and 8 are ignored, which make it easy to build version number programs that compare properly with past versions. And in some cases, adding 1 or 8 will be needed to make a particular program be a properly formatted version number.

For example, an infinite loop version number is:

1+69

A nice short hello world is:

1+6-6336+6-8-1-29-6333999222-92-1.1-1-1-8.2.8.2.3333-1.3+1.22222.2.33.3-1.1

Licensing: Yes, there should also be a way to embed a license in a version ... Oh, I mean to say, the Wikipedia excerpt above is CC-BY-SA, and the hello world is based on https://esolangs.org/wiki/Hello_world_program_in_esoteric_languages

Previously: a brainfuck monad

Matthew Garrett: Do we need to rethink what free software is?

Friday 27th of September 2019 05:47:56 PM
Licensing has always been a fundamental tool in achieving free software's goals, with copyleft licenses deliberately taking advantage of copyright to ensure that all further recipients of software are in a position to exercise free software's four essential freedoms. Recently we've seen people raising two very different concerns around existing licenses and proposing new types of license as remedies, and while both are (at present) incompatible with our existing concepts of what free software is, they both raise genuine issues that the community should seriously consider.

The first is the rise in licenses that attempt to restrict business models based around providing software as a service. If users can pay Amazon to provide a hosted version of a piece of software, there's little incentive for them to pay the authors of that software. This has led to various projects adopting license terms such as the Commons Clause that effectively make it nonviable to provide such a service, forcing providers to pay for a commercial use license instead.

In general the entities pushing for these licenses are VC backed companies[1] who are themselves benefiting from free software written by volunteers that they give nothing back to, so I have very little sympathy. But it does raise a larger issue - how do we ensure that production of free software isn't just a mechanism for the transformation of unpaid labour into corporate profit? I'm fortunate enough to be paid to write free software, but many projects of immense infrastructural importance are simultaneously fundamental to multiple business models and also chronically underfunded. In an era where people are becoming increasingly vocal about wealth and power disparity, this obvious unfairness will result in people attempting to find mechanisms to impose some degree of balance - and given the degree to which copyleft licenses prevented certain abuses of the commons, it's likely that people will attempt to do so using licenses.

At the same time, people are spending more time considering some of the other ethical outcomes of free software. Copyleft ensures that you can share your code with your neighbour without your neighbour being able to deny the same freedom to others, but it does nothing to prevent your neighbour using your code to deny other fundamental, non-software, freedoms. As governments make more and more use of technology to perform acts of mass surveillance, detention, and even genocide, software authors may feel legitimately appalled at the idea that they are helping enable this by allowing their software to be used for any purpose. The JSON license includes a requirement that "The Software shall be used for Good, not Evil", but the lack of any meaningful clarity around what "Good" and "Evil" actually mean makes it hard to determine whether it achieved its aims.

The definition of free software includes the assertion that it must be possible to use the software for any purpose. But if it is possible to use software in such a way that others lose their freedom to exercise those rights, is this really the standard we should be holding? Again, it's unsurprising that people will attempt to solve this problem through licensing, even if in doing so they no longer meet the current definition of free software.

I don't have solutions for these problems, and I don't know for sure that it's possible to solve them without causing more harm than good in the process. But in the absence of these issues being discussed within the free software community, we risk free software being splintered - on one side, with companies imposing increasingly draconian licensing terms in an attempt to prop up their business models, and on the other side, with people deciding that protecting people's freedom to life, liberty and the pursuit of happiness is more important than protecting their freedom to use software to deny those freedoms to others.

As stewards of the free software definition, the Free Software Foundation should be taking the lead in ensuring that these issues are discussed. The priority of the board right now should be to restructure itself to ensure that it can legitimately claim to represent the community and play the leadership role it's been failing to in recent years, otherwise the opportunity will be lost and much of the activist energy that underpins free software will be spent elsewhere.

If free software is going to maintain relevance, it needs to continue to explain how it interacts with contemporary social issues. If any organisation is going to claim to lead the community, it needs to be doing that.

[1] Plus one VC firm itself - Bain Capital, an investment firm notorious for investing in companies, extracting as much value as possible and then allowing the companies to go bankrupt

comments

Thomas Lange: Read-only nfsroot with NFS v4 and overlayfs

Thursday 26th of September 2019 08:10:16 AM

The Fully Automatic Installation (FAI) is using a read-only nfsroot since it's very beginning. This is also used in diskless clients enviroments and in the LTSP (Linux Terminal Server Project).

During a network installation the clients are running as diskless clients, so the installation has full access to the local hard disk which is not in use. But we need some files to be writable on the read-only nfsroot. In the past we've created symlinks to a ram disk. Later we used aufs (another union fs), a kernel module for doing union mounts of several file systems. Putting a ram disk on top of the read-only nfsroot with aufs makes the nfsroot writable. But aufs was not available in kernel 4.X any more. It was replaced by overlayfs.

The initrd of FAI mounts the nfsroot read only and then puts a tmpfs ram disk on top of it using overlayfs. The result is a new merged file system which is writable. This works nicely since several years when using NFSv3. But when using NFSv4 we can read from a file, but writing always reported

openat(AT_FDCWD,....) = -1 EOPNOTSUPP (Operation not supported)

After some days of debugging overlayfs and NFS v4, I found that it's a complicated mixture of NFS and acl support (POSIX and nfs4 acl) and what overlayfs expects from the file systems in respect to certain xattr. Overlayfs uses calls like

setxattr(work/work, "trusted.overlay.opaque", "0", 1, 0x0) = 0

and writing to a file used

getxattr("/b/lower/etc/test1", "system.nfs4_acl", ....) = 80

without any errors. When talking to some overlayfs guys they ask me to disable acl for the exported NFS file system. There's an noacl option listed on nfs(5), but it's for NFS version 2 and 3 only, not for NFS v4. You cannot disable ACL on a NFS v4 mount.

In the end the solution was to disable ACL on the whole file system on the NFS server, which is exported to the clients. If you have a ext4 file system this works on the NFS server by doing

# mount -oremount,noacl $EXPORTED_FS

After that, overlayfs will detect that ACL's are not support on the NFS mount and behaves as expected allowing writes to a file.

You will need to use dracut instead of initramfs-tools for creating the initrd. The later is using busybox or klibc tools inside the initrd. Both do not support NFS v4 mounts (https://bugs.debian.org/409271).

Dracut is using the normal libc based executables. The Debian package of dracut supports the kernel cmdline option rootovl. This is an example of the kernel cmdline options:

rootovl ip=dhcp root=11.22.33.44:/srv/fai/nfsroot

This mounts a read only nfsroot and puts a tmpfs on top for making it writable.

NFSv4 nfsroot

Shirish Agarwal: Life, Liberty and Kashmir

Wednesday 25th of September 2019 11:15:57 PM

I was going to write about history of banking today but because the blockade is still continuing in Kashmir, I am forced to write my opinions on it and clear at least some ideas and myths various people have about Kashmir. Before I start though, I hope the Goa Debian Utsav was good. While I haven’t seen any reports, I hope it went well. Frankly, I was in two minds whether I should apply for the Debutsav in Goa or not. While there is a possibility that I could have applied and perhaps even got the traveling sponsorship, I was unsure as to what to tell the students. With recovery of the economy in India at least 6 quarters away if not more, it would have been difficult for me to justify to the students as to how to look for careers in I.T. when salaries of most professionals have been stagnant, lowered and even retention happening in Pune, Bangalore and other places it would have been difficult to say that.

Anyways, this would be a long one. I would like to start with a lawsuit filed in Kerala which was shared and the judgement which was given which at least in my view was a progressive decision. The case I am reciting is ”Right To Access Internet Is Part Of Right To Privacy And Right To Education‘ which was given by Kerala HC recently. The judgement of the case is at https://www.livelaw.in/pdf_upload/pdf_upload-364655.pdf which I reproduce below as well.

Right-to-access-internet-is-right-to-privacy-judgement
Download

So let us try to figure out what the suit/case was all about and how it involves the larger question of communication blockades and other things in Kashmir. The case involves a woman student of 18 years of age, a Faheema shirin (Petitioner) who came to Kerala for higher studies (B.Ed) at an institute called Narayanguru College located in Kozhikhode District. Incidentally, I have been fortunate to visit Kerala and Khozikhode District and they are beautiful places but we can have that conversation some other day. Now apparently, she was expelled from the college hostel for using the mobile phone during study time. The College is affiliated to University of Calicut. Now according to statements from the hostel matron, the petitioner and others, it became clear that inmates of the hostel were not allowed to use mobile phones from 10 p.m. to 6.a.m. -i.e. 22:00 hrs. to 0600 hrs. Apparently, this rule was changed to 1800 hrs – 2000 hrs. arbitrarily. The petitioner’s house is 150 kms. from the place. When she said it is not possible to follow the rules because of the subjects she was studying as well as she needed to connect to home anytime she wanted or her father or relatives may feel to call her or in case of any help. She alleged discrimination as these rules were only made for the girl’s hostel and not for the boy’s hostel. I had also seen and felt the same but as shared that’s for another day altogether.

The petitioner invoked the Conventions on Eliminations of all forms of Discrimination against Women, 1979, the Beijing Declaration and Universal Declaration of Human Rights, to which GOI is a signatory and hence had to abide by its rules. She further contended that her education depended on her using digital technology with access to web as given in her textbook. She needed to scan the QR codes in various places in her textbooks and use the link given therein to see videos, animations etc. on a digital platform called swayam. Incidentally, it seems swayam runs on closed source software as shared by SFLC.in on their website. Now if it is closed, commercial software than most probably the only the content can be viewed is via streaming rather than downloading, going offline and seeing it as that would attract provisions of the IT ACT and perhaps would constitute piracy. While this point was not argued, it seemed pertinent for me to point out as few people on social media have asked about. In several such cases it is either impossible or you have to be an expert in order to manipulate and download such data (like Snowdem did) but then that’s again a story for another day. Interestingly, the father in the case above was also in the favor of the girl using mobile phone for whatever purpose as he trusts her implicitly and she is adult enough to make her own life choices.

Thankfully, the petitioner had presence of mine throughout the journey that she did all her correspondence through letters instead of orally and had documentary evidence to back up all her claims. The State Govt. of Kerala has been on the forefront of digital technology for a long while and me and many of my friends have been both witness and played our small parts in whichever way to see Kerala become an IT hub. While they still do need to do a lot more but that again is a story for another day. While there was lot of back and forth between her, the hostel authorities, the father and the hostel authorities, she, her father, the hostel authorities and the college but they were unable to resolve the issues amicably. Her grounds for the fight were –

a. She is an adult and of rational mind so she can make decisions on her own.
b. She has right of privacy ( as shared by the Honorable Supreme Court in its 2017 landmark judgement)
c. She needs the mobile and the laptop for studying as her studies demand her using Internet.
d. She also relied and used the budget speech made by Minister of Finance and State Government for making internet accessible to all citizens and recognizing the right to Internet as a human right.
e. Her violation to right of property under Article 300 A.

In order to further bolster her case, through her lawyers she cited further judgements and studies which show how women are disadvantaged to Internet access, in particular she cited a UNESCO study which tells the same.

The judge, Honorable Jutice P.V. Asha guided herself with the arguments and counter-arguments by both parties, she also delved into Calicut University First Ordinances under which the University, the college and the hostel come in to see how thngs fare there. She had also asked the respondent that by using Internet has she or any other student in the hostel ever caused disturbance to any of the other inmates to which the reply was negative. The Judge also determined that if a miuse of a mobile phone or laptop has to happen, it can happen any time, anywhere and you cannot and should not control adult behavior especially when it collides with dignity and freedom of an adult. The learned counsel for the petitioner also shared resolution 23/2 in the UN General Assembly held on 24th June 2013 which talks of freedom of expression and opinion for women’s empowerment to which India is a signatory. There is also resolution 20/8 of 5th July 2012 which also underscores the point. Both the portions of the resolution can be found on page 18 of the judgement. The judge also cited few other judgements which were pointed out by the learned counsel for the petitioner, the Vishaka Judgement (1997) , the Beijing Statement and several other cases and judgement which showed how women are discriminated against under society. In the end she set aside the expulsion citing various judgements and her rationale for the same and asked the matron to take the student back and also asked the student to not humiliate the teacher or warden and she be allowed to use phone in any way she feels fit as far as she doesn’t create any disturbance to other students.

Observations – It opens up several questions which are part of society’s issues even today and probably for sometime.

a. I have been part of quite a few workshops where while I was supposed to share about GNU/Linux, more often than not I ended up sharing about how to use web access rather than advanced technologies. In this I found women to be more backward and take more time to understand and use the concepts than men. Whether it is due to just access issues or larger societal reasons ( the hidden alleged misuse of web) I just don’t know. While I do wish we could do more I don’t have any solutions.

b. As correctly pointed by Honorable Justice Asha, if a women who is pursuing B.Ed. it would harm the career of the young woman. I would opine and put one step more, wouldn’t it also be endangering her proteges, her students from getting a better teacher who is able to guide her students to the best of her ability. As we all know, rightly or wrongly almost all information is available on the net. The role of the teacher or guide is not to show information but probably more as to how to inquire and interpret information in different ways.

Kashmir

In light of the above judgement would not the same principles apply to Kashmir. There are two points shared by various people who are in favor of the lockdown. The first is National Security, National Interest and the second is Kashmiri Pandits. Let us take them one by one –

a. National Interest or/and National Security – I find this reason porous on many grounds. This Govt. is ruled by one of the richest political parties that India ever has. Without divulging further, there is such a huge range of hardware and software for the Government to surveil. With AFSA in-place and all sorts of technologies available off-the-shelf to surveil on residents that argument looks weak. Further, the Minister’s statement tells that the issue is not security of the state but something else. Of course the majoratian view is that they deserve it because they are muslims. If this is not hate, I dunno what is. A person on twitter did a social experiment where a daughter and a mother had the same conflict. The daughter’s view is that it is not right, the mother’s view being the opposite. The daughter disallowed the mother any contact with her, her husband and her daughter for 2 weeks, the mother was in tears. Then how can you think of people being blocked for 2 months.

Another variation of the argument is that militants will come and kill. Now I find it hard to believe that even after having half a million soldiers in the valley they still feel miitants can do something and they cannot. I find it a little hard to digest. There has been news now that the Taliban are involved. If this is true then they have troubled U.S. also, so if one of the most powerful armies on the earth can be stale-mated for what 19 years, are we going to put Kashmiris in lockdown for 19 years ? In fact the prejudcial face can be seen even more at https://www.youtube.com/watch?v=kXWZnnD6JFY-

Kashmiri Pandits – There is no doubt that there was a mass exodus of Kashmiri Hindus from the valley. Nobody disputes that. But just like the process followed in NRC, whether rightly or wrongly couldn’t the Kashmiri Pandits be sent back home. I would argue this is the best time. You have a huge contigent of forces in the valley, you can start the process, get the documents, get them back into the valley, otherwise this will continue to be something like Palestine is in Israel which has continued to an issue for both Israelis and Palestinians with no end in sight. The idea that Pakistan will not harass or do something in Kashmir in fool’s paradise. They have been doing it since 90’s, for that to have a huge population blocked from communicating is nothing but harassment. And hate will never get you anywhere. While this is more greyer than I am making it out, feel free to read this interview as well as the series called The Family Man which I found to be pretty truthful as to the greyishness of the situation out there. While most of the mainstream media gave it an average score, I found it thought-provoking. The fact is mainstream media in India no longer questions the Government excesses. Some people do and they are often targeted. I do hope to share the banking scenario and a sort of mini-banking crisis soon. Till later.

Mike Gabriel: IServ Schulserver - Insecure Setup Strategy allows Hi-Jacking of User Accounts

Wednesday 25th of September 2019 04:02:44 PM

"IServ Schulserver" [1] is a commercial school server developed by a company in Braunschweig, Germany. The "IServ Schulserver" is a product based on Debian. The whole project started as a students' project.

The "IServ" is an insular school server (one machine for everything + backup server) that provides a web portal / communication platform for the school (reachable from the internet), manages the school's MS Windows® clients via OPSI [2] and provides other features like chatrooms, mail accounts, etc.

The "IServ Schulserver" has written quite a success story in various areas of Germany, recently. IServ has been deployed at many many schools in Northrhein-Westfalia, Lower Saxony and Schleswig-Holstein. You can easily find those schools on the internet, if you search the web for "IServ IDesk".

The company that is developing "IServ" has various IT partner businesses all over Germany that deploy the IServ environment at local schools and are also the first point of contact for support.

It's all hear-say...

So, last night, I heard about a security design flaw not having been fixed / addressed since I had first heard about it. That was in 2014, when one of the Debian Edu schools I supported back then migrated over to IServ. At that time, the below could be confirmed. Last night, I learned that the following is still an issue on an IServ machine deployed recently here in Schleswig-Holstein (its deployment dates only a few weeks back). It's all hear-say, you know. But alas, ...

Mass User Creation Modes

If IServ admins mass create user accounts or (updated 20190930) perform user import from CSV-like data following the product's documentation [3a, 3b], they can opt for user accounts to be created and made active immediately, or they can opt for creating user accounts that are initially deactivated.

If the site admins uses the user import tool on the other hand, they also can opt for activated or deactivated accounts ot be created and they can choose one of the available password creation strategies (password := login (default), password from CSV, password generated via pwgen).

The password creation strategy of the local supplier of IServ Schulserver in Schleswig Holstein (around the area of city of Kiel) seems to be creating these initial user accounts (that is, all contemporary teachers and students) with immediately activated accounts and the default password creation strategy (password := login). (Cough cough...)

Initial Login

If you are a teacher (or student) at a school and have been notified about your initial IServ account being set up for you, you will get the instruction to initially log into the IServ web portal. The school provides each teacher with a URL and a login name. The default scheme for login names is <firstname>.<lastname>.

The password is not explicitly mentioned, as it is easy to remember. It is also <firstname>.<lastname> (i.e. initial_password := login_name). Conveniently as it is, people can do these logins from anywhere. When doing the initial login, the users are guided to a change-password dialog in their web browser session and finally, they can set their own password.

Pheeeww.... one account less that is just too dumb easy to hack.

Getting to know People at your New School

Nowadays, most schools have a homepage. On that homepage, they always present the core teacher staff group (people with some sort of a leadership position) with full names. Sometimes they even list all teachers with their full names. More rarely, but also quite common, all teachers are listed with a portrait photo (and/or the subjects they teach). Wanna be a teacher at that school? Hacky-sign up for an account then...

Update (20190930): To be fully clear on this: IServ does not provide a Sign-Up Feature byitself, all user accounts get created via an import of school data taken out of the school's administration database. However, picking an existing account that is likely to be still fresh and untouched by its user, is pretty much as easy as signing up for an account on.

How to Get In

If you are a nasty hacker, you can now go to some school's homepage, pick a teacher/face (or subject combination) that makes you assume that that person is not an IT-affiliated-kind-of-person and try to login as that person. If you are a neat hacker, you do this via Tor (or similar), of course.

Seriously!

If our imaginery hackers succeed with logging in using initial credentials, they can set a password for the impersonated teacher and they are in.

Many schools, I have seen, distribute documents and information to their teachers via the schools communication platform. If that platform is "IServ Schulserver", then you can easily gain access to those documents [4].

My personal guess is, that schools also use their school communication platform for distributing personal data, which is probably not allowed on the educational network of a school anyway (the "IServ Schulserver" is not an E-Mail server on the internet, it is the core server, firewall, mail gateway, Windows Network Server, etc. of the school's educational network).

Now, sharing those information via a system that is so easy to get unauthorized access to, is IMHO highly negligent and a severe violation of the GDPR.

Securing Mass User Creation

There are several ways, to fix this design flaw:

  • mass create users with accounts being initially deactivated and come up with some internal social workflow for enabling and setting up accounts and user passwords
  • talk to the developers and ask them to add credential imports (i.e. mass setting passwords for a list of given usernames)
  • Obsolete 20190930: use some other school server solution
  • Update 20190930: the previous statement about just using another school server solution is not really leading to better security by itself. The problem here in this blog post is not so much about IServ's user import code, but about the combination of software-featured setup strategies and that service providers deploy IServ in such an insecure manner (although more secure features are available, but not the default). So, I could also say: get another service provider. People who, when setting up school IT, are aware of the security impact of their doings.
Other Security Issues?

If people like to share their observations about school IT and security, I'd be interested. Let me know (see the imprint page [5] on my blog for my mail address).

light+love
Mike Gabriel (aka sunweaver at debian.org)

References & Footnotes Update 20190930:

Last Friday, I received feedback from Sören Wendhauen (IServ GmbH). He provided some more background information about IServ user import. Thanks a lot for that.

Admin coaches at IServ GmbH do in fact make there service partner businesses aware of what I have depicted above. So, service providers should be in the loop of the security weakness (and act accordingly, I'd expect).

However, (and that was the essence of my reply), they (IServ GmbH developers) nonetheless developed this "password := login" feature in the first place, made it the default password generation strategy and even now that they have more secure password creation methods at hand, they leave the "password := login" method the default method.

Another alternative: If user accounts are activated at creation time and if the "password := login" password creation method had been used during creation, the IServ WebUI could e.g. prohibit a world-wide login, but restrict the user login to the computer labs of the school. Not a good solution, but drastically shrinking the attack vector, while keeping the wanted usability. However, this only works at schools where computer lab access is always monitored by teacher staff.

With Dürrenmatt's "Die Physiker" in mind, as a software developer I am responsible for the features I give people at hand to use and/or misuse.

Andrej Shadura: Rust-like enums in Kotlin

Wednesday 25th of September 2019 01:31:43 PM

Rust has an exciting concept of enumeration types, which is much more powerful than enums in other languages. Notably C has the weakest type of enum, since there’s no type checking of any kind, and enum values can be used interchangeably with integers:

enum JobState { PENDING, STARTED, FAILED, COMPLETED };

You can opt for manually assigning integers instead of leaving this to the compiler, but that’s about it.

Higher level languages like Python and Java treat enumeration types as classes, bringing stricted type checking and better flexibility, since they can be extended nearly as any other classes. In both Python and Java individual enumerated values are singleton instances of the enumeration class.

class JobState(Enum): PENDING = auto() STARTED = auto() FAILED = auto() COMPLETED = auto() enum JobState { PENDING, STARTED, FAILED, COMPLETED; }

Since enumerations are classes, they can define extra methods, but because the enum values are singletons, they can’t be coupled with any extra data, and no new instances of the enum class can be created.

In contrast with Python and Java, Rust allows attaching data to enumerations:

enum JobState { Pending, Started, Failed(String), Completed }

This allows us to store the error message in the same value as the job state, without having to declare a structure with an extra field which would be used only when the state in Failed.

So, what Kotlin has to offer? Kotlin has a language feature called sealed classes. A sealed class is an abstract class with limited interitance: all of its subclasses have to be declated in the same file. In a way, this is quite close to the Rust enums, even though sealed classed look and behave a bit differently.

sealed class JobState { object Pending : JobState() object Started : JobState() object Completed : JobState() data class Failed(val errorMessage: String) : JobState() }

Declared this way, JobState can be used in a way similar to Rust’s enums: a single variable of this type can be assigned singletons Pending, Started or Completed, or any instance of Failed with a mandatory String member:

val state: JobState = JobState.Failed("I/O error") when (state) { is JobState.Completed -> println("Job completed") is JobState.Failed -> println("Job failed with an error: ${state.errorMessage}") }

This usage resembles the regular Java/Kotlin enums quite a bit, but alternatively, Pending and friends can be declared outside of the sealed class, allowing them to be used directly without the need to add a JobState qualifier.

A slightly simplified real life example from a Kotlin project I’m working on, where a separate coroutine handles I/O with a Bluetooth or a USB device:

sealed class Result object Connected : Result() data class Failed(val error: String) : Result() sealed class CommServiceMsg data class Connect(val response: CompletableDeferred<Result>) : CommServiceMsg() object Disconnect : CommServiceMsg() data class Write(val data: ByteArray) : CommServiceMsg() fun CoroutineScope.bluetoothServiceActor(device: BluetoothDevice) = actor<CommServiceMsg>(Dispatchers.IO) { val socket: BluetoothSocket = device.createSocket() process@ for (msg in channel) { when (msg) { is Connect -> { with(socket) { msg.response.complete(try { connect() Connected } catch (e: IOException) { val error = e.message ?: "" Failed(error) } } } is Disconnect -> break@process is Write -> { socket.outputStream.write(msg.data) } } } socket.outputStream.flush() socket.close() }

Here, we can talk to bluetoothServiceActor using messages each carrying extra data; if the coroutine needs to talk back (in this example, the result of a connection attempt), it uses a CompletableDeferred<> value of the Result type, which can hold an error message when needed.

With that in place, we can write something like this:

val bluetoothService = bluetoothServiceActor(device) val response = CompletableDeferred<Result>() bluetoothService.send(Connect(response)) var result = response.await() when (result) { is Connected -> { bluetoothService.send(Write(byteArrayOf(42, 0x1e, 0x17))) bluetoothService.send(Disconnect) } is Failed -> println("error occurred: ${result.error}") }

Enrico Zini: xtypeinto: type text into X windows

Tuesday 24th of September 2019 01:27:12 PM

Several sites have started disabling paste in input fields, mostly password fields, but also other fields for no apparent reason.

Random links on the topic:

  • https://developers.google.com/web/tools/lighthouse/audits/password-pasting
  • https://www.ncsc.gov.uk/blog-post/let-them-paste-passwords
  • https://www.troyhunt.com/the-cobra-effect-that-is-disabling/
  • https://www.wired.com/2015/07/websites-please-stop-blocking-password-managers-2015/

This said, I am normally uneasy about copy-pasting passwords, as any X window can sniff the clipboard contents at any time, and I like password managers like impass that would type it for you instead of copying it to the clipboard.

However, today I got out way more frustrated than I could handle after illing in 17-digits nonsensical, always-slightly-different INPS payment codelines inside input fields that disabled paste for no reason whatsoever (they are not secret).

I thought "never again", I put together some code from impass and wmctrl and created xtypeinto:

$ ./xtypeinto --help usage: xtypeinto [-h] [--verbose] [--debug] [string] Type text into a window positional arguments: string string to type (default: stdin) optional arguments: -h, --help show this help message and exit --verbose, -v verbose output --debug debug output

Pass a string to xtypeinto as an argument, or as standard input.

xtypeinto will show a crosshair to pick a window, and the text will be typed into that window.

Please make sure that you focus on the right field before running xtypeinto, to make sure things are typed where you need them.

Keith Packard: picolibc

Tuesday 24th of September 2019 06:18:12 AM
Picolibc Version 1.0 Released

I wrote a couple of years ago about the troubles I had finding a good libc for embedded systems, and for the last year or so I've been using something I called 'newlib-nano', which was newlib with the stdio from avrlibc bolted on. That library has worked pretty well, and required very little work to ship.

Now that I'm doing RISC-V stuff full-time, and am currently working to improve the development environment on deeply embedded devices, I decided to take another look at libc and see if a bit more work on newlib-nano would make it a good choice for wider usage.

One of the first changes was to switch away from the very confusing "newlib-nano" name. I picked "picolibc" as that seems reasonably distinct from other projects in the space and and doesn't use 'new' or 'nano' in the name.

Major Changes

Let's start off with the big things I've changed from newlib:

  1. Replaced stdio. In place of the large and memory-intensive stdio stack found in newlib, picolibc's stdio is derived from avrlibc's code. The ATmel-specific assembly code has been replaced with C, and the printf code has seen significant rework to improve standards conformance. This work was originally done for newlib-nano, but it's a lot cleaner looking in picolibc.

  2. Switched from 'struct _reent' to TLS variables for per-thread values. This greatly simplifies the library and reduces memory usage for all applications -- per-thread data from unused portions of the library will not get allocated for any thread. On RISC-V, this also generates smaller and faster code. This also eliminates an extra level of function call for many code paths.

  3. Switched to the 'meson' build system. This makes building the library much faster and also improves the maintainability of the build system as it eliminates a maze of twisty autotools configure scripts.

  4. Updated the math test suite to use glibc as a reference instead of some ancient Sun machine.

  5. Manually verified the test results to see how the library is doing; getting automated testing working will take a lot more effort as many (many) tests still have invalid 'correct' values resulting in thousands of failure.

  6. Remove unused code with non-BSD licenses. There's still a pile of unused code hanging around, but all non-BSD licensed bits have been removed to make the licensing situation clear. Picolibc is BSD licensed.

Picocrt

Starting your embedded application requires initializing RAM as appropriate and calling initializers/constructors before invoking main(). Picocrt is designed to do that part for you.

Building Simplified

Using newlib-nano meant specifying the include and library paths very carefully in your build environment, and then creating a full custom linker script. With Picolibc, things are much easier:

  • Compile with -specs=picolibc.specs. That and the specification of the target processor are enough to configure include and library paths. The Debian package installs this in the gcc directory so you don't need to provide a full path to the file.

  • Link with picolibc.ld (which is used by default with picolibc.specs). This will set up memory regions and include Picocrt to initialize memory before your application runs.

Debian Packages

I've uploaded Debian packages for this version; they'll get stuck in the new queue for a while, but should eventually make there way into the repository. I'll plan on removing newlib-nano at some point in the future as I don't plan on maintaining both.

More information

You can find the source code on both my own server and over on github:

You'll find some docs and other information linked off the README file

Dirk Eddelbuettel: RcppAnnoy 0.0.13

Tuesday 24th of September 2019 12:48:00 AM

A new release of RcppAnnoy is now on CRAN.

RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours—originally developed to drive the famous Spotify music discovery algorithm.

This release brings several updates. First and foremost, the upstream Annoy C++ code was updated from version 1.12 to 1.16 bringing both speedier code thanks to AVX512 instruction (where available) and new functionality. Which we expose in two new functions of which buildOnDisk() may be of interest for some using the file-back indices. We also corrected a minor wart in which a demo file was saved (via example()) to a user directory; we now use tempfile() as one should, and contributed two small Windows build changes back to Annoy.

Detailed changes follow below.

Changes in version 0.0.13 (2019-09-23)
  • In example(), the saved and loaded filename is now obtained via tempfile() to not touch user directories per CRAN Policy (Dirk).

  • RcppAnnoy was again synchronized with Annoy upstream leading to enhanced performance and more features (Dirk #48).

  • Minor changes made (and send as PRs upstream) to adapt both annoylib.h and mman.h changes (Dirk).

  • A spurious command was removed from one vignette (Peter Hickey in #49).

  • Two new user-facing functions onDiskBuild() and unbuild() were added (Dirk in #50).

  • Minor tweaks were made to two tinytest-using test files (Dirk).

Courtesy of CRANberries, there is also a diffstat report for this release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Molly de Blanc: Freedoms and Rights

Monday 23rd of September 2019 10:38:48 AM

I want to talk a bit about the relationship between rights and freedoms, and what they are. I think building a mutual understanding around this is important as I dig deeper into conversations around digital rights and software, user, and technology freedom.

A right is like a privilege in as much is that it’s something you’re allowed to do, however rights are innate and not earned. They are things to which everyone is entitled. A freedom expresses a lack of constraints related to an action. When we have a particular freedom (freedom X), we have an unrestrained ability to do X — we can do whatever we want in relation to X. You can also have the right to a certain kind of freedom (e.g. freedom of speech). I talk about both digital rights and digital freedoms. I view digital rights are the extension of our rights into digital spaces, and digital freedoms as the freedoms we have in those spaces. We have the right to free expression when speaking in a room; we have the right to free expression when speaking on the Internet.

Typically, we frame rights and freedoms in terms of government restrictions: governments are not allowed to keep you from exercising your freedoms, and they are there to protect and ensure your rights. It is becoming increasingly relevant (and common) to also talk about these in relation to companies and technology. It is important to also shift this discussion to include companies and technologies — especially computing software. As computing becomes more pervasive, we need to make sure that the software we’re writing is freedom protecting and rights respecting. These freedoms include the freedoms we typically associate with free and open source software: the unbridaled ability to use, study, modify, and share. it also includes freedoms like expression (to express ourselves without constraint) and the freedom to assemble (to get together without constraint). All of these freedoms are freedoms we have the right to, in addition to other rights including the right to digital autonomy and the right to consent.

I want to dig a little into a specific example, of the play between freedoms and rights, and the way we see computing fits in.

We have the right to freedom of speech — to communicate unfettered with one another. Free expression is something to which everyone is entitled, and there is a societal, social, and moral imperative to protect that right. Computers connect us to one another and enable us to express ourselves. They also give us safe spaces to develop the ideas we want to express in public ones, which is a necessary part of freedom of speech. However, computers can also infringe upon that right. Home surveillance devices, like home assistants, that are listening to and recording everything you say are stepping on your right and restricting your freedom. They are taking away your safe space to develop ideas and creating an environment where you cannot express yourself without restriction for fear of possible repercussions.

This is just one example of how computers play with the things we traditionally consider our rights and freedoms. Computers also force us to consider rights and freedoms in new contexts, and push the boundaries of what we consider to “count.” Our right to bodily autonomy now includes which medical devices, which computers, we allow to be implanted into our bodies; what happens with our medical and biometric data; and when and how our bodies are being monitored in public (and private) spaces. This includes the near future, where we see an increase in wearable computers and recreational and elective implants.

We have freedoms, we have rights, and we have the rights to certain freedoms because it is moral, ethical, and necessary for a just world. Our digital rights and digital freedoms are necessary for our digital autonomy, to borrow a phrase from Karen Sandler. Digital autonomy is necessary to move forward into a world of justice, equity, and equality.

Special thanks for Christopher Lemmer Webber.

William (Bill) Blough: Free Software Activities (August 2019)

Sunday 22nd of September 2019 09:13:40 PM
Debian
  • Fixed bug 933422: passwordsafe — Switch to using wxgtk3

    Versions:

    • unstable/testing: 1.06+dfsg-3
  • Upgraded passwordsafe package to latest upstream version (1.08.2)

    Versions:

    • unstable/testing: 1.08.2+dfsg-1
    • buster-backports: 1.08.2+dfsg-1~bpo10+1
  • Updated python-django-cas-client to latest upstream version (1.5.1) and did some miscellaneous cleanup/maintenance of the packaging.

    Versions:

    • unstable/testing: 1.5.1-1
  • Discovered an issue with sbuild where the .changes file output by the build was different from the .changes file passed to lintian. This meant that the lintian results were sometimes different when lintian was run via sbuild vs when it was run manually. Patch submitted.

  • Provided a patch for NuSOAP to update deprecated class constructors.

  • Submitted a merge request to update the ftp-master website and replace a reference to Buster as testing with Bullseye.

Axis2-C
  • Fixed bug AXIS2C-1619: CVE-2012-6107: SSL/TLS Hostname validation

    Commits:

    • r1866225 - Perform SSL hostname validation
    • r1866245 - Add SSL host validation check to X509_V_OK code path

Colin Watson: Porting Storm to Python 3

Sunday 22nd of September 2019 07:56:42 AM

We released Storm 0.21 on Friday (the release announcement seems to be stuck in moderation, but you can look at the NEWS file directly). For me, the biggest part of this release was adding Python 3 support.

Storm is a really nice and lightweight ORM (object-relational mapper) for Python, developed by Canonical. We use it for some major products (Launchpad and Landscape are the ones I know of), and it’s also free software and used by some other folks as well. Other popular ORMs for Python include SQLObject, SQLAlchemy and the Django ORM; we use those in various places too depending on the context, but personally I’ve always preferred Storm for the readability of code that uses it and for how easy it is to debug and extend it.

It’s been a problem for a while that Storm only worked with Python 2. It’s one of a handful of major blockers to getting Launchpad running on Python 3, which we definitely want to do; stoq ended up with a local fork of Storm to cope with this; and it was recently removed from Debian for this and other reasons. None of that was great. So, with significant assistance from a large patch contributed by Thiago Bellini, and with patient code review from Simon Poirier and some of my other colleagues, we finally managed to get that sorted out in this release.

In many ways, Storm was in fairly good shape already for a project that hadn’t yet been ported to Python 3: while its internal idea of which strings were bytes and which text required quite a bit of untangling in the way that Python 2 code usually does, its normal class used for text database columns was already Unicode which only accepted text input (unicode in Python 2), so it could have been a lot worse; this also means that applications that use Storm tend to get at least this part right even in Python 2. Aside from the bytes/text thing, many of the required changes were just the usual largely-mechanical ones that anyone who’s done 2-to-3 porting will be familiar with. But there were some areas that required non-trivial thought, and I’d like to talk about some of those here.

Exception types

Concrete database implementations such as psycopg2 raise implementation-specific exception types. The inheritance hierarchy for these is defined by the Python Database API (DB-API), but the actual exception classes aren’t in a common place; rather, you might get an instance of psycopg2.errors.IntegrityError when using PostgreSQL but an instance of sqlite3.IntegrityError when using SQLite. To make things easier for applications that don’t have a strict requirement for a particular database backend, Storm arranged to inject its own virtual exception types as additional base classes of these concrete exceptions by patching their __bases__ attribute, so for example, you could import IntegrityError from storm.exceptions and catch that rather than having to catch each backend-specific possibility.

Although this was always a bit of a cheat, it worked well in practice for a while, but the first sign of trouble even before porting to Python 3 was with psycopg2 2.5. This release started implementing its DB-API exception types in a C extension, which meant that it was no longer possible to patch __bases__. To get around that, a few years ago I landed a patch to Storm to use abc.ABCMeta.register instead to register the DB-API exceptions as virtual subclasses of Storm’s exceptions, which solved the problem for Python 2. However, even at the time I landed that, I knew that it would be a porting obstacle due to Python issue 12029; Django ran into that as well.

In the end, I opted to refactor how Storm handles exceptions: it now wraps cursor and connection objects in such a way as to catch DB-API exceptions raised by their methods and properties and re-raise them using wrapper exception types that inherit from both the appropriate subclass of StormError and the original DB-API exception type, and with some care I even managed to avoid this being painfully repetitive. Out-of-tree database backends will need to make some minor adjustments (removing install_exceptions, adding an _exception_module property to their Database subclass, adjusting the raw_connect method of their Database subclass to do exception wrapping, and possibly implementing _make_combined_exception_type and/or _wrap_exception if they need to add extra attributes to the wrapper exceptions). Applications that follow the usual Storm idiom of catching StormError or any of its subclasses should continue to work without needing any changes.

SQLObject compatibility

Storm includes some API compatibility with SQLObject; this was from before my time, but I believe it was mainly because Launchpad and possibly Landscape previously used SQLObject and this made the port to Storm very much easier. It still works fine for the parts of Launchpad that haven’t been ported to Storm, but I wouldn’t be surprised if there were newer features of SQLObject that it doesn’t support.

The main question here was what to do with StringCol and its associated AutoUnicodeVariable. I opted to make these explicitly only accept text on Python 3, since the main reason for them to accept bytes was to allow using them with Python 2 native strings (i.e. str), and on Python 3 str is already text so there’s much less need for the porting affordance in that case.

Since releasing 0.21 I realised that the StringCol implementation in SQLObject itself in fact accepts both bytes and text even on Python 3, so it’s possible that we’ll need to change this in the future, although we haven’t yet found any real code using Storm’s SQLObject compatibility layer that might rely on this. Still, it’s much easier for Storm to start out on the stricter side and perhaps become more lenient than it is to go the other way round.

inspect.getargspec

Storm had some fairly complicated use of inspect.getargspec on Python 2 as part of its test mocking arrangements. This didn’t work in Python 3 due to some subtleties relating to bound methods. I switched to the modern inspect.signature API in Python 3 to fix this, which in any case is rather simpler with the exception of a wrinkle in how method descriptors work.

(It’s possible that these mocking arrangements could be simplified nowadays by using some more off-the-shelf mocking library; I haven’t looked into that in any detail.)

What’s next?

I’m working on getting Storm back into Debian now, which will be with Python 3 support only since Debian is in the process of gradually removing Python 2 module support. Other than that I don’t really have any particular plans for Storm at the moment (although of course I’m not the only person with an interest in it), aside from ideally avoiding leaving six years between releases again. I expect we can go back into bug-fixing mode there for a while.

From the Launchpad side, I’ve recently made progress on one of the other major Python 3 blockers (porting Bazaar code hosting to Breezy, coming soon). There are still some other significant blockers, the largest being migrating to Mailman 3, subvertpy fixes so that we can port code importing to Breezy as well, and porting the lazr.restful stack; but we may soon be able to reach the point where it’s possible to start running interesting subsets of the test suite using Python 3 and categorising the failures, at which point we’ll be able to get a much better idea of how far we still have to go. Porting a project with the best part of a million lines of code and around three hundred dependencies is always going to take a while, but I’m happy to be making progress there, both due to Python 2’s impending end of upstream support and so that eventually we can start using new language facilities.

Joey Hess: how to detect chef

Saturday 21st of September 2019 10:29:04 PM

If you want your program to detect when it's being run by chef, here's one way to do that.

sleep 1 while $ENV{PATH} =~ m#chef[^:]+/bin#;

This works because Chef's shell_out adds Gem.bindir to PATH, which is something like /opt/chefdk/embedded/bin.

You may want to delete the "sleep", which will make it run faster.

Would I or anyone ever really do this? Chef Inc's management seems determined to test the question, don't they.

Dirk Eddelbuettel: digest 0.6.21

Friday 20th of September 2019 11:38:00 PM

A new version of digest is just now arriving at CRAN (following a slight holdup over one likely spurious reverse dependency error), and I will send an updated package to Debian shortly as well.

digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, and spookyhash algorithms) permitting easy comparison of R language objects. It is a fairly widely-used package (currently listed at 795k downloads) as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation.

Every now and then open source work really surprises you. Out of nowhere arrived a very fine pull request by Matthew de Queljoe which adds a very clever function getVDigest() supplying a (much faster) vectorized wrapper for digest creation. We illustrate this in a quick demo vectorized.R that is included too. So if you call digest() in bulk, this will most likely be rather helpful to you. Matthew even did further cleanups and refactorings but we are saving that for a subsequent pull request or two.

CRANberries provides the usual summary of changes to the previous version.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Ben Hutchings: Linux Plumbers Conference 2019, part 2

Friday 20th of September 2019 11:09:28 PM

Here's the second chunk of notes I took at Linux Plumbers Conference earlier this month. Part 1 covered the Distribution kernels track.

Kernel Debugging Tools BoF

Moderators: George Wilson and Serapheim Dimitropoulos from Delphix; Omar Sandoval from Facebook

Details: https://linuxplumbersconf.org/event/4/contributions/539/

Problem: ability to easily anlyse failures in production (live system) or post-mortem (crash dump).

Debuggers need to:

  • Get consistent stack traces
  • Traverse and pretty-print memory structures
  • Easily introduce, extend. combine commands

Most people present use crash; one mentioned crash-python (aka pycrash) and one uses kgdb.

Pain points:

  • Tools not keeping up with kernel changes
  • Poor scripting support in crash

crash-python is a Python layer on top of a gdb fork. Uses libkdumpfile to decode compressed crash-dumps.

drgn (aka Dragon) is a debugger-as-a-library. Excels in introspectiion of live systems and crash-dumps, and covers both kernel and user-space. It can be extended through Python. As a library it can be imported and used from the Python REPL.

sdb is Deplhix's front-end to drgn, providing a more shell-like interactive interface. Example of syntax:

> modules | filter obj.refcnt.counter > 10 | member name

Currently it doesn't always have good type information for memory. A raw virtual address can be typed using the "cast" command in a pipeline. Hoping that BTF will allow doing better.

Allows defining pretty-print functions, though it appears these have to be explciitly invoked.

Answering tough questions:

  • Can I see any stacks with a specific function in? (bpftrace can do that on a live system, but there's no similar facility for crash dumps.)
  • What I/O is currently being issued?
  • Which files are currently being written?

Some discussion around the fact that drgn has a lot of code that's dependent on kernel version, as internal structures change. How can it be kept in sync with the kernel? Could some of that code be moved into the kernel tree?

Omar (I think) said that his approach was to make drgn support multiple versions of structure definitions.

Q: How does this scale to the many different kernel branches that are used in different distributions and different hardware platforms?

A: drgn will pick up BTF structure definitions. When BTF is available the code only needs to handle addition/removal of members it accesses.

Brendan Gregg made a plea to distro maintainers to enable BTF. (CONFIG_DEBUG_INFO_BTF).

Wayland BoF

Moderator: Hans de Goede of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/533/

Pain points and missing pieces with Wayland, or specifically GNOME Shell:

  • GNOME Shell is slower
  • Synergy doesn't work(?) - needs to be in the compositor
  • With Nvidia proprietary driver, mutter and native Wayland clients get GPU acceleration but X clients don't
  • No equivalent to ssh -X. Pipewire goes some way to the solution. The whole desktop can be remoted over RDP which can be tunnelled over SSH.
  • No remote login protocol like XDMCP
  • No Xvfb equivalent
  • Various X utilities that grab hot-keys don't have equivalents for Wayland
  • Not sure if all X's video acceleration features are implemented. Colour format conversion and hardware scaling are implemented.
  • Pointer movement becomes sluggish after a while (maybe related to GC in GNOME Shell?)
  • Performance, in general. GNOME Shell currently has to work as both a Wayland server and an X compositor, which limits the ability to optimise for Wayland.
IoT from the point of view of view of a generic and enterprise distribution

Speaker: Peter Robinson of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/439/

The good

Can now use u-boot with UEFI support on most Arm hardware. Much easier to use a common kernel on multiple hardware platforms, and UEFI boot can be assumed.

The bad

"Enterprise" and "industrial" IoT is not a Raspberry Pi. Problems result from a lot of user-space assuming the world is an RPi.

Is bluez still maintained? No user-space releases for 15 months! Upstream not convinced this is a problem, but distributions now out of synch as they have to choose between last release and arbitrary git snapshot.

Wi-fi and Bluetooth firmware fixes (including security fixes) missing from linux-firmware.git. RPi Foundation has improved Bluetooth firmware for the chip they use but no-one else can redistribute it.

Lots of user-space uses /sys/class/gpio, which is now deprecated and can be disabled in kconfig. libgpiod would abstract this, but has poor documentation. Most other GPIO libraries don't work with new GPIO UAPI.

Similar issues with IIO - a lot of user-space doesn't use it but uses user-space drivers banging GPIOs etc. libiio exists but again has poor documentation.

For some drivers, even newly added drivers, the firmware has not been added to linux-firmware.git. Isn't there a policy that it should be? It seems to be an unwritten rule at present.

Toolchain track

Etherpad: https://etherpad.net/p/LPC2019_TC/timeslider#5767

Security feature parity between GCC and Clang

Speaker: Kees Cook of Google

Details: https://linuxplumbersconf.org/event/4/contributions/398/

LWN article: https://lwn.net/Articles/798913/

Analyzing changes to the binary interface exposed by the Kernel to its modules

Speaker: Dodji Seketeli of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/399/

Wrapping system calls in glibc

Speakers: Maciej Rozycki of WDC

Details: https://linuxplumbersconf.org/event/4/contributions/397/

LWN article: https://lwn.net/Articles/799331/

More in Tux Machines

Red Hat Enterprise Linux 7 and CentOS 7 Get Important Kernel Security Update

Marked as important by Red Hat Product Security, the new Linux kernel security patch is here to fix a use-after-free flaw (CVE-2018-20856) discovered in the __blk_drain_queue() function in block/blk-core.c, as well as a heap overflow issue (CVE-2019-3846) discovered in the mwifiex_update_bss_desc_with_ie function in marvell/mwifiex/scan.c. It also addresses a heap overflow issue (CVE-2019-10126) discovered in the mwifiex_uap_parse_tail_ies function in drivers/net/wireless/marvell/mwifiex/ie.c and a Bluetooth flaw (CVE-2019-9506) that may lead to BR/EDR encryption key negotiation attacks (KNOB). Read more

Purism: Supplying the Demand

Thank you all for the continued support and remarkable demand for the Librem 5. As we’ve shared earlier, we are iterating through shipping batches. The purpose of doing so is to increment and improve with each batch toward mass production and share that story publicly. As a result, these earlier batches are limited in quantity as we move toward mass production. Publicly releasing iterated hardware at this level of transparency is extremely uncommon, but in nearly everything we do we try to lead by example. Forming as a Social Purpose Corporation, open sourcing all our software, having PureOS be FSF endorsed, securing the lower layers of computing, or manufacturing a revolutionary mobile phone from scratch… all have required sacrifice but are well worth it to provide people with a values-driven alternative to Big Tech. Read more Also: Purism Provides Update On Librem 5 Shipping, Known Issues

KDE Plasma 5.17 Desktop Environment Gets First Point Release with 40 Bug Fixes

Released last week on October 15th, the KDE Plasma 5.17 desktop environment introduces Night Color support on X11, fractional scaling on Wayland, HiDPI and multi-screen improvements, as well as the ability to support for managing and configuring Thunderbolt devices in System Settings. It also improves the notification system with a new Do Not Disturb mode that automatically detects presentations, Breeze GTK theme support for the Google Chrome and Chromium web browsers, Nvidia GPU stats in System Settings, and color scheme support for GTK and GNOME apps in the Breeze GTK theme. Read more

Ubuntu Touch OTA-11 Release

Ubuntu Touch is the privacy and freedom respecting mobile operating system by UBports. Today we are happy to announce the release of Ubuntu Touch OTA-11! OTA-11 is immediately available for all supported Ubuntu Touch devices. You can skip to How to get OTA-11 to get it right away if you're impatient, or read on to learn more about this release. We were calling this a "small release" originally. Our plan was to cover the backlog of pull requests that weren't quite ready for OTA-10. It turns out, that made this "small" update not small at all. Read more Also: Ubuntu Touch OTA-11 for Ubuntu Phones Brings Smarter Keyboard, Better Browsing UBports' Ubuntu Touch OTA-11 Released