Language Selection

English French German Italian Portuguese Spanish

Debian

Syndicate content
Planet Debian - https://planet.debian.org/
Updated: 7 hours 1 min ago

Colin Watson: Porting Storm to Python 3

Sunday 22nd of September 2019 07:56:42 AM

We released Storm 0.21 on Friday (the release announcement seems to be stuck in moderation, but you can look at the NEWS file directly). For me, the biggest part of this release was adding Python 3 support.

Storm is a really nice and lightweight ORM (object-relational mapper) for Python, developed by Canonical. We use it for some major products (Launchpad and Landscape are the ones I know of), and it’s also free software and used by some other folks as well. Other popular ORMs for Python include SQLObject, SQLAlchemy and the Django ORM; we use those in various places too depending on the context, but personally I’ve always preferred Storm for the readability of code that uses it and for how easy it is to debug and extend it.

It’s been a problem for a while that Storm only worked with Python 2. It’s one of a handful of major blockers to getting Launchpad running on Python 3, which we definitely want to do; stoq ended up with a local fork of Storm to cope with this; and it was recently removed from Debian for this and other reasons. None of that was great. So, with significant assistance from a large patch contributed by Thiago Bellini, and with patient code review from Simon Poirier and some of my other colleagues, we finally managed to get that sorted out in this release.

In many ways, Storm was in fairly good shape already for a project that hadn’t yet been ported to Python 3: while its internal idea of which strings were bytes and which text required quite a bit of untangling in the way that Python 2 code usually does, its normal class used for text database columns was already Unicode which only accepted text input (unicode in Python 2), so it could have been a lot worse; this also means that applications that use Storm tend to get at least this part right even in Python 2. Aside from the bytes/text thing, many of the required changes were just the usual largely-mechanical ones that anyone who’s done 2-to-3 porting will be familiar with. But there were some areas that required non-trivial thought, and I’d like to talk about some of those here.

Exception types

Concrete database implementations such as psycopg2 raise implementation-specific exception types. The inheritance hierarchy for these is defined by the Python Database API (DB-API), but the actual exception classes aren’t in a common place; rather, you might get an instance of psycopg2.errors.IntegrityError when using PostgreSQL but an instance of sqlite3.IntegrityError when using SQLite. To make things easier for applications that don’t have a strict requirement for a particular database backend, Storm arranged to inject its own virtual exception types as additional base classes of these concrete exceptions by patching their __bases__ attribute, so for example, you could import IntegrityError from storm.exceptions and catch that rather than having to catch each backend-specific possibility.

Although this was always a bit of a cheat, it worked well in practice for a while, but the first sign of trouble even before porting to Python 3 was with psycopg2 2.5. This release started implementing its DB-API exception types in a C extension, which meant that it was no longer possible to patch __bases__. To get around that, a few years ago I landed a patch to Storm to use abc.ABCMeta.register instead to register the DB-API exceptions as virtual subclasses of Storm’s exceptions, which solved the problem for Python 2. However, even at the time I landed that, I knew that it would be a porting obstacle due to Python issue 12029; Django ran into that as well.

In the end, I opted to refactor how Storm handles exceptions: it now wraps cursor and connection objects in such a way as to catch DB-API exceptions raised by their methods and properties and re-raise them using wrapper exception types that inherit from both the appropriate subclass of StormError and the original DB-API exception type, and with some care I even managed to avoid this being painfully repetitive. Out-of-tree database backends will need to make some minor adjustments (removing install_exceptions, adding an _exception_module property to their Database subclass, adjusting the raw_connect method of their Database subclass to do exception wrapping, and possibly implementing _make_combined_exception_type and/or _wrap_exception if they need to add extra attributes to the wrapper exceptions). Applications that follow the usual Storm idiom of catching StormError or any of its subclasses should continue to work without needing any changes.

SQLObject compatibility

Storm includes some API compatibility with SQLObject; this was from before my time, but I believe it was mainly because Launchpad and possibly Landscape previously used SQLObject and this made the port to Storm very much easier. It still works fine for the parts of Launchpad that haven’t been ported to Storm, but I wouldn’t be surprised if there were newer features of SQLObject that it doesn’t support.

The main question here was what to do with StringCol and its associated AutoUnicodeVariable. I opted to make these explicitly only accept text on Python 3, since the main reason for them to accept bytes was to allow using them with Python 2 native strings (i.e. str), and on Python 3 str is already text so there’s much less need for the porting affordance in that case.

Since releasing 0.21 I realised that the StringCol implementation in SQLObject itself in fact accepts both bytes and text even on Python 3, so it’s possible that we’ll need to change this in the future, although we haven’t yet found any real code using Storm’s SQLObject compatibility layer that might rely on this. Still, it’s much easier for Storm to start out on the stricter side and perhaps become more lenient than it is to go the other way round.

inspect.getargspec

Storm had some fairly complicated use of inspect.getargspec on Python 2 as part of its test mocking arrangements. This didn’t work in Python 3 due to some subtleties relating to bound methods. I switched to the modern inspect.signature API in Python 3 to fix this, which in any case is rather simpler with the exception of a wrinkle in how method descriptors work.

(It’s possible that these mocking arrangements could be simplified nowadays by using some more off-the-shelf mocking library; I haven’t looked into that in any detail.)

What’s next?

I’m working on getting Storm back into Debian now, which will be with Python 3 support only since Debian is in the process of gradually removing Python 2 module support. Other than that I don’t really have any particular plans for Storm at the moment (although of course I’m not the only person with an interest in it), aside from ideally avoiding leaving six years between releases again. I expect we can go back into bug-fixing mode there for a while.

From the Launchpad side, I’ve recently made progress on one of the other major Python 3 blockers (porting Bazaar code hosting to Breezy, coming soon). There are still some other significant blockers, the largest being migrating to Mailman 3, subvertpy fixes so that we can port code importing to Breezy as well, and porting the lazr.restful stack; but we may soon be able to reach the point where it’s possible to start running interesting subsets of the test suite using Python 3 and categorising the failures, at which point we’ll be able to get a much better idea of how far we still have to go. Porting a project with the best part of a million lines of code and around three hundred dependencies is always going to take a while, but I’m happy to be making progress there, both due to Python 2’s impending end of upstream support and so that eventually we can start using new language facilities.

Joey Hess: how to detect chef

Saturday 21st of September 2019 10:29:04 PM

If you want your program to detect when it's being run by chef, here's one way to do that.

sleep 1 while $ENV{PATH} =~ m#chef[^:]+/bin#;

This works because Chef's shell_out adds Gem.bindir to PATH, which is something like /opt/chefdk/embedded/bin.

You may want to delete the "sleep", which will make it run faster.

Would I or anyone ever really do this? Chef Inc's management seems determined to test the question, don't they.

Dirk Eddelbuettel: digest 0.6.21

Friday 20th of September 2019 11:38:00 PM

A new version of digest is just now arriving at CRAN (following a slight holdup over one likely spurious reverse dependency error), and I will send an updated package to Debian shortly as well.

digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, and spookyhash algorithms) permitting easy comparison of R language objects. It is a fairly widely-used package (currently listed at 795k downloads) as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation.

Every now and then open source work really surprises you. Out of nowhere arrived a very fine pull request by Matthew de Queljoe which adds a very clever function getVDigest() supplying a (much faster) vectorized wrapper for digest creation. We illustrate this in a quick demo vectorized.R that is included too. So if you call digest() in bulk, this will most likely be rather helpful to you. Matthew even did further cleanups and refactorings but we are saving that for a subsequent pull request or two.

CRANberries provides the usual summary of changes to the previous version.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Ben Hutchings: Linux Plumbers Conference 2019, part 2

Friday 20th of September 2019 11:09:28 PM

Here's the second chunk of notes I took at Linux Plumbers Conference earlier this month. Part 1 covered the Distribution kernels track.

Kernel Debugging Tools BoF

Moderators: George Wilson and Serapheim Dimitropoulos from Delphix; Omar Sandoval from Facebook

Details: https://linuxplumbersconf.org/event/4/contributions/539/

Problem: ability to easily anlyse failures in production (live system) or post-mortem (crash dump).

Debuggers need to:

  • Get consistent stack traces
  • Traverse and pretty-print memory structures
  • Easily introduce, extend. combine commands

Most people present use crash; one mentioned crash-python (aka pycrash) and one uses kgdb.

Pain points:

  • Tools not keeping up with kernel changes
  • Poor scripting support in crash

crash-python is a Python layer on top of a gdb fork. Uses libkdumpfile to decode compressed crash-dumps.

drgn (aka Dragon) is a debugger-as-a-library. Excels in introspectiion of live systems and crash-dumps, and covers both kernel and user-space. It can be extended through Python. As a library it can be imported and used from the Python REPL.

sdb is Deplhix's front-end to drgn, providing a more shell-like interactive interface. Example of syntax:

> modules | filter obj.refcnt.counter > 10 | member name

Currently it doesn't always have good type information for memory. A raw virtual address can be typed using the "cast" command in a pipeline. Hoping that BTF will allow doing better.

Allows defining pretty-print functions, though it appears these have to be explciitly invoked.

Answering tough questions:

  • Can I see any stacks with a specific function in? (bpftrace can do that on a live system, but there's no similar facility for crash dumps.)
  • What I/O is currently being issued?
  • Which files are currently being written?

Some discussion around the fact that drgn has a lot of code that's dependent on kernel version, as internal structures change. How can it be kept in sync with the kernel? Could some of that code be moved into the kernel tree?

Omar (I think) said that his approach was to make drgn support multiple versions of structure definitions.

Q: How does this scale to the many different kernel branches that are used in different distributions and different hardware platforms?

A: drgn will pick up BTF structure definitions. When BTF is available the code only needs to handle addition/removal of members it accesses.

Brendan Gregg made a plea to distro maintainers to enable BTF. (CONFIG_DEBUG_INFO_BTF).

Wayland BoF

Moderator: Hans de Goede of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/533/

Pain points and missing pieces with Wayland, or specifically GNOME Shell:

  • GNOME Shell is slower
  • Synergy doesn't work(?) - needs to be in the compositor
  • With Nvidia proprietary driver, mutter and native Wayland clients get GPU acceleration but X clients don't
  • No equivalent to ssh -X. Pipewire goes some way to the solution. The whole desktop can be remoted over RDP which can be tunnelled over SSH.
  • No remote login protocol like XDMCP
  • No Xvfb equivalent
  • Various X utilities that grab hot-keys don't have equivalents for Wayland
  • Not sure if all X's video acceleration features are implemented. Colour format conversion and hardware scaling are implemented.
  • Pointer movement becomes sluggish after a while (maybe related to GC in GNOME Shell?)
  • Performance, in general. GNOME Shell currently has to work as both a Wayland server and an X compositor, which limits the ability to optimise for Wayland.
IoT from the point of view of view of a generic and enterprise distribution

Speaker: Peter Robinson of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/439/

The good

Can now use u-boot with UEFI support on most Arm hardware. Much easier to use a common kernel on multiple hardware platforms, and UEFI boot can be assumed.

The bad

"Enterprise" and "industrial" IoT is not a Raspberry Pi. Problems result from a lot of user-space assuming the world is an RPi.

Is bluez still maintained? No user-space releases for 15 months! Upstream not convinced this is a problem, but distributions now out of synch as they have to choose between last release and arbitrary git snapshot.

Wi-fi and Bluetooth firmware fixes (including security fixes) missing from linux-firmware.git. RPi Foundation has improved Bluetooth firmware for the chip they use but no-one else can redistribute it.

Lots of user-space uses /sys/class/gpio, which is now deprecated and can be disabled in kconfig. libgpiod would abstract this, but has poor documentation. Most other GPIO libraries don't work with new GPIO UAPI.

Similar issues with IIO - a lot of user-space doesn't use it but uses user-space drivers banging GPIOs etc. libiio exists but again has poor documentation.

For some drivers, even newly added drivers, the firmware has not been added to linux-firmware.git. Isn't there a policy that it should be? It seems to be an unwritten rule at present.

Toolchain track

Etherpad: https://etherpad.net/p/LPC2019_TC/timeslider#5767

Security feature parity between GCC and Clang

Speaker: Kees Cook of Google

Details: https://linuxplumbersconf.org/event/4/contributions/398/

LWN article: https://lwn.net/Articles/798913/

Analyzing changes to the binary interface exposed by the Kernel to its modules

Speaker: Dodji Seketeli of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/399/

Wrapping system calls in glibc

Speakers: Maciej Rozycki of WDC

Details: https://linuxplumbersconf.org/event/4/contributions/397/

LWN article: https://lwn.net/Articles/799331/

Bernhard R. Link: Firefox 69 dropped support for <keygen>

Friday 20th of September 2019 07:46:29 PM
With version 69, firefox removed the support for the <keygen> feature to easily deploy TLS client certificates.
It's kind of sad how used I've become to firefox giving me less and less reasons to use it...

Enrico Zini: Upgrading LineageOS 14 to 16

Friday 20th of September 2019 02:18:44 PM

The LineageOS updater notified me that there will be no more updates for LineageOS 14, because now development on my phone happens on LineageOS 16, so I set aside some time and carefully followed the upgrade instructions.

I now have a phone with Lineageos 16, but the whole modem subsystem does not work.

Advice on #lineageos was that "the wiki instructions are often a bit generic.. offical thread often has the specific details".

Official thread is here, and the missing specific detail was "Make sure you had Samsung's Oreo firmware bootloader and modem before installing this.".

It looks like nothing ever installed firmware updates, since the Android that came with my phone ages ago. I can either wipe everything and install a stock android to let it do the upgrade, then replace it with LineageOS, or try a firmware upgrade.

This link has instructions for firmware upgrades using haimdall, which is in Debian, instead of Odin, which is in Windows.

Finding firmwares is embarassing. They only seem to be available from links on shady download sites, or commercial sites run by who knows whom. I verify sha256sums on LineageOS images, F-Droid has reproducible builds, but at the base of this wonderful stack there's going to be a blob downloaded off some forum on the internet.

In this case, this link points to some collection of firmware blobs.

I downloaded the pack and identified the ones for my phone, then unpacked the tar files and uncompressed the lz4 blobs.

With heimdall, I identified the mapping from partition names to blob names:

heimdall print-pit --no-reboot

Then I did the flashing:

heimdall flash --resume --RADIO modem.bin --CM cm.bin --PARAM param.bin --BOOTLOADER sboot.bin

The first time flashing didn't work, and I got stuck in download mode. This explains how to get out of download mode (power + volume down for 10s).

Second attempt worked fine, and now I have a working phone again:

heimdall flash --RADIO modem.bin --CM cm.bin --PARAM param.bin --BOOTLOADER sboot.bin

Louis-Philippe Véronneau: Praise Be CUPS Driverless Printing

Thursday 19th of September 2019 04:30:13 PM

Last Tuesday, I finally got to start updating $work's many desktop computers to Debian Buster. I use Puppet to manage them remotely, so major upgrades basically mean reinstalling machines from scratch and running Puppet.

Over the years, the main upgrade hurdle has always been making our very large and very complicated printers work on Debian. Unsurprisingly, the blog posts I have written on that topic are very popular and get me a few 'thank you' emails per month.

I'm very happy to say, thanks to CUPS Driverless Printing (CUPS 2.2.2+), all those trials and tribulations are finally over. Printing on Buster just works. Yes yes, even color booklets printed on 11x17 paper folded in 3 stapled in the middle.

Xerox Altalink C8045 and Canon imageRUNNER ADVANCE C5550i

Although by default the Xerox Altalink C8045 comes with IPP Everywhere enabled, I wasn't able to print in color until I enabled AirPrint. I also had to update the printer to firmware version 101.002.008.274001 to make the folding and stapling features more stable.

As for the Canon imageRUNNER ADVANCE C5550i, it seems it doesn't support IPP Everywhere. After enabling AirPrint manually, everything worked perfectly.

Both printers now work wonderfully with all our computers, without the need to resort to strange (and broken) proprietary drivers or aweful 32bit libraries.

Note that if you run a firewall locally, you will need to open port 5353 UDP for machines to resolve .local addresses via mDNS. This had me bumped for a while.

Praises

Packaging CUPS and all the related CUPS bits for Debian isn't an easy task. I'm so glad I don't have to touch that side of CUPS. Three cheers to Didier Raboud, Till Kamppeter and to the Debian Printing Team.

Many, many thanks to Brian Potkin for the work he did to document CUPS Driverless printing and AirPrint on the Debian wiki. If we ever meet, I definitely owe you a pint.

Finally, well, thanks to Apple. (I never thought I'd ever say that)

  1. That took me a few hours. Yes. 

Thomas Lange: FAI 5.8.7 and new ISO images using Debian 10

Thursday 19th of September 2019 03:01:42 PM

The new FAI release 5.8.7 now supports apt keys in files called package_config/CLASS.gpg. Before we only supported .asc files. fai-mirror has a new option -V, which checks if variables are used in package_config/ and uses variable definitions from class/.var.

I've also created new ISO images, which now install Debian 10 by default. They are available from

https://fai-project.org/fai-cd

If you need a newer kernel, checkout the FAI.me service which can also build ISO images using the kernel from backports which currently is 5.2.

FAI

Rapha&#235;l Hertzog: Freexian’s report about Debian Long Term Support, August 2019

Thursday 19th of September 2019 10:04:27 AM

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In August, 212.5 work hours have been dispatched among 13 paid contributors. Their reports are available:

  • Adrian Bunk got 8h assigned (plus 18 extra hours from July), but did nothing, thus he is carrying over 26h to September.
  • Ben Hutchings did 20 hours (out of 20 hours allocated).
  • Brian May did 10 hours (out of 10 hours allocated).
  • Chris Lamb did 18 hours (out of 18 hours allocated).
  • Emilio Pozuelo Monfort did 31 hours (out of 21.75h assigned plus 14.5 extra hours from July), thus he is carrying over 5.25h to September).
  • Hugo Lefeuvre did 30.5 hours (out of 21.75 hours allocated, plus 8.75 extra hours from July).
  • Jonas Meurer did 0.5 hours (out of 10, thus carrying 9.5h to September).
  • Markus Koschany did 21.75 hours (out of 21.75 hours allocated).
  • Mike Gabriel did 24 hours (out of 21.75 hours allocated plus 10 extra hours from July, thus carrying over 7.75h to September).
  • Ola Lundqvist got 8h assigned (plus 8 extra hours from August), but did nothing and gave back 8h, thus he is carrying over 8h to September.
  • Roberto C. Sanchez did 8 hours (out of 8 hours allocated).
  • Sylvain Beucler did 21.75 hours (out of 21.75 hours allocated).
  • Thorsten Alteholz did 21.75 hours (out of 21.75 hours allocated).
Evolution of the situation

August was more or less a normal month, a bit still affected by summer in the area where most contributors live: so one contributor is still taking a break (thus we only had 13, not 14), two contributors were distracted by summer events and another one is still in training.

It’s been a while that we haven’t welcomed a new LTS sponsors. Nothing worrisome at this point as few sponsors are stopping, but after 5 years, some have moved on so it would be nice to keep finding new ones as well. We are still at 215 hours sponsored by month.

The security tracker currently lists 42 packages with a known CVE and the dla-needed.txt file has 39 packages needing an update.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

Louis-Philippe Véronneau: Archiving 20 years of online content

Thursday 19th of September 2019 04:00:00 AM

Last Spring at work I was tasked with archiving all of the digital content made by Association pour une Solidarité Syndicale Étudiante (ASSÉ), a student union federation that was planned to shut down six months later.

Now that I've done it, I have to say it was quite a demanding task: ASSÉ was founded in 2001 and neither had proper digital archiving policies nor good web practices in general.

The goal was not only archiving those web resources, but also making sure they were easily accessible online too. I thus decided to create a meta site regrouping and presenting all of them.

All in all, I archived:

  • a Facebook page
  • a Twitter account
  • a YouTube account
  • multiple ephemeral websites
  • old handcrafted PHP4 websites that I had to partially re-write
  • a few crummy Wordpress sites
  • 2 old phpBB Forum using PHP3
  • a large Mailman2 mailing list
  • a large Yahoo! Group mailing list

Here are the three biggest challenges I faced during this project:

The Twitter API has stupid limitations

The Twitter API won't give you more than an account's last 3000 posts. When you need to automate the retrieval of more than 5500 tweets, you know you're entering a world of pain.

Long story short, I ended up writing this crummy shell script to parse the HTML, statify all the Twitter links and push the resulting code to a Wordpress site using Ozh' Tweet Archive Theme. The URL list was generated using the ArchiveTeam's web crawler.

Of course, once done I made the Wordpress into a static website. I personally think the result looks purty.

Here's the shell script I wrote - displayed here for archival purposes only. Let's pray I don't ever have to do this again. Please don't run this, as it might delete your grandma.

cat $1 | while read line do # get the ID id=$(echo $line | sed 's@https://mobile.twitter.com/.\+/status/@@') # download the whole HTML page html=$(curl -s $line) # get the date date=$(echo "$html" | grep -A 1 '<div class="metadata">' | grep -o "[0-9].\+20[0-9][0-9]" | sed 's/ - //' | date -f - +"%F %k:%M:%S") # extract the tweet tweet=$(echo "$html" | grep '<div class="dir-ltr" dir="ltr">') # we strip the HTML tags for the title title=$(echo $tweet | sed -e 's/<[^>]*>//g') # get CSV list of tags tags=$(echo "$tweet" | grep -io "\#[a-z]\+" | sed ':a;N;$!ba;s/\n/,/g') # get a CSV list of links links=$(echo "$tweet" | grep -Po "title=\"http.*?>" | sed 's/title=\"//; s/">//' | sed ':a;N;$!ba;s/\n/,/g') # get a CSV list of usernames usernames=$(echo "$tweet" | grep -Po ">\@.*?<" | sed 's/>//; s/<//' | sed ':a;N;$!ba;s/\n/,/g') image_link=$(echo "$html" | grep "<img src=\"https://pbs.twimg.com/media/" | sed 's/:small//') # remove twitter cruft tweet=$(echo $tweet | sed 's/<div class="dir-ltr" dir="ltr"> /<p>/' | perl -pe 's@<a href="/hashtag.*?dir="ltr">@<span class="hashtag hashtag_local">@g') # expand links if [ ! -z $links ] then IFS=',' read -ra link <<< "$links" for i in "${link[@]}" do tweet=$(echo $tweet | perl -pe "s@<a href=\"*.+?rel=\"nofollow noopener\"dir=\"ltr\"data-*.+?</a>@<a href='$i'>$i</a>@") done fi # replace hashtags by links if [ ! -z $tags ] then IFS=',' read -ra tag <<< "$tags" for i in "${tag[@]}" do plain=$(echo $i | sed -e 's/#//') tweet=$(echo $tweet | sed -e "s@$i@#<a href=\"https://oiseau.asse-solidarite.qc.ca/index.php/tag/$plain\">$plain@") done fi # replace usernames by links if [ ! -z $usernames ] then IFS=',' read -ra username <<< "$usernames" for i in "${username[@]}" do plain=$(echo $i | sed -e 's/\@//') tweet=$(echo $tweet | perl -pe "s@<a href=\"/$plain.*?</a>@<span class=\"username username_linked\">\@<a href=\"https://twitter.com/$plain\">$plain</a></span>@i") done fi # replace images tweet=$(echo $tweet | perl -pe "s@<a href=\"http://t.co*.+?data-pre-embedded*.+?</a>@<span class=\"embed_image embed_image_yes\">$image_link</span>@") echo $tweet | sudo -u twitter wp-cli post create - --post_title="$title" --post_status="publish" --tags_input="$tag" --post_date="$date" > tmp post_id=$(grep -Po "[0-9]{4}" tmp) sudo -u twitter wp-cli post meta add $post_id ozh_ta_id $id echo "$post_id created" rm tmp done Does anyone ever update phpBBs?

What's worse than a phpBB forum? Two phpBB 2.0.x forums using PHP3 and last updated in 2006.

I had to resort to unholy methods just to be able to get those things running again to be able to wget the crap out of them.

By the way, the magic wget command to grab a whole website looks like this:

wget --mirror -e robots=off --page-requisites --adjust-extension -nv --base=./ --convert-links --directory-prefix=./ -H -D www.foo.org,foo.org http://www.foo.org/

Depending on the website you are trying to archive, you might have to play with other obscure parameters. I sure had to. All the credits for that command goes to Koumbit's wiki page on the dark arts of website statification.

Archiving mailing lists

mailman2 is pretty great. You can get a dump of an email list pretty easily and mailman3's web frontend, the lovely hyperkitty, is well, lovely. Importing a legacy mailman2 mbox went without a hitch thanks to the awesome hyperkitty_import importer. Kudos to the Debian Mailman Team for packaging this in Debian for us.

But what about cramming a Yahoo! Group mailing list in hyperkitty? I wouldn't recommend it. After way too many hours spent battling character encoding errors I just decided people that wanted to read obscure emails from 2003 would have to deal with broken accents and shit. But hey, it kinda works!

Oh, and yes, archiving a Yahoo! Group with an old borken Perl script wasn't an easy task. Hell, I kept getting blacklisted by Yahoo! for scraping too much data to their liking. I ended up patching together the results of multiple runs over a few weeks to get the full mbox and attachments.

By the way, if anyone knows how to tell hyperkitty to stop at a certain year (i.e. not display links for 2019 when the list stopped in 2006), please ping me.

Shirish Agarwal: Dissonance, Rakhigiri, one-party system

Tuesday 17th of September 2019 10:23:51 PM

I heard the term dissonance or Cognitive dissonance about a year, year and a half back. This is the strategy that BJP used to win in 2014 and in 2019. How much part did EVM’s play and how much part did the cognitive dissonance work is hard to tell and define but both have made irreperable damage to not just the economy, social fabric, institutions, the constitution but the idea of India itself. India, the country which has thousands of different cultures, castes, are for one reason or the other put into hate which seems to have no end. But before going into the politics, let me share one beautiful video which got played on social media and also quickly forgotten.

Dhol Tasha in Ganesh Chaturthi

The gentleman playing that drum is Mr. Greg Ellis. While we had Ganesh Chaturthi just a while back, the most laid-back I have seen in my whole 40+ years. While there were other dhol tasha plays which played for e.g. these two videos

and

I do like the first one though which represents what is and used to be call jugalbandi. This is where two artists talk to each other having conversation while playing their instruments. For e.g. Zakir Hussain Sahab and Shivamani and many others. but that perhaps is a story for another day.

The Science and Politics of the 4500 year old Rakhigarhi woman.

This story was broken couple of weeks back. The best way or person to perhaps explain it in plain english would perhaps be Shekhar Gupta . From the discussions you see it is much heated and contested topic.

The study that was done can be found at https://reich.hms.harvard.edu/sites/reich.hms.harvard.edu/files/inline-files/2019_Cell_ShindeNarasimhan_Rakhigarhi.pdf and https://reich.hms.harvard.edu/sites/reich.hms.harvard.edu/files/inline-files/2019_Science_NarasimhanPatterson_CentralSouthAsia.pdf . In fact you can see whole series of papers on the same topic https://reich.hms.harvard.edu/publications . Now the Indian nationalist narrative movement is that Aryan migration does not take place while others say it’s the opposite. One can look at a bit more background on the story in the Indian Express article which came a few months before, before the whole controversy erupted.

One of the best things that has happened is the ability to get the DNA from a 4500 year old specimen. This is Priya Moorani’s github repo. which helped them get the genome data. I am sure some of the Debian folks may be interested in such kind of forensic anthropology. I know I had shared bits of its previously as well but it didn’t fit well with me as I had hadn’t shared the whole narrative behind it and the positions that people have taken.

The Economy

The Economy is in a bad shape in India is in a bad shape is not a hidden thing anymore but India Inc. had been keeping quiet. Apart from Ratan Tata and Mahindra who quipped about the R-word (Recession) only two ladies seems to be able a spade, a spade and both of them called on it on NDTV . Both these ladies are wealth-creators and have made India proud, one of them even benefiting from the rupee crash but even then their heart in the right place.

The Madame above is Ms. Kiran Majumdar Shaw who is a billionaire who makes her money selling generic active pharmaceutical ingredients and other bio products to advanced countries. Businessworld had done stories on pharmaceutical development, engineering about 6-7 years back when it was an emerging industry and lot of startups had come into the field. While it is a pretty competitive field with about 60 odd companies and biocon itself shedding quite a bit of its price on the stock exchange and still she has the guts to criticize the Modi Govt.

The other is Shobhana Kamineni of Apollo Hospitals, which is in superspeciality hospitals. Both the ladies are into medicine field and both are in the high-risk and high-reward game and still they are able to speak their mind. What is and was shocking for many of us is the kind of numbers she had shared of people emigrating to elsewhere in the last 5 odd years which perhaps wasn’t the case before. In fact, couple of days back both Indian Express and the Wire ran stories showing how millionaires and billionaires are moving out of the country. This hasn’t happened in a day but over a 5 year period. While people are saying that the money is being spent on trael and education, in fact it is due to tax harrassment and structural issues in the broader economy that wealth-creators are leaving. Of course for the rest of us it is the Hindu-Muslim topics but that is another story altogether.

The Hindu-Muslim polarization

When I talk to most of the Bhakts or people who have a militant belief of the way BJP is going, one of the questions I ask is, who are these muslims, are they the ones who came from Arab or are these converted Muslims, Christians, Buddhists, Sikhs etc. Most of them were and our own people who choose to convert to a religion because they saw upward mobility in that religion. So in a way the whole debate collapses. The saddest part is whether it is Hindu-Muslim polarization or Hindu-Sikh polarization it is always one brother beating the other. Also, studies showed that in the 2002 Gujarat riots, the ones who took part in the riots were from the lowest strata of the society because by killing the Muslims, they got prestige and money in the society. While I’m sure I have shared the studies before on the site, but if not shared, please let me know and will update the same accordingly. It is somewhere in my bookmarks somewhere

Bits from Debian: New Debian Developers and Maintainers (July and August 2019)

Tuesday 17th of September 2019 03:30:00 PM

The following contributors got their Debian Developer accounts in the last two months:

  • Keng-Yu Lin (kengyu)
  • Judit Foglszinger (urbec)

The following contributors were added as Debian Maintainers in the last two months:

  • Hans van Kranenburg
  • Scarlett Moore

Congratulations!

Antoine Beaupré: FSF resignations

Tuesday 17th of September 2019 02:58:00 PM

I have been hesitant in renewing my membership to the Free Software Foundation for a while, but now I never want to deal with the FSF until Richard Stallman, president and founder of the free software movement, resigns. So, like many people and organizations, I have written this letter to cancel my membership. (Update: RMS resigned before I even had time to send this letter, but I publish here to share my part of this story.)

My encounters with a former hero

I had the (mis)fortune of meeting rms in person a few times in my life. The first time was at an event we organized for his divine visit to Montreal in 2005. I couldn't attend the event myself, but I had the "privilege" of having dinner with rms later during the week. Richard completely shattered any illusion I had about him as a person. He was arrogant, full of himself, and totally uninterested in the multitude of young hackers he was meeting in his many travels, apart from, of course, arguing with them about proper wording and technicalities. Even though we brought him to the fanciest vegetarian restaurant in town, he got upset because the restaurant was trying to make "fake meat" meals. Somehow my hero, who wrote the GNU manifesto that inspired me to make free software a life goal, has spoiled a delicious meal by being such an ungrateful guest. I would learn later that Stallman has rock star level requirements, with "vegetarian meals served just so" being only one exception out of many. (I don't mind vegetarians of course: I've been a vegetarian for more than 20 years now, but I will never refuse vegetarian food given to me.)

The second time was less frustrating: it was in 2006 during the launch of the GPLv3 discussion draft, an ambitious project to include the community in the rewrite of the GPLv2. Even though I was deeply interested in the legal implications of the changes, everything went a bit over my head and I felt left out of a process that was supposedly designed to include legal geeks like me. At best, I was able to assist Stallman's assistant as she skidded over icy Boston sidewalks with a stereotypical (and maybe a little machismo, I must admit) Canadian winter assurance. At worst, I burned liters of fuel to drive me and some colleagues over the border to see idols speak on a stage.

Finally, I somehow got tangled up with rms in a hallway conversation about open hardware and wireless chipsets at LibrePlanet 2017, the FSF's yearly conference. I forgot the exact details, but we were debating whether or not legislation that forbids certain wireless chipsets to be open was legitimate or not.

(For some reason, rms has ambiguous opinions about "hardware freedom" and sees a distinction between software that runs on a computer (as "in the CPU") and software that is embedded in the hardware, etched into electronic circuits. The fact that this is a continuum that has various in-between incarnations ("firmware", ASIC, FPGA) seems to escape his analysis. But that is besides the point here.)

We "debated" this for a while, but for people who don't know, debating with rms is a little bit like talking with a three year old: they have their deeply rooted opinion, they might recognize you have one as well (if your lucky), but they will generally ignore whatever it is you non-sensical adult are saying because it's incomprehensible anyways. With a three year old, it's kind of hilarious (until they spill an bottle full of vanilla on the floor), but with an adult, it's kind of aggravating and makes you feel like an idiot for even trying.

I mention this anecdote because it's a good example of how Stallman doesn't think rules apply to him. Simple, informal rules like listening to people you're talking to seem like basic courtesy, but rms is above such mundane things. If this was just a hallway conversation, I wouldn't mind that much: after all, I don't need to talk to Richard Stallman. But at LibrePlanet (and in fact anywhere), he believes it is within his prerogative to interrupt any discussion or talk around him . I was troubled by the FSF's silence on Eric Schultz's request for safety at Libre Planet: while I heard the FSF privately reached out to Eric, nothing seemed to have been done to curb Stallman's attitude in public. This is the reason why I haven't returned to Boston for LibrePlanet since then, even though I have dear friends that live there and were deeply involved in the organization.

The final straw before this week's disclosurse was an event in Quebec city where Stallman was speaking at a conference. A friend of mine asked a question involving his daughter as an example user. Stallman responded to the question by asking my friend if he could meet his (underage) daughter, with obvious obscene undertones. Everyone took this as a joke, but, in retrospect, it was just horrible and I had come to conclude that Stallman was now a liability to the free software movement. I just didn't know what to do back then. I wish I had done something.

Why I am resigning from the FSF

Those events at LibrePlanet were the first reason why I haven't renewed my membership yet. But now I want to formally cancel my membership with the FSF because its president went over his usual sexism and weird pedophilia justification from the past. I first treated those as an abhorrent eccentricity or at best an unfortunate intellectual posture, but rms has gone way beyond this position now. Now rms has joined the rank of rape apologists in the Linux kernel development community, an inexcusable position in our community that already struggles too much with issues of inclusion, respect, and just being nice with each other. I am not going to go into details that are better described by this courageous person, but needless to say that this kind of behavior is inexcusable from anyone, and particularly from an historical leader. Stallman did respond to the accusations, but far from issuing an apology, he said his statements were "mischaracterised"; something that looks to me like a sad caricature.

I do not want to have anything to do with the FSF anymore. I don't know if they would be able to function without Stallman, and frankly at this point, I don't care: they have let this gone on for too long. I know how much rms contributed to the free software movement: he wrote most of Emacs, GCC and large parts of the GNU system so many people use on their desktops. I am grateful for that work, but that was a long time ago and this is now. As others have said, we don't need to replace rms. We need a world where such leaders are not necessary, because rock stars too easily become abusers.

Stallman is just the latest: our community is filled with obnoxious leaders like this. It seems our community leaders are (among other things) either assholes, libertarian gun freaks, or pedophilia apologists and sexists. We tolerate their abuse because we somehow believe they are technically exceptional. They aren't: they're just hard-working and privileged. But even if they would be geniuses, but as selamie says:

For a moment, let’s assume that someone like Stallman is truly a genius. Truly, uniquely brilliant. If that type of person keeps tens or even hundreds of highly intelligent but not ‘genius’ people out of science and technology, then they are hindering our progress despite the brilliance.

Or, as Banksy says:

We don't need any more heroes.

We just need someone to take out recycling.

I wish Stallman would just retire already. He's done enough good work for a lifetime, now he's bound to just do more damage.

Update: Richard Stallman resigned from the FSF and from MIT ("due to pressure on MIT and me"), still dodging responsability and characterizing the problem as "a series of misunderstandings and mischaracterizations". Obviously, this man cannot be reformed and we need to move on. Those events happened before I even had time to actually send this letter to the FSF, so I guess I might renew my membership after all. I'll hold off until LibrePlanet, however, we'll see what happens there... In the meantime, I'll see how I can help my friends left the FSF because they must be living through hell now.

Molly de Blanc: Thinkers

Tuesday 17th of September 2019 09:16:17 AM

Free and open source software, ethical technology, and digital autonomy have a number of great thinkers, inspiring leaders, and hard working organizations. I see two discussions occurring now that I feel the need to address: What will we do next? Who will our new great leader be?

The thing is, we don’t need to do something new next, and we don’t need to find new leader.

Organizations and individuals have been doing amazing work in our sphere for more than thirty years. We only need to look at the works of groups like Public Labs, OpenStreetMap, and Wikimedia to see where the future of our work lies: applying the principles of user freedom to create demonstrable change, build equity, and fight for justice. I am positively inspired by the GNOME community and their dedication to building software for people in every country, of every ability, and of every need. Outreachy and projects and companies that participate in Outreachy internships are working hard to build the future of community that we want to see.

Deb Nicholson recently reminded me that we cannot build a principled future where people are excluded from the process of building it. She also pointed out that once we’ve have a techno-utopia, it will include everyone, because it needs to. This utopia is built on ideas, but it is also built by plumbers — by people doing work on the ground with those ideas.

Deb Nicholson is another inspiration to me. I’ve been lucky enough to know her since 2010, when she graciously began to mentor me. I now consider her both a mentor and a dear friend. Her ideas are innovative, her principles hard, and her vision wide.

Deb is one of the many  people who have helped and continue to help shape my ideas, teach me things. Allison Randall, Asheesh Laroia, Christopher Lemmer-Webber, Daniel Khan Gilmore, Elana Hashman, Gabriella Coleman, Jeffrey Warren, Karen Sandler, Karl Fogel, Stefano Zacchiroli — these are just a few of the individuals who have been necessary figures in my life.

We don’t need to find new leaders and thinkers because they’re already here. They’ve been here, thinking, writing, speaking, and doing for years.

What we need to do is listen to their voices.

As I see people begin to discuss the next president of the Free Software Foundation, they do so in a context of asking who will be leading the free software movement. The free software movement is more than the FSF and it’s more than any given individual. We don’t need to go in search of the next leader, because there are leaders who work every day not just for our digital rights, but for a better world. We don’t need to define a movement by one man, nor should we do so. We instead need to look around us and listen to what is already happening.

Steve Kemp: A slack hack

Monday 16th of September 2019 09:00:00 PM

So recently I've been on-call, expected to react to events around the clock. Of course to make it more of a challenge alerts are usually raised via messages to a specific channel in slack which come from a variety of sources. Let's pretend I'm all retro/hip and I'm using IRC instead.

Knowing what I'm like I knew there was essentially zero chance a single beep on my phone, from the slack/irc app, would wake me up. So I spent a couple of hours writing a simple bot:

  • Connect to the server.
  • Listen for messages.
  • When an alert is posted in the channel:
    • Trigger a voice-call via the twilio API.

That actually worked out really, really, really well. Twilio would initiate a call to my mobile which absolutely would, could, and did wake me up. I did discover a problem pretty quickly though; too many phone-calls!

Imagine something is broken. Imagine a notice goes to your channel, and then people start replying to it:

Some Bot: Help! Stuff is broken! I'm on Fire!! :fire: :hot: :boom: Colleague Bob: Is this real? Colleague Ann: Can you poke Chris? Colleage Chris: Oh dears, woe is me.

The first night I was on call I got a phone call. Then another. Then another. Even I replied to the thread/chat to say "Yeah I'm on it". So the next step was to refine my alerting:

  • If there is a message in the channel
    • Which is not from Bob
    • Which is not from Steve
    • Which is not from Ann
    • Which is not from Chris
    • Which doesn't contain the text "common false-positive"
    • Which doesn't contain the text "backup completed"
  • Then make a phone-call.

Of course the next problem was predictable enough, so the rules got refined:

  • If the time is between 7PM and 7AM raise the alert.
  • Unless it is the weekend in which case we alert regardless of the time of day.

So I had a growing set of rules. All encoded in my goloang notification application. I moved some of them to JSON (specificially a list of users/messages to ignore) but things like the time of day were harder to move.

I figured I shouldn't be hardwiring these things. So last night put together a simple filter-library, an evaluation engine, in golang to handle them. Now I can load a script and filter things out much more dynamically. For example assume I have the following struct:

type Message struct { Author string Channel string Message string .. }

And an instance of that struct named message, I can run a user-written script against that object:

// Create a new eval-filter eval, er := evalfilter.New( "script goes here ..." ) // Run it against the "message" object out, err := eval.Run( message )

The logic of reacting now goes inside that script, which is hopefully easy to read - but more importantly can be edited without recompiling the application:

// // This is a filter script: // // return false means "do nothing". // return true means initiate a phone-call. // // // Ignore messages on channels that we don't care about // if ( Channel !~ "_alerts" ) { return false; } // // Ignore messages from humans who might otherwise write in our channels // of interest. // if ( Sender == "USER1" ) { return false; } // Steve if ( Sender == "USER2" ) { return true; } // Ann if ( Sender == "USER3" ) { return false; } // Bob // // Is it a weekend? Always alert. // if ( IsWeekend() ) { return true ; } // // OK so it is not a weekend. // // We only alert if 7pm-7am // // The WorkingHours() function returns `true` during working hours. // if ( WorkingHours() ) { return false ; } // // OK by this point we should raise a call: // // * The message was NOT from a colleague we've filtered out. // * The message is upon a channel with an `_alerts` suffix. // * It is not currently during working hours. // * And we already handled weekends by raising calls above. // return true ;

If the script returns true I initiate a phone-call. If the script returns false we ignore the message/event.

The alerting script itself is trivial, and probably non-portable, but the filtering engine is pretty neat. I can see a few more uses for it, even without it having nested blocks and a real grammar. So take a look, if you like:

Neil McGovern: GNOME relationship with GNU and the FSF

Monday 16th of September 2019 06:05:47 PM

On Saturday, I wrote an email to the FSF asking them to cancel my membership. Other people who I greatly respect are doing the same. This came after the president of the FSF made some pretty reprehensible remarks saying that the “most plausible scenario is that [one of Epstein’s underage victims] presented themselves as entirely willing” while being trafficked. This isn’t the only incident, but it is the straw that broke the camel’s back.

In my capacity as the Executive Director of the GNOME Foundation, I have also written to the FSF. One of the most important parts of my role is to think of the well being of our community and the GNOME mission. One of the GNOME Foundation’s strategic goals is to be an exemplary community in terms of diversity and inclusion. I feel we can’t continue to have a formal association with the FSF or the GNU project when its main voice in the world is saying things that hurt this aim.

I greatly admire the work of FSF staffers and volunteers, but have now reached the point of concluding that the greatest service to the mission of software freedom is for Richard to step down from FSF and GNU and let others continue in his stead. Should this not happen in a timely manner, then I believe that severing the historical ties between GNOME, GNU and the FSF is the only path forward.

Edit: I’ve also cross-posted this to the GNOME discourse instance.

Sven Hoexter: ansible scp_if_ssh: smart debugging

Monday 16th of September 2019 04:16:25 PM

I guess that is just one of the things you've to know, so maybe it helps someone else.

We saw some warnings in our playbook rollouts like

[WARNING]: sftp transfer mechanism failed on [192.168.23.42]. Use ANSIBLE_DEBUG=1 to see detailed information

They were actually reported for sftp and scp usage. If you look at the debug output it's not very helpful for the average user, similar if you go to verbose mode with -vvv. The later one at least helped to see parameters passed to sftp and scp, but you still see no error message. But if you set

scp_if_ssh: True

or

scp_if_ssh: False

you will suddenly see the real error message

fatal: [docker-023]: FAILED! => {"msg": "failed to transfer file to /home/sven/testme.txt /home/sven/ .ansible/tmp/ansible-tmp-1568643306.1439135-27483534812631/source:\n\nunknown option -- A\r\nusage: scp [-346BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]\n [-l limit] [-o ssh_option] [-P port] [-S program] source ... target\n"}

Lesson learned, as long as ansible is running in "smart" mode it will hide all error messages from the user. Now we could figure out that the culprit is the -A for AgentForwarding, which is for obvious reasons not available in sftp and scp. One can move it to group_vars ansible_ssh_extra_args. The best documentation regarding this, beside of the --help output, seems to be the commit message of 3ad9b4cba62707777c3a144677e12ccd913c79a8.

Sam Hartman: Free as in Sausage Making: Inside the Debian Project

Monday 16th of September 2019 02:30:18 AM

Recently, we’ve been having some discussion around the use of non-free software and services in doing our Debian work. In judging consensus surrounding a discussion of Git packaging, I said that we do not have a consensus to forbid the use of non-free services like Github. I stand behind that consensus call. Ian Jackson, who initially thought that I misread the consensus later agreed with my call.


I have been debating whether it would be wise for me as project leader to say more on the issue. Ultimately I have decided to share my thoughts. Yes, some of this is my personal opinion. Yet I think my thoughts resonate with things said on the mailing list; by sharing my thoughts I may help facilitate the discussion.


We are bound together by the Social Contract. Anyone is welcome to contribute to Debian so long as they follow the Social Contract, the DFSG, and the rest of our community standards. The Social Contract talks about what we will build (a free operating system called Debian). Besides SC #3 (we will not hide problems), the contract says very little about how we will build Debian.


What matters is what you do, not what you believe. You don’t even need to believe in free software to be part of Debian, so long as you’re busy writing or contributing to free software. Whether it’s because you believe in user freedom or because your large company has chosen Debian for entirely pragmatic reasons, your free software contributions are welcome.


I think that is one of our core strengths. We’re an incredibly diverse community. When we try to tie something else to what it means to be Debian beyond the quality of that free operating system we produce, judged by how it meets the needs of our users, we risk diminishing Debian. Our diversity serves the free software community well. We have always balanced pragmatic concerns against freedom. We didn’t ignore binary blobs and non-free firmware in the kernel, but we took the time to make sure we balanced our users’ needs for functional systems against their needs for freedom. By being so diverse, we have helped build a product that is useful both to people who care about freedom and other issues. Debian has been pragmatic enough that our product is wildly popular. We care enough about freedom and do the hard work of finding workable solutions that many issues of software freedom have become mainstream concerns with viable solutions.


Debian has always taken a pragmatic approach to its own infrastructure and to how Debian is developed. The Social Contract requires that the resulting operating system be 100% free software. But that has never been true of the Debian Project nor of our developers.



  • At the time the Social contract was adopted, uploading a package to Debian involved signing it with the non-free PGP version 2.6.3. It was years later that GnuPG became commonly used.

  • Debian developers of the day didn’t use non-free tools to sign the Social Contract. They didn’t digitally sign it at all. Yet their discussions used the non-free Qmail because people running the Debian infrastructure decided that was the best solution for the project’s mailing lists.


“That was then,” you say.



  • Today, some parts of security.debian.org redirect to security-cdn.debian.org, a non-free web service

  • Our recommended mirror (deb.debian.org) is backed by multiple non-free CDN web services.

  • Some day we may be using more non-free services. If trends in email handling continue, we may find that we need to use some non-free service to get the email we send accepted by major email providers. I know of no such plan in Debian today, but I know other organizations have faced similar choices.


Yet these choices to use non-free software and non-free services in the production of Debian have real costs. Many members of our community prefer to use free software. When we make these choices, we can make it harder for people to contribute to Debian. When we decline to use free software we may also be missing out on an opportunity to improve the free software community or to improve Debian itself. Ian eloquently describes the frustrations those who wish to use only free software face when faced with choices to use non-free services.


As alternatives to non-free software or services have become available, we as a project have consistently moved toward free options.


Normally, we let those doing the work within Debian choose whether non-free services or software are sufficiently better than the free alternatives that we will use them in our work. There is a strong desire to prefer free software and self-hosted infrastructure when that can meet our needs.


For individual maintainers, this generally means that you can choose the tools you want to do your Debian work. The resulting contributions to Debian must themselves be free. But if you want to go write all your Debian packaging in Visual Studio on Windows, we’re not going to stop you, although many of us will think your choices are unusual.


And my take is that if you want to store Debian packages on Github, you can do that too. But if you do that, you will be making it harder for many Debian contributors to contribute to your packages. As Ian discussed, even if you listen to the BTS, you will create two classes of contributors: those who are comfortable with your tools and those who are not. Perhaps you’ve considered this already. Perhaps you value making things easier for yourself or for interacting with an upstream community on Github over making it easier for contributors who want to use only free tools. Traditionally in Debian, we’ve decided that the people doing the work generally get to make that decision. Some day perhaps we’ll decide that all Debian packaging needs to be done in a VCS hosted on Debian infrastructure. And if we make that decision, we will almost certainly choose a free service to host. We’re not ready to make that change today.


So, what can you do if you want to use only free tools?



  • You could take Ian’s original approach and attempt to mandate project policy. Yet each time we mandate such policy, we will drive people and their contributions away. When the community as a whole evaluates such efforts we’ll need to ask ourselves whether the restriction is worth what we will lose. Sometimes it is. But unsurprisingly in my mind, Debian often finds a balance on these issues.


  • You could work to understand why people use Github or other non-free tools. As you take the time to understand and value the needs of those who use non-free services, you could ask them to understand and value your needs. If you identify gaps in what free software and services offer, work to fix those gaps.


  • Specifically in this instance, I think that setting up easy ways to bidirectionally mirror things between Github and services like Salsa could really help.



Conclusions

  1. We have come together to make a free operating system. Everything else is up for debate. When we shut down that debate—when we decide there is one right answer—we risk diluting our focus and diminishing ourselves.

  2. We and the entire free software community win through the Debian Project’s diversity.

  3. Freedom within the Debian Project has never been simple. Throughout our entire history we’ve used non-free bits in the sausage making, even though the result consists (and can be built from) entirely free bits.

  4. This complexity and diversity is part of what allows us to advocate for software freedom more successfully. Over time, we have replaced non-free software that we use with free alternatives, but those decisions are nuanced and ever-changing.

Shirish Agarwal: Freedom, Chandrayaan 2 and Corporations in Space.

Sunday 15th of September 2019 09:15:36 PM

Today will be a longish blogpost so please excuse if you do not want to read a long article.

While today is my birthday, I don’t feel at all like celebrating. When 8 million Kashmiris are locked down in Kashmir and 19 million to be sent in detention camp, the number may increase, how one one feel happy? Sadly, many people disregard that illegal immigration is everywhere. Whether it is UK or US, Indians too have illegally immigrated. If you look at the comments either in US or UK papers is just as toxic as you would find in twitter in India. Most of the people are uninformed of the various reasons that people choose to take a dangerous path to make home a new country. Alliances are also divided because the children grow up in another culture and then they will be ‘corrupted’ especially if women are sent back to India. The situation in India have never been as similar as they are today, see this from Najam Sethi, an internationally known left-leaning journalist
https://www.youtube.com/watch?v=OCcrobZMy7A
and similarly you can see how Indian and US Investigative journalism is having a slow death in both India and U.S.
https://www.youtube.com/watch?v=65P44plUCng

You can also see how similar the societies are going into with this conversations
https://www.youtube.com/watch?v=ieWZi4gm_yE
https://www.youtube.com/watch?v=g_1oJui2Zq8

There are good moments too as can be seen here –

People going to Ganesh Visarjan and Muharram, Hyderabad.

We always say we are better than Pakistan but we seem to be going down to the same road and that can’t be good. Forget politics, even human right issues are not being tackled sensitively by our Supreme Court. Just today there came an incident involving one of the victims of the Muzaffarpur Shelter Home being allegedlly raped in a moving car. The case has been pending in the Supreme Court for quite sometime but no action being taken so far. Journalists either in Uttar Pradesh or Haryana are being booked for showing the truth. I have been trying to engage with people from across the political divide, i.e. the ones who support BJP. Majority of them are don’t have jobs and this is the only way they can get their frustration out. Also the dissonance in political message is such they feel that their jobs are being taken by outsiders. Ironically the Ministers get away with saying things like ‘North Indians lack qualifications’ . It shows lack of empathy on the Minister’s part. If they are citizens of the state, then it is the state’s responsibility of making sure they are skilled. If they are not skilled, then it is the Central Government and State Governments responsibility. Most of the States in the North are governed by BJP. I could share more but then it will all be about BJP only and nothing about the Chandrayaan 2 mission.

Chandrayaan 2 and Corporate Interests

Before we get to Chandrayaan 2, there are few interesting series I want to talk about, share about. The first one is AltBalaji’s Mission Over Mars which in some ways is similar to Mars 6-part series Docu-drama made by National Geographic and lot of movies, books etc. read over years. In both these and other books, movies etc. it has been shown how Corporate Interests win over science and exploration which the motives of such initiatives were and are. The rich become richer and richer while the poor become more poorer.

There has been also lot of media speculation that ISRO should be privatized similar to how NASA is and people saying that NASA’s importance has not lessened even though they couldn’t have been more wrong. Take the Space Launch System . It was first though of in the 2010 after the NASA Authorization Act of 2010 came into being. When it was shared or told it was told that it would be ready somewhere in 2016. Now it seems it won’t be ready until 2025. And who is responsible for this, the same company which has been responsible for lot of bad news in the international aviation business Boeing. The auditor’s report for NASA while blaming NASA for oversight also blames Boeing for not doing things right. And from what we have come to know that in the american system of self-regulation leaves much to be desired. More so, when an ex-employee of Boeing is exercising his fifth Amendment rights which raises the suspicion that there is more than just simply an oversight issue. Boeing also is a weapons manufacturer but that’s another story altogether. For people interested in the arms stuff, a wired article published 2 years back gives enough info. as to how America is good bad or Arms sale.

I know the whole thing gives a rather complex picture but that is the nature of things. The only thing I would say is we should be very careful in privatizing ISRO as the same issues are bound to happen sooner or later, the more private and non-transparent things become. We, the common citizens would never come to know if any sort of national or industrial espionage is happening and of course all profits would be of corporates while losses will be public as seen can be nowadays. Doesn’t leave a good taste in the mouth.

Vikram Lander

Now while the jury is still out as to what happened or might have happened and we hope that Vikram does connect within the 14 days there are lots of theories as to what could have gone wrong. While I’m no expert, I do find it hard to take the statement that probably ISRO saw an image of Chandrayaan 2 lander, at least not a single image has been released in the public. What ISRO has shared in its updates is that it located a lander which doesn’t tell much. While the Chandrayaan 2 orbitor started at 100 km. lunar orbit it probably would have some deviations to make sure that the orbiter itself doesn’t get into Moon’s gravity and crash-lands on the moon itself. The lens which probably would have been used would be to take panaromic shots and not telescopic in nature. As to what happened, we just don’t know as of yet. There are probably a dozen or two probabilities. One of the most simplest explanation to my mind could be some space rock could have crashed into the lander when it was landing. The dark side of the moon has more impacts than the one which we face so it’s entirely possible that the lander got hit by a space rock or lava. From what little I have been able to learn, the lander doesn’t seem to have any A.I. to manoeuver if such a scenario happens. Also any functioning A.I. would probably need more energy and for space missions energy, weight, electrical interference, contamination are all issues that Space Agencies have to deal with it. The other is of course, sensor failure, wrong calculation or a rough spot where it landed and broke the antennae. Till ISRO doesn’t share more details with us, we have only conjecture to help us.

Chandrayaan 2 Imaging

While we await news about the lander, would be curious to know about the images that Chandrayaan 2 is getting. Sadly, none of the images have made it to the public domain as of yet. Whether the images are in FITS or RAW format and whatever spectrum, (Chandrayaan 2 is going to image in a wide range of spectrum.) . Like Spirit and Opportunity did for NASA, I hope ISRO does show renderings of Moon as captured by the Orbitor, even though its lifeless so people, especially children get enthused about getting into Space Sciences .

Molly de Blanc: Free software activities (August 2019)

Sunday 15th of September 2019 02:34:38 PM

August was really marked by traveling too much. I took the end of the month off from non-work activities in order to focus on GUADEC and GUADEC-follow up.

Personal

  • The Debian Community Team (CT) had a meeting where we discussed some of our activities, including potential new team members!
  • CT team members went on VAC, so we took a bit of a break in the second half of the month.
  • The OSI standing committee and board had meetings.
  • I handled some paperwork in my capacity as president.
  • I had regular meetings with the OSI general manager.
  • I gave a keynote at FrOSCon on “Open source citizenship for everyone!” TL;DR: We have rights and responsibilities as people participating in free software and the open source ecosystem — “we” here includes corporate actors.
  • I bought a really sweet pair of GNOME socks. Do recommend.

Professional

  • The LAS sponsorship team met and handled the creation of some important paperwork, and discussed fundraising strategy for the event.
  • I attended the GNOME Advisory Board meeting, where I got to meet and speak with the Foundation Board and the Advisory Board about activities over the past year, plans for the future, and the needs of the communities of AdBoard members. It was really educational and a lot of fun.
  • I attended my first GUADEC! it was amazing. I wrote a trip report over on the GNOME Engagement Blog.
  • At GUADEC, I spent some time helping out with basic operations, including keeping time in sessions.
  • We, the staff and board, did a Q&A at the Annual General Meeting.
  • I drank a lot of coffee. Like, a lot.

More in Tux Machines

A Look at KDE Plasma 5.17 Beta and Report From Akademy 2019

  • KDE Plasma 5.17 Beta Run Through

    In this video, we look at KDE Plasma 5.17 Beta, enjoy!

  • TSDgeos' blog: Akademy 2019

    It's 10 days already since Akademy 2019 finished and I'm already missing it :/ Akademy is a week-long action-packed conference, talks, BoFs, daytrip, dinner with old and new friends, it's all a great combination and shows how amazing KDE (yes, the community, that's our name) is. On the talks side i missed some that i wanted to attend because i had to extend my time at the registration booth helping fellow KDE people that had forgotten to register (yes, our setup could be a bit easier, doesn't help that you have to register for talks, for travel support and for the actual conference in three different places), but I am not complaining, you get to interact with lots of people in the registration desk, it's a good way to meet people you may not have met otherwise, so please make sure you volunteer next year ;) One of the talks i want to highlight is Dan VrĂĄtil's talk about C++, I agree with him that we could do much better in making our APIs more expressive using the power of "modern" C++ (when do we stop it calling modern?). It's a pity that the slides are not up so you'll have to live with KĂŠvin Ottens sketch of it for now.

Programming Leftovers

  • DevNation Live: Event-driven business automation powered by cloud-native Java

    DevNation Live tech talks are hosted by the Red Hat technologists who create our products. These sessions include real solutions and code and sample projects to help you get started. In this talk, presented by Red Hat’s Maciej Swiderski, Principal Software Engineer, and Burr Sutter, Chief Developer Evangelist, you’ll learn about event-driven business automation using Kogito, Quarkus, and more. Kogito is a new Java toolkit, based on Drools and jBPM, that’s made to bring rules and processes to the Quarkus world. This DevNation Live presentation shows how Kogito can be used to build cloud-ready, event-driven business applications, and it includes a demo of implementing the business logic of a complex domain. Kogito itself is defined as a cloud-native business automation toolkit that helps you to build intelligent applications. It’s way more than just a business process or a single business rule—it’s a bunch of business rules, and it’s based on battle-tested capabilities.

  • NVIDIA Video Codec SDK 9.1 Brings CUDA CUStream Support, Other Encoder Improvements

    Following the February release of Video Codec SDK 9.0, NVIDIA recently did a quiet release of the Video Codec SDK 9.1 update that furthers along this cross-platform video encode/decode library.

  • Mike Driscoll: PyDev of the Week: Peter Farrell

    This week we welcome Peter Farrell (@hackingmath) as our PyDev of the Week! Peter is the author Math Adventures with Python and two other math related Python books. You can learn more about Peter by visiting his website.

  • Mutation testing by example: How to leverage failure
  • Reuven Lerner: Looking for Python podcast co-hosts

    As you might know, I’m a panelist on the weekly “Freelancers Show” podcast, which talks about the business of freelancing. The good news: The same company that’s behind the Freelancers Show, Devchat.tv, is putting together a weekly podcast about Python, and I’m going to be on that, too! We’ll have a combination of discussion, interviews with interesting people in the Python community, and (friendly) debates over the current and future state of the language.

  • Getting started with data science using Python

    Data science is an exciting new field in computing that's built around analyzing, visualizing, correlating, and interpreting the boundless amounts of information our computers are collecting about the world. Of course, calling it a "new" field is a little disingenuous because the discipline is a derivative of statistics, data analysis, and plain old obsessive scientific observation. But data science is a formalized branch of these disciplines, with processes and tools all its own, and it can be broadly applied across disciplines (such as visual effects) that had never produced big dumps of unmanageable data before. Data science is a new opportunity to take a fresh look at data from oceanography, meteorology, geography, cartography, biology, medicine and health, and entertainment industries and gain a better understanding of patterns, influences, and causality. Like other big and seemingly all-inclusive fields, it can be intimidating to know where to start exploring data science. There are a lot of resources out there to help data scientists use their favorite programming languages to accomplish their goals, and that includes one of the most popular programming languages out there: Python. Using the Pandas, Matplotlib, and Seaborn libraries, you can learn the basic toolset of data science.

Excellent Utilities: Liquid Prompt – adaptive prompt for Bash & Zsh

This is a new series highlighting best-of-breed utilities. We’re covering a wide range of utilities including tools that boost your productivity, help you manage your workflow, and lots more besides. There’s a complete list of the tools in this series in the Summary section. The Command Line Interface (CLI) is a way of interacting with your computer. And if you ever want to harness all the power of Linux, it’s highly recommended to master it. It’s true the CLI is often perceived as a barrier for users migrating to Linux, particularly if they’re grown up using GUI software exclusively. While Linux rarely forces anyone to use the CLI, some tasks are better suited to this method of interaction, offering inducements like superior scripting opportunities, remote access, and being far more frugal with a computer’s resources. For anyone spending time at the CLI, they’ll rely on the shell prompt. My favorite shell is Bash. By default, the configuration for Bash on popular distributions identifies the user name, hostname, and the current working directory. All essential information. But with Liquid Prompt you can display additional information such as battery status, CPU temperature, and much more. Read more

today's howtos