Language Selection

English French German Italian Portuguese Spanish

Planet Debian

Syndicate content
Planet Debian - https://planet.debian.org/
Updated: 1 hour 26 min ago

Utkarsh Gupta: Joining Debian LTS!

5 hours 43 min ago

Hey,

(DPL Style):
TL;DR: I joined Debian LTS as a trainee in July (during DebConf) and finally as a paid contributor from this month onward! :D

Here’s something interesting that happened last weekend!
Back during the good days of DebConf19, I finally got a chance to meet Holger! As amazing and inspiring a person he is, it was an absolute pleasure meeting him and also, I got a chance to talk about Debian LTS in more detail.

I was introduced to Debian LTS by Abhijith during his talk in MiniDebConf Delhi. And since then, I’ve been kinda interested in that project!
But finally it was here that things got a little “official” and after a couple of mail exchanges with Holger and Raphael, I joined in as a trainee!

I had almost no idea what to do next, so the next month I stayed silent, observing the workflow as people kept committing and announcing updates.
And finally in September, I started triaging and fixing the CVEs for Jessie and Stretch (mostly the former).

Thanks to Abhijith who explained the basics of what DLA is and how do we go about fixing bugs and then announcing them.

With that, I could fix a couple of CVEs and thanks to Holger (again) for reviewing and sponsoring the uploads! :D

I mostly worked (as a trainee) on:

  • CVE-2019-10751, affecting httpie, and
  • CVE-2019-16680, affecting file-roller.

And finally this happened:
(Though there’s a little hiccup that happened there, but that’s something we can ignore!)

So finally, I’ll be working with the team from this month on!
As Holger says, very much yay! :D

Until next time.
:wq for today.

Iustin Pop: Actually fixing a bug

6 hours 41 min ago

One of the outcomes of my recent (last few years) sports ramp-up is that my opensource work is almost entirely left aside. Having an office job makes it very hard to spend more time sitting at the computer at home too…

So even my travis dashboard was red for a while now, but I didn’t look into it until today. Since I didn’t change anything recently, just travis builds started to fail, I was sure it’s just environment changes that need to be taken into account.

And indeed it was so, for two out of three projects. The third one… I actually got to fix a bug, introduced at the beginning of the year, but for which gcc (same gcc that originally passed) started to trip on a while back. I even had to read the man page of snprintf! Was fun ☺, too bad I don’t have enough time to do this more often…

My travis dashboard is green again, and “test suite” (if you can call it that) is expanded to explicitly catch this specific problem in the future.

Shirish Agarwal: Social media, knowledge and some history of Banking

Saturday 12th of October 2019 10:58:37 PM

First of all Happy Dusshera to everybody. While Dusshera is India is a symbol of many things, it is a symbol of forgiveness and new beginnings. While I don’t know about new beginnings I do feel there is still lot of baggage which needs to be left I would try to share some insights I uncovered over last few months and few realizations I came across.

First of all thank you to the Debian-gnome-team to keep working at new version of packages. While there are still a bunch of bugs which need to be fixed especially #895990 and #913978 among others, still kudos for working at it. Hopefully, those bugs and others will be fixed soon so we could install gnome without a hiccup. I have not been on IRC because my riot-web has been broken for several days now. Also most of the IRC and telegram channels at least related to Debian become mostly echo chambers one way or the other as you do not get any serious opposition. On twitter, while it’s highly toxic, you also get the urge to fight the good fight when either due to principles or for some other reason (usually paid trolls) people fight, While I follow my own rules on twitter apart from their TOS, I feel at least new people who are going on social media in India or perhaps elsewhere as well could use are –

  1. It is difficult to remain neutral and stick to the facts. If you just stick to the facts, you will be branded as urban naxal or some such names.
  2. I find many times, if you are calm and don’t react, many a times, they are curious and display ignorance of knowledge which you thought everyone knew is not there. Now whether that is due to either due to lack of education, lack of knowledge or pretensions, although if its pretentious, you are caught sooner or later.
  3. Be civil at all times, if somebody harassess you, calls you names, report them and block them, although twitter still needs to fix the reporting thing a whole lot more. Although, when even somebody like me (bit of understanding of law, technology, language etc.) had a hard time figuring out twitter’s reporting ways, I dunno how many people would be able to use it successfully ? Maybe they make it so unhelpful so the traffic flows no matter what. I do realize they still haven’t figured out their business model but that’s a question for another day. In short, they need to make it far more simpler than it is today.
  4. You always have an option to block people but it has its own consequences.
  5. Be passive-aggressive if the situation demands it.
  6. Most importantly though, if somebody starts making jokes about you or start abusing you, it is sure that the person on the other side doesn’t have any more arguments and you have won.
Banking

Before I start, let me share why I am putting a blog post on the topic. The reason is pretty simple. It seems a huge number of Indians don’t either know the history of how banking started, the various turns it took and so on and so forth. In fact, nowadays history is being so hotly contested and perhaps even being re-written. Hence for some things I would be sharing some sources but even within them, there is possibiity of contestations. One of the contestations for a long time is when ancient coinage and the technique of smelting, flattening came to India. Depending on whom you ask, you have different answers. Lot of people are waiting to get more insight from the Keezhadi excavation which may also give some insight to the topic as well. There are rumors that the funding is being stopped but hope that isn’t true and we gain some more insight in Indian history. In fact, in South India, there seems to be lot of curiousity and attraction towards the site. It is possible that the next time I get a chance to see South India, I may try to see if there is a chance to see this unique location if a museum gets built somewhere nearby. Sorry from deviating from the topic, but it seems that ancient coinage started anywhere between 1st millenium BCE to 6th century BCE so it could be anywhere between 1500 – 2000 years old in India. While we can’t say anything for sure, but it’s possible that there was barter before that. There has also been some history about sharing tokens in different parts of the world as well. The various timelines get all jumbled up hence I would suggest people to use the wikipedia page of History of Money as a starting point. While it may not be give a complete, it would probably broaden the understanding a little bit. One of the reasons why history is so hotly contested could also perhaps lie because of the destruction of the Ancient Library of Alexandria. Who knows what more we would have known of our ancients if it was not destroyed

Dirk Eddelbuettel: GitHub Streak: Round Six

Saturday 12th of October 2019 03:53:00 PM

Five ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery:

This is sometimes called Jerry Seinfeld’s secret to productivity: Just keep at it. Don’t break the streak.

and then showed the first chart of GitHub streaking

github activity october 2013 to october 2014

And four year ago a first follow-up appeared in this post:

github activity october 2014 to october 2015

And three years ago we had a followup

github activity october 2015 to october 2016

And two years ago we had another one

github activity october 2016 to october 2017

And last year another one

github activity october 2017 to october 2018

As today is October 12, here is the newest one from 2018 to 2019:

github activity october 2018 to october 2019

Again, special thanks go to Alessandro Pezzè for the Chrome add-on GithubOriginalStreak.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Louis-Philippe Véronneau: Alpine MusicSafe Classic Hearing Protection Review

Saturday 12th of October 2019 04:00:00 AM

Yesterday, I went to a punk rock show and had tons of fun. One of the bands playing (Jeunesse Apatride) hadn't played in 5 years and the crowd was wild. The other bands playing were also great. Here's a few links if you enjoy Oi! and Ska:

Sadly, those kind of concerts are always waaaaayyyyy too loud. I mostly go to small venue concerts and for some reason the sound technicians think it's a good idea to make everyone's ears bleed. You really don't need to amplify the drums when the whole concert venue is 50m²...

So I bough hearing protection. It was the first time I wore earplugs at a concert and it was great! I can't really compare the model I got (Alpine MusicSafe Classic earplugs) to other brands since it's the only one I tried out, but:

  • They were very comfortable. I wore them for about 5 hours and didn't feel any discomfort.

  • They came with two sets of plastic tips you insert in the silicone earbuds. I tried the -17db ones but I decided to go with the -18db inserts as it was still freaking loud.

  • They fitted very well in my ears even tough I was in the roughest mosh pit I've ever experienced (and I've seen quite a few). I was sweating profusely from all the heavy moshing and never once I feared loosing them.

  • My ears weren't ringing when I came back home so I guess they work.

  • The earplugs didn't distort sound, only reduce the volume.

  • They came with a handy aluminium carrying case that's really durable. You can put it on your keychain and carry them around safely.

  • They only cost me ~25 CAD with taxes.

The only thing I disliked was that I found it pretty much impossible to sing while wearing them. as I couldn't really hear myself. With a bit of practice, I was able to sing true but it wasn't great :(

All in all, I'm really happy with my purchase and I don't think I'll ever go to another concert without earplugs.

Molly de Blanc: Conferences

Friday 11th of October 2019 11:23:44 PM

I think there are too many conferences.

Are there too many FLOSS conferences?

— Molly dBoo (@mmillions) October 7, 2019

I conducted this very scientific Twitter poll and out of 52 respondants, only 23% agreed with me. Some people who disagreed with me pointed out specifically what they think is lacking:  more regional events, more in specific countries, and more “generic” FLOSS events.

Many projects have a conference, and then there are “generic” conferences, like FOSDEM, LibrePlanet, LinuxConfAU, and FOSSAsia. Some are more corporate (OSCON), while others more community focused (e.g. SeaGL).

There are just a lot of conferences.

I average a conference a month, with most of them being more general sorts of events, and a few being project specific, like DebConf and GUADEC.

So far in 2019, I went to: FOSDEM, CopyLeft Conf, LibrePlanet, FOSS North, Linux Fest Northwest, OSCON, FrOSCon, GUADEC, and GitLab Commit. I’m going to All Things Open next week. In November I have COSCon scheduled. I’m skipping SeaGL this year. I am not planning on attending 36C3 unless my talk is accepted. I canceled my trip to DebConf19. I did not go to Camp this year. I also had a board meeting in NY, an upcoming one in Berlin, and a Debian meeting in the other Cambridge. I’m skipping LAS and likely going to SFSCon for GNOME.

So 9 so far this year,  and somewhere between 1-4 more, depending on some details.

There are also conferences that don’t happen every year, like HOPE and CubaConf. There are some that I haven’t been to yet, like PyCon, and more regional events like Ohio Linux Fest, SCALE, and FOSSCon in Philadelphia.

I think I travel too much, and plenty of people travel more than I do. This is one of the reasons why we have too many events: the same people are traveling so much.

When you’re nose deep in it, when you think that you’re doing is important, you keep going to them as long as you’re invited. I really believe in the messages I share during my talks, and I know by speaking I am reaching audiences I wouldn’t otherwise. As long as I keep getting invited places, I’ll probably keep going.

Finding sponsors is hard(er).

It is becoming increasingly difficult to find sponsors for conferences. This is my experience, and what I’ve heard from speaking with others about it. Lower response rates to requests and people choosing lower sponsorship levels than they have in past years.

CFP responses are not increasing.

I’m yet to hear of any established community-run tech conferences who’ve had growth in their CFP response rate this year.

Peak conference?

— Christopher Neugebauer

Markus Koschany: My Free Software Activities in September 2019

Thursday 10th of October 2019 08:49:21 PM

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games
  • Reiner Herrmann investigated a build failure of supertuxkart on several architectures and prepared an update to link against libatomic. I reviewed and sponsored the new revision which allowed supertuxkart 1.0 to migrate to testing.
  • Python 3 ports: Reiner also ported bouncy, a game for small kids, to Python3 which I reviewed and uploaded to unstable.
  • Myself upgraded atomix to version 3.34.0 as requested although it is unlikely that you will find a major difference to the previous version.
Debian Java Misc
  • I packaged new upstream releases of ublock-origin and privacybadger, two popular Firefox/Chromium addons and
  • packaged a new upstream release of wabt, the WebAssembly Binary Toolkit.
Debian LTS

This was my 43. month as a paid contributor and I have been paid to work 23,75 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • From 11.09.2019 until 15.09.2019 I was in charge of our LTS frontdesk. I investigated and triaged CVE in libonig, bird, curl, openssl, wpa, httpie, asterisk, wireshark and libsixel.
  • DLA-1922-1. Issued a security update for wpa fixing 1 CVE.
  • DLA-1932-1. Issued a security update for openssl fixing 2 CVE.
  • DLA-1900-2. Issued a regression update for apache fixing 1 CVE.
  • DLA-1943-1. Issued a security update for jackson-databind fixing 4 CVE.
  • DLA-1954-1. Issued a security update for lucene-solr fixing 1 CVE. I triaged CVE-2019-12401 and marked Jessie as not-affected because we use the system libraries of woodstox in Debian.
  • DLA-1955-1. Issued a security update for tcpdump fixing 24 CVE by backporting the latest upstream release to Jessie. I discovered several test failures but after more investigation I came to the conclusion that the test cases were simply created with a newer version of libpcap which causes the test failures with Jessie’s older version.
ELTS

Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 “Wheezy”. This was my sixteenth month and I have been assigned to work 15 hours on ELTS plus five hours from August. I used 15 of them for the following:

  • I was in charge of our ELTS frontdesk from 30.09.2019 until 06.10.2019 and I triaged CVE in tcpdump. There were no reports of other security vulnerabilities for supported packages in this week.
  • ELA-163-1. Issued a security update for curl fixing 1 CVE.
  • ELA-171-1. Issued a security update for openssl fixing 2 CVE.
  • ELA-172-1. Issued a security update for linux fixing 23 CVE.
  • ELA-174-1. Issued a security update for tcpdump fixing 24 CVE.

Norbert Preining: R with TensorFlow 2.0 on Debian/sid

Thursday 10th of October 2019 06:15:18 AM

I recently posted on getting TensorFlow 2.0 with GPU support running on Debian/sid. At that time I didn’t manage to get the tensorflow package for R running properly. It didn’t need much to get it running, though.

The biggest problem I faced was that the R/TensorFlow package recommends using install_tensorflow, which can use either auto, conda, virtualenv, or system (at least according to the linked web page). I didn’t want to set up neither a conda nor virtualenv environment, since TensorFlow was already installed, so I thought system would be correct, but then, I had it already installed. Anyway, the system option is gone and not accepted, but I still got errors. In particular because the code mentioned on the installation page is incorrect for TF2.0!

It turned out to be a simple error on my side – the default is to use the program python which in Debian is still Python2, while I have TF only installed for Python3. The magic incantation to fix that is use_python("/usr/bin/python3") and one is set.

So here is a full list of commands to get R/TensorFlow running on top of an already installed TensorFlow for Python3 (as usual either as root to be installed into /usr/local or as user to have a local installation):

devtools::install_github("rstudio/tensorflow")

And if you want to run some TF program:

library(tensorflow) use_python("/usr/bin/python3") tf$math$cumprod(1:5)

This gives lots of output but mentioning that it is running on my GPU.

At least for the (probably very short) time being this looks like a workable system. Now off to convert my TF1.N code to TF2.0.

Louis-Philippe Véronneau: Trying out Sourcehut

Thursday 10th of October 2019 04:00:00 AM

Last month, I decided it was finally time to move a project I maintain from Github1 to another git hosting platform.

While polling other contributors (I proposed moving to gitlab.com), someone suggested moving to Sourcehut, a newish git hosting platform written and maintained by Drew DeVault. I've been following Drew's work for a while now and although I had read a few blog posts on Sourcehut's development, I had never really considered giving it a try. So I did!

Sourcehut is still in alpha and I'm expecting a lot of things to change in the future, but here's my quick review.

Things I like Sustainable FOSS

Sourcehut is 100% Free Software. Github is proprietary and I dislike Gitlab's Open Core business model.

Sourcehut's business model also seems sustainable to me, as it relies on people paying a monthly fee for the service. You'll need to pay if you want your code hosted on https://sr.ht once Sourcehut moves into beta. As I've written previously, I like that a lot.

In comparison, Gitlab is mainly funded by venture capital and I'm afraid of the long term repercussions this choice will have.

Continuous Integration

Continuous Integration is very important to me and I'm happy to say Sourcehut's CI is pretty good! Like Travis and Gitlab CI, you declare what needs to happen in a YAML file. The CI uses real virtual machines backed by QEMU, so you can run many different distros and CPU archs!

Even nicer, you can actually SSH into a failed CI job to debug things. In comparison, Gitlab CI's Interactive Web Terminal is ... web based and thus not as nice. Worse, it seems it's still somewhat buggy as Gitlab still hasn't enabled it on their gitlab.com instance.

Here's what the instructions to SSH into the CI look like when a job fails:

This build job failed. You may log into the failed build environment within 10 minutes to examine the results with the following command: ssh -t builds@foo.bar connect NUMBER

Sourcehut's CI is not as feature-rich or as flexible as Gitlab CI, but I feel it is more powerful then Gitlab CI's default docker executor. Folks that run integration tests or more complicated setups where Docker fails should definitely give it a try.

From the few tests I did, Sourcehut's CI is also pretty quick (it's definitely faster than Travis or Gitlab CI).

No JS

Although Sourcehut's web interface does bundle some Javascript, all features work without it. Three cheers for that!

Things I dislike Features division

I'm not sure I like the way features (the issue tracker, the CI builds, the git repository, the wikis, etc.) are subdivided in different subdomains.

For example, when you create a git repository on git.sr.ht, you only get a git repository. If you want an issue tracker for that git repository, you have to create one at todo.sr.ht with the same name. That issue tracker isn't visible from the git repository web interface.

That's the same for all the features. For example, you don't see the build status of a merged commit when you look at it. This design choice makes you feel like the different features aren't integrated to one another.

In comparison, Gitlab and Github use a more "centralised" approach: everything is centered around a central interface (your git repository) and it feels more natural to me.

Discoverability

I haven't seen a way to search sr.ht for things hosted there. That makes it hard to find repositories, issues or even the Sourcehut source code!

Merge Request workflow

I'm a sucker for the Merge Request workflow. I really like to have a big green button I can click on to merge things. I know some people prefer a more manual workflow that uses git merge and stuff, but I find that tiresome.

Sourcehut chose a workflow based on sending patches by email. It's neat since you can submit code without having an account. Sourcehut also provides mailing lists for projects, so people can send patches to a central place.

I find that workflow harder to work with, since to me it makes it more difficult to see what patches have been submitted. It also makes the review process more tedious, since the CI isn't ran automatically on email patches.

Summary

All in all, I don't think I'll be moving ISBG to Sourcehut (yet?). At the moment it doesn't quite feel as ready as I'd want it to be, and that's OK. Most of the things I disliked about the service can be fixed by some UI work and I'm sure people are already working on it.

Github was bought by MS for 7.5 billion USD and Gitlab is currently valued at 2.7 billion USD. It's not really fair to ask Sourcehut to fully compete just yet :)

With Sourcehut, Drew DeVault is fighting the good fight and I wish him the most resounding success. Who knows, maybe I'll really migrate to it in a few years!

  1. Github is a proprietary service, has been bought by Microsoft and gosh darn do I hate Travis CI. 

Dirk Eddelbuettel: RcppArmadillo 0.9.800.1.0

Thursday 10th of October 2019 12:59:00 AM

Another month, another Armadillo upstream release! Hence a new RcppArmadillo release arrived on CRAN earlier today, and was just shipped to Debian as well. It brings a faster solve() method and other goodies. We also switched to the (awesome) tinytest unit test frameowrk, and Min Kim made the configure.ac script more portable for the benefit of NetBSD and other non-bash users; see below for more details. One again we ran two full sets of reverse-depends checks, no issues were found, and the packages was auto-admitted similarly at CRAN after less than two hours despite there being 665 reverse depends. Impressive stuff, so a big Thank You! as always to the CRAN team.

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 665 other packages on CRAN.

Changes in RcppArmadillo version 0.9.800.1.0 (2019-10-09)
  • Upgraded to Armadillo release 9.800 (Horizon Scraper)

    • faster solve() in default operation; iterative refinement is no longer applied by default; use solve_opts::refine to explicitly enable refinement

    • faster expmat()

    • faster handling of triangular matrices by rcond()

    • added .front() and .back()

    • added .is_trimatu() and .is_trimatl()

    • added .is_diagmat()

  • The package now uses tinytest for unit tests (Dirk in #269).

  • The configure.ac script is now more careful about shell portability (Min Kim in #270).

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Adnan Hodzic: Hello world!

Wednesday 9th of October 2019 03:55:10 PM

Welcome to WordPress. This is your first post. Edit or delete it, then start writing!

Enrico Zini: Fixed XSS issue on debtags.debian.org

Wednesday 9th of October 2019 08:51:57 AM

Thanks to Moritz Naumann who found the issues and wrote a very useful report, I fixed a number of Cross Site Scripting vulnerabilities on https://debtags.debian.org.

The core of the issue was code like this in a Django view:

def pkginfo_view(request, name): pkg = bmodels.Package.by_name(name) if pkg is None: return http.HttpResponseNotFound("Package %s was not found" % name) # …

The default content-type of HttpResponseNotFound is text/html, and the string passed is the raw HTML with clearly no escaping, so this allows injection of arbitrary HTML/<script> code in the name variable.

I was so used to Django doing proper auto-escaping that I missed this place in which it can't do that.

There are various things that can be improved in that code.

One could introduce escaping (and while one's at it, migrate the old % to format):

from django.utils.html import escape def pkginfo_view(request, name): pkg = bmodels.Package.by_name(name) if pkg is None: return http.HttpResponseNotFound("Package {} was not found".format(escape(name))) # …

Alternatively, set content_type to text/plain:

def pkginfo_view(request, name): pkg = bmodels.Package.by_name(name) if pkg is None: return http.HttpResponseNotFound("Package {} was not found".format(name), content_type="text/plain") # …

Even better, raise Http404:

from django.utils.html import escape def pkginfo_view(request, name): pkg = bmodels.Package.by_name(name) if pkg is None: raise Http404(f"Package {name} was not found") # …

Even better, use standard shortcuts and model functions if possible:

from django.shortcuts import get_object_or_404 def pkginfo_view(request, name): pkg = get_object_or_404(bmodels.Package, name=name) # …

And finally, though not security related, it's about time to switch to class-based views:

class PkgInfo(TemplateView): template_name = "reports/package.html" def get_context_data(self, **kw): ctx = super().get_context_data(**kw) ctx["pkg"] = get_object_or_404(bmodels.Package, name=self.kwargs["name"]) # … return ctx

I proceeded with a review of the other Django sites I maintain in case I reproduced this mistake also there.

Chris Lamb: Tour d'Orwell: Southwold

Wednesday 9th of October 2019 12:29:35 AM

I recently read that that during 1929 George Orwell returned to his family home in the Suffolk town of Southwold but when I further learned that he had acquired a motorbike during this time to explore the surrounding villages I could not resist visiting myself on such a transport mode.

Orwell would end up writing his first novel here ("Burmese Days") followed by his first passable one ("A Clergyman's Daughter") but unfortunately the local bookshop was only to have the former in stock. He moved back to London in 1934 to work in a bookshop in Hampstead, now a «Le Pain Quotidien».

If you are thinking of visiting, Southwold has some lovely quaint beach huts, a brewery and the officially signposted A1120 "Scenic Route" I took on the way out was neither as picturesque nor as fun to ride as the A1066

Steve Kemp: A blog overhaul

Tuesday 8th of October 2019 06:00:00 PM

When this post becomes public I'll have successfully redeployed my blog!

My blog originally started in 2005 as a Wordpress installation, at some point I used Mephisto, and then I wrote my own solution.

My project was pretty cool; I'd parse a directory of text-files, one file for each post, and insert them into an SQLite database. From there I'd initiate a series of plugins, each one to generate something specific:

  • One plugin would output an archive page.
  • Another would generate a tag cloud.
  • Yet another would generate the actual search-results for a particular month/year, or tag-name.

All in all the solution was flexible and it wasn't too slow because finding posts via the SQLite database was pretty good.

Anyway I've come to realize that freedom and architecture was overkill. I don't need to do fancy presentation, I don't need a loosely-coupled set of plugins.

So now I have a simpler solution which uses my existing template, uses my existing posts - with only a few cleanups - and generates the site from scratch, including all the comments, in less than 2 seconds.

After running make clean a complete rebuild via make upload (which deploys the generated site to the remote host via rsync) takes 6 seconds.

I've lost the ability to be flexible in some areas, but I've gained all the speed. The old project took somewhere between 20-60 seconds to build, depending on what had changed.

In terms of simplifying my life I've dropped the remote installation of a site-search which means I can now host this site on a static site with only a single handler to receive any post-comments. (I was 50/50 on keeping comments. I didn't want to lose those I'd already received, and I do often find valuable and interesting contributions from readers, but being 100% static had its appeal too. I guess they stay for the next few years!)

Antoine Beaupré: Tip of the day: batch PDF conversion with LibreOffice

Tuesday 8th of October 2019 04:28:09 PM

Someone asked me today why they couldn't write on the DOCX document they received from a student using the pen in their Onyx Note Pro reader. The answer, of course, is that while the Onyx can read those files, it can't annotate them: that only works with PDFs.

Next question then, is of course: do I really need to open each file separately and save them as PDF? That's going to take forever, I have 30 students per class!

Fear not, shell scripting and headless mode flies in to the rescue!

As it turns out, one of the Libreoffice parameters allow you to run batch operations on files. By calling:

libreoffice --headless --convert-to pdf *.docx

LibreOffice will happily convert all the *.docx files in the current directory to PDF. But because navigating the commandline can be hard, I figured I could push this a tiny little bit further and wrote the following script:

#!/bin/sh exec libreoffice --headless --convert-to pdf "$@"

Drop this in ~/.local/share/nautilus/scripts/libreoffice-pdf, mark it executable, and voilà! You can batch-convert basically any text file (or anything supported by LibreOffice, really) into PDF.

Now I wonder if this would be a useful addition to the Debian package, anyone?

Jonas Meurer: debian lts report 2019.09

Tuesday 8th of October 2019 02:34:42 PM
Debian LTS report for September 2019

This month I was allocated 10 hours and carried over 9.5 hours from August. Unfortunately, again I didn't find much time to work on LTS issues, partially because I was travelling. I spent 5 hours on the task listed below. That means that I carry over 14.5 hours to October.

Links

Jamie McClelland: Editing video without a GUI? Really?

Tuesday 8th of October 2019 01:19:19 PM

It seems counter intuitive - if ever there was a program in need of a graphical user interface, it's a non-linear video editing program.

However, as part of the May First board elections, I discovered otherwise.

We asked each board candidate to submit a 1 - 2 minute video introduction about why they want to be on the board. My job was to connect them all into a single video.

I had an unrealistic thought that I could find some simple tool that could concatenate them all together (like mkvmerge) but I soon realized that this approach requires that everyone use the exact same format, codec, bit rate, sample rate and blah blah blah.

I soon realized that I needed to actually make a video, not compile one. I create videos so infrequently, that I often forget the name of the video editing software I used last time so it takes some searching. This time I found that I had openshot-qt installed but when I tried to run it, I got a back trace (which someone else has already reported).

I considered looking for another GUI editor, but I wasn't that interested in learning what might be a complicated user interface when what I need is so simple.

So I kept searching and found melt. Wow.

I ran:

melt originals/* -consumer avformat:all.webm acodec=libopus vcodec=libvpx

And a while later I had a video. Impressive. It handled people who submitted their videos in portrait mode on their cell phones in mp4 as well as web cam submissions using webm/vp9 on landscape mode.

Thank you melt developers!

Jonathan McDowell: Ada Lovelace Day: 5 Amazing Women in Tech

Tuesday 8th of October 2019 07:00:00 AM

It’s Ada Lovelace day and I’ve been lax in previous years about celebrating some of the talented women in technology I know or follow on the interwebs. So, to make up for it, here are 5 amazing technologists.

Allison Randal

I was initially aware of Allison through her work on Perl, was vaguely aware of the fact she was working on Ubunutu, briefly overlapped with her at HPE (and thought it was impressive HP were hiring such high calibre of Free Software folk) when she was working on OpenStack, and have had the pleasure of meeting her in person due to the fact we both work on Debian. In the continuing theme of being able to do all things tech she’s currently studying a PhD at Cambridge (the real one), and has already written a fascinating paper about about the security misconceptions around virtual machines and containers. She’s also been doing things with home automation, properly, with local speech recognition rather than relying on any external assistant service (I will, eventually, find the time to follow her advice and try this out for myself).

Alyssa Rosenzweig

Graphics are one of the many things I just can’t do. I’m not artistic and I’m in awe of anyone who is capable of wrangling bits to make computers do graphical magic. People who can reverse engineer graphics hardware that would otherwise only be supported by icky binary blobs impress me even more. Alyssa is such a person, working on the Panfrost driver for ARM’s Mali Midgard + Bifrost GPUs. The lack of a Free driver stack for this hardware is a real problem for the ARM ecosystem and she has been tirelessly working to bring this to many ARM based platforms. I was delighted when I saw one of my favourite Free Software consultancies, Collabora, had given her an internship over the summer. (Selfishly I’m hoping it means the Gemini PDA will eventually be able to run an upstream kernel with accelerated graphics.)

Angie McKeown

The first time I saw Angie talk it was about the user experience of Virtual Reality, and how it had an entirely different set of guidelines to conventional user interfaces. In particular the premise of not trying to shock or surprise the user while they’re in what can be a very immersive environment. Obvious once someone explains it to you! Turns out she was also involved in the early days of custom PC builds and internet cafes in Northern Ireland, and has interesting stories to tell. These days she’s concentrating on cyber security - I’ve her to thank for convincing me to persevere with Ghidra - having looked at Bluetooth security as part of her Masters. She’s also deeply aware of the implications of the GDPR and has done some interesting work on thinking about how it affects the computer gaming industry - both from the perspective of the author, and the player.

Claire Wilgar

I’m not particularly fond of modern web design. That’s unfair of me, but web designers seem happy to load megabytes of Javascript from all over the internet just to display the most basic of holding pages. Indeed it seems that such things now require all the includes rather than being simply a matter of HTML, CSS and some graphics, all from the same server. Claire talked at Women Techmakers Belfast about moving away from all of this bloat and back to a minimalistic approach with improved performance, responsiveness and usability, without sacrificing functionality or presentation. She said all the things I want to say to web designers, but from a position of authority, being a front end developer as her day job. It’s great to see someone passionate about front-end development who wants to do things the right way, and talks about it in a way that even people without direct experience of the technologies involved (like me) can understand and appreciate.

Karen Sandler

There aren’t enough people out there who understand law and technology well. Karen is one of the few I’ve encountered who do, and not only that, but really, really gets Free software and the impact of the four freedoms on users in a way many pure technologists do not. She’s had a successful legal career that’s transitioned into being the general counsel for the Software Freedom Law Center, been the executive director of GNOME and is now the executive director of the Software Freedom Conservancy. As someone who likes to think he knows a little bit about law and technology I found Karen’s wealth of knowledge and eloquence slightly intimidating the first time I saw her speak (I think at some event in San Francisco), but I’ve subsequently (gratefully) discovered she has an incredible amount of patience (and ability) when trying to explain the nuances of free software legal issues.

Julien Danjou: Python and fast HTTP clients

Monday 7th of October 2019 09:30:00 AM

Nowadays, it is more than likely that you will have to write an HTTP client for your application that will have to talk to another HTTP server. The ubiquity of REST API makes HTTP a first class citizen. That's why knowing optimization patterns are a prerequisite.

There are many HTTP clients in Python; the most widely used and easy to
work with is requests. It is the de-factor standard nowadays.

Persistent Connections

The first optimization to take into account is the use of a persistent connection to the Web server. Persistent connections are a standard since HTTP 1.1 though many applications do not leverage them. This lack of optimization is simple to explain if you know that when using requests in its simple mode (e.g. with the get function) the connection is closed on return. To avoid that, an application needs to use a Session object that allows reusing an already opened connection.

import requests session = requests.Session() session.get("http://example.com") # Connection is re-used session.get("http://example.com")Using Session with requests

Each connection is stored in a pool of connections (10 by default), the size of
which is also configurable:

import requests session = requests.Session() adapter = requests.adapters.HTTPAdapter( pool_connections=100, pool_maxsize=100) session.mount('http://', adapter) response = session.get("http://example.org")Changing pool size

Reusing the TCP connection to send out several HTTP requests offers a number of performance advantages:

  • Lower CPU and memory usage (fewer connections opened simultaneously).
  • Reduced latency in subsequent requests (no TCP handshaking).
  • Exceptions can be raised without the penalty of closing the TCP connection.

The HTTP protocol also provides pipelining, which allows sending several requests on the same connection without waiting for the replies to come (think batch). Unfortunately, this is not supported by the requests library. However, pipelining requests may not be as fast as sending them in parallel. Indeed, the HTTP 1.1 protocol forces the replies to be sent in the same order as the requests were sent – first-in first-out.

Parallelism

requests also has one major drawback: it is synchronous. Calling requests.get("http://example.org") blocks the program until the HTTP server replies completely. Having the application waiting and doing nothing can be a drawback here. It is possible that the program could do something else rather than sitting idle.

A smart application can mitigate this problem by using a pool of threads like the ones provided by concurrent.futures. It allows parallelizing the HTTP requests in a very rapid way.

from concurrent import futures import requests with futures.ThreadPoolExecutor(max_workers=4) as executor: futures = [ executor.submit( lambda: requests.get("http://example.org")) for _ in range(8) ] results = [ f.result().status_code for f in futures ] print("Results: %s" % results)Using futures with requests

This pattern being quite useful, it has been packaged into a library named requests-futures. The usage of Session objects is made transparent to the developer:

from requests_futures import sessions session = sessions.FuturesSession() futures = [ session.get("http://example.org") for _ in range(8) ] results = [ f.result().status_code for f in futures ] print("Results: %s" % results)Using futures with requests

By default a worker with two threads is created, but a program can easily customize this value by passing the max_workers argument or even its own executor to the FuturSession object – for example like this: FuturesSession(executor=ThreadPoolExecutor(max_workers=10)).

Asynchronicity

As explained earlier, requests is entirely synchronous. That blocks the application while waiting for the server to reply, slowing down the program. Making HTTP requests in threads is one solution, but threads do have their own overhead and this implies parallelism, which is not something everyone is always glad to see in a program.

Starting with version 3.5, Python offers asynchronicity as its core using asyncio. The aiohttp library provides an asynchronous HTTP client built on top of asyncio. This library allows sending requests in series but without waiting for the first reply to come back before sending the new one. In contrast to HTTP pipelining, aiohttp sends the requests over multiple connections in parallel, avoiding the ordering issue explained earlier.

import aiohttp import asyncio async def get(url): async with aiohttp.ClientSession() as session: async with session.get(url) as response: return response loop = asyncio.get_event_loop() coroutines = [get("http://example.com") for _ in range(8)] results = loop.run_until_complete(asyncio.gather(*coroutines)) print("Results: %s" % results)Using aiohttp

All those solutions (using Session, threads, futures or asyncio) offer different approaches to making HTTP clients faster.

Performances

The snippet below is an HTTP client sending requests to httpbin.org, an HTTP API that provides (among other things) an endpoint simulating a long request (a second here). This example implements all the techniques listed above and times them.

import contextlib import time import aiohttp import asyncio import requests from requests_futures import sessions URL = "http://httpbin.org/delay/1" TRIES = 10 @contextlib.contextmanager def report_time(test): t0 = time.time() yield print("Time needed for `%s' called: %.2fs" % (test, time.time() - t0)) with report_time("serialized"): for i in range(TRIES): requests.get(URL) session = requests.Session() with report_time("Session"): for i in range(TRIES): session.get(URL) session = sessions.FuturesSession(max_workers=2) with report_time("FuturesSession w/ 2 workers"): futures = [session.get(URL) for i in range(TRIES)] for f in futures: f.result() session = sessions.FuturesSession(max_workers=TRIES) with report_time("FuturesSession w/ max workers"): futures = [session.get(URL) for i in range(TRIES)] for f in futures: f.result() async def get(url): async with aiohttp.ClientSession() as session: async with session.get(url) as response: await response.read() loop = asyncio.get_event_loop() with report_time("aiohttp"): loop.run_until_complete( asyncio.gather(*[get(URL) for i in range(TRIES)]))Program to compare the performances of different requests usage

Running this program gives the following output:

Time needed for `serialized' called: 12.12s Time needed for `Session' called: 11.22s Time needed for `FuturesSession w/ 2 workers' called: 5.65s Time needed for `FuturesSession w/ max workers' called: 1.25s Time needed for `aiohttp' called: 1.19s

Without any surprise, the slower result comes with the dumb serialized version, since all the requests are made one after another without reusing the connection — 12 seconds to make 10 requests.

Using a Session object and therefore reusing the connection means saving 8% in terms of time, which is already a big and easy win. Minimally, you should always use a session.

If your system and program allow the usage of threads, it is a good call to use them to parallelize the requests. However threads have some overhead, and they are not weight-less. They need to be created, started and then joined.

Unless you are still using old versions of Python, without a doubt using aiohttp should be the way to go nowadays if you want to write a fast and asynchronous HTTP client. It is the fastest and the most scalable solution as it can handle hundreds of parallel requests. The alternative, managing hundreds of threads in parallel is not a great option.

Streaming

Another speed optimization that can be efficient is streaming the requests. When making a request, by default the body of the response is downloaded immediately. The stream parameter provided by the requests library or the content attribute for aiohttp both provide a way to not load the full content in memory as soon as the request is executed.

import requests # Use `with` to make sure the response stream is closed and the connection can # be returned back to the pool. with requests.get('http://example.org', stream=True) as r: print(list(r.iter_content()))Streaming with requestsimport aiohttp import asyncio async def get(url): async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.content.read() loop = asyncio.get_event_loop() tasks = [asyncio.ensure_future(get("http://example.com"))] loop.run_until_complete(asyncio.wait(tasks)) print("Results: %s" % [task.result() for task in tasks])Streaming with aiohttp

Not loading the full content is extremely important in order to avoid allocating potentially hundred of megabytes of memory for nothing. If your program does not need to access the entire content as a whole but can work on chunks, it is probably better to just use those methods. For example, if you're going to save and write the content to a file, reading only a chunk and writing it at the same time is going to be much more memory efficient than reading the whole HTTP body, allocating a giant pile of memory, and then writing it to disk.

I hope that'll make it easier for you to write proper HTTP clients and requests. If you know any other useful technic or method, feel free to write it down in the comment section below!

Antoine Beaupré: This is why native apps matter

Monday 7th of October 2019 12:15:59 AM

I was just looking a web stream on Youtube today and was wondering why my CPU was so busy. So I fired up top and saw my web browser (Firefox) took up around 70% of a CPU to play the stream.

I thought, "this must be some high resolution crazy stream! how modern! such wow!" Then I thought, wait, this is the web, there must be something insane going on.

So I did a little experiment: I started chromium --temp-profile on the stream, alongside vlc (which can also play Youtube streams!). Then I took a snapshot of the top(1) command after 5 minutes. Here are the results:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16332 anarcat 20 0 1805160 269684 102660 S 60,2 1,7 3:34.96 chromium 16288 anarcat 20 0 974872 119752 87532 S 33,2 0,7 1:47.51 chromium 16410 anarcat 20 0 2321152 176668 80808 S 22,0 1,1 1:15.83 vlc 6641 anarcat 20 0 21,1g 520060 137580 S 13,8 3,2 55:36.70 x-www-browser 16292 anarcat 20 0 940340 83980 67080 S 13,2 0,5 0:41.28 chromium 1656 anarcat 20 0 1970912 18736 14576 S 10,9 0,1 4:47.08 pulseaudio 2256 anarcat 20 0 435696 93468 78120 S 7,6 0,6 16:03.57 Xorg 16262 anarcat 20 0 3240272 165664 127328 S 6,2 1,0 0:31.06 chromium 920 message+ 20 0 11052 5104 2948 S 1,3 0,0 2:43.37 dbus-daemon 17915 anarcat 20 0 16664 4164 3276 R 1,3 0,0 0:02.07 top

To deconstruct this, you can see my Firefox process (masquerading as x-www-browser) which has been started for a long time. It's taken 55 hours of CPU time, but let's ignore that for now as it's not in the benchmark. What I find fascinating is there are at least 4 chromium processes running here, and they collectively take up over 7 minutes of CPU time.

Compare this a little over one (1!!!11!!!) minute of CPU time for VLC, and you realize why people are so ranty about everything being packaged as web apps these days. It's basically using up an order of magnitudes more processing power (and therefore electric power and slave labor) to watch those silly movies in your web browsers than in a proper video player.

Keep that in mind next time you let Youtube go on a "autoplay Donald Drumpf" playlist...

More in Tux Machines

OSS Leftovers

  • New Vector to scale open-source alternative to WhatsApp and Slack, where users own their data

    New Vector has announced $8.5 million in funding to scale its open-source, secure communication network, a bid to revolutionise data privacy and ownership in the messaging app space. The investments come from European VCs who specialize in enterprise tech: Notion Capital, Dawn and firstminute capital. Necessary for understanding New Vector’s business is to first understand Matrix. Matrix is an open-source project, building a global network for decentralised communication. Users can collaborate securely via end-to-end encryption, and notably, they retain all ownership and control over their data.

  • New Vector raises $8.5 million to develop an open source Slack and WhatsApp

    Tech giants like Facebook, Google, Apple, and Microsoft needn’t be gatekeepers to communication. That’s the idea upon which Matrix, an open standard and decentralized protocol for real-time communication, was formulated. It’s designed to allow users of one service provider to communicate with users of different providers via online chat, voice over IP, and videotelephony, ideally as seamlessly as SMTP (Simple Mail Transfer Protocol) facilitates email exchanges across clients and services. Implementing the Matrix protocol at scale requires infrastructure and technical expertise, however — and that’s where startups like New Vector have carved out a niche for themselves. In a little over two years, the startup has helped to grow the Matrix network 400% to 11 million users across 40,000 deployments, including French and U.S. government agencies, Wikipedia parent Wikimedia, KDE, RedHat, and more.

  • Paris uses open source to get closer to the citizen

    Around 35 per cent of Paris’ 1,000 IT applications are Lutece-driven and 15 per cent are based on other open-source software, with the remaining 50 per cent using proprietary systems. As applications are upgraded or new ones added, Lutece and open-source tools will be deployed as much as possible, Lanouar said, noting that this approach enables greater autonomy and agility for the City, as well as the ability to be more transparent and create a better user experience for the citizen.

  • After Dallas County's TechShare software failure, the future must be open source

    There has been plenty of coverage of the very expensive failures of TechShare, Dallas County's attempt to create case-tracking software that could be used in any Texas criminal court. Like many battles over operations-level issues, it is easy to miss the forest for the trees. One basic principle of good governing was flagrantly violated in this instance: Government shouldn't be involved in a for-profit operation. TechShare's leadership sought profit, rather than to merely recoup costs. I hope members of both parties can agree this is a principle we should consciously adopt. A public discussion will help avoid future misadventures that cost the county $30 million for a hot plate of nothing. The term "crony capitalism" gets tossed around a lot, and it sometimes unfairly tarnishes good models of public-private partnerships. Crony capitalism usually means the government gives preference to certain favored private firms without seeking the best price (or quality) for a service or good. That preference is odious because it denies taxpayers the best price. Crony capitalism props up firms that would otherwise fail, using taxpayer money as insurance.

  • AI Researchers' Open-Source Model Explanation Toolkit AllenNLP Interpret

    Although the techniques are generic, AllenNLP Interpret is intended for use in NLP. Inputs to NLP systems are strings of text, usually sentences or whole documents, and the text is parsed into its constituent words or tokens. AllenNLP Interpret includes saliency maps that show each token's contribution to the model prediction; a use case for this might be explaining which words in a sentence caused its sentiment to be classified as positive or negative. The toolkit also includes two adversarial methods that show how changing the tokens in the input could affect the output. The first, HotFlip, replaces the input word that has the highest gradient with other words until the model output changes. The other attack, input reduction, iteratively removes the word with the smallest gradient without changing the output; this results in input texts that are "usually nonsensical but cause high confidence predictions."

  • The best open source software of 2019
  • InfoWorld Identifies the Most Innovative Products Available to Developers, Data Analysts, and IT Organizations

    InfoWorld — the technology media brand committed to keeping IT decision-makers ahead of the technology curve — announces the winners of its 2019 Best of Open Source Software Awards, better known as the Bossies. The annual Bossie awards recognize the most important and innovative open source projects for businesses and the IT professionals who serve them. The 26 winners in this year’s Bossie Awards are the next-generation tools and technologies that are enabling digital transformation, allowing businesses to succeed and IT organizations to excel at a time when the technology is more complex than ever.

  • Open Source Rules the World

    Not too long ago I attended Linux Foundation’s Open Source Summit in San Diego, and this declaration of world dominance (tongue in cheek) was a fairly prominent refrain throughout. From best practices in OS development to emerging technologies to getting started—how to create an open source strategy, sustain it, and the right path to developing an Open Source Program Office (OSPO). All open source all the time. What became abundantly clear to me through the cacophony of voices representing developers, technologists and enthusiasts is that at the center of all that is open source are three key components critical to ultimate success (however you define it): people, processes, and technology. [...] The entire tech space is being redesigned by a digital transformation and the emergence of new open source technology platforms. It’s a revolution of sorts, led by groundbreaking innovations in machine learning, open source IoT, cyber security, virtual reality, big data analytics, blockchain and open source development tools. Additionally, there’s technology to help you know what’s in your code and automate the detection and remediation of license compliance and security issues in your DevOps life cycle.

  • Extreme Networks Transitions StackStorm to the Linux Foundation

    Extreme Networks, Inc. (EXTR) today announced it has turned governance of StackStorm™ platform, its popular open-source workflow automation platform, over to The Linux Foundation. In making this transition, Extreme expects the Foundation's open source community to accelerate development and adoption of the platform so enterprises everywhere can reap the benefits of new applications and use cases.

  • ExpressionEngine Under New Ownership, Will Remain Open Source for Now

    EllisLab founder Rick Ellis announced yesterday that ExpressionEngine has been acquired by Packet Tide, the parent company of EEHarbor, one of the most successful EE add-on providers and development agencies in the community. A year ago EllisLab, the developers of EE core, was acquired by Digital Locations but Ellis said the company ended up not being a good fit for the future of the CMS...

  • Open Source Seed, a Hoax or a Wake-Up Call?

    “Open source” is a trend in various industries. It started to take root in the software industry (Mozilla), followed by biotechnology (CAMBIA) and publishing, where the creative commons concepts have taken root. Several of these trends are based in an opposition against corporate power generated by exclusive rights provided by patents and copyright. Others have a positive goal, i.e. to enhance participation by a much wider population to generate, validate and share information (e.g. Wikipedia). The seed sector has a very good story to tell with regard to its contributions to societal goals, but in parts of society, the corporate image and the use of patents create questions, so we could expect that also our sector would be challenged. It is there now. The University of Wisconsin developed an Open Source Seed Initiative several years ago, which was followed in Germany more recently. Access to “freed” plant genetic resources is made conditional to users making them available under the same “open source” conditions – that no IP is vested. The system should thus go “viral” and “force” breeders to join and thus stop protecting their products through IP.

  • Satellite images and open-source programs for mapping during disasters

    A few weeks ago, the states of Assam and Bihar were reeling under floods. Over 200 people were reported dead, with at least 10 million (one crore) of the states’ residents estimated to have been displaced. To save more lives and prevent further infrastructural damage, search and rescue missions during such disasters need to be effective, and more importantly, need to be rapid. The answer to this may lie in space. Open-source access to satellite images and new technologies to process these images have been a significant breakthrough to help document the true extent of flooding. Getting this information in time is key to plan and conduct evacuation missions, response operations and damage assessments. The European Space Agency (ESA)’s Sentinel-1 mission and the web-based Google Earth Engine (GEE) platform are two recent developments that have helped timely capture and analysis of satellite information. A research team from the Indian Institute for Human Settlements (IIHS) used this combination (Sentinel and GEE) to come up with an illustrative example of how such mapping can be used in the future to help in rescue missions, through accurate mapping of flood extents.

Events: Fibre Optic Conference, All Things Open and HacktoberFest

  • Andile Ngcaba urges embracing open source

    Given the growth of data and the Internet of things, insofar as data is concerned, the fibre industry must adopt open source architecture in terms of designing and building networks. This is the sentiment shared by Andile Ngcaba, president of the FTTx Council Africa, at the annual Fibre Optic Conference that kicked-off at the Sandton Convention Centre yesterday. Ngcaba was speaking about the future of the industry and how to be part of it, pointing out that modern businesses are being built on open source, while modern telcos are going to be built on open source.

  • All Things Open: The ‘hidden tech gem in the Triangle’ that draws thousands

    In its seventh year, All Things Open is preparing for more than 5,000 attendees. The conference will feature more than 250 talks from some of the top technologists and decision-makers discussing open source technology during three days of programming at the Raleigh Convention Center.

  • Six reasons why you should attend All Things Open in Raleigh

    Haven’t decided whether to attend the All Things Open conference in Raleigh? Well, Open Source is growing more important in technology so you might want to keep an open mind about attending. And more than 4,500 people are already scheduled to attend. Action begins Sunday.

  • Tech Village Hosting HacktoberFest Open-Source Meetup This Weekend

    The event will be hosted in Bulawayo in the 1st floor of the NetOne Building, Corner Fife Street and L.Takawira. Opposite Central Police Station. Maintainers -the guys/girls who build source code into a binary package for distribution, commit patches, or organize code in a source repository– will be present to help out would-be contributors to help move open-source projects forward.

FOSS in SaaS/Back End/Databases

  • What to expect from Scylla Summit 2019

    Scylla (the company) takes its name directly from Scylla [pronounced: sill-la], a Greek god sea monster whose mission was to haunt and torment the rocks of a narrow strait of water opposite the Charybdis whirlpool. Outside of Greek history, Scylla is an open source essentially distributed NoSQL data store that uses a sharded design on each node, meaning each CPU core handles a different subset of data.

  • Licence to grill: A year on, MongoDB's Eliot Horowitz talks to The Reg about SSPL

    A year after its controversial switch to the Server Side Public License (SSPL), and with new products livening up the summer, MongoDB remains unrepentant. The change was aimed at making vendors selling a service using the company's code share the source of applications used to run the service as well as any tweaks. The move appeared to be aimed squarely at cloud vendors, content to "capture all the value and give nothing back to the community," as Dev Ittycheria, CEO of MongoDB, told us at the time. Elements of the open source community were less than impressed. The Open Source Initiative (OSI) rejected the company's attempts to get the licence approved and eventually MongoDB withdrew the thing from the process, although the company continued to use it for its own products. Indeed, at MongoDB's London .Local event, where we met co-founder and CTO Eliot Horowitz, the company was trumpeting the opening up of its Compass GUI for MongoDB under the SSPL.

  • From Russia with OLAP: Percona uses ClickHouse analytics

    At Percona Live Europe last week, one such example came up around the open source scene that is developing in Russia and how one of the projects that is now starting to open up to international use.

  • The love and the lament: Percona CEO details state of open source data

    Open source has changed, obviously it has. Starting from its origins among the hobbyist programmers and hackers who dared to defy the proprietary Silicon Valley behemoths, the open community-centric model for software development has now been widely adopted by the commercial software sector. In many cases, open source has become the norm for modern platforms, tools and applications. But how has this affected the nature of open development and what impact has this shift left in its wake on the data landscape that we view today?

  • GraphDB 9.0 Open Sources Its Front End and Engine Plugins to Support Knowledge Graph Solutions

    Ontotext has announced GraphDB 9.0, which is aimed at lowering the effort required for development and continuous operation of knowledge graphs by opening multiple integration extension points for its users and developers. GraphDB is a database for managing semantic information with more than 30 large production installations in big enterprises. With the growing complexity of enterprise data integration, many organizations are starting the journey of building knowledge graphs.

  • Ververica Announces Open Source Framework to Enable Lightweight, Stateful Applications at Scale

    Ververica, the original creators of Apache Flink, today announced at Flink Forward Europe the launch of Stateful Functions (statefun.io), an open source framework that reduces the complexity of building and orchestrating stateful applications at scale. Stateful Functions enables users to define loosely coupled, independent functions with a low footprint that can interact consistently and reliably in a shared pool of resources. Ververica will propose the project, licensed under Apache 2.0, to the Apache Flink community as an open source contribution.

  • DataStax offers bidirectional data dexterity for Apache Kafka

    DataStax has opened up ‘early access’ to its DataStax Change Data Capture (CDC) Connector for Apache Kafka, the open source stream-processing (where applications can use multiple computational units, similar to parallel processing) software platform. As a company, DataStax offers a commercially supported ‘enterprise-robust’ database built on open source Apache Cassandra. Stream processing is all about speed and cadence, so, the DataStax CDC Connector for Apache Kafka gives developers ‘bidirectional data movement’ between DataStax, Cassandra and Kafka clusters.

Security: WireGuard, SafeBreach and More

  • WireGuard Snapshot `0.0.20191012` Available
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA256
    
    Hello,
    
    A new snapshot, `0.0.20191012`, has been tagged in the git repository.
    
    Please note that this snapshot is a snapshot rather than a final
    release that is considered secure and bug-free. WireGuard is generally
    thought to be fairly stable, and most likely will not crash your
    computer (though it may).  However, as this is a snapshot, it comes
    with no guarantees; it is not applicable for CVEs.
    
    With all that said, if you'd like to test this snapshot out, there are a
    few relevant changes.
    
    == Changes ==
    
      * qemu: bump default version
      * netns: add test for failing 5.3 FIB changes
      
      Kernels 5.3.0 - 5.3.3 crash (and are probably exploitable) via this one liner:
      
      unshare -rUn sh -c 'ip link add dummy1 type dummy && ip link set dummy1 up && ip -6 route add default dev dummy1 && ip -6 rule add table main suppress_prefixlength 0 && ping -f 1234::1'
      
      We fixed this upstream here:
      
      https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ca7a03c4175366a92cee0ccc4fec0038c3266e26
      
      This is relevant to WireGuard because a very similar sequence of commands is
      used by wg-quick(8).
      
      So, we've now added some tests to catch this code path in the future. While
      the bug here was a random old use-after-free, the test checks the general
      policy routing setup used by wg-quick(8), so that we make sure this continues
      to work with future kernels.
      
      * noise: recompare stamps after taking write lock
      
      We now recompare counters while holding a write lock.
      
      * netlink: allow preventing creation of new peers when updating
      
      This is a small enhancement for wg-dynamic, so that we can update peers
      without readding them if they've already been removed.
      
      * wg-quick: android: use Binder for setting DNS on Android 10
      
      wg-quick(8) for Android now supports Android 10 (Q). We'll be releasing a new
      version of the app for this later today.
    
    This snapshot contains commits from: Jason A. Donenfeld and Nicolas Douma.
    
    As always, the source is available at https://git.zx2c4.com/WireGuard/ and
    information about the project is available at https://www.wireguard.com/ .
    
    This snapshot is available in compressed tarball form here:
      https://git.zx2c4.com/WireGuard/snapshot/WireGuard-0.0.20191012.tar.xz
      SHA2-256: 93573193c9c1c22fde31eb1729ad428ca39da77a603a3d81561a9816ccecfa8e
      BLAKE2b-256: d7979c453201b9fb6b1ad12092515b27ea6899397637a34f46e74b52b36ddf56
    
    A PGP signature of that file decompressed is available here:
      https://git.zx2c4.com/WireGuard/snapshot/WireGuard-0.0.20191012.tar.asc
      Signing key: AB9942E6D4A4CFC3412620A749FC7012A5DE03AE
    
    If you're a snapshot package maintainer, please bump your package version. If
    you're a user, the WireGuard team welcomes any and all feedback on this latest
    snapshot.
    
    Finally, WireGuard development thrives on donations. By popular demand, we
    have a webpage for this: https://www.wireguard.com/donations/
    
    Thank you,
    Jason Donenfeld
    
  • WireGuard 0.0.20191012 Released With Latest Fixes

    WireGuard is still working on transitioning to the Linux kernel's existing crypto API as a faster approach to finally make it into the mainline kernel, but for those using the out-of-tree WireGuard secure VPN tunnel support, a new development release is available.

  • SafeBreach catches vulnerability in controversial HP Touchpoint Analytics software

    Now the feature is embroiled in another minor controversy after security researchers at SafeBreach said they uncovered a new vulnerability. HP Touchpoint Analytics comes preinstalled on many HP devices that run Windows. Every version below 4.1.4.2827 is affected by what SafeBreach found. In a blog post, SafeBreach Labs security researcher Peleg Hadar said that because the service is executed as "NT AUTHORITY\SYSTEM," it is afforded extremely powerful permissions that give it wide access. "The CVE-2019-6333 vulnerability gives attackers the ability to load and execute malicious payloads using a signed service. This ability might be abused by an attacker for different purposes such as execution and evasion, for example: Application Whitelisting Bypass Signature Validation Bypassing," Hadar wrote. [...] The company has long had to defend HP Touchpoint Analytics against critics who say it gives HP unnecessary access to users' systems. When it first became widely noticed in 2017, dozens of users complained that they had not consented to adding the system.

  • Security Tool Sprawl Reaches Tipping Point
  • How trusted digital certificates complement open source security

    Application developers incorporating open source software into their designs may only discover later that elements of this software have left them (and their customers) exposed to cyber-attacks.

  • Securing the Container Supply Chain