Language Selection

English French German Italian Portuguese Spanish

Debian

Syndicate content
Planet Debian - https://planet.debian.org/
Updated: 3 hours 5 min ago

Jelmer Vernooij: Improvements to Merge Proposals by the Janitor

Saturday 8th of August 2020 12:30:23 AM

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor.

Since the original post, merge proposals created by the janitor now include the debdiff between a build with and without the changes (showing the impact to the binary packages), in addition to the merge proposal diff (which shows the impact to the source package).

New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds.

This is part of the effort to keep the changes from the janitor high-quality.

The rollout surfaced some bugs in lintian-brush; these have been either fixed or mitigated (e.g. by disabling specified fixers).

For more information the Janitor's lintian-fixes efforts, see the landing page

Antonio Terceiro: When community service providers fail

Friday 7th of August 2020 10:00:00 PM

I'm starting a new blog, and instead of going into the technical details on how it's made, or on how I migrated my content from the previous ones, I want to focus on why I did it.

It's been a while since I have written a blog post. I wanted to get back into it, but also wanted to finally self-host my blog/homepage because I have been let down before. And sadly, I was not let down by a for-profit, privacy invading corporation, but by a free software organization.

The sad story of wiki.softwarelivre.org

My first blog that was hosted in a blog engine written by me, which was hosted in a TWiki, and later Foswiki instance previously available at wiki.softwarelivre.org, hosted by ASL.org.

I was the one who introduced the tool to the organization in the first place. I had come from a previous, very fruitful experience on the use of wikis for creation of educational material while in university, which ultimately led me to become a core TWiki, and then Foswiki developer.

In 2004, I had just moved to Porto Alegre, got involved in ASL.org, and there was a demand for a tool like that. 2 years later, I left Porto Alegre, and some time after that the daily operations of ASL.org when it became clear that it was not really prepared for remote participation. I was still maintaing the wiki software and the OS for quite some years after, until I wasn't anymore.

In 2016, the server that hosted it went haywire, and there were no backups. A lot of people and free software groups lost their content forever. My blog was the least important content in there. To mention just a few examples, here are some groups that lost their content in there:

  • The Brazilian Foswiki user group hosted a bunch of getting started documentation in Portuguese, organized meetings, and coordinated through there.
  • GNOME Brazil hosted its homepage there until the moment the server broke.
  • The Inkscape Brazil user group had an amazing space there where they shared tutorials, a gallery of user-contributed drawings, and a lot more.

Some of this can still be reached via the Internet Archive Wayback Machine, but that is only useful for recovering content, not for it to be used by the public.

The announced tragedy of softwarelivre.org

My next blog after that was hosted at softwarelivre.org, a Noosfero instance also hosted by ASL.org. When it was introduced in 2010, this Noosfero instance became responsible for the main domain softwarelivre.org name. This was a bold move by ASL.org, and was a demonstration of trust in a local free software project, led by a local free software cooperative (Colivre).

I was a lead developer in the Noosfero project for a long time, and I was also involved in maintaining that server as well.

However, for several years there is little to no investment in maintaining that service. I already expect that it will probably blow up at some point as the wiki did, or that it may be just shut down on purpose.

On the responsibility of organizations

Today, a large part of wast mot people consider "the internet" is controlled by a handful of corporations. Most popular services on the internet might look like they are gratis (free as in beer), but running those services is definitely not without costs. So when you use services provided by for-profit companies and are not paying for them with money, you are paying with your privacy and attention.

Society needs independent organizations to provide alternatives.

The market can solve a part of the problem by providing ethical services and charging for them. This is legitimate, and as long as there is transparency about how peoples' data and communications are handled, there is nothing wrong with it.

But that only solves part of the problem, as there will always be people who can't afford to pay, and people and communities who can afford to pay, but would rather rely on a nonprofit. That's where community-based services, provided by nonprofits, are also important. We should have more of them, not less.

So it makes me worry to realize ASL.org left the community in the dark. Losing the wiki wasn't even the first event of its kind, as the listas.softwarelivre.org mailing list server, with years and years of community communications archived in it, broke with no backups in 2012.

I do not intend to blame the ASL.org leadership personally, they are all well meaning and good people. But as an organization, it failed to recognize the importance of this role of service provider. I can even include myself in it: I was member of the ASL.org board some 15 years ago; I was involved in the deployment of both the wiki and Noosfero, the former as a volunteer and the later professionally. Yet, I did nothing to plan the maintenance of the infrastructure going forward.

When well meaning organizations fail, people who are not willing to have their data and communications be exploited for profit are left to their own devices. I can afford a virtual private server, and have the technical knowledge to import my old content into a static website generator, so I did it. But what about all the people who can't, or don't?

Of course, these organizations have to solve the challenge of being sustainable, and being able to pay professionals to maintain the services that the community relies on. We should be thankful to these organizations, and their leadership needs to recognize the importance of those services, and actively plan for them to be kept alive.

Antonio Terceiro: When community service providers fail

Friday 7th of August 2020 10:00:00 PM

I'm starting a new blog, and instead of going into the technical details on how it's made, or on how I migrated my content from the previous ones, I want focus on why I did it.

It's been a while since I have written a blog post. I wanted to get back into it, but also wanted to finally self-host my blog/homepage because I have been let down before. And sadly, I was not let down by a for-profit, privacy invading corporation, but by a free software organization.

The sad story of wiki.softwarelivre.org

My first blog that was hosted in a blog engine written by me, which was hosted in a TWiki, and later Foswiki instance previously available at wiki.softwarelivre.org, hosted by ASL.org.

I was the one who introduced the tool to the organization in the first place. I had come from a previous, very fruitful experience on the use of wikis for creation of educational material while in university, which ultimately led me to become a core TWiki, and then Foswiki developer.

In 2004, I had just moved to Porto Alegre, got involved in ASL.org, and there was a demand for a tool like that. 2 years later, I left Porto Alegre, and some time after that the daily operations of ASL.org when it became clear that it was not really prepared for remote participation. I was still maintaing the wiki software and the OS for quite some years after, until I wasn't anymore.

In 2016, the server that hosted it went haywire, and there were no backups. A lot of people and free software groups lost their content forever. My blog was the least important content in there. To mention just a few examples, here are some groups that lost their content in there:

  • The Brazilian Foswiki user group hosted a bunch of getting started documentation in Portuguese, organized meetings, and coordinated through there.
  • GNOME Brazil hosted its homepage there until the moment the server broke.
  • The Inkscape Brazil user group had an amazing space there where they shared tutorials, a gallery of user-contributed drawings, and a lot more.

Some of this can still be reached via the Internet Archive Wayback Machine, but that is only useful for recovering content, not for it to be used by the public.

The announced tragedy of softwarelivre.org

My next blog after that was hosted at softwarelivre.org, a Noosfero instance also hosted by ASL.org. When it was introduced in 2010, this Noosfero instance became responsible for the main domain softwarelivre.org name. This was a bold move by ASL.org, and was a demonstration of trust in a local free software project, led by a local free software cooperative (Colivre).

I was a lead developer in the Noosfero project for a long time, and I was also involved in maintaining that server as well.

However, for several years there is little to no investment in maintaining that service. I already expect that it will probably blow up at some point as the wiki did, or that it may be just shut down on purpose.

On the responsibility of organizations

Today, a large part of wast mot people consider "the internet" is controlled by a handful of corporations. Most popular services on the internet might look like they are gratis (free as in beer), but running those services is definitely not without costs. So when you use services provided by for-profit companies and are not paying for them with money, you are paying with your privacy and attention.

Society needs independent organizations to provide alternatives.

The market can solve a part of the problem by providing ethical services and charging for them. This is legitimate, and as long as there is transparency about how peoples' data and communications are handled, there is nothing wrong with it.

But that only solves part of the problem, as there will always be people who can't afford to pay, and people and communities who can afford to pay, but would rather rely on a nonprofit. That's where community-based services, provided by nonprofits, are also important. We should have more of them, not less.

So it makes me worry to realize ASL.org left the community in the dark. Losing the wiki wasn't even the first event of its kind, as the listas.softwarelivre.org mailing list server, with years and years of community communications archived in it, broke with no backups in 2012.

I do not intend to blame the ASL.org leadership personally, they are all well meaning and good people. But as an organization, it failed to recognize the importance of this role of service provider. I can even include myself in it: I was member of the ASL.org board some 15 years ago; I was involved in the deployment of both the wiki and Noosfero, the former as a volunteer and the later professionally. Yet, I did nothing to plan the maintenance of the infrastructure going forward.

When well meaning organizations fail, people who are not willing to have their data and communications be exploited for profit are left to their own devices. I can afford a virtual private server, and have the technical knowledge to import my old content into a static website generator, so I did it. But what about all the people who can't, or don't?

Of course, these organizations have to solve the challenge of being sustainable, and being able to pay professionals to maintain the services that the community relies on. We should be thankful to these organizations, and their leadership needs to recognize the importance of those services, and actively plan for them to be kept alive.

Jonathan Dowland: Vimwiki

Friday 7th of August 2020 10:55:42 AM

At the start of the year I begun keeping a daily diary for work as a simple text file. I've used various other approaches for this over the years, including many paper diaries and more complex digital systems. One great advantage of the one-page text file was it made assembling my weekly status report email very quick, nearly just a series of copies and pastes. But of course there are drawbacks and room for improvement.

vimwiki is a personal wiki plugin for the vim and neovim editors. I've tried to look at it before, years ago, but I found it too invasive, changing key bindings and display settings for any use of vim, and I use vim a lot.

I decided to give it another look. The trigger was actually something completely unrelated: Steve Losh's blog post "Coming Home to vim". I've been using vim for around 17 years but I still learned some new things from that blog post. In particular, I've never bothered to Use The Leader for user-specific shortcuts.

The Leader, to me, feels like a namespace that plugins should not touch: it's like the /usr/local of shortcut keys, a space for the local user only. Vimwiki's default bindings include several incorporating the Leader. Of course since I didn't use the leader, those weren't the ones that bothered me: It turns out I regularly use carriage return and backspace for moving the cursor around in normal mode, and Vimwiki steals both of those. It also truncates the display of (what it thinks are) URIs. It turns out I really prefer to see exactly what's in the file I'm editing. I haven't used vim folds since I first switched to it, despite them being why I switched.

Disabling all the default bindings and URI concealing stuff and Vimwiki is now much less invasive and I can explore its features at my own pace:

let g:vimwiki_key_mappings = { 'all_maps': 0, } let g:vimwiki_conceallevel = 0 let g:vimwiki_url_maxsave = 0

Followed by explicitly configuring the bindings I want. I'm letting it steal carriage return. And yes, I've used some Leader bindings after all.

nnoremap <leader>ww :VimwikiIndex<cr> nnoremap <leader>wi :VimwikiDiaryIndex<cr> nnoremap <leader>wd :VimwikiMakeDiaryNote<cr> nnoremap <CR> :VimwikiFollowLink<cr> nnoremap <Tab> :VimwikiNextLink<cr> nnoremap <S-Tab> :VimwikiPrevLink<cr> nnoremap <C-Down> :VimwikiDiaryNextDay<cr> nnoremap <C-Up> :VimwikiDiaryPrevDay<cr>

,wd (my leader) now brings me straight to today's diary page, and I can create separate, non-diary pages for particular work items (e.g. a Ticket reference) that will span more than one day, and keep all the relevant stuff in one place.

Reproducible Builds (diffoscope): diffoscope 155 released

Friday 7th of August 2020 12:00:00 AM

The diffoscope maintainers are pleased to announce the release of diffoscope version 155. This version includes the following changes:

[ Chris Lamb ] * Bump Python requirement from 3.6 to 3.7 - most distributions are either shipping3.5 or 3.7, so supporting 3.6 is not somewhat unnecessary and also more difficult to test locally. * Improvements to setup.py: - Apply the Black source code reformatter. - Add some URLs for the site of PyPI.org. - Update "author" and author email. * Explicitly support Python 3.8. [ Frazer Clews ] * Move away from the deprecated logger.warn method logger.warning. [ Mattia Rizzolo ] * Document ("classify") on PyPI that this project works with Python 3.8.

You find out more by visiting the project homepage.

Dirk Eddelbuettel: nanotime 0.3.0: Yuge New Features!

Thursday 6th of August 2020 11:53:00 PM

A fresh major release of the nanotime package for working with nanosecond timestamps is hitting CRAN mirrors right now.

nanotime relies on the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from work by Leonardo Silvestri who rejigged internals in S4—and now added new types for periods, intervals and durations. This is what is commonly called a big fucking deal!! So a really REALLY big thank you to my coauthor Leonardo for all these contributions.

With all these Yuge changes patiently chisseled in by Leonardo, it took some time since the last release and a few more things piled up. Matt Dowle corrected something we borked for integration with the lovely and irreplacable data.table. We also switched to the awesome yet minimal tinytest package by Mark van der Loo, and last but not least we added the beginnings of a proper vignette—currently at nine pages but far from complete.

The NEWS snippet adds full details.

Changes in version 0.3.0 (2020-08-06)
  • Use tzstr= instead of tz= in call to RcppCCTZ::parseDouble()) (Matt Dowle in #49).

  • Add new comparison operators for nanotime and charcters (Dirk in #54 fixing #52).

  • Switch from RUnit to tinytest (Dirk in #55)

  • Substantial functionality extension in with new types nanoduration, nanoival and nanoperiod (Leonardo in #58, #60, #62, #63, #65, #67, #70 fixing #47, #51, #57, #61, #64 with assistance from Dirk).

  • A new (yet still draft-ish) vignette was added describing the four core types (Leonardo and Dirk in #71).

  • A required compilation flag for Windows was added (Leonardo in #72).

  • RcppCCTZ function are called in new 'non-throwing' variants to not trigger exeception errors (Leonardo in #73).

We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Chris Lamb: The Bringers of Beethoven

Thursday 6th of August 2020 09:48:27 PM

This is a curiously poignant work to me that I doubt I would ever be able to communicate. I found it about fifteen years ago, along with a friend who I am quite regrettably no longer in regular contact with, so there was some complicated nostalgia entangled with rediscovering it today.

What might I say about it instead? One tell-tale sign of 'good' art is that you can find something new in it, or yourself, each time. In this sense, despite The Bringers of Beethoven being more than a little ridiculous, it is somehow 'good' music to me. For example, it only really dawned on me now that the whole poem is an allegory for a GDR-like totalitarianism.

But I also realised that it is not an accident that it is Beethoven himself (quite literally the soundtrack for Enlightenment humanism) that is being weaponised here, rather than some fourth-rate composer of military marches or one with a problematic past. That is to say, not only is the poem arguing that something universally recognised as an unalloyed good can be subverted for propagandistic ends, but that is precisely the point being made by the regime. An inverted Clockwork Orange, if you like.

Yet when I listen to it again I can't help but laugh. I think of the 18th-century poet Alexander Pope, who first used the word bathos to refer to those abrupt and often absurd transitions from the elevated to the ordinary, contrasting it with the concept of pathos, the sincere feeling of sadness and tragedy. I can't think of two better words.

Joey Hess: Mr Process's wild ride

Thursday 6th of August 2020 08:02:28 PM

When a unix process is running in a directory, and that directory gets renamed, the process is taken on a ride to a new location in the filesystem. Suddenly, any "../" paths it might be using point to new, and unexpected locations.

This can be a source of interesting behavior, and also of security holes.

Suppose root is poking around in ~user/foo/bar/ and decides to vim ../../etc/conffile

If the user notices this process is running, they can mv ~/foo/bar /tmp and when vim saves the file, it will write to /tmp/bar/../../etc/conffile AKA /etc/conffile.

(Vim does warn that the file has changed while it was being edited. Other editors may not. Or root may be feeling especially BoFH and decide to overwrite the user's changes to their file. Or the rename could perhaps be carefully timed to avoid vim's overwrite protection.)

Or, suppose root, in the same place, decides to archive ../../etc with tar, and then delete it:

tar cf etc.tar ../../etc; rm -rf ../../etc

Now the user has some time to take root's shell on a ride, before the rm starts ... and make it delete all of /etc!

Anyone know if this class of security hole has a name?

Christian Kastner: My new favorite utility: autojump

Thursday 6th of August 2020 02:41:08 PM

Like any developer, I have amassed an impressive collection of directory trees both broad and deep. Navigating these trees became increasingly cumbersome, and setting CDPATH, using auto-completion, and working with the readline history search alleviated this only somewhat.

Enter autojump, from the package of the same name.

Whatever magic it uses is unbelievably effective. I estimate that in at least 95% of my cases, typing j <name-fragment> changes to the directory I was actually thinking of.

Say I'm working on package scikit-learn. My clone of the Salsa repo is in ~/code/pkg-scikit-learn/scikit-learn. Changing to that directory is trivial, I only need to specify a name fragment:

$ j sci /home/christian/code/pkg-scikit-learn/scikit-learn christian@workstation:~/code/pkg-scikit-learn/scikit-learn

But what if I want to work on scikit-learn upstream, to prepare a patch, for example? That repo has been cloned to ~/code/github/scikit-learn. No problem at all, just add another name fragment:

$ j gi sci /home/christian/code/github/scikit-learn christian@workstation:~/code/github/scikit-learn

The magic, however, is most evident with directory trees I rarely enter. As in: I have a good idea of the directory name I wish to change to, but I don't really recall its exact name, nor where (in the tree) it is located. I used to rely on autocomplete to somehow get there which can involve hitting the [TAB] key far too many times, and falling back to find in the worst case, but now, autojump always seems gets me there on first try.

I can't believe that this has been available in Debian for 10 years and I only discovered it now.

Sam Hartman: Good Job Debian: Compatibility back to 1999

Thursday 6th of August 2020 12:54:58 PM
So, I needed a container of Debian Slink (2.1), released back in 1999. I expected this was going to be a long and involved process. Things didn't look good from the start:
root@mount-peerless:/usr/lib/python3/dist-packages/sqlalchemy# debootstrap slink /build/slink2 http://archive.debian.org/debian E: No such script: /usr/share/debootstrap/scripts/slink
Hmm, I thought I remembered slink support for debootstrap--not that slink used debootstrap by default--back when I was looking through the debootstrap sources years ago. Sure enough looking through the changelogs, slink support was dropped back in 2005.
Okay, well, this isn't going to work either, but I guess I could try debootstrapping sarge and from there go back to slink.
Except it worked fine.
Go us!

Holger Levsen: 20200805-debconf7

Wednesday 5th of August 2020 10:27:00 PM
DebConf7

This tshirt is 13 years old and from DebConf7.

DebConf7 was my 5th DebConf and took place in Edinburgh, Scotland.

And finally I could tell people I was a DD Though as you can guess, that's yet another story to be told. So anyway, Edinburgh.

I don't recall exactly whether the video team had to record 6 or 7 talk rooms on 4 floors, but this was probably the most intense set up we ran. And we ran a lot, from floor to floor, and room to room.

DebConf7 was also special because it had a very special night venue, which was in an ex-church in a rather normal building, operated as sort of community center or some such, while the old church interior was still very much visible as in everything new was build around the old stuff.

And while the night venue was cool, it also ment we (video team) had no access to our machines over night (or for much of the evening), because we had to leave the university over night and the networking situation didn't allow remote access with the bandwidth needed to do anything video.

The night venue had some very simple house rules, like don't rearrange stuff, don't break stuff, don't fix stuff and just a few little more and of course we broke them in the best possible way: Toresbe with the help of people I don't remember fixed the organ, which was broken for decades. And so the house sounded in some very nice new old tune and I think everybody was happy we broke that rule.

I believe the city is really nice from the little I've seen of it. A very nice old town, a big castle on the hill I'm not sure whether I missed the day trip to Glasgow to fix video things or to rest or both...

Another thing I missed was getting a kilt, for which Phil Hands made a terrific design (update: the design is called tartan and was made by Phil indeed!), which spelled Debian in morse code. That was pretty cool and the kilts are really nice on DebConf group pictures since then. And if you've been wearing this kilt regularily for the last 13 years it was probably also a sensible investment.

It seems I don't have that many more memories of this DebConf, British power plugs and how to hack them comes to my mind and some other stuff here and there, but I remember less than previous years. I'm blaming this on the intense video setup and also on the sheer amount of people, which was the hightest until then and for some years, I believe maybe even until Heidelberg 8 years later. IIRC there were around 470 people there and over my first five years of DebConf I was incredible lucky to make many friends in Debian, so I probably just hung out and had good times.

Holger Levsen: 20200805-debconf7.jpg

Wednesday 5th of August 2020 09:46:21 PM

Dirk Eddelbuettel: RcppCCTZ 0.2.8: Minor API Extension

Wednesday 5th of August 2020 01:25:00 AM

A new minor release 0.2.8 of RcppCCTZ is now on CRAN.

RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. And while CCTZ is made by Google(rs), it is not an official Google product. The RcppCCTZ page has a few usage examples and details. This package was the first CRAN package to use CCTZ; by now at least three others do—using copies in their packages which remains less than ideal.

This version adds three no throw variants of three existing functions, contributed again by Leonardo. This will be used in an upcoming nanotime release which we are finalising now.

Changes in version 0.2.8 (2020-08-04)
  • Added three new nothrow variants (for win32) needed by the expanded nanotime package (Leonardo in #37)

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Holger Levsen: 20200804-debconf6

Wednesday 5th of August 2020 12:24:27 AM
DebConf6

This tshirt is 14 years old and from DebConf6.

DebConf6 was my 4th DebConf and took place in Oaxtepec, Mexico.

I'm a bit exhausted right now which is probably quite fitting to write something about DebConf6... many things in life are a question of perception, so I will mention the waterfall and the big swirl and the band playing with the fireworks during the conference dinner, the joy that we finally could use the local fiber network (after asking for months) just after discovering that the 6h shopping tour forgot to bring the essential pig tail connectors to connect the wireless antennas to the cards, which we needed to provide network to the rooms where the talks would take place.

DebConf6 was the first DebConf with live streaming using dvswitch (written by Ben Hutchings and removed from unstable in 2015 as the world had moved to voctomix, which is yet another story to be told eventually). The first years (so DebConf6 and some) the videoteam focussed on getting the post processing done and the videos released, and streaming was optional, even though it was an exciting new feature and we still managed to stream mostly all we recorded and sometimes more...

Setting up the network uplink also was very challenging and took, I don't remember exactly, until day 4 or 5 of DebCamp (which lasted 7 days), so there were group of geeks in need of network, and mostly unable to fix it, because for fixing it we needed to communicate and IRC was down. (There was no mobile phone data at that time, the first iphone wasn't sold yet, it were the dark ages.)

I remember literally standing on a roof to catch the wifi signal and excitingly shouting "I got one ping back! ... one ping back ...", less excitingly. I'll spare you the details now (and me writing them down) but I'll say that the solution involved Neil McGovern climbing an antenna and attaching a wifi antenna up high, probably 15m or 20m or some such. Finally we had uplink. I don't recall if that pig tail connector incident happened before of after, but in the end the network setup worked nicely on the wide area we occupied. Even though in some dorms the cleaning people daily removed one of our APs to be able to watch TV while cleaning (Which kind of was ok, but still... they could have plugged it back in.)

I also joyfully remember a certain vegetarian table, a most memorable bus ride (I'll just say 5 or rather cinco, and, unrelated except on the same bus ride, "Jesus" (and "Maria" for sure..)!) and talking with Jim Gettys and thus learning about the One Laptop per Child (OLPC) project.

As for any DebConf, there's sooo much more to be told, but I'll end here and just thank Gunnar Wolf (as he masterminded much of this DebConf) and go to bed now

Osamu Aoki: exim4 configuration for Desktop (better gmail support)

Tuesday 4th of August 2020 03:03:46 PM
Since gmail rewrites "From:" address now (2020) and keep changing access limitation, it is wise not  to use it as smarthost any more.  (If you need to access multiple gmail addresses from mutt etc, use esmtp etc.)
---For most of our Desktop PC running with stock exim4 and mutt, I think sending out mail is becoming a bit rough since using random smarthost causes lots of trouble due to the measures taken to prevent spams.

As mentioned in Exim4 user FAQ , /etc/hosts should have FQDN with external DNS resolvable domain name listed instead of localdomain to get the correct EHLO/HELO line.  That's the first step.

The stock configuration of exim4 only allows you to use single smarthost for all your mails.  I use one address for my personal use which is checked by my smartphone too.  The other account is for subscribing to the mailing list.  So I needed to tweak ...

Usually, mutt is smart enough to set the From address since my .muttrc has

# Set default for From: for replyes for alternates.
set reverse_name

So how can I teach exim4 to send mails depending on the  mail accounts listed in the From header.

For my gmail accounts, each mail should be sent to the account specific SMTP connection matching your From header to get all the modern SPAM protection data in right state.  DKIM, SPF, DMARC...  (Besides, they overwrite From: header anyway if you use wrong connection.)

For my debian.org mails, mails should be sent from my shell account on people.debian.org so it is very unlikely to be blocked.  Sometimes, I wasn't sure some of these debian.org mails sent through my ISP's smarthost are really getting to the intended person.

To these ends, I have created small patches to the /etc/exim4/conf.d files and reported it to Debian BTS: #869480 Support multiple smarthosts (gmail support).  These patches are for the source package.

To use my configuration tweak idea, you have easier route no matter which exim version you are using.  Please copy and read pertinent edited files from my github site to your installed /etc/exim4/conf.d files and get the benefits.
If you really wish to keep envelope address etc. to match From: header, please rewite agressively using the From: header using eddited rewrite/31_exim4-config_rewriting as follows:
.ifndef NO_EAA_REWRITE_REWRITE
*@+local_domains "${lookup{${local_part}}lsearch{/etc/email-addresses}\
                   {$value}fail}" f
# identical rewriting rule for /etc/mailname
*@ETC_MAILNAME "${lookup{${local_part}}lsearch{/etc/email-addresses}\
                   {$value}fail}" f
.endif
* "$h_from:" Frs

So far its working fine for me but if you find bug, let me know.

Osamu

Holger Levsen: 20200803-debconf5

Monday 3rd of August 2020 10:13:34 PM
DebConf5

This tshirt is 15 years old and from DebConf5. It still looks quite nice!

DebConf5 was my 3rd DebConf and took place in Helsinki, or rather Espoo, in Finland.

This was one of my most favorite DebConfs (though I basically loved them all) and I'm not really sure why, I guess it's because of the kind of community at the event. We stayed in some future dorms of the universtity, which were to be first used by some European athletics chamopionship and which we could use even before that, guests zero. Being in Finland there were of course saunas in the dorms, which we frequently used and greatly enjoyed. Still, one day we had to go on a trip to another sauna in the forest, because of course you cannot visit Finland and only see one sauna. Or at least, you should not.

Another aspect which increased community bonding was that we had to authenticate using 802.10 (IIRC, please correct me) which was an authentication standard mostly used for wireless but which also works for wired ethernet, except that not many had used it on Linux before. Thus quite some related bugs were fixed in the first days of DebCamp...

Then my powerpc ibook also decided to go bad, so I had to remove 30 screws to get the harddrive out and 30 screws back in, to not have 30 screws laying around for a week. Then I put the harddrive into a spare (x86) laptop and only used my /home partition and was very happy this worked nicely. And then, for travelling back, I had to unscrew and screw 30 times again. (I think my first attempt took 1.5h and the fourth only 45min or so Back home then I bought a laptop where one could remove the harddrive using one screw.

Oh, and then I was foolish during the DebConf5 preparations and said, that I could imagine setting up a team and doing video recordings, as previous DebConfs mostly didn't have recordings and the one that had, didn't have releases of them...

And so we did videos. And as we were mostly inexperienced we did them the hard way: during the day we recorded on tape and then when the talks were done, we used a postprocessing tool called 'cinelerra' and edited them. And because Eric Evans was on the team and because Eric worked every night almost all night, all nights, we managed to actually release them all when DebConf5 was over. I very well remember many many (23 or 42) Debian people cleaning the dorms thoroughly (as they were brand new..) and Eric just sitting somewhere, exhausted and watching the cleaners. And everybody was happy Eric was idling there, cause we knew why. In the aftermath of DebConf5 Ben Hutchings then wrote videolink (removed from sid in 2013) which we used to create video DVDs of our recordings based on a simple html file with links to the actual videos.

There were many more memorable events. The boat ride was great. A pirate flag appeared. One night people played guitar until very late (or rather early) close to the dorms, so at about 3 AM someone complained about it, not in person, but on the debian-devel mailinglist. And those drunk people playing guitar, replied immediatly on the mailinglist. And then someone from the guitar group gave a talk, at 9 AM, and the video is online... (It's a very slowwwwwww talk.)

If you haven't been to or close to the polar circles it's almost impossible to anticipate how life is in summer there. It get's a bit darker after midnight or rather after 1 AM and then at 3 AM it get's light again, so it's reaaaaaaally easy to miss the night once and it's absolutly not hard to miss the night for several nights in a row. And then I shared a room with 3 people who all snore quite loud...

There was more. I was lucky to witness the first (or second?) cheese and whine party which at that time took place in a dorm room with, dunno 10 people and maybe 15 kinds of cheese. And, of course, I met many wonderful people there, to mention a few I'll say Jesus, I mean mooch or data, Amaya and p2. And thanks to some bad luck which turned well, I also had my first time ever Sushi in Helsinki.

And and and. DebConfs are soooooooo good! I'll stop here as I originally planned to only write a paragraph or two about each and there are quite some to be written!

Oh, and as we all learned, there are probably no mosquitos in Helsinki, just in Espoo. And you can swim naked through a lake and catch a taxi on the other site, with no clothes and no money, no big deal. (And you might not believe it, but that wasn't me. I cannot swim that well.)

Giovanni Mascellani: Bye bye Python 2!

Monday 3rd of August 2020 07:00:00 PM

And so, today, while I was browsing updates for my Debian unstable laptop, I noticed that aptitude wouldn't automatically upgrade python2 and related packages (I don't know why, and at this point don't care). So I decided to dare: I removed the python2 package to see what the dependency solver would have proposed me. It turned out that there was basically nothing I couldn't live without.

So, bye bye Python 2. It was a long ride and I loved programming with you. But now it's the turn of your younger brother.

$ python bash: python: comando non trovato

(guess what "comando non trovato" means?)

And thanks to all those who made this possible!

Sylvain Beucler: Debian LTS and ELTS - July 2020

Monday 3rd of August 2020 01:52:10 PM

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In July, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 25.25h for LTS (out of 30 max; all done) and 13.25h for ELTS (out of 20 max; all done).

We shifted suites: welcome Stretch LTS and Jessie ELTS. The LTS->ELTS switch happened at the start of the month, but the oldstable->LTS switch happened later (after finalizing and flushing proposed-updates to a last point release), causing some confusion but nothing major.

ELTS - Jessie

  • New local build setup
  • ELTS buildds: request timezone harmonization
  • Reclassify in-progress updates from jessie-LTS to jessie-ELTS
  • python3.4: finish preparing update, security upload ELA 239-1
  • net-snmp: global triage: bisect CVE-2019-20892 to identify affected version, jessie/stretch not-affected
  • nginx: global triage: clarify CVE-2013-0337 status; locate CVE-2020-11724 original patch and regression tests, update MITRE
  • nginx: security upload ELA-247-1 with 2 CVEs

LTS - Stretch

  • Reclassify in-progress/needed updates from stretch/oldstable to stretch-LTS
  • rails: upstream security: follow-up on CVE-2020-8163 (RCE) on upstream bug tracker and create pull request for 4.x (merged), hence getting some upstream review
  • rails: global security: continue coordinating upload in multiple Debian versions, prepare fixes for common stretch/buster vulnerabilities in buster
  • rails: security upload DLA-2282 fixing 3 CVEs
  • python3.5: security upload DLA-2280-1 fixing 13 pending non-critical vulnerabilities, and its test suite
  • nginx: security upload DLA-2283 (cf. common ELTS work)
  • net-snmp: global triage (cf. common ELTS work)
  • public IRC monthly team meeting
  • reach out to clarify the intro from last month's report, following unsettled feedback during meeting

Documentation/Scripts

  • ELTS/README.how-to-release-an-update: fix typo
  • ELTS buildd: attempt to diagnose slow perfs, provide comparison with Debian and local builds
  • LTS/Meetings: improve presentation
  • SourceOnlyUpload: clarify/de-dup pbuilder doc
  • LTS/Development: reference build logs URL, reference proposed-updates issue during dists switch, reference new-upstream-versioning discussion, multiple jessie->stretch fixes and clean-ups
  • LTS/Development/Asan: drop wheezy documentation
  • Warn about jruby mis-triage
  • Provide feedback for ksh/CVE-2019-14868
  • Provide feedback for condor update
  • LTS/TestsSuites/nginx: test with new request smuggling test cases

Arnaud Rebillout: GoAccess 1.4, a detailed tutorial

Monday 3rd of August 2020 12:00:00 AM

GoAccess v1.4 was just released a few weeks ago! Let's take this chance to write a loooong tutorial. We'll go over every steps to install and operate GoAccess. This is a tutorial aimed at those who don't play sysadmin every day, and that's why it's so long, I did my best to provide thorough explanations all along, so that it's more than just a "copy-and-paste" kind of tutorial. And for those who do play sysadmin everyday: please try not to fall asleep while reading, and don't hesitate to drop me an e-mail if you spot anything inaccurate in here. Thanks!

Introduction

So what's GoAccess already? GoAccess is a web log analyzer, and it allows you to visualize the traffic for your website, and get to know a bit more about your visitors: how many visitors and hits, for which pages, coming from where (geolocation, operating system, web browser...), etc... It does so by parsing the access logs from your web server, be it Apache, NGINX or whatever.

GoAccess gives you different options to display the statistics, and in this tutorial we'll focus on producing a HTML report. Meaning that you can see the statistics for your website straight in your web browser, under the form of a single HTML page.

For an example, you can have a look at the stats of my blog here: http://goaccess.arnaudr.io.

GoAccess is written in C, it has very few dependencies, it had been around for about 10 years, and it's distributed under the MIT license.

Assumptions

This tutorial is about installing and configuring, so I'll assume that all the commands are run as root. I won't prefix each of them with sudo.

I use the Apache web server, running on a Debian system. I don't think it matters so much for this tutorial though. If you're using NGINX it's fine, you can keep reading.

Also, I will just use the name SITE for the name of the website that we want to analyze with GoAccess. Just replace that with the real name of your site.

I also assume the following locations for your stuff:

  • the website is at /var/www/SITE
  • the logs are at /var/log/apache2/SITE (yes, there is a sub-directory)
  • we're going to save the GoAccess database in /var/lib/goaccess-db/SITE.

If you have your stuff in /srv/SITE/{log,www} instead, no worries, just adjust the paths accordingly, I bet you can do it.

Installation

The latest version of GoAccess is v1.4, and it's not yet available in the Debian repositories. So for this part, you can follow the instructions from the official GoAccess download page. Install steps are explained in details, so there's nothing left for me to say :)

When this is done, let's get started with the basics.

We're talking about the latest version v1.4 here, let's make sure:

$ goaccess --version GoAccess - 1.4. ...

Now let's try to create a HTML report. I assume that you already have a website up and running.

GoAccess needs to parse the access logs. These logs are optional, they might or might not be created by your web server, depending on how it's configured. Usually, these log files are named access.log, unsurprisingly.

You can check if those logs exist on your system by running this command:

find /var/log -name access.log

Another important thing to know is that these logs can be in different formats. In this tutorial we'll assume that we work with the combined log format, because it seems to be the most common default.

To check what kind of access logs your web server produces, you must look at the configuration for your site.

For an Apache web server, you should have such a line in the file /etc/apache2/sites-enabled/SITE.conf:

CustomLog ${APACHE_LOG_DIR}/SITE/access.log combined

For NGINX, it's quite similar. The configuration file would be something like /etc/nginx/sites-enabled/SITE, and the line to enable access logs would be something like:

access_log /var/log/nginx/SITE/access.log

Note that NGINX writes the access logs in the combined format by default, that's why you don't see the word combined anywhere in the line above: it's implicit.

Alright, so from now on we assume that yes, you have access log files available, and yes, they are in the combined log format. If that's the case, then you can already run GoAccess and generate a report, for example for the log file /var/log/apache2/access.log

goaccess \ --log-format COMBINED \ --output /tmp/report.html \ /var/log/apache2/access.log

It's possible to give GoAccess more than one log files to process, so if you have for example the file access.log.1 around, you can use it as well:

goaccess \ --log-format COMBINED \ --output /tmp/report.html \ /var/log/apache2/access.log \ /var/log/apache2/access.log.1

If GoAccess succeeds (and it should), you're on the right track!

All is left to do to complete this test is to have a look at the HTML report created. It's a single HTML page, so you can easily scp it to your machine, or just move it to the document root of your site, and then open it in your web browser.

Looks good? So let's move on to more interesting things.

Web server configuration

This part is very short, because in terms of configuration of the web server, there's very little to do. As I said above, the only thing you want from the web server is to create access log files. Then you want to be sure that GoAccess and your web server agree on the format for these files.

In the part above we used the combined log format, but GoAccess supports many other common log formats out of the box, and even allows you to parse custom log formats. For more details, refer to the option --log-format in the GoAccess manual page.

Another common log format is named, well, common. It even has its own Wikipedia page. But compared to combined, the common log format contains less information, it doesn't include the referrer and user-agent values, meaning that you won't have it in the GoAccess report.

So at this point you should understand that, unsurprisingly, GoAccess can only tell you about what's in the access logs, no more no less.

And that's all in term of web server configuration.

Configuration to run GoAccess unprivileged

Now we're going to create a user and group for GoAccess, so that we don't have to run it as root. The reason is that, well, for everything running unattended on your server, the less code runs as root, the better. It's good practice and common sense.

In this case, GoAccess is simply a log analyzer. So it just needs to read the logs files from your web server, and there is no need to be root for that, an unprivileged user can do the job just as well, assuming it has read permissions on /var/log/apache2 or /var/log/nginx.

The log files of the web server are usually part of the adm group (though it might depend on your distro, I'm not sure). This is something you can check easily with the following command:

ls -l /var/log | grep -e apache2 -e nginx

As a result you should get something like that:

drwxr-x--- 2 root adm 20480 Jul 22 00:00 /var/log/apache2/

And as you can see, the directory apache2 belongs to the group adm. It means that you don't need to be root to read the logs, instead any unprivileged user that belongs to the group adm can do it.

So, let's create the goaccess user, and add it to the adm group:

adduser --system --group --no-create-home goaccess addgroup goaccess adm

And now, let's run GoAccess unprivileged, and verify that it can still read the log files:

setpriv \ --reuid=goaccess --regid=goaccess \ --init-groups --inh-caps=-all \ -- \ goaccess \ --log-format COMBINED \ --output /tmp/report2.html \ /var/log/apache2/access.log

setpriv is the command used to drop privileges. The syntax is quite verbose, it's not super friendly for tutorials, but don't be scared and read the manual page to learn what it does.

In any case, this command should work, and at this point, it means that you have a goaccess user ready, and we'll use it to run GoAccess unprivileged.

Integration, option A - Run GoAccess once a day, from a logrotate hook

In this part we wire things together, so that GoAccess processes the log files once a day, adds the new logs to its internal database, and generates a report from all that aggregated data. The result will be a single HTML page.

Introducing logrotate

In order to do that, we'll use a logrotate hook. logrotate is a little tool that should already be installed on your server, and that runs once a day, and that is in charge of rotating the log files. "Rotating the logs" means moving access.log to access.log.1 and so on. With logrotate, a new log file is created every day, and log files that are too old are deleted. That's what prevents your logs from filling up your disk basically :)

You can check that logrotate is indeed installed and enabled with this command (assuming that your init system is systemd):

systemctl status logrotate.timer

What's interesting for us is that logrotate allows you to run scripts before and after the rotation is performed, so it's an ideal place from where to run GoAccess. In short, we want to run GoAccess just before the logs are rotated away, in the prerotate hook.

But let's do things in order. At first, we need to write a little wrapper script that will be in charge of running GoAccess with the right arguments, and that will process all of your sites.

The wrapper script

This wrapper is made to process more than one site, but if you have only one site it works just as well, of course.

So let me just drop it on you like that, and I'll explain afterward. Here's my wrapper script:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50#!/bin/bash # Process log files /var/www/apache2/SITE/access.log, # only if /var/lib/goaccess-db/SITE exists. # Create HTML reports in $1, a directory that must exist. set -eu OUTDIR= LOGDIR=/var/log/apache2 DBDIR=/var/lib/goaccess-db fail() { echo >&2 "$@"; exit 1; } [ $# -eq 1 ] || fail "Usage: $(basename $0) OUTPUT_DIRECTORY" OUTDIR=$1 [ -d "$OUTDIR" ] || fail "'$OUTDIR' is not a directory" [ -d "$LOGDIR" ] || fail "'$LOGDIR' is not a directory" [ -d "$DBDIR" ] || fail "'$DBDIR' is not a directory" for d in $(find "$LOGDIR" -mindepth 1 -maxdepth 1 -type d); do site=$(basename "$sitedir") dbdir=$DBDIR/$site logfile=$d/access.log outfile=$OUTDIR/$site.html if [ ! -d "$dbdir" ] || [ ! -e "$logfile" ]; then echo "‣ Skipping site '$site'" continue else echo "‣ Processing site '$site'" fi setpriv \ --reuid=goaccess --regid=goaccess \ --init-groups --inh-caps=-all \ -- \ goaccess \ --agent-list \ --anonymize-ip \ --persist \ --restore \ --config-file /etc/goaccess/goaccess.conf \ --db-path "$dbdir" \ --log-format "COMBINED" \ --output "$outfile" \ "$logfile" done

So you'd install this script at /usr/local/bin/goaccess-wrapper for example, and make it executable:

chmod +x /usr/local/bin/goaccess-wrapper

A few things to note:

  • We run GoAccess with --persist, meaning that we save the parsed logs in the internal database, and --restore, meaning that we include everything from the database in the report. In other words, we aggregate the data at every run, and the report grows bigger every time.
  • The parameter --config-file /etc/goaccess/goaccess.conf is a workaround for #1849. It should not be needed for future versions of GoAccess > 1.4.

As is, the script makes the assumption that the logs for your site are logged in a sub-directory /var/log/apache2/SITE/. If it's not the case, adjust that in the wrapper accordingly.

The name of this sub-directory is then used to find the GoAccess database directory /var/lib/goaccess-db/SITE/. This directory is expected to exist, meaning that if you don't create it yourself, the wrapper won't process this particular site. It's a simple way to control which sites are processed by this GoAccess wrapper, and which sites are not.

So if you want goaccess-wrapper to process the site SITE, just create a directory with the name of this site under /var/lib/goaccess-db:

mkdir -p /var/lib/goaccess-db/SITE chown goaccess:goaccess /var/lib/goaccess-db/SITE

Now let's create an output directory:

mkdir /tmp/goaccess-reports chown goaccess:goaccess /tmp/goaccess-reports

And let's give a try to the wrapper script:

goaccess-wrapper /tmp/goaccess-reports ls /tmp/goaccess-reports

Which should give you:

SITE.html

At the same time, you can check that GoAccess populated the database with a bunch of files:

ls /var/lib/goaccess-db/SITE Setting up the logrotate prerotate hook

At this point, we have the wrapper in place. Let's now add a pre-rotate hook so that goaccess-wrapper runs once a day, just before the logs are rotated away.

The logrotate config file for Apache2 is located at /etc/logrotate.d/apache2, and for NGINX it's at /etc/logrotate.d/nginx. Among the many things you'll see in this file, here's what is of interest for us:

  • daily means that your logs are rotated every day
  • sharedscripts means that the pre-rotate and post-rotate scripts are executed once total per rotation, and not once per log file.

In the config file, there is also this snippet:

prerotate if [ -d /etc/logrotate.d/httpd-prerotate ]; then \ run-parts /etc/logrotate.d/httpd-prerotate; \ fi; \ endscript

It indicates that scripts in the directory /etc/logrotate.d/httpd-prerotate/ will be executed before the rotation takes place. Refer to the man page run-parts(8) for more details...

Putting all of that together, it means that logs from the web server are rotated once a day, and if we want to run scripts just before the rotation, we can just drop them in the httpd-prerotate directory. Simple, right?

Let's first create this directory if it doesn't exist:

mkdir -p /etc/logrotate.d/httpd-prerotate/

And let's create a tiny script at /etc/logrotate.d/httpd-prerotate/goaccess:

1 2#!/bin/sh exec goaccess-wrapper /tmp/goaccess-reports

Don't forget to make it executable:

chmod +x /etc/logrotate.d/httpd-prerotate/goaccess

As you can see, the only thing that this script does is to invoke the wrapper with the right argument, ie. the output directory for the HTML reports that are generated.

And that's all. Now you can just come back tomorrow, check the logs, and make sure that the hook was executed and succeeded. For example, this kind of command will tell you quickly if it worked:

journalctl | grep logrotate Integration, option B - Run GoAccess once a day, from a systemd service

OK so we've just seen how to use a logrotate hook. One downside with that is that we have to drop privileges in the wrapper script, because logrotate runs as root, and we don't want to run GoAccess as root. Hence the rather convoluted syntax with setpriv.

Rather than embedding this kind of thing in a wrapper script, we can instead run the wrapper script from a [systemd][] service, and define which user runs the wrapper straight in the systemd service file.

Introducing systemd niceties

So we can create a systemd service, along with a systemd timer that fires daily. We can then set the user and group that execute the script straight in the systemd service, and there's no need for setpriv anymore. It's a bit more streamlined.

We can even go a bit further, and use systemd parameterized units (also called templates), so that we have one service per site (instead of one service that process all of our sites). That will simplify the wrapper script a lot, and it also looks nicer in the logs.

With this approach however, it seems that we can't really run exactly before the logs are rotated away, like we did in the section above. But that's OK. What we'll do is that we'll run once a day, no matter the time, and we'll just make sure to process both log files access.log and access.log.1 (ie. the current logs and the logs from yesterday). This way, we're sure not to miss any line from the logs.

Note that GoAccess is smart enough to only consider newer entries from the log files, and discard entries that are already in the database. In other words, it's safe to parse the same log file more than once, GoAccess will do the right thing. For more details see "INCREMENTAL LOG PROCESSING" from man goaccess.

systemd]: https://freedesktop.org/wiki/Software/systemd/

Implementation

And here's how it all looks like.

First, a little wrapper script for GoAccess:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32#!/bin/bash # Usage: $0 SITE DBDIR LOGDIR OUTDIR set -eu SITE=$1 DBDIR=$2 LOGDIR=$3 OUTDIR=$4 LOGFILES=() for ext in log log.1; do logfile="$LOGDIR/access.$ext" [ -e "$logfile" ] && LOGFILES+=("$logfile") done if [ ${#LOGFILES[@]} -eq 0 ]; then echo "No log files in '$LOGDIR'" exit 0 fi goaccess \ --agent-list \ --anonymize-ip \ --persist \ --restore \ --config-file /etc/goaccess/goaccess.conf \ --db-path "$DBDIR" \ --log-format "COMBINED" \ --output "$OUTDIR/$SITE.html" \ "${LOGFILES[@]}"

This wrapper does very little. Actually, the only thing it does is to check for the existence of the two log files access.log and access.log.1, to be sure that we don't ask GoAccess to process a file that does not exist (GoAccess would not be happy about that).

Save this file under /usr/local/bin/goaccess-wrapper, don't forget to make it executable:

chmod +x /usr/local/bin/goaccess-wrapper

Then, create a systemd parameterized unit file, so that we can run this wrapper as a systemd service. Save it under /etc/systemd/system/goaccess@.service:

[Unit] Description=Update GoAccess report - %i ConditionPathIsDirectory=/var/lib/goaccess-db/%i ConditionPathIsDirectory=/var/log/apache2/%i ConditionPathIsDirectory=/tmp/goaccess-reports PartOf=goaccess.service [Service] Type=oneshot User=goaccess Group=goaccess Nice=19 ExecStart=/usr/local/bin/goaccess-wrapper \ %i \ /var/lib/goaccess-db/%i \ /var/log/apache2/%i \ /tmp/goaccess-reports

So, what is a systemd parameterized unit? It's a service to which you can pass an argument when you enable it. The %i in the unit definition will be replaced by this argument. In our case, the argument will be the name of the site that we want to process.

As you can see, we use the directive ConditionPathIsDirectory= extensively, so that if ever one of the required directories does not exist, the unit will just be skipped (and marked as such in the logs). It's a graceful way to fail.

We run the wrapper as the user and group goaccess, thanks to User= and Group=. We also use Nice= to give a low priority to the process.

At this point, it's already possible to test. Just make sure that you created a directory for the GoAccess database:

mkdir -p /var/lib/goaccess-db/SITE chown goaccess:goaccess /var/lib/goaccess-db/SITE

Also make sure that the output directory exists:

mkdir /tmp/goaccess-reports chown goaccess:goaccess /tmp/goaccess-reports

Then reload systemd and fire the unit to see if it works:

systemctl daemon-reload systemctl start goaccess@SITE.service journalctl | tail

And that should work already.

As you can see, the argument, SITE, is passed in the systemctl start command. We just append it after the @, in the name of the unit.

Now, let's create another GoAccess service file, which sole purpose is to group all the parameterized units together, so that we can start them all in one go. Note that we don't use a systemd target for that, because ultimately we want to run it once a day, and that would not be possible with a target. So instead we use a dummy oneshot service.

So here it is, saved under /etc/systemd/system/goaccess.service:

[Unit] Description=Update GoAccess reports Requires= \ goaccess@SITE1.service \ goaccess@SITE2.service [Service] Type=oneshot ExecStart=true

As you can see, we simply list the sites that we want to process in the Requires= directive. In this example we have two sites named SITE1 and SITE2.

Let's ensure that everything is still good:

systemctl daemon-reload systemctl start goaccess.service journalctl | tail

Check the logs, both sites SITE1 and SITE2 should have been processed.

And finally, let's create a timer, so that systemd runs goaccess.service once a day. Save it under /etc/systemd/system/goaccess.timer.

[Unit] Description=Daily update of GoAccess reports [Timer] OnCalendar=daily RandomizedDelaySec=1h Persistent=true [Install] WantedBy=timers.target

Finally, enable the timer:

systemctl daemon-reload systemctl enable --now goaccess.timer

At this point, everything should be OK. Just come back tomorrow and check the logs with something like:

journalctl | grep goaccess

Last word: if you have only one site to process, of course you can simplify, for example you can hardcode all the paths in the file goaccess.service instead of using a parameterized unit. Up to you.

Daily operations

So in this part, we assume that you have GoAccess all setup and running, once a day or so. Let's just go over a few things worth noting.

Serve your report

Up to now in this tutorial, we created the reports in /tmp/goaccess-reports, but that was just for the sake of the example. You will probably want to save your reports in a directory that is served by your web server, so that, well, you can actually look at it in your web browser, that was the point, right?

So how to do that is a bit out of scope here, and I guess that if you want to monitor your website, you already have a website, so you will have no trouble serving the GoAccess HTML report.

However there's an important detail to be aware of: GoAccess shows all the IP addresses of your visitors in the report. As long as the report is private it's OK, but if ever you make your GoAccess report public, then you should definitely invoke GoAccess with the option --anonymize-ip.

Keep an eye on the logs

In this tutorial, the reports we create, along with the GoAccess databases, will grow bigger every day, forever. It also means that the GoAccess processing time will grow a bit each day.

So maybe the first thing to do is to keep an eye on the logs, to see how long it takes to GoAccess to do its job every day. Also, maybe you'd like to keep an eye on the size of the GoAccess database with:

du -sh /var/lib/goaccess-db/SITE

If your site has few visitors, I suspect it won't be a problem though.

You could also be a bit pro-active in preventing this problem in the future, and for example you could break the reports into, say, monthly reports. Meaning that every month, you would create a new database in a new directory, and also start a new HTML report. This way you'd have monthly reports, and you make sure to limit the GoAccess processing time, by limiting the database size to a month.

This can be achieved very easily, by including something like YEAR-MONTH in the database directory, and in the HTML report. You can handle that automatically in the wrapper script, for example:

sfx=$(date +'%Y-%m') mkdir -p $DBDIR/$sfx goaccess \ --db-path $DBDIR/$sfx \ --output "$OUTDIR/$SITE-$sfx.html" \ ...

You get the idea.

Further notes Migration from older versions

With the --persist option, GoAccess keeps all the information from the logs in a database, so that it can re-use it later. In prior versions, GoAccess used the Tokyo Cabinet key-value store for that. However starting from v1.4, GoAccess dropped this dependency and now uses its own database format.

As a result, the previous database can't be used anymore, you will have to remove it and restart from zero. At the moment there is no way to convert the data from the old database to the new one. If you're interested, this is discussed upstream at [#1783][bug-1783].

Another thing that changed with this new version is the name for some of the command-line options. For example, --load-from-disk was dropped in favor of --restore, and --keep-db-files became --persist. So you'll have to look at the documentation a bit, and update your script(s) accordingly.

Other ways to use GoAccess

It's also possible to do it completely differently. You could keep GoAccess running, pretty much like a daemon, with the --real-time-html option, and have it process the logs continuously, rather than calling it on a regular basis.

It's also possible to see the GoAccess report straight in the terminal, thanks to libncurses, rather than creating a HTML report.

And much more, GoAccess is packed with features.

Conclusion

I hope that this tutorial helped some of you folks. Feel free to drop an e-mail for comments.

Enrico Zini: Toxic positivity links

Sunday 2nd of August 2020 10:00:00 PM
The Oppression of the Positivity Movement politics privilege archive.org 2020-08-03 “That which we do not bring to consciousness appears in our lives as fate.” — Carl Jung The Tyrannical Culture of Positivity health privilege archive.org 2020-08-03 Emotional support of others can take the form of surface-level consolation. But compassion means being willing to listen and feel, even when it's uncomfortable. How Positive Thinking Can Do More Harm Than Good health privilege archive.org 2020-08-03 Ultimately, the driving force behind the “power of positive thinking” meme is the word “power.” But what about those whose bodies are not powerful? What about those who are vulnerable? What about those who are tired, isolated, and struggling? What about those who are ill? What about those who lack Toxic Positivity - Tanglaw Mental Health — hindsight is so 2020 health privilege archive.org 2020-08-03 I have often been dismissive or unhelpful when someone close to me was dealing with painful circumstances, having learned to “accentuate the positive.” In the more recent past, I have recognized these behavioral patterns as part of what some mental health professionals term, “toxic positivity.” Toxic Positivity: The Dark Side of Positive Vibes health privilege archive.org 2020-08-03 Toxic positivity is the overgeneralization of a happy, optimistic state resulting in the denial & invalidation of the authentic human emotional experience.

More in Tux Machines

Go 1.15 Release Notes

The latest Go release, version 1.15, arrives six months after Go 1.14. Most of its changes are in the implementation of the toolchain, runtime, and libraries. As always, the release maintains the Go 1 promise of compatibility. We expect almost all Go programs to continue to compile and run as before. Read more Also: Go 1.15 Released With Much Improved Linker, New CPU Mitigations

Intel and Linux: Mesa, mOS, SERIALIZE and IWD

  • Intel Iris Gallium3D Driver Adds Compute Kernel Support In Mesa 20.3

    While Mesa 20.2 isn't even releasing for a few weeks, Mesa 20.3 is already seeing new feature work that will debut next quarter.  Intel's Jason Ekstrand has landed a set of patches for handling of kernels within Iris, Intel's modern Gallium3D driver. He commented, "This MR contains most of the patches required to handle kernels in iris. I've had them lying around in a branch in some form or another for a while. We should upstream what we can." 

  • Intel Making Progress On Their "mOS" Modified Linux Kernel Running Lightweight Kernels

    For a while now Intel has been quietly been working on "mOS" as the "multi-OS" that is a modified version of the Linux kernel that in turn is running lightweight kernels for high-performance computing purposes.

  •          
  • POWER10 Virtualization, Intel SERIALIZE Come For KVM On Linux 5.9

    Sent in last week for the Linux 5.9 kernel merge window were the initial batch of changes to the Kernel-based Virtual Machine (KVM) while today some additional interesting changes were sent out.  This latest material for KVM in Linux 5.9 includes:  - Support for the SERIALIZE instruction on KVM x86/x86_64. Intel's SERIALIZE ensures all flags/register/memory modifications are complete and all buffered writes drained before moving on to execute the next instruction. This can be used for stopping speculative execution and prefetching of modified kernel. The first CPUs expected with SERIALIZE are Sapphire Rapids and Alder Lake next year while Linux has already begun preparing for SERIALIZE where relevant. 

  • Ubuntu Is Looking At Offering Better WiFi Support By Using Intel's IWD

    Ubuntu developers are looking at using Intel IWD as the iNET wireless daemon to potentially replace WPA_Supplicant for offering a better WiFi experience. Intel's open-source team has always been working on IWD as a potential replacement to WPA_Supplicant while recently the Ubuntu folks have found it has "mostly reached feature parity" now to WPA_Supplicant albeit is in need of more testing on the desktop side.

Games: Terminal Phase, Imperator: Rome and More

  • Terminal Phase in Linux Magazine (Polish edition)

    Hey look at that! My terminal-space-shooter-game Terminal Phase made an appearance in the Polish version of Linux Magazine. I had no idea, but Michal Majchrzak both tipped me off to it and took the pictures. (Thank you!) I don't know Polish but I can see some references to Konami and SHMUP (shoot-em-up game). The screenshot they have isn't the one I published, so I guess the author got it running too... I hope they had fun!

  • Imperator: Rome gets a major free update, new DLC and cross-store multiplayer

    Paradox Interactive and Paradox Development Studio put out a massive upgrade for Imperator: Rome which includes a free update, an expansion and cross-platform / cross-store online play. There's quite a lot to dissect here, so let's start with the free content update. The 1.5 "Menander" update went out, as part of their focus on smaller and more regular updates to various systems. With the main point being to add greater depth to cultural management in the game.

  • Prepare your hard drive as another Steam Game Festival is coming in October

    After a massive success with the most recent Steam Game Festival back in June, it's going to return for another round later this year in October. This is where Steam users get to play through a ton of limited-time demos, which originally started back in December 2019 to go along with The Game Awards. From a post on the Steamworks Development group on Steam, the date is confirmed to be October 7 - 13. Valve mentioned in the announcement that they will soon open up the developer opt-in for the event, giving developers another chance to get a demo out there and get more eyes on their game. Developers don't have long, as the opt-in date is only open from between August 19 - 26.

Android Leftovers