Accurate Conclusions from Bogus Data: Methodological Issues in “Collaboration in the open-source arena: The WebKit case”

Nearly five years ago, when I was in grad school, I stumbled across the paper Collaboration in the open-source arena: The WebKit case when trying to figure out what I would do for a course project in network theory (i.e. graph theory, not computer networking; I’ll use the words “graph” and “network” interchangeably). The paper evaluates collaboration networks, which are graphs where collaborators are represented by nodes and relationships between collaborators are represented by edges. Our professor had used collaboration networks as examples during lecture, so it seemed at least mildly relevant to our class, and I wound up writing a critique on this paper for the class project. In this paper, the authors construct collaboration networks for WebKit by examining the project’s changelog files to define relationships between developers. They perform “community detection” to visually group developers who work closely together into separate clusters in the graphs. Then, the authors use those graphs to arrive at various conclusions about WebKit (e.g. “[e]ven if Samsung and Apple are involved in expensive patent wars in the courts and stopped collaborating on hardware components, their contributions remained strong and central within the WebKit open source project,” regarding the period from 2008 to 2013). At the time, I contacted the authors to let them know about some serious problems I found with their work. Then I left the paper sitting in a short-term to-do pile on my desk, where it has been sitting since Obama was president, waiting for me to finally write this blog post. Unfortunately, nearly five years later, the authors’ email addresses no longer work, which is not very surprising after so long — since I’m no longer a student, the email I originally used to contact them doesn’t work anymore either — so I was unable to contact them again to let them know that I was finally going to publish this blog post. Anyway, suffice to say that the conclusions of the paper were all correct; however, the networks used to arrive at those conclusions suffered from three different mistakes, each of which was, on its own, serious enough to invalidate the entire work. So if the analysis of the networks was bogus, how did the authors arrive at correct conclusions anyway? The answer is confirmation bias. The study was performed by visually looking at networks and then coming to non-rigorous conclusions about the networks, and by researching the WebKit community to learn what is going on with the major companies involved in the project. The authors arrived at correct conclusions because they did a good job at the later, then saw what they wanted to see in the graphs. I don’t want to be too harsh on the authors of this paper, though, because they decided to publish their raw data and methodology on the internet. They even published the python scripts they used to convert WebKit changelogs into collaboration graphs. Had they not done so, there is no way I would have noticed the third (and most important) mistake that I’ll discuss below, and I wouldn’t have been able to confirm my suspicions about the second mistake. You would not be reading this right now, and likely nobody would ever have realized the problems with the paper. The authors of most scientific papers are not nearly so transparent: many researchers today consider their source code and raw data to be either proprietary secrets to be guarded, or simply not important enough to merit publication. The authors of this paper deserve to be commended, not penalized, for their openness. Mistakes are normal in research papers, and open data is by far the best way for us to be able to detect mistakes when they happen.

today's howtos

• How to bind keyboard controls to controller on Linux

  Controller support for Linux games is excellent, thanks to Steam and the Linux kernel. However, not every video game available on the Linux platform brings controller support. Some games require a keyboard and mouse, and that’s a bummer if you prefer a gamepad.

• How to try out the new Vivaldi email client on Linux

  The Vivaldi web browser has a new feature: a built-in email client. With it, users can read their email directly through their web browser without needing to deal with webmail.

• How to install Cinnamon 4.8 on Ubuntu 20.04 - YouTube

  In this video, we are looking at how to install Cinnamon 4.8 on Ubuntu 20.04.

Audiocasts/Shows: Sad, GNU World Order, Linux Action News

• Sad: Bring Sed Into The Future With Space Age Sed - YouTube

  Search and replace is a function you'll frequently need for text processing so today we're looking at another one of the many rust replacements out their, today's tool is a rust replacement for sed called sad otherwise known as space age sed, one of the standard tools on your unix like linux system.

• GNU World Order 382

  **seejpeg** , **slackpkg** , and related hacks.

• Linux Action News 165

  What caused the recent major AWS outage, the breaking changes that just arrived upstream, and a new mail client for Linux.

Xilinx Continues Their Open-Source FPGA Upstreaming Push For The Linux Kernel

Earlier this month we covered the news of Xilinx is looking to upstream their open-source "AI Engine" driver to the Linux kernel. This comes as Xilinx and AMD are working on Radeon Open eCosystem (ROCm) support for their FPGAs with AMD being in the process of acquiring the FPGA giant. Now more open-source code is looking to be included in the Linux kernel tree. Sent out on Saturday were more patches around the Xilinx Alveo accelerator and the Xilinx Runtime (XRT) open-source stack.