Language Selection

English French German Italian Portuguese Spanish

New LWN Kernel (Linux) Articles, Paywall Just Removed

Filed under
Linux
  • Some unreliable predictions for 2019

    Kernel development will become more formal. One of the things that has traditionally attracted a certain type of developer to kernel work is the fact that many of the normal rules don't apply. Kernel development often requires working with high levels of complexity, combined with the ups and downs of dealing with real-world hardware; in that setting, pulling together any sort of solution can be an accomplishment. The result is a sort of cowboy culture that emphasizes working solutions over formal designs.

    The increasing level of complexity in the kernel and in the hardware it drives has made that approach less tenable over the years. The kernel community has responded in a number of ways, including better documentation and better testing. One real harbinger of the future, though, may be the work that has been quietly happening to develop a formal memory-ordering model that makes it possible to reason about concurrency and ensure that kernel code is correct. If the kernel is going to continue to scale, this kind of approach will have to spread to other areas. There will be grumbling, since adding formality may slow the pace of development. But, with luck, it should also slow the issuance of urgent bug fixes and security updates.

    More kernel APIs will be created for BPF programs rather than exported as traditional system calls; we are heading toward a world where a significant amount of kernel functionality is only available via BPF. The result will be a significant increase in flexibility and efficiency, but some growing pains should also be expected. The BPF API sees even less review than other kernel interfaces, and the community's record with the latter is decidedly less than perfect. This may be the year when we realize that we haven't yet figured out how to provide such low-level access to the kernel in ways that can be supported indefinitely.

    Somebody will attempt to test the kernel community's code of conduct and its enforcement processes in the coming year. The community will handle that test without trouble, though, just as it has been handling the constant stream of trolling emails attempting to stir up strife. At the end of the year, the code of conduct will look pretty much the way it does now: a set of expectations that helps to improve behavior in the community, but not a big deal in general.

  • Some 4.20 development statistics

    This year's holiday gifts will include the 4.20 kernel; that can only mean that it is time for another look at where the code going into this release has come from. This development cycle was typically busy and brought a lot of new code into the kernel. There are some new faces showing up in the statistics this time around, but not a lot of surprises otherwise.
    As of this writing, 13,856 non-merge changesets have found their way into the mainline repository for the 4.20 release; they were contributed by 1,743 developers. That makes 4.20 the busiest cycle since 4.15, but only by a little bit; both numbers are essentially in line with recent release history. Of those 1,743 developers, 283 were first-time contributors this time around.

  • What's coming in the next kernel release (part 1)

    When the 4.20 kernel was released on December 23, Linus Torvalds indicated that he would try to keep to the normal merge window schedule despite the presence of the holidays in the middle of it. Thus far, he seems to be trying to live up to that; just over 8,700 changesets have been merged for the next release, which seems likely to be called 5.0. A number of long-awaited features are finally landing in the kernel with this release.

  • Live patching for CPU vulnerabilities

    The kernel's live-patching (KLP) mechanism can apply a wide variety of fixes to a running kernel but, at a first glance, the sort of highly intrusive changes needed to address vulnerabilities like Meltdown or L1TF would not seem like likely candidates for live patches. The most notable obstacles are the required modifications of global semantics on a running system, as well as the need for live patching the kernel's entry code. However, we at the SUSE live patching team started working on proof-of-concept live patches for these vulnerabilities as a fun project and have been able to overcome these hurdles. The techniques we developed are generic and might become handy again when fixing future vulnerabilities.
    For completeness, it should be noted that these two demo live patches have been implemented for kGraft, but kGraft is conceptually equivalent to KLP.

    At the heart of the Meltdown vulnerability is the CPU speculating past the access rights encoded in the page table entries (PTEs) and thereby enabling malicious user-space programs to extract data from any kernel mapping. The kernel page-table isolation (KPTI) mechanism blocks such attacks by switching to stripped-down "shadow" page tables whenever the kernel returns to user space. These mirror the mappings from the lower, user-space half of the address space, but lack almost anything from the kernel region except for the bare minimum needed to reenter the kernel and switch back to the fully populated page tables. The difficulty, from a live-patching perspective, is to keep the retroactively introduced shadow page tables consistent with their fully populated counterparts at all times. Furthermore, the entry code has to be made to switch back and forth between the full and shadow page table at kernel entries and exits, but that is outside of the scope of what is live patchable with KLP.

    For the L1TF vulnerability, recall that each PTE has a _PAGE_PRESENT bit that, when clear, causes page faults upon accesses to the corresponding virtual memory region. The PTE bits designated for storing a page's frame number are architecturally ignored in this case. The Linux kernel swapping implementation exploits this by marking the PTEs corresponding to swapped-out pages as non-present and reusing the physical address part to store the page's swap slot number. Unfortunately, CPUs vulnerable to L1TF do not always ignore the contents of these "swap PTEs", but can instead speculatively misinterpret the swap slot identifiers as physical addresses. These swap slot identifiers, being index-like in nature, tend to alias with valid physical page-frame numbers, so this speculation allows for extraction of the corresponding memory contents. The Linux kernel mitigation is to avoid this aliasing by bit-wise inverting certain parts of the swap PTEs. Unfortunately, this change of representation is again something which is not safely applicable to a running system with KLP's consistency guarantees alone.

  • Improving idle behavior in tickless systems

    Most processors spend a great deal of their time doing nothing, waiting for devices and timer interrupts. In these cases, they can switch to idle modes that shut down parts of their internal circuitry, especially stopping certain clocks. This lowers power consumption significantly and avoids draining device batteries. There are usually a number of idle modes available; the deeper the mode is, the less power the processor needs. The tradeoff is that the cost of switching to and from deeper modes is higher; it takes more time and the content of some caches is also lost. In the Linux kernel, the cpuidle subsystem has the task of predicting which choice will be the most appropriate. Recently, Rafael Wysocki proposed a new governor for systems with tickless operation enabled that is expected to be more accurate than the existing menu governor.

More in Tux Machines

today's leftovers

  • Hardware Review - The ZaReason Virtus 9200 Desktop
  • Chrome OS 76 will disable Crostini Linux backups by default
    Essentially, this is still a work in progress feature. And I shouldn’t be terribly surprised by that, even though in my experience, the functionality hasn’t failed me yet. That’s because we know that the Chromium team is considering on a way to backup and restore Linux containers directly from the Files app on a Chromebook. That proposal is targeted for Chrome OS 78, so this gives the team more time to work that out, as well as any other nits that might not be quite right with the current implementation.
  • Andrei Lisita: Something to show for
    Unfortunately along with the progress that was made we also encountered a bug with the NintendoDS core that causes Games to crash if we attempt to load a savestate. We are not yet 100% sure if the bug is caused by my changes or by the NintendoDS core itself. I hope we are able to fix it by the end of the summer although I am not even sure where to start since savestates are working perfectly fine with other cores. Another confusing matter about this is that the Restart/Resume Dialog works fine with the NintendoDS core and it also uses savestates. This led me to believe that perhaps cores can be used to load savestates only once, but this can’t be the problem since we re-instantiate the core every time we load a savestate. In the worst case we might just have to make a special case for the NintendoDS core and not use savestates with it, except for the Resume/Restart dialog. This would sadden me deeply since there are plenty of NintendoDS games which could benefit from this feature.
  • OSMC's June update is here with Kodi v18.3
    Team Kodi recently announced the 18.3 point release of Kodi Leia. We have now prepared this for all supported OSMC devices and added some improvements and fixes. Here's what's new:

OSS Leftovers

  • A comparison of open source, real-time data streaming platforms
    A variety of open source, real-time data streaming platforms are available today for enterprises looking to drive business insights from data as quickly as possible. The options include Spark Streaming, Kafka Streams, Flink, Hazelcast Jet, Streamlio, Storm, Samza and Flume -- some of which can be used in tandem with each other. Enterprises are adopting these real-time data streaming platforms for tasks such as making sense of a business marketing campaign, improving financial trading or recommending marketing messages to consumers at critical junctures in the customer journey. These are all time-critical areas that can be used for improving business decisions or baked into applications driven by data from a variety of sources.
  • Amphenol’s Jason Ellison on Signal Integrity Careers and His Free, Open Source PCB Design Software
    Ellison, Senior Staff Signal Integrity Engineer at Amphenol ICC, gives his insight on the importance of networking, giving to the EE community, and his open-source signal integrity project. How does signal integrity engineering compare to other EE fields? What are open-source resources worth these days? What makes for a good work life for an engineer? Learn this and more in this Engineer Spotlight! Jason Ellison started down the path to becoming an electrical engineer because someone told him it was "fun and easy if you're good at math." In this interview with AAC's Mark Hughes, Ellison—a Senior Staff Signal Integrity Engineer at Amphenol ICC—describes how his career has grown from these beginnings into the rewarding and diverse work of signal integrity engineering.
  • Cruise open-sources Webviz, a tool for robotics data analysis [Ed: Releasing a little tool that's part of proprietary software so that it 'feels' more "open"]
    Cruise, the self-driving startup that General Motors acquired for nearly $1 billion in 2016, generates an enormous amount of data by any measure. It orchestrates 200,000 hours of driving simulation jobs daily in Google Cloud Platform, spread across 30,000 virtual cars in an environment running on 300,000 processor cores and 5,000 graphics cards. Both those cars and Cruise’s fleet of over 180 real-world autonomous Chevrolet Bolts make thousands of decisions every second, and they base these decisions on observations captured in binary format from cameras, microphones, radar sensors, and lidar sensors.
  • EWF launches world’s first open source blockchain for the energy industry
    The Energy Web Foundation this week announced that it has launched the world’s first public, open-source, enterprise-grade blockchain tailored to the energy sector: the Energy Web Chain (EW Chain). More than ten Energy Web Foundation (EWF) Affiliates — including utilities, grid operators, and blockchain developers — are hosting validator nodes for the live network, according to the company.
  • Pimcore Releases Pimcore 6.0, Amplifying User-Friendly Digital Experiences Through Open Source
    Pimcore, the leading open-source platform for data and customer experience management, has released the most powerful version of the Pimcore platform, Pimcore 6.0. The updated platform includes a new user interface that seamlessly connects MDM/PIM, DAM, WCM, and digital commerce capabilities to create more advanced and user-friendly experiences quickly and efficiently.
  • VCV Rack reaches version 1.0.0: free and open-source modular synth gets a full release
    VCV Rack is a free, open-source modular software synth that’s been gaining ground for a couple of years, but only now has it reached the significant milestone of version 1.0. Designed to replicate the feeling of having a hardware modular synth on your desktop, VCV Rack enables you to add both free and paid-for modules, and now supports polyphony of up to 16 voices. There’s MIDI Output, too with CV-Gate, CV-MIDI and CV-CC modules enabling you to interface with drum machines, desktop synths and Eurorack gear.
  • Flying Above the Shoulders of Giants
    Thanks to open-source platforms, developers can stand on the shoulders of software giants to build bigger and better things. Linux is probably the biggest...
  • MIT Researchers Open-Source AutoML Visualization Tool ATMSeer
    A research team from MIT, Hong Kong University, and Zhejiang University has open-sourced ATMSeer, a tool for visualizing and controlling automated machine-learning processes. Solving a problem with machine learning (ML) requires more than just a dataset and training. For any given ML tasks, there are a variety of algorithms that could be used, and for each algorithm there can be many hyperparameters that can be tweaked. Because different values of hyperparameters will produce models with different accuracies, ML practitioners usually try out several sets of hyperparameter values on a given dataset to try to find hyperparameters that produce the best model. This can be time-consuming, as a separate training job and model evaluation process must be conducted for each set. Of course, they can be run in parallel, but the jobs must be setup and triggered, and the results recorded. Furthermore, choosing the particular values for hyperparameters can involve a bit of guesswork, especially for ones that can take on any numeric value: if 2.5 and 2.6 produce good results, maybe 2.55 would be even better? What about 2.56 or 2.54?
  • Open-Source Cybersecurity Tool to Enhance Grid Protection
    A revolutionary new cybersecurity tool that can help protect the electric power grid has been released to the public on the code-hosting website GitHub.
  • Quick notes for Mozilla Whistler All Hands 2019
  • Deeper into the data fabric with MongoDB
    However, to gain access to rich search functionality, many organisations pair their database with a search engine such as Elasticsearch or Solr, which MongoDB claims can complicate development and operations — because we end up with two entirely separate systems to learn, maintain and scale.

Raspberry Pi 4 is here!

The latest version of the Raspberry Pi—Raspberry Pi 4—was released today, earlier than anticipated, featuring a new 1.5GHz Arm chip and VideoCore GPU with some brand new additions: dual-HDMI 4K display output; USB3 ports; Gigabit Ethernet; and multiple RAM options up to 4GB. The Raspberry Pi 4 is a very powerful single-board computer and starts at the usual price of $35. That gets you the standard 1GB RAM, or you can pay $45 for the 2GB model or $55 for the 4GB model—premium-priced models are a first for Raspberry Pi. Read more

Open Data, Open Access and Open Hardware

  • DoD’s Joint AI Center to open-source natural disaster satellite imagery data set
    As climate change escalates, the impact of natural disasters is likely to become less predictable. To encourage the use of machine learning for building damage assessment this week, Carnegie Mellon University’s Software Engineering Institute and CrowdAI — the U.S. Department of Defense’s Joint AI Center (JAIC) and Defense Innovation Unit — open-sourced a labeled data set of some of the largest natural disasters in the past decade. Called xBD, it covers the impact of disasters around the globe, like the 2010 earthquake that hit Haiti. “Although large-scale disasters bring catastrophic damage, they are relatively infrequent, so the availability of relevant satellite imagery is low. Furthermore, building design differs depending on where a structure is located in the world. As a result, damage of the same severity can look different from place to place, and data must exist to reflect this phenomenon,” reads a research paper detailing the creation of xBD. [...]

    xBD includes approximately 700,000 satellite images of buildings before and after eight different kinds of natural disasters, including earthquakes, wildfires, floods, and volcanic eruptions. Covering about 5,000 square kilometers, it contains images of floods in India and Africa, dam collapses in Laos and Brazil, and historic deadly fires in California and Greece.

    The data set will be made available in the coming weeks alongside the xView 2.0 Challenge to unearth additional insights from xBD, coauthor and CrowdAI machine learning lead Jigar Doshi told VentureBeat. The data set collection effort was informed by the California Air National Guard’s approach to damage assessment from wildfires.

  • Open-source textbooks offer free alternative for UC Clermont students
    Some UC Clermont College students are avoiding paying hundreds of dollars for textbooks — and getting the content for free — thanks to online open-source textbooks, a growing trend among faculty at the college and throughout higher education. UC Clermont Dean Jeff Bauer, who is also a professor of business, said the benefits of open textbooks are many. “All students have the book on the first day of class, it saves them a lot of money, and the information can be accessed anywhere, anytime, without carrying around a heavy textbook,” Bauer said. “They don’t need to visit the bookstore before or after each semester to buy or sell back books, either.”
  • Open Source Computer Controlled Loom Knits Pikachu For You
    The origin story of software takes us back past punch card computers and Babbage's Difference Engine to a French weaver called Joseph Marie Jacquard.
  • Successful open-source RISC-V microcontroller launched through crowdfunding
    X-FAB Silicon Foundries, together with crowd-sourcing IC platform partner Efabless Corporation, launched the first-silicon availability of the Efabless RISC-V SoC reference design. This open-source semiconductor project went from start of design to tape-out in less than three months employing the Efabless design flow produced on open-source tools. The mixed-signal SoC, called Raven, is based on the community developed ultra-low power PicoRV32 RISC-V core. Efabless has bench-tested the Raven at 100MHz, and based on simulations, the solution should operate at up to 150MHz.
  • Open Hardware: Open-Source MRI Scanners Could Bring Enormous Cost Savings
    Wulfsberg explore the possibilities of open source MRI scanning. As open-source technology takes its place around the world—everywhere from makerspaces to FabLabs, users on every level have access to design and innovation. In allowing such access to MRI scanning, the researchers realize the potential for ‘technological literacy’ globally—and with MRIs specifically, astronomical sums could be saved in healthcare costs. The authors point out that medical technology is vital to the population of the world for treating not only conditions and illnesses, but also disabilities. As so many others deeply involved in the world of technology and 3D printing realize, with greater availability, accessibility, and affordability, huge strides can be made to improve and save lives. Today, with so many MRI patents expiring, the technology is open for commercialization.