Guillaume's 🧠

Search

SearchSearch

Not my usual Open Source Software Rant pt1

Jul 29, 2024, 11 min read

This is the first part of something I decided to write following an event that occurred during the Easter weekend of 2024 and specifically some of the reactions I saw in the tech world in response. This is not something related to me personally or that affected me, but something that is very dear to my heart and values. It is about Open Source Software. At first, I wanted to write a few things that I had on my mind, but it quickly became a pretty extensive text that I decided to split in two parts.

In the first part, I go over the event that occurred on Easter 2024 that will most likely be referred to as the XZ backdoor in the future. I then begin to unravel and explain what I consider the more profound problems that led to this situation.

In the second part, I discuss a fallacy that crops up every time something like this occurs, whether it be log4j, leftpad or xz, namely the misconception of the software supply chain. I continue by broadening the discussion to the more general Open Source ecosystem and try to explain to laymen what it is, and especially how important and prevalent it is.

As my usual custom, I believe that education is always a central part of any solution to a problem. As such, I included as many facts and references as I could without making this text too long or spending the next weeks or months working on it. I hope you’ll get something out of it, even if just the story of an unsung hero named Andres.

Happy Easter Everyone

Something happened during this year’s Easter holiday. Something really bad. Something so incredibly problematic that most people, maybe none of you, have even heard of it unless it got some media attention from your favorite news network. Are you ready? Here it is.

A backdoor exploit was injected in the git repository of the XZ project.

Underwhelmed?

Let me explain. XZ is a compression tool and library used for releasing tarballs, software packages, kernel images and initramfs images. It is a piece of code used to package and compress other pieces of code for sharing and deployment on the systems that keep the world turning (more on that later).1 An understandable reaction at this point if you are not familiar with what you just read would be a shrug of shoulders followed by an innocuous “so what?“. Well, let me put it this way… A system infected with the XZ backdoor could be accessed and controlled, as root, remotely, from the public internet via ssh. I should specify further. Root is the name we use for the superuser on a Unix/Linux system… The admin… and if you did not know, there is nothing you cannot do, on Unix/Linux, as root. Nothing is restricted or off-limits for root.2 So ponder this for a second. An exploit that can potentially allow control of a computer, as root, through the public internet, was (almost) successfully injected into a tool that is used to package and distribute a very large number of other packages without anyone noticing …3 Still at “so what”?

Ok, so how serious is it?

As Brodie said : “Everybody and their dog has released security reports, basically immediately.”4 In addition to RedHat enterprises releasing CVE-2024-3094. CVE is short for Common Vulnerabilities and Exposures, they are basically reports on publicly disclosed computer security flaws.5 RedHat,6 Debian,7 Arch,8 openSuse9 and others published security reports about the situation. This CVE was scored on the CVSS (Common Vulnerability Scoring System) at a 10.0 out of 10. This is as bad as it gets. So bad, in fact, that RedHat recommended immediately stopping use of any Fedora Rawhide instances currently in operation. That would be akin to Microsoft releasing a statement that says “Yeah… Just stop all computers currently running the latest revision of Windows…”

Ok, so what should you do?

Nothing. But, not because there is nothing to be done, there is plenty that needs doing, just not by you. Your system is most likely safe, as long as you make sure you update frequently to always have the latest security patches. Even if you were to use one of the Linux distributions with the specific package version that got infected, you’d most likely know about it by now and have taken the necessary actions. In short, this was only another entry in the long list of problems you never heard of. This time, we only got lucky because Andres Freund, a curious engineer, investigated a 0.5 second latency issue that occurred during a software update.10 With enough digging, he was able to trace it back to the xz package and eventually unveiled the backdoor and notified the relevant people about it. The issue was then addressed within hours and reduced to nothing but a whisper of a dying catastrophe that could have been. So, for the vast majority of people, there is simply nothing to be done and nothing to worry about. Just be very glad to know Andres was there, that he, single-handedly and without a doubt, prevented what could have been a disaster, and that we have the robust infrastructures in place to find vulnerabilities like this one as soon as they are introduced. Or do we?.. Keep reading.

A Bigger Problem

Why did I scare you like this?

Because everything above is only a tiny part of a much bigger problem and although, in this instance, much like many others you never heard of, we were incredibly lucky that the issue was detected promptly and disclosed publicly. Things could have gone much worse, but it didn’t… this time.

So what’s the problem?

People Don’t Realize that the World Runs on Linux

A vulnerability that affects a handful of bleeding-edge Linux distributions can be a problem for everyone.

Let’s take a step back. I concede that many might not feel particularly vulnerable to this specific exploit, even had it not been detected and dealt with right away, even if it infected many systems that were vulnerable. As I put it myself, it would have affected primarily some bleeding-edge Linux distributions, but people use Windows or Mac in their professional and personal life. I am well aware of this, and there is no question about that. Of course, I could point out that in 2023, Microsoft introduced an incredibly popular new feature on Windows called WSL (Windows subsystem for Linux), which is intended to run a Linux environment on Windows machines and that MacOS is part of the Unix family and has a lot in common with Linux, but never mind that. Let us simply focus on the fact that most of the world’s digital infrastructure runs on Linux-based systems. This is not my usual Linux preaching or banter for desktop and personal use, for which the Linux market share is generally below a few percent and not worth even considering here. I am referring to the web servers where we put an egregious and ever-increasing amount of personal information, the mainframes that run banks, airlines, retailers, hospitals, city infrastructure and the supercomputers that are used for research and this little new toy called “AI” for which Linux is the overwhelming majority.11 What about phones? Android, which worldwide market share has continuously remained above 60% since 2015,12 uses the Linux kernel.13 Here are a few statistics,

  • Android has a market share of 71.77% 14
  • The top 500 most powerful supercomputers in the world all use Linux 15
  • 94% of Amazon’s EC2 cloud computing platform runs on Linux 15
  • Microsoft is now heavily reliant on the operating system as evidenced by the fact that over 60% of cloud instance found on Azure run on Linux 15
  • Linux dominates cloud computing, powering over 90% of cloud infrastructure. It is the backbone for major public cloud providers like AWS, Google Cloud Platform, and Microsoft Azure due to its scalability, security, and cost-effectiveness. 16
  • 96.3% of the top one million web servers run on Linux 16
  • 60% of auto shipments rely on Automotive Grade Linux, a project designed for carmakers, suppliers, and tech companies. 17
  • 71.8% of IoT developers choose Linux as their preferred operating system. 17

It should seem pretty obvious at this point why a vulnerability as problematic as the xz backdoor that affects Linux systems could and most likely would have had an effect on your personal, and most likely your professional life if you interact with computers at all. But the problem is not that the world runs on Linux, it is that most are not aware of it. Then it becomes all too easy for institutions, governments, companies and generally the people in charge to also forget that these systems, the systems that run so much of our modern lives, were not built and are not maintained by sunshine and rainbows. Actually, it was and still is, but that’s something I’ll get into in part 2. For now, keep being damn glad that Andres was there.

It Keeps Happening

Back in 2009, a company named RockYou got hacked and an unencrypted list of its user’s login credentials (usernames and passwords) was downloaded by the attacker .18 A similar thing happened to LinkedIn in 2012 .19 Both lists of passwords are now ubiquitous in every penetration tester (the good guys) and hacker (the bad guys) toolboxes 20 and have a measurable impact on people’s digital life even years later .21 Both incidents could have been nullified or outright avoided if these not so tiny companies followed basic cybersecurity guidelines from the time (e.g. not storing passwords in plain text and salting the hashed passwords). To add insult to injury, RockYou then waited several days after the breach had occurred to notify its 32 million users that their credentials had been stolen .22 LinkedIn, on the other hand, immediately apologized and asked its users to change their passwords .23 Neither offered additional compensations other than their PR-styled mea culpa prompted by user’s outrage, a pattern that will become blatantly obvious as similar breach keep happening.

Now ask yourselves. Considering that you have little to no power over this, are you confident that the big platforms, on which our modern lives rely, are doing everything dutifully, during the Easter holiday, to ensure that none of their systems have been compromised? And if they have, that you will be dutifully and transparently informed in a timely manner and that they will take the necessary steps to compensate you accordingly should you be affected? Recent history leaves me skeptical for many reasons. Curious? Feel free to go through this long list of Data Breaches that have happened in 2022, 2023 and 2024 so far. Here are a few of them that I selected because I remember hearing about them when they occured,

  • The profiles of nearly 7 million 23andMe (a saliva DNA testing company) were accessed by hackers. 24
  • Lastpass, a password management service where users aggregate numerous passwords and credentials, had a security breach. This series of multiple events began on August 8th 2022. The first time they notified users their data was compromised was on November 30th 2022. 25
  • Numerous Health organizations had healthcare data stolen through via the MOVEit exploit 26 including the birth registry from 3.4 millions in Ontario ,27 4.1 million patients in Colorado 28 and medicaid recipients from more than 610 companies in Missouri ,29
  • Trello was compromised in January 2024 and the personal details of roughly 15 million users have been stolen and are being sold on hacking forums online. 30

Footnotes

  1. https://en.wikipedia.org/wiki/XZ_Utils ↩

  2. https://www.howtogeek.com/737563/what-is-root-on-linux/ ↩

  3. https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/ ↩

  4. Broadie’s video on the XZ Linux Backdoor ↩

  5. https://www.redhat.com/en/topics/security/what-is-cve ↩

  6. https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users ↩

  7. https://lists.debian.org/debian-security-announce/2024/msg00057.html ↩

  8. https://archlinux.org/news/the-xz-package-has-been-backdoored/ ↩

  9. https://news.opensuse.org/2024/03/29/xz-backdoor/ ↩

  10. https://www.akamai.com/blog/security-research/critical-linux-backdoor-xz-utils-discovered-what-to-know ↩

  11. https://en.wikipedia.org/wiki/Usage_share_of_operating_systems#Market_share_by_category ↩

  12. https://www.statista.com/statistics/272698/global-market-share-held-by-mobile-operating-systems-since-2009/ ↩

  13. https://en.wikipedia.org/wiki/Android_(operating_system) ↩

  14. https://techjury.net/blog/android-market-share/ ↩

  15. https://www.blackdown.org/linux-facts-and-statistics/ ↩ ↩2 ↩3

  16. https://www.enterpriseappstoday.com/stats/linux-statistics.html ↩ ↩2

  17. https://kommandotech.com/statistics/linux-statistics/ ↩ ↩2

  18. https://www.keepersecurity.com/blog/2023/08/04/understanding-rockyou-txt-a-tool-for-security-and-a-weapon-for-hackers/ ↩

  19. https://en.wikipedia.org/wiki/2012_LinkedIn_hack ↩

  20. https://github.com/danielmiessler/SecLists ↩

  21. https://therecord.media/hackers-leak-linkedin-700-million-data-scrape ↩

  22. https://techcrunch.com/2009/12/14/rockyou-hack-security-myspace-facebook-passwords/ ↩

  23. https://www.linkedin.com/blog/member/archive/linkedin-member-passwords-compromised ↩

  24. https://www.cnn.com/2023/12/05/tech/hackers-access-7-million-23andme-profiles/index.html ↩

  25. https://www.cybersecuritydive.com/news/lastpass-cyberattack-timeline/643958/ ↩

  26. https://arstechnica.com/information-technology/2023/06/mass-exploitation-of-critical-moveit-flaw-is-ransacking-orgs-big-and-small/ ↩

  27. https://globalnews.ca/news/9983285/ontario-birth-registry-data-breach-born/ ↩

  28. https://techcrunch.com/2023/08/14/millions-americans-health-data-moveit-hackers-clop-ibm/?guccounter=1 ↩

  29. https://www.hipaajournal.com/missouri-dss-medicaid-recipients-moveit-hack/ ↩

  30. https://www.forbes.com/sites/barrycollins/2024/01/23/personal-details-of-15-million-trello-users-up-for-sale/?sh=58e39ee24d94 ↩

Graph View

Backlinks

  • No backlinks found