I’ve got the last issue “Admin - Network and Security” magazine. In the short news section they write about the newest State of Open Source Report 2024. According to it 95% of organizations increased or maintained their use of open source software. Almost every organization in the world uses open source. The rest 5% of organizations simply don’t know that they use open source software. Let be honest. In today’s world even if you use Windows, you use open source software or software based on open source or software developed with the help of open source software.
Open source won 🎉 Looks like at least.
Look at the excerpt from the news above. I trimmed it at the top reason using open source software. It is not ideology that companies would like to make a better collaborative world. It is not features, open source software provides. It is not innovation, security, open source code, name your favourite open source feature here.
The most favourite feature of open source is its cost. Or visible absence of costs.
Do you develop a new software product? Don’t invent a bike - take an open source project and reuse its code in your product.
Do you make a new solution for your customer? Take an open source project and implement it. The customer saves license costs.
Do you want to increase velocity of your business projects? Take open source! Same as many years ago everyone knew, that nobody got fired for buying IBM, nowadays - nobody gets fired for choosing open source!
Looks great and everyone profits from the open source development.
Oh, really?
There is one guy who suffers from this trend. It is open source developer of a small unknown open source project. His (or her) project is not funded by any company and most of the users uses the project without knowing of its existence.
Did you see the picture below?
It is XKCD comic from 2020. Funny, isn’t it?
In reality - no.
Minor performance problem. Huge infrastructure problem.
End of March 2023 one german guy mentions that his SSH login takes 0,5 (or other sources say 0,75) seconds more than usual. What looked like a small performance issue was indeed the biggest security problem of the last decade. If you didn’t hear about the problem, SFTW. But for me it is not a security problem, but an open source infrastructure problem, shown in the comic above. Let me explain.
Many modern Linux open source projects rely on each other. If you want to login to your remote Linux server, you must have ssh service running there. To start the ssh service, systemd must be running. To start systemd, the server must be started. This is the very well known path. Same in all operating systems but the names can be a little bit different. What a “normal user” doesn’t see here, is a dependencies hell. Each software requires some library or some binary which should be there. Sometimes the library or the binary is required only for some minor function of the software, but it still must be there.
What you don’t see in the path is xz requirement. XZ is a small compression library and set of utilities. Nothing spectacular. Since some years Linux kernel can be compressed using xz. It means you can find xz everywhere meanwhile - on each Linux installation. If it can be found everythere, it can be used by other software. Another part of our path which uses xz is systemd. Systemd uses one of the libraries from the xz project. Sshd which is systemd-agnostic is patched by distribution maintainers to work with systemd. Now we have sshd which loads systemd libraries, which load liblzma library from the xz project. A user has no chance to see it and change it. Users must blindly trust their distribution vendors.
The attack on xz started 2 (TWO) years ago. You can see the attack’s development in the picture below. It is not my picture. The picture’s author has signed it and I found it at https://cdn.arstechnica.net/wp-content/uploads/2024/04/xz-backdoor-graphic-thomas-roccia-scaled.jpg.
This problem has non-technical nature.
The project was started and maintained by a single developer since 2009. Many big companies use the project. Some of them earn money on it. Do you think any of such companies invested some money into the project? Do you really think they helped the only developer of the project by any means? NO!
Lasse Collin, the xz developer, wrote 2022 (same year as the attack begins), that he has health issues:
I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things.
It’s also good to keep in mind that this is an unpaid hobby project.
Did any company jump in to help him? NO!
The only guy who helped him was the attacker. He helped him to maintain the project, to develop it further and to check the patches. He gained the trust and implemented the backdoor one year later.
Don’t blame Lasse, blame your Linux vendor!
OK, xz is a small one man project. You can say, we shouldn’t rely on such one man projects and should use projects which have real history, many developers and are sustainable.
Let’s take on another project - ffmpeg.
ffmpeg started 2000 by the same programmer, who developed QEmu and is used by many people and software since that time. It is also very good to proof that IBM Power has superior performance comparing to x86 systems. Try to recode some longer movie. Where I needed several days on my x86 hardware, took several hours on a POWER8 S822LC system.
FFmpeg project has many developers, many users and code of conduct. A very sustainable project you can find in every Linux distribution and used by Blender, VLC, Kodi, Youtube, Google Chrome and … Microsoft.
Majority of my readers works in corporate environments and are familiar with this famous Microsoft product called Teams. I’ve heard it will be more expensive this year. The Teams had (or still has) an issue with captions for some users. After investigation they (Microsoft developers) found that it is FFmpeg issue:
One FFmpeg developer helped and noticed, that the setting’s name they need is changed in the newer FFmpeg version.
But why in all hell such a huge company like Microsoft selling tons of MS Teams licenses to their customers can’t find bunch of bucks to pay to FFmpeg developers if they rely on their work?
Yes, it is open source and it is free to use. We even don’t talk if Microsoft violates GPL if they use FFmpeg in their closed source product. We talk only about one thing in the world -
if you use open source project in your production, consider paying to the project developers.
Did Microsoft pay to FFmpeg? No. They offered one payment and didn’t pay it.
The picture above is courtesy of Brian Correa (@_briancorrea on X).
You can contradict me that
there are successful examples of converting open source projects into commercial profit bringing products.
Yes, there are. Let’s talk about them.
The biggest example is Red Hat and their flagship product Red Hat Enterprise Linux.
Successful? Yes, of course!
Last year Red Hat closed the access to the source code of Red Hat Enterprise Linux for all non-customers. It was accused by many open source users in license violations. It is not true but we will leave the discussion to lawyers and let’s think about the reasons of such move.
The reason was named in one word - “freeloaders”. Many CentOS, Rocky, Almalinux users felt to be accused to steal Red Hat’s IP even if they just follow normal open source practices. No, sorry guys, you as a user of one of this distribution is not a “freeloader”. If you want to use Red Hat Enterprise Linux you always have a choice to get it for free - in Red Hat Developer Subscription. You have a small company and develop software? No problem at all. You can get Red Hat Enterprise Linux for free.
The “freeloader” is for example Oracle and some other Linux distributions which use freely available code to build and to sell their own products without paying a cent back to Red Hat for the work Red Hat did for them. But not only they are the problem.
I know a cloud provider. They bought several RHEL premium subscriptions to be able to open a ticket in case of a problem. They recompiled RHEL code and used it as their own Linux distribution almost on every server. What is this? Budget optimization or greed?
I’ve heard about a very big and famous financial giant, one of the world’s biggest banks. They saved a lot of money by rebuilding RHEL and renaming it in “my internal Linux distribution”.
The whole world missed the money which could be used to implement new features in FFmpeg or to make a security audit in xz.
Red Hat is not the only loser of the open source. The latest example is Redis. The first two examples for me were Elastic and Mongo. All three are successful open source projects used by many enterprises and users. All three wanted to develop their projects further and with help of cloud services. All three failed because major cloud providers used their open source projects and sell them as their own services under their own names cheaper. All three changed their software code licenses and all three were accused in betraying open source ideals.
The picture is courtesy of Kat Scott (@kscottz on X).
The problem is not open source and not huge companies wanting to earn more money.
The problem is we, human beings.
We all want to consume more by doing less and paying less for our consume. It is normal. But it makes the modern open source development model unsustainable.
Rivals like Microsoft, Oracle and Google can co-develop Linux kernel together. They need it and they invest a lot of money into the development. But if we talk about smaller projects even such famous like Elastic, Redis, Mongo, FFmpeg and many others, nobody is there to help them. I know many more open source projects struggling to find any financial help and used by almost everyone in the world.
The modern era of open source is over.
Does it mean that we will not have open source in the future? Of course not. Open source projects will be created and maintained as they were before Richard Stallman. Of course you know that the first operating system in the world was developed by users, not by a vendor.
The open source will evolve. But how? Do you have any ideas? Feel free to comment.
Have fun using your (still) open source projects
Andrey