The Sustainability of Open Source
Open Source software plays a critical role in organizations of all sizes. When open source fails, the natural response is to question why the failure occurred and how to prevent the problem from happening again. Increasingly, failures stem from lack of support and/or misalignment of interests.
Unfortunately, this is a difficult problem to assess, and even harder to solve. There are answers, but they’re often underwhelming and sometimes unwelcome.
Open Source Fragility
Recently, the Apache Foundation disclosed a vulnerability known as Log4Shell. It affects the popular Log4j project (a logging utility). Every major cloud provider has built services on this tool. Millions of smaller providers have too. The public disclosure of vulnerabilities alert everyone so protective updates can be applied to their systems.
In this case, hackers are abusing the vulnerability faster than organizations can patch them. While bigger companies race to fix these issues, many smaller organizations do not have the resources to be so nimble.
The Log4Shell vulnerability is one of the higher profile exploits seen in open source recently, but there are many others.
In September 2021, a tactic described as Trojan Source was detailed by the University of Cambridge. The attack uses hidden characters to alter the programming logic, allowing hackers to inject their own code into a program. In open source projects, code can be contributed by anyone, but must be accepted/approved by the project maintainers. With Trojan Source, an attacker submits a seemingly helpful patch with hidden characters that alter the logic. Open source maintainers usually review code submissions, but cannot see this exploit (it is literally invisible). Maintainers unwittingly approve malicious code without ever knowing a danger exists. This code is then distributed to anyone and everyone using the project.
Trojan source is a more sophisticated attack model that blends physical and social hacking to exploit software. It extends beyond code contributions to open source projects. For example, popular developer help sites like Stack Overflow are a breeding ground for Trojan Source exploits. This developer Q&A site is well-known as a source of “copy/paste” code snippets developers use in their own software. Sites like these make it incredibly easy to copy/paste Trojan Source into software. This is part of a greater copy/paste behavioral problem promoted (often inadvertently) by the tech industry itself, and it reaches far beyond Stack Overflow.
The “open” nature of open source software is being exploited by malicious people, yet this is not the only threat open source faces. Many popular projects are severely understaffed and/or vulnerable to turnover.
The JavaScript community is one of the largest development communities in the world. This community has suffered several staffing problems that have impacted millions of people. The “core-js” project is a good example. This project is commonly used in developer tooling and automation services. It is downloaded 26 million times every week. It was maintained by one person, who was sentenced to 18 months in prison for vehicular manslaughter. In the blink of an eye, all maintenance stopped.
In the 2016 “left-pad” incident, the project maintainer “rage-quit” over a naming/branding dispute. He deleted his open source code from a popular code package manager (273 modules), immediately impacting millions of organizations. The “left-pad” code was only 17 lines long and was quickly replaced within two hours, but what happens when more complex software is abruptly removed? The incident highlights the fragility of open source governance and the trickle down effect it can have on all software developers.
Open source software is clearly vulnerable to increasingly complex attacks. Its transparent nature makes it easy to abuse. Most open source authors and maintainers manage their software as side projects. They do not have the resources of big tech companies (or even little tech companies). Their focus is on simpler and more benign decisions such as fixing something people won’t pay for versus spending the evening with their spouse and children. As the weight of the world’s tech problems pile onto open source maintainers, many feel exploited. Evidence already shows this leads to abandonment.
A Problem of Choice
Organizations keep asking what the world can do to sustain “critical open source infrastructure”. However; the problem isn’t open source infrastructure. It’s business infrastructure. The fact that organizations use open source projects for their critical infrastructure is a choice.
People want reliable software for free. Personally, I’d also like to live, eat, travel the world, and golf for free. I’d like to not pay taxes too. Unfortunately, that’s unrealistic. Without value, there is no incentive to produce. Perhaps people need to adjust their expectations of what they’re getting compared to what they’re providing in exchange for that value. Changing perspectives of open source would change the choices people make about using/supporting open source.
Conceptual Conflicts
One of the biggest challenges to open source sustainability is the inner conflict between the core principles of open source and the audience it services.
Open source is based on the idea that community members can receive greater value by sharing and contributing to a common goal under a shared governance model. This differs from capitalism, which is based on profit through private ownership.
Prior to Github/SourceForge, open source was nascent. Companies didn’t rely on open source. Some even saw it as a threat to their existence. Instead, software was purchased with an understanding of the exchange of value.
Open source fails to provide adequate exchange of value to its creators. There are a few open source developers making a living on their own, but most of the open source community is employed by corporations. Employers typically dictate what employees will work on, meaning a project is still susceptible and influenced by the needs of employers.
For the very few making a living as independent open source developers, it is proving to be a dead end job. There aren’t promotions, raises, or benefits. There isn’t much room for advancement. Remember, those who produce innovative open source projects are often more ambitious — a character trait that doesn’t align well with routines that don’t often change. The lure of jobs, startups, and even consulting often trumps what may have previously been a benevolent effort to better the world through open source.
There is an argument that open source creates career opportunities for developers. If a project becomes popular and is relied upon by companies, it bolsters the profile of the developer. This may make it easier to get a better job, but it wouldn’t equate to the value given to the community. Being “open source popular” is the equivalent of getting a new job title without a pay raise. It makes people feel good amongst their peers, but doesn’t do anything tangible for their life.
Open Source Made us Lazy
Prior to 2010, when open source was less common, companies did their own due diligence before purchasing software. Today, technical choices are often based on the popularity of a project. Many decision makers delegate their tech decisions to others with statements like “if it’s good enough for Facebook, it’s good enough for us”.
There is an argument that open source saves time and “smart people” have already done the hard work. There’s also a strong “don’t recreate the wheel” attitude. However; this often disguises the fact many people don’t understand the problems they’re trying to solve. This excuse isn’t much different from the old adage “Nobody got fired for buying IBM”. In the open source community, this attitude leads to frustration when consumers complain about problems they don’t take the time to understand.
These days, it’s not uncommon to see tech decisions based on “Github stars”, tweets, or blog posts describing “the next big thing in open source”. It boils down to the fact people are less discerning than they used to be.
Not everyone is lazy, but it’s much more common than it used to be. Pre-open source corporate “technology selection” meetings often ignored software because vendors weren’t established enough. There wasn’t enough confidence those companies would be around in a few years. They may not have had enough customers. They only generated $5M/yr in revenue. They’d have only been in business for a few years. Today, companies will select software backed by one weekend developer and no funding.
Devalued Technology
Technology selection meetings of yesteryear included budgeting for maintenance contracts. People valued reliability. Today, they don’t have to pay for anything. Consider the state of sponsorship for open source projects. People can volunteer to support projects, but very few actually do.
In my own work, I’ve built open source software used daily by millions of people. Over 3.5M users leverage a single project. Another project is used to power 1M+ enterprise applications. The most complaints come from people who want changes “for their work”. None of their employers sponsor the projects.
Independent sponsors are different. There are developers who sponsor other developers. However; it’s usually not enough to change the sustainability of a project. The highest contribution I’ve ever seen was from an individual who wanted to help several of his clients. That person single handedly contributed 3X that of everyone else combined. It’s not uncommon for individual developers to recognize day to day value of open source Nevertheless, but they have limited means to support others.
Tech managers face a conundrum when deciding whether or not to donate to a person making software when there is no direct benefit they can communicate to their leadership. Developers know there is a time and reliability benefit, but that doesn’t immediately translate to bottom line revenues.
Sponsorship works for moving some features forward on open source projects, but it is not proving to be sustainable over the long term.
Support Organizations
Organizations like the Apache Software Foundation, the Linux Foundation, and others are not proving to be sufficient enough to support open source software reliably. The Log4Shell exploit is a prime example of a foundation-supported open source project that still struggles to meet critical business needs.
Foundations also struggle with a governance problem. They often work more like a government elected by lobbyists. Sponsors get board seats to vote on proposals and set the general tone for the foundation’s goals. The more you sponsor, the more representation/influence you get. In other words, there is a pay-to-play concept that happens. Influence and bias is inevitable in anything volunteer-driven.
The standards authorities (IETF, W3C/WHATG, IANA, etc) are less prone to this, but still susceptible.
These organizations often struggle from the same talent shortages employers struggle with. For industry experts, it’s more lucrative to take a high salary from a company. Additionally, many roles within these organizations don’t really offer much career mobility. As a result, people come and go as their careers grow.
Foundations can help sustain some software, but they drive markets towards commoditization. As the Log4j situation shows, commodity software can acquire a large user base. Open source projects don’t maintain lists of users who download their software. Disclosing security vulnerabilities is something that must be done publicly. Upgrades/fixes are the responsibility of users. Project maintainers only recourse is to encourage users to make fixes.
Sustainable Software
There is a proven sustainability model for software: commercial software. It’s time to abandon the idea that sustainable software can be free. The concept of “sustainable open source” is an unrealistic have-your-cake-and-eat-it-too dream. That said, open source can still play an important role in producing sustainable software people can rely on.
The Graduation Process
A growing trend is the graduation of open source software into commercial ventures. Open source already serves as a solid market-building activity - even for individual developers who aren’t trying to build a market. Successful open source projects will see continued growth. As more organizations rely on these projects, greater demands are placed on the developers. At this time, projects should “graduate” to commercial ventures.
The venture capital world is already seeing a variation of this:
The venture capital world may view the open source movement as a means for saving underperforming investments, albeit not necessarily a good one. Still, there are many similarities between open source and the more traditional startup playbook that shouldn’t be dismissed.
Most startup incubators/accelerators preach the importance of identifying markets before building a product. The idea is to get founders to think about inciting hunger for a product before spending money to produce it. This “proof of market” concept is used to justify a “proof of concept” (POC) effort. If markets respond positively to POC efforts, an MVP (minimum viable product) is produced — i.e., the bare minimum to people will pay for.
In software development, open source services like Github provide a means to test ideas through code. The difference between a startup and a new code venture is programming code needs to be produced before the market will talk to the creators. Project popularity isn’t a perfect metric for predicting success, but there is a point where it’s hard to ignore. It serves as proof of market. Sponsorship helps validate a degree of the monetization viability. Again, these aren’t perfect metrics, but they illustrate the parallels with the typical startup process. Ultimately, open source can be a tool for proving business viability.
In recent years, open source has produced several new public companies. MongoDB ($MDB), Gitlab ($GTLB), and HashiCorp ($HCP) are good examples of public companies built on their own original open source software. Private companies, like Postman, have seen multibillion dollar valuations. These companies all share a common origin in open source, sustained by commercial business practices.
Commercial Software
Opponents of software commercialization argue that it creates gatekeepers. Those who control the software control its direction. This is a valid criticism, but it also exposes a contradictory notion. First, this happens in open source, often times by accident (i.e. solo developer vs organization). Second, tech leaders who delegate decisions to popular consensus and outside developers are the most common to seek “protection”. That’s essentially what gatekeepers do. In commercial software, customers have leverage and a voice with the gatekeepers. If customers aren’t happy, they stop paying. That’s strong incentive for developers to keep customers happy. In open source, a “star” is about the only compensation offered.
As the world moves towards “decentralized everything” and democratization of software (Web3 & beyond), commercial software and the inherent gatekeeper problem becomes a hotter topic. The question to ask is whether it’s actually a problem.
The world can’t completely democratize software and still serve commercial interests - especially for core infrastructure. In software, democratization is the same as commoditization. In business, commodities aren’t lucrative and don’t promote innovation. Capital markets thrive due to competition. Competition requires diversity, which is the exact opposite of commoditization.
Commercialization can solve certain security problems and it is sustainable. It’s just not open source or democratized.
Professional Open Source
Another solution is to pay open source maintainers as professionals. This practice has been active in the software development community for many years now. Companies will hire employees whose full-time job is working on an open source project.
This has provided mixed success. There are certainly projects which have accelerated growth and provided value to the software community. Contributions delivered by full-time salaried support are the direct result of company support. Still, most companies don’t sponsor salaried open source developers. Those who do are usually tech companies with a vested interest in the industry.
Professional open source is less susceptible to having a single gatekeeper, but still prone to the influence of the biggest sponsors.
The biggest challenge with professional open source is it’s still a dead end job. When core developers leave a project for a promotion/change, it impacts the entire project. Funded projects have to compete in the tech talent wars. However; when companies only back developers, the projects suffer. Product continuity and replacement staff are problems companies think about, but individuals typically don’t.
The idea of professional open source may not be a perfect solution, but it hasn’t proven to be an utter failure either.
What’s Next?
It’s hard to tell if the industry will find alternative ways to sustain open source software. However; the success of commercial software provides a proven path to keep software developers motivated.
*** Remember to clap if you liked this article. ***
Thanks to Miguel Paraz for suggesting this as a blog post. The original inspiration for this piece can be found in his question at How should critical open source infrastructure such as Log4j be maintained and funded to avoid security problems? — Quora.
