Secure open source foundations: Flux case
Open source projects usually have a large network of dependencies connected to a code base that will eventually build and compile into a package that may, in turn become the dependency of another project. That is why concerns about open source as a potential vector for critical vulnerabilities have always existed despite Linus’ law. Flux was able to reach a CVE count of zero due to its maintainers limiting the amount of dependencies used....
Open source was to be shared from the beginning. It is only natural to the communities that build, maintain and release it to use other open source projects on their own. However, concerns about open source as a potential vector for critical vulnerabilities have always existed despite Linus’ law. Initially, the main underlying idea held up and it went like this: the open nature of open source software would put enough eyes on it to surface any bug or vulnerability eventually. However, it turns out this is not entirely true. The current open source supply chain vulnerability surge is a proof of it. Open source is everywhere to be found and its usage will only increase, so this problem will likely not go away anytime soon, quite the contrary.
The problem is remarkably acute in open source used in the enterprise. Open source is making a significant impact in such types of companies that are hiring more and more OSPOs and kicking off innersourcing programs. However, such mastodons are considerably wary of security threats. They are expected to be compliant with a myriad of regulations such as PCI DSS, HIPAA, or Sarbanes-Oxley, and are liable for considerable fines for breaches of their massive client bases and other huge threats. If not properly audited, an open source value chain is a potential threat to any enterprise. Ask Equifax (a vulnerability in Apache Struts) about it if in doubt.
So where does this problem come from and how do we tackle it at Weaveworks, a company that co-founded the CNCF, created several enterprise ready popular open source projects like Flux, Flagger, Cortex and more?
Software supply chain attacks and dependency sprawl: a trust nightmare
Almost every single open source project is a massive network of dependencies connected to a code base that will eventually build and compile into a package that may, in turn become the dependency of another project. Just look at this beautiful image of the nixverse, the NixOS dependency universe, one of the biggest in the open source world.
An overview of the nix dependency graph
Dependency and package managers
Dependency managers or package managers exist for this very reason: to manage the download of packages and their versions. While this makes 3rd party software consumption easier and cheaper it can also streamline the spread of a vulnerability. If an attacker gains control of NPM he or she is in control of the distribution of 750,000 packages. If that number is in itself mind blowing it dwarfs compared to the number of programs that depend on those packages to execute themselves.
The CVE age in code until report graph [in the cURL project] shows that in general, reported vulnerabilities were introduced into the code base many years before they are found and fixed. In fact, the all time average time suggests they are present for more than 2,700 – more than seven years. Daniel Stenberg, maintainer of cURL
There are many heuristics to follow when considering the reliability of a certain package like code quality, reputation, dependencies of the dependency, testing, debugging… and yet it is unrealistic to assume that vulnerabilities are completely absent from them.
Know, Prevent and Fix
Google’s proposal to fix vulnerabilities in open source is the framework Know, Prevent and Fix. While their research and proposed framework will likely burden more maintainers and stewards of open source projects, it has also surfaced the lesser known fact that most vulnerabilities are preventable and non ill-intended.
Taking a step back, although supply-chain attacks are a risk, the vast majority of vulnerabilities are mundane and unintentional—honest errors made by well-intentioned developers. Know, Prevent, Fix: A framework for shifting the discussion around vulnerabilities in open source
Another approach to the problem, this time industry-wide, is the creation of the Open Source Security Foundation. On a personal note, I am happy that OpenUK, an organization of which I am an ambassador of, is part of it. It’s industry wide organizations like this that can help mitigate broad and complex problems like securing the open source supply chain security. Their charter includes the promotion and evangelism of security frameworks for reviews or metrics.
Securing downstream: Secure Continuous Delivery
These approaches, albeit well intended, are partially flawed because they burden the already overloaded open source maintainers. It’s not like every open source project has enough infosec maintainers to keep vulnerabilities at bay.
There’s a ratio of one infosec person per 10 infrastructure people per 100 developers in large companies. James Wickett, Head of Research at Signal Sciences. DORA State of DevOps 2018.
But those of us in the Continuous Delivery space know that the software we built is used in large enterprises and we are committed to deliver it securely. These are Flux’s latest data on CVE count:
October (Flux 1.21.0) - 27 CVE issues
January (Flux 1.21.1) - 15 CVE issues
February (Flux 1.21.2) - 15 CVE issues
March (Flux 1.22.0) - 10 CVE issues
March (Flux 1.22.1) - 5 CVE issues
April (Flux 1.22.2) - 0 CVE issues
In fact, one of Flux’s latest releases was solely devoted to correcting a CVE. The counter reached 0 in the following weeks as we continued monitoring the scan results with Snyk, making fixes available as quickly as possible to mitigate potential upstream CVE issues.
How we did it: Flux next big release.
As a company, Weaveworks is committed to deliver to its users and customers secure software. Flux engineers – those that work at Weaveworks and those who don’t – have several ways to tackle the dependency sprawl surface attack and deliver secure software to its users.
First and most importantly is to limit the amount of dependencies used. It’s the dead simple solutions that are usually ignored and, let’s be honest, sometimes we just abuse our dependency addiction. Since Flux is built in Go, what Flux engineers will leverage is the Go-native solution first rather than a third party package or dependency.
This approach though, puts the burden on Go. Trust has moved from third party packages and dependencies to built in, Go-native features. Thus, the Flux team follow the bleeding edge of Go dependencies by upgrading often. Latest release is always the latest crowd-reviewed version of Go and thus the most reliable version of it.
Flux’s GitOps Toolkit currently has 0 CVEs. That’s mainly because all project repos follow the Linux Foundation’s set of best practices encapsulated in the CII Best Practices badge program. Each pull request is thoroughly code reviewed and once approved it kicks in GitHub’s CodeQL and Snyk scans to detect any vulnerabilities.
What it means in terms of security to use Flux
Open source software constitutes around 80 to 90 percent of all software. We are committed to make that figure bigger but, most importantly, by making the projects we fund more secure to our users. We want our customers to trust Weave Kubernetes Platform (WKP) and the open source software that runs beneath. We believe that open source is much more secure than closed source and that’s mainly because contributors and maintainers do a great job at implementing best practices shared across the industry.
The Department of Defense recently moved to Flux to meet compliance and security requirements. It doesn’t get more “battle tested” than this but the community will always strive to make Flux a secure continuous delivery platform.
The fact that WKP runs already critical software delivery through GitOps in different enterprises does not curb our acknowledgement of the critical importance of open source and the opportunities and challenges related to it. We strive to make WKP the most secure software every day.
References
- Surviving Software Dependencies, Russ Cox (Google), ACM Queue, January/February 2021.
- How we’re helping to reshape the software supply chain ecosystem securely, Heather Atkins (Google).
- Know, Prevent, Fix: A framework for shifting the discussion around vulnerabilities in open source, Eric Brewer (Google), Rob Pike (Google) et al.
