[tl;dr sec] Now With More Deep Dives

Hello there! A quick note from me (Clint Gibler), the creator of tl;dr sec.

You may already be familiar with how tl;dr sec is a free weekly newsletter round-up of the best articles, tools, and general cybersecurity research.

But we’re also getting more into long form, deep dive content (like how to securely build product features using AI APIs, a Staff Security Engineer Guide, or how to do a risk analysis of Kubernetes clusters).

This time, I’d like to introduce you to Francis Odum, who will be sharing a two part overview of software supply chain security.

Francis is the author of the software analyst blog. He teaches a cybersecurity & SaaS bootcamp on Maven. He previously led cybersecurity research at the venture capital firm, Contrary, and helped build Contrary Research. Follow him on Twitter @InvestiAnalyst or LinkedIn at Francis Odum.

Actionable Summary

• High-profile attacks such as Microsoft Windows HCP, Solarwinds attack, Kaseya ransomware, and CodeCov attack, Log4shell have raised public awareness of the importance of securing software supply chains.

• Software supply-chain (SSC) attacks have existed for many years. However, these large breaches have created more awareness of the importance of securing every aspect of the software development lifecycle.

• Today, more than 85% of enterprises leverage some form of open-source software. Over 27M developers have leveraged over 37M open-source components and packages.

• Gartner predicts that by 2025, 45% of organizations worldwide will have experienced attacks on their software supply chains, a 3x rise from 2021. Equally, a recent study by Juniper estimates cyberattacks targeting software supply-chain could cost organizations an estimated $81 billion in lost revenue and damages annually by 2026.

• This is an issue for enterprises, governments, and developers. SSC has become so critical that it has been driven by the government. The Biden Administration, in its second year in office, released an executive order on SSC risks. This has created a tailwind that has led to a proliferation of companies aiming to protect the supply chain and enable companies to comply with this legislation.

• According to data from NightDragon’s software supply chain report, 70% of CISOs said software supply chain is a top investment priority for them in 2023, and over 96% of CISOs said they are using or considering implementing SSC solutions in the next 12 months.

• We define software supply chain attacks as those attacks and vulnerabilities that occur during the source, build and package stage of delivering software.

• This report is the first of two series exploring software supply chain components. Part 2 is a comprehensive analysis of 12+ SSC vendors and their core differentiators within the software development lifecycle.

This Report In One Minute

Companies are embracing open-source third-party dependencies to ship features to users faster.

Software supply chain security includes using third-party code securely as well as securing the development process from beginning to end.

This report focuses on the following three phases of the software supply chain. Across these phases, we discuss what constitutes software supply chain risks and processes for securing each stage.

1. Source: This stage constitutes creating the actual code used to build an app. We discuss source code review, managing access to code environments and IDE extensions, and source code management systems (SCM).

2. Build: This stage compiles and transforms the source code into a deployable form, typically binaries or executable files and includes the CI/CD pipeline. We discuss risk due to dependencies (unintentionally vulnerable, malicious, transitive dependencies, and pipeline dependencies), the CI/CD pipeline, and containers and registries.

3. Deployment and Package: Bundling software components and dependencies into a deployable format and distributing it for installation on target systems. We discuss Software Bill of Materials (SBOM), code provenance and signatures, and artifact repositories.

By the end of this post, you’ll have a solid overview of the core components of the software supply chain, risks across the chain and how to deploy secure processes at each stage.

If you want to make sure you receive Part 2 of this report, in which we provide a comprehensive analysis of 12+ SSC vendors and their core differentiators, make sure to subscribe to tl;dr sec (if you’re not already), a free weekly update of the best tools, blog posts, and talks in cybersecurity across AppSec, cloud security, AI, supply chain security, and more!

Alright, let’s get into it!

Understanding The Buzz

The demand and supply of open-source tools continue to rise. Similarly, the volume of open-source downloads is accelerating, creating a massive increase in consumption. Across project versions in 2022, there was a 33% growth in downloaded projects to 3T across major open-source ecosystems. JavaScript npm (2M projects) and Maven.js (492K) remain the most popular.

This proliferation of open-source solutions in enterprises has created an opportunity for attackers to use open-source solutions as a means to inflict harm on companies. For example, known attacks on the software supply in 2022 grew over 633% YoY compared to 2021. There was over 742% average growth in these attacks. Similarly, in 2022, the number of malicious packages has continued to increase significantly, growing to over 88K instances.

Introduction To Software Supply Chain

Supply chain attacks are difficult to manage because of the multiple links, inter-dependencies and processes that make up current software development practices. This report aims to categorize the software supply chain attacks into three broad challenges that need solutions that may be intertwined in between them. These include source, build and deployment and package.

The problem of software supply chain is not entirely new. However, there has been a renewed focus on areas of the software development where traditional software security platforms never captured in the past. Many of the legacy providers missed on the following problems.

First, many of these companies focused on scanning for vulnerabilities within source code. That focus was only on one segment of the development lifecycle which created more noise, but never pointed to actionable feedback points for developers. Most of these past solutions focused on source code level compared to understanding the deeper intricacies of dependencies within the build and pipeline stages.

Secondly, many of the legacy players were founded long before the cloud evolved or before open-source became extremely popular amongst developers.

The Rise In Open-Source Software (OSS)

Today, open source is used in 98% of applications. According to the 2023 State of Open Source Report from OpenLogic by Perforce Software, and 80% of organizations increased their use of open-source software over the last 12 months. Open source has grown in popularity because they enable developers build and ship products faster. However, these open source packages have increasingly become a popular vector for software supply chain attacks, creating more risks for organizations. Packages are sometimes uploaded by developers into their registries with little to no security oversight, and are downloaded into codebases without scrutiny. Popular package registries such as JavaScript npm, Python PyPI and RubyGems have become easy targets used to distribute malicious packages.

What Defines Software Supply Chain (SSC)

To understand SSC security, we must first define SSC and clear up any misconceptions. SSC involves building and producing software, much like assembling ingredients for a burrito wrap or raw materials for a manufacturing plant. The software supply chain includes all the processes, steps and components you need to create an application. Just like a traditional supply chain where raw materials are sourced, assembled, and transformed into finished goods before they are distributed to retailers or customers. This framework applies to how software supply chain works as well.

Applying it to software development

Modern ways of building applications can be categorized into three phases: source, build, and package stages. Each of these stages could be vulnerable to attackers. During the source stage, developers write custom code. The build stage is responsible for assembling the raw files of source code and transforming them into binaries (output used for deployment). Finally, the deployment and package stage gets the final software ready for deployment to production use.

Just as car manufacturers outsource many parts to suppliers or offshore teams to accelerate production, software development also involves using external tools from suppliers, which are open-source packages and libraries, to accelerate their delivery cadence.

The three most important phases of the software supply chain as it relates to software development are source, build and deployment and package. At a basic level, here are some explanations for each category.

1. Source: Involves creating the actual code used to build an app. It consists of an assembly of a longer list of components, including custom code, open-source dependencies, publicly hosted code repositories, build and packaging scripts, containers, and more.

2. Build: The process of compiling and transforming the source code into a deployable form, typically binaries or executable files. Agile has changed how software is developed, and many organizations utilize Continuous Integration (CI)/Continuous Development (CD), a practice for frequently and automatically building, testing, and deploying software changes. Examples include GitHub Actions or GitLab runners.

3. Deployment and Package: This involves bundling software components and dependencies, like libraries and resources into a deployable format and distributing it for installation on target systems, streamlining application deployment and updates.

Layers of Software Supply Chain Attacks

After building upon a basic understanding of each of these areas, let’s understand how risks can be introduced at each layer.

According to the NIST guide, an SSC attack is when a malicious party tampers with steps, artifacts, or actors within the chain to compromise the consumers of a software artifact down the line. In order to carry out an SSC attack, an attacker needs to subvert, remove, or introduce a step within the SSC process to modify the resulting software product. The chart below shows the various areas where an attacker could launch an attack against the supply chain.

The Supply Chain Levels for Software Artifacts (SLSA) framework is another important framework used to understand each layer of attack. The SLSA (pronounced “salsa”) framework provides a checklist of standards and controls to prevent tampering, improve integrity, and secure open-source packages. SLSA breaks down the three categories (source, build and deployment) into eight distinct types of threats that could occur at each stage.

Using both definitions from the NIST and SLSA framework around the source, build and deployment/package stage. The rest of the report is built around exploring each of these areas in more depth.

1. The Source Code Layer

2. The Build and Pipeline Layer

3. The Packaging and Deployment Layer

Source Code Layer

Everything begins at the source code where developers write custom code using languages like Javascript, C++ or Python. This section discusses SSC risks that could occur at the source code layer as well as the three solutions used to prevent these attacks.

1. Source code review

2. Managing access to code environments and IDE extensions

3. Source code management systems (SCM)

Source code review

Code is the first and core ingredient in the software supply chain used to build the end product, an application. This code can be developed in-house or from external sources. As code moves across different hands, there are a number of vulnerabilities that occur here.

First, as developers make changes to source code in a version control system (VCS) during code commit, there is always a risk that a bad actor internally could make modifications to the source code or the controls that host them in a code repository. Some of these issues could be resolved with companies implementing code reviews whereby one or more developers analyze a teammate’s code to identify any poorly written code, bugs, or errors. Code changes can be submitted as pull requests (or merge requests), and reviews can be conducted before the changes are merged into the main codebase.

In addition to code reviews, many companies leverage a static application security testing (SAST) tool that continuously scans all new code and existing code for vulnerabilities. You can think of SAST tools, when used properly, as scaling secure code review across every pull request on every code repository.

Historically, companies used heavyweight, legacy SAST tools to try to “find every vulnerability,” leading to large amounts of false positives and heavy operational cost for security teams and/or development teams. Over the past few years, some modern, engineering-focused security teams have been leaning into the concept of “secure defaults” (also called “secure defaults”, or to use Netflix’s term, building a “paved road”). The idea is that the build tools, libraries, and processes make it easy for developers to do things the secure way, and hard to proceed insecurely, thus eliminating entire vulnerability classes from occurring in the first place. See Figma Head of Security Dev Akhawe’s Modern Static Analysis blog post, this Global AppSec SF talk, and this BSidesSF talk for more.

Securing source code management (SCM) systems

Source code leakage has become a major issue, such as the attack in 2022 where Microsoft had its source code leaked by the hacking group Lapsus$. The group alleges they released source code for Bing, Cortana, and other projects stolen from Microsoft's internal Azure DevOps server.

With source code management systems (SCM) like GitHub or GitLab acting as the central hub for the developer’s software development lifecycle, securing these SCMs has become paramount. Developers use these code repositories to store and organize their code while using version control systems like Git to manage and track changes to the source code. By compromising a code repository through unauthorized access, an attacker can gain access to the software’s source code, inject malicious code, hijack credentials, or even provision vulnerable infrastructure.

Managing access to code environments and IDE extensions

Integrated Development Environments (IDEs) and Visual Studio Code (VS Code) like Microsoft’s VS Code have become indispensable tools for developers today. They enable developers to customize their development environments and streamline various aspects of the development process. While IDEs and VS have become popular, developers may unknowingly install malicious extensions like an infected VSCode extension. Some of these extensions could be from open-source packages that rely on dependencies with third-party libraries. Risks could occur when these open-source tools gain access to code repos and compromise the security of the IDE, such as the ability to steal sensitive data or inject malware into a codebase.

The Build & Pipeline Layer

The build stage is responsible for pulling together various code components and transforming the source code that has been compiled into a deployable form, such as binaries, executable files, or code libraries. During this stage, all dependencies, configuration files, and assets are pulled into a distributable form for a target environment.

Across the build pipeline, development teams have to ensure they protect containers and the build automation tools like Apache Maven or GitLab CI/CD. These tools facilitate the building, testing, and deployment of applications. If the build environment is not secure or there are weaknesses in the build scripts, attackers can exploit vulnerabilities in build tools, servers, or the CI/CD pipeline itself to compromise the build process leading to future vulnerabilities in the final software.

Software supply chain attacks could happen during the build stage when the following are incorporated:

1. Dependencies (Unintentionally vulnerable, malicious, transitive and pipeline)

2. CI/CD pipeline

3. Containers and registries

Dependencies

Dependencies play a key role across most stages of SDLC, but primarily in the build pipeline stage since they help speed up development of new applications. Dependencies in the context of software supply chain refer to external software components, such as libraries, modules, packages or frameworks that a particular software application or project relies on to function properly.

Categories of third-party dependencies

Dependencies are third-party software ‘borrowed’ and downloaded through open-source tools to provide functionality for an application. At a high level, there are two ways third-party dependencies can introduce risk:

1. Unintentionally vulnerable code

2. Intentionally malicious code.

Though in both cases, the source of risk is third-party dependencies, how one detects and mitigates these risks is meaningfully different, thus, we’ll discuss them separately. We’ll also provide an in-depth discussion on the importance of SCA tools and their relationship to dependencies.

1) Unintentionally Vulnerable Code

It refers to code within a third-party dependency that contains security vulnerabilities or weaknesses that were not intentionally inserted by the developer of the dependency. These vulnerabilities are often the result of coding errors or poor development practices. They may include issues like buffer overflows, injection vulnerabilities, or incorrect data validation. The industry term for this class of tools is Software Composition Analysis (SCA). SCA plays an integral role in identifying vulnerable dependencies. In part 2 of our report, we provide an extensive discussion on the importance of SCA solutions and their role in preventing vulnerabilities within dependencies.

Transitive dependencies

Transitive dependencies are a form of dependency that could fall into the category of unintentionally vulnerable risk. These are dependencies that are indirectly required by a software project through its direct dependencies. In other words, when a project like JavaScript (Node.js) relies on a specific library or package like React (a direct dependency), that library or package may, in turn, depend on other libraries or packages (transitive dependencies) to function correctly. The two charts below illustrate the differences between direct vs. transitive dependency. The relationship between A and B is a direct dependency, but the reliance on C makes it transitive in nature. Hence, this complicates the risk profile but makes it easy for attackers to infiltrate harm going unnoticed.

In some cases, developers may download a library that have vulnerable or malicious transitive dependencies, but may be unaware of the risks because the direct libraries do not explicitly mention them in their configuration files. Some of the biggest complaints security advocates have about the state of SBOMs today is that they still struggle to communicate transitive dependencies related to the enumerated software, which we’ll discuss further in part 2 of this report.

2) Malicious Dependencies / Intentionally malicious code

This refers to code that has been intentionally inserted into a third-party dependency with the purpose of compromising the security or functionality of the software using that dependency.

This particular risk has become popular in recent years. This could be perpetrated by hackers with financial motives or nation-states who want to steal company IP, access national secrets, obtain personal information on many individuals (which could later be used for blackmail or espionage), or even preemptively gain the upper hand in future military conflict, for example, gaining privileged access to the electric grid or other industrial control systems.

Pipeline Dependencies

Attackers can use a variety of methods to inject malicious code into the code that defines the build process. These malicious items could be injected into CI scripts, or the build tooling configurations through 3rd party plugins, or tooling binaries. If this happens, the build process itself is compromised and used to distribute malicious code to downstream consumers.

CI/CD Pipeline

A CI/CD (Continuous Integration/Continuous Delivery) pipeline is an automated process that helps software development teams in the build stage to test, and deploy code changes efficiently and consistently. It automates these tasks ensuring that software is continuously updated and delivered to users with minimal manual intervention. CI/CD pipelines use processes called workflows to transform source code to deployable packages in production environments. Popular solutions include GitHub Actions workflows or GitLab Runner.

Although, the goal of CI/CD is to ensure that code changes are thoroughly tested and quickly released to users, developers often make costly mistakes that expose their company to vulnerabilities such as introducing vulnerabilities in 1st party code, third party vulnerability code, or have insecure build systems.

Attackers target CI/CD pipelines to introduce vulnerabilities or malicious code in workflows which can then be deployed to production environments. If build artifacts (compiled code, binaries, or packages) are not properly secured and validated, attackers could tamper with them during the deployment process. Also unauthorized access to CI/CD pipeline can lead to code tampering or unauthorized deployments.

NIST published some guidelines on protecting CI/CD:

1. Incorporate a range of defensive measures to ensure that attackers cannot tamper with software production processes or introduce malicious software updates (e.g., secure platform for build process).

2. Ensure the integrity of the CI/CD pipeline artifacts (e.g., repositories) and activities through role definitions and authorizations for actors.

Containers and Registries

Containers play a pivotal role in the software supply chain, particularly in modern application build and deployment workflows. Developers use them to build, test and package their applications in portable ways, so they can work across multiple environments. Developers have also recently faced challenges around many container images having no provenance information, making it difficult to verify where they came from or if someone has tampered with them. Well-intentioned developers sometimes install these packages and run images with known vulnerabilities.

Whether it’s a Docker image or Kubernetes, these solutions have become an indispensable core component of CI/CD pipelines especially for testing in various environments whether in staging or production. Container registries like Docker Hub or Amazon's Elastic Container Registry (ECR) which are centralized locations for storing, distributing and downloading container images have become a key focus for malicious actors. If attackers gain credentials for a registry, they can create and upload malicious container images to public or private registries. Unsuspecting developers may inadvertently use these images in their applications, leading to security breaches.

Developers who want to build containerized applications will generally start by using a base image like Ubuntu or Red Hat OpenShift and consume them from Docker Hub. Attackers can compromise these dependencies, leading to vulnerabilities in the containerized application. For example, as discussed earlier, it was discovered that over 1600 publicly available Docker Hub images were hiding malicious behavior including embedded secrets that could be used as backdoors for attackers.

Finally, it’s important to note that if teams don’t leverage reproducible and deterministic builds that focus on ensuring the process of building software from source code results in identical binary outputs each time, developers may be vulnerable to supply chain attacks. Without using this build protocol, teams run the risk of having malicious actors inject vulnerabilities that lead to misleading binaries. By ensuring that the resulting binaries are consistent and verifiable, these practices enhance trust so the software can be independently verified by third parties to match the claimed source code.

The Packaging & Deployment Layer

The final stages of software development involves bundling the software and its dependencies into a distributable package for deployment into production environments. Alternatively, most cloud-native companies build applications into a container that they then run on some sort of container orchestration platform, like Kubernetes.

Software is packaged into distributable artifacts, which can be in the form of installer packages like Caphyon, container images, or other formats. so it can be easily distributed, deployed, and installed on different systems. At this stage, it’s critical that development teams are able to implement tools that give them clear transparency and integrity into tracing every line of code. This is why we’ve focused on discussing SBOMs and code provenance in this section.

Software supply chain attacks may occur when an attacker tries to tamper with the packaging process such as a yarn package manager to insert malicious code or alter package metadata, leading to compromised software distributions. There are also possible attacks on package mirrors that are running on popular package repos. For example, as part of the CodeCov attacks, leaked credentials were used to upload malicious artifacts to a GCS bucket.

The package and deploy stage covers the following:

1. Software Bill of Materials (SBOM)

2. Code provenance and signatures

3. Artifact repository

Software Bill of Materials (SBOM)

Closer to the packaging and deployment stage, teams may use SBOMs to filter for bad dependencies and vulnerabilities either from the source code to the build stage. An SBOM is a machine-readable inventory of software components, build tools and dependencies derived from the use of open source and third-party libraries.

A common analogy is reviewing the list of ingredients on food packaging or determining what is inside a burrito package. An SBOM is meant to provide details on the different components included within a software product: including name, version, license, and vulnerabilities, etc.

SBOM evolved as a result of licensing and infringements against people who would develop proprietary software leveraging open-source, but would not give credits to creators of dependencies they leveraged when building their software. The OWASP CycloneDX and SPDX license list, were developed to create open standardization for SBOMs. However, in 2021 the Biden executive order on the formalization of software supply-chain standards played a key role in the adoption and standards for SBOMs.

SBOM enrichment and aggregation: In recent years, some of the complaints against SBOM include the lack of robustness especially when it comes to keeping up with newer open-source dependencies. Some other issues include how to aggregate and enrich these SBOM artifacts. Lastly, aggregating SBOM information across software portfolios and lines of business will be a growing concern for security leaders. Hence, some organizations create their own SBOMs together with ingesting SBOMs from their vendors. Some organizations add vulnerability exploitability exchange (VEX), a guide that allows developers to know if certain software products are affected by a known vulnerability supplier. SBOM providers will generally integrate this into their solution as it serves as a companion. SBOMs only provide you with a list of components within software, but might not detect ‘known’ vulnerable risks which are exploitable.

Alternatively, OpenSSF Scorecard, an automated scanning tool can perform a number of checks on the security posture of an open source project (e.g. is it using security tools like SAST or fuzzing, does it follow build pipeline best practices like requiring code reviews and uses branch protection, etc.) for contextualizing and enriching SBOMs.

Code provenance and signatures

Without knowing the origin of code components, it's challenging to verify the integrity of software. Code from unknown sources may have poor quality, lack documentation, or vulnerabilities leading to software defects and reliability issues. This is where code provenance plays a critical role.

Code provenance allows developers to trace the origin and history of a piece of code or software component, including its authorship, modifications, and any intermediaries it passed through during its development lifecycle. Essentially, it's about knowing where your code comes from and understanding its journey from inception to deployment. In the context of the software supply chain, code provenance is crucial for security, traceability and verifying the integrity of code. It helps ensure that the code you incorporate into your software is trustworthy and hasn't been tampered with or compromised by malicious actors. Heavily regulated industries like financial or healthcare sectors have to adopt these software for compliance requirements.

Code signatures and signing are increasingly becoming the best practice for ensuring the integrity of code. As developers commit and deploy software over its lifecycle, code signing certificates are a favored target for software supply chain attackers. Solutions like Sigstore sign & verification have become standards for ensuring supply chain integrity both across developers and users. There are code-signing vendors such as Garantir, CircleCI, GitHub Cosign, and Venafi.

Artifact repository

Artifacts in software development refer to the output of a build process, such as compiled code, libraries, or executable files. An artifacts repository, like Jfrog is a centralized storage location that securely stores and manages these artifacts. Their role is to provide a reliable and organized way to store, version, and distribute software components, making it easier for developers to collaborate, build, and deploy software.

In the software supply chain, artifacts and artifacts Repositories play a crucial role in version control, dependency management, and in ensuring builds are reproducible. They ensure that software components are consistently built, tested, and delivered to ensure a streamlined deployment process.

Software supply chain attacks generally occur if an artifact repo is not properly secured and unauthorized users gain access to tamper with artifacts. Attackers might inject malware into artifacts, especially if the repository lacks proper scanning and validation mechanisms. If artifacts are not treated as immutable (unchangeable), it's possible for them to be altered after they are stored in the repository.

Part 2 of the SSC Overview: Vendor Analysis

Many of the software supply chain vendors look similar on their websites or how people view them. However, each of them are approaching the SSC problem from a different angle or core competency.

We have analyzed over 12+ software supply chain vendors. In Part 2 of our SSC report, we provide a detailed analysis of each of the key software supply chain vendors.

To make sure you see Part 2 when it’s published, make sure to subscribe to tl;dr sec, a free weekly update of the best tools, blog posts, and talks in cybersecurity across AppSec, cloud security, AI, supply chain security, and more!