Security Scorecards for Open Source Software

Open source software (OSS) adoption has accelerated in recent years. OSS is now pervasive across large enterprises, with organisations keen to avoid the software lock-in brought about by closed-source, proprietary code as far as possible.

The security of OSS projects, however, has not kept pace with adoption. From vulnerabilities like Heartbleed in critical software like OpenSSL, to dependency confusion attacks, and the more recent supply chain attacks against SolarWinds and Codecov, weaknesses in OSS security have shone a bright light on our collective blind spot. Open source project maintainers often support the projects on a voluntary basis, and do not have the resources required for full scale software security audits.

With the increase in software supply chain attacks, there is a growing awareness and need within enterprises to review the security health of OSS packages before integrating them into applications. There is a cold-start problem though. OSS projects don't necessarily follow a common set of best practices, and there are no ready metrics to evaluate the security health of these projects. This has led to a bunch of organisations like GitHub, Google, IBM, Microsoft, OWASP, Red Hat and others to band together and form the OpenSSF.

What is OpenSSF Scorecards?

Hosted by The Linux Foundation, The Open Source Security Foundation (OpenSSF) is a cross-industry initiative to improve OSS security, with a focus on metrics, tooling, best practices, developer identity validation and vulnerability disclosures best practices. OpenSSF kickstarted their foray into software supply chain security with Scorecards, an initiative to auto-generate a security score for open source projects. From their GitHub page:

Scorecards is an automated tool that assesses a number of important heuristics ("checks") associated with software security and assigns each check a score of 0-10. You can use these scores to understand specific areas to improve in order to strengthen the security posture of your project. You can also assess the risks that dependencies introduce, and make informed decisions about accepting these risks, evaluating alternative solutions, or working with the maintainers to make improvements.

Using Scorecards, you can ascertain whether OSS projects adhere to minimum security defaults e.g. Is branch protection used? Are fuzzing tools used? Is code review required? Are releases cryptographically signed? Are there unfixed vulnerabilities? And so on. Here are the currently available checks.

The Scorecards team runs weekly scans on critical open source projects and publishes the results in the Google BigQuery public dataset openssf:scorecardcron.scorecard-v2, with the latest results available in the view openssf:scorecardcron.scorecard-v2_latest. The scan results are also available on the OpenSSF metrics dashboard.

Scan GitHub Repositories against Scorecards

At the time of writing, Scorecards only supports GitHub repositories. For repositories that you own, the easiest way to use Scorecards is with the native GitHub Action. See detailed installation instructions here.

To run a scan on repositories you do not own, install the Scorecards package first. I'm going to use a DigitalOcean droplet for this post - if you don't have an account, sign up here (if you sign up with my link, you’ll receive a $100, 60-day credit when you add a valid payment method to your account). Next, create a droplet - choose a plan (4GB RAM / 2 CPU), an image (Ubuntu 20.04), the data center region, an authentication option (root password for now, but non-root user in a real environment) and the hostname.

Before you install Scorecards, you must create a GitHub personal access token or a GitHub App to work around GitHub API rate limits on unauthenticated requests. Once you have done so, install the Scorecards docker container and run the scan.

# Update package metadata
apt update

# Option 1: For GitHub personal access token,
export GITHUB_AUTH_TOKEN=<your_access_token>

# Option 2: For GitHub App and key file,
GITHUB_APP_KEY_PATH=<app_key_file_path>
GITHUB_APP_INSTALLATION_ID=<app_installation_id>
GITHUB_APP_ID=<app_id>

# Install docker
apt install docker.io

# Install and run Scorecards against the code repository
docker run -e GITHUB_AUTH_TOKEN=<your_access_token> gcr.io/openssf/scorecard:stable --show-details --repo=https://github.com/ossf/scorecard

Here is a sample output (albeit truncated) from the scan results. You can change the default output format to json or csv using the --format flag. You can also get more details about a check failure using the --show-details option.

Each check returns a score between 0-10, with 10 being the best possible score. Scorecards produce an aggregated score too, which is a risk-weighted average of the individual check scores.

• Critical risk checks are weighted at 10
• High risk checks are weighted at 7.5
• Medium risk checks are weighted at 5, and
• Low risk checks are weighted at 2.5

A final note - scorecards are not meant to be an end onto themselves. Rather, they should be integrated into your CI/CD pipelines, with the score determining whether the integration moves forward or gets terminated for a manual/detailed risk analysis into the vulnerable packages and checks. Scorecards are only being used by a handful of projects today but, with more widespread adoption, we can hope to see a future where software supply chain attacks are no longer the norm.