In the previous blog post, I introduced Fosstars, an open source Java-based framework, that allows you to create ratings for open source projects. Such ratings help to assess various properties of open source projects such as activity, maintenance and so on. Security often becomes one of the properties that are quite difficult to assess unless one has significant background in this field. To address this, Fosstars offers a security rating for open source projects. In this blog post, you will learn how it works.
Open source components can extend an application’s functionality, save time and reduce costs. Unfortunately, they can also introduce security risks. Like any other software, open source components may have vulnerabilities. It is quite sad but should not be surprising. What makes the difference is that open source projects can take different steps for addressing and preventing vulnerabilities. One project can use multiple tools for detecting security issues early, it can have guidelines for reporting security problems, and apply other best practices. Another project can ignore these measures. As a result, the second project brings more security risk than the first one. How can we identify projects that don’t care about security enough? This is where Fosstars security ratings comes into play.
The open source security rating assesses how well an open source project takes care of security. It helps to answer the following question: How is it safe to use a particular open source component in the long term?
The rating procedure takes into account multiple factors:
While calculating a security rating for an open source project, the above factors get assessed separately by dedicated scoring functions. Each scoring function outputs a sub-score. Then, a top level scoring function takes all the sub-scores and calculates an overall security score. Not all sub scores contribute equally to the overall security score. The top-level scoring function weights the input sub-score which makes some of them more important than others. For example, the score for security testing is a bit more important than the scores for activity and popularity, therefore the security testing score has bigger impact on the overall security score.
The rating procedure outputs one of the following labels:
The “Unclear” label is usually returned when the rating procedure didn’t get enough data about the project. Let’s have a look at the data that are used for calculating security ratings and how the data can be gathered.
Fosstars gathers various publicly available information about an open source project in order to calculate a security rating. There are several main groups of data.
The first group contains data about security testing in the project:
Whether the project uses:
The second group describes security practices that are used in the project:
The third group contains data that describes the project’s community:
The last group contains other useful information about the project:
Fosstars gathers most information about open source project in an automated way without involving a user.
An open source project receives one of the labels (Good, Moderate or Bad) depending on a calculated security score. If the score is above a certain threshold X, then the project gets the label “Good”. If the score is below X but still is above Y, then the project gets the label “Moderate”. Otherwise, the project receives the label “Bad”. Let’s see how X and Y can be selected.
On the one hand, the thresholds X and Y can be set by a security expert. That looks like the simplest way. The disadvantage of this approach is that the thresholds are completely based on one individual’s opinion. That introduces bias. Having multiple experts may reduce the bias.
Fosstars takes another way. The thresholds X and Y depends on security scores for multiple well-known open source projects:
The thresholds are updated on every Fosstars release. Here is how it is done:
The main goal of this method is to reduce potential bias that may be introduced by those who set the thresholds. Instead of setting the thresholds directly, the experts provide a list of well-known projects and specify a desired fraction for the labels. Currently, the fraction is 20/50/30 which looks like a normal distribution.
As a result, the thresholds don't set a bar that is totally based on someone's opinion. Instead, open source projects are compared with real, well-known and trusted ones.
If you have an open source project that stays on GitHub, you can use Fosstars GitHub action to calculate a security rating for it.
There is also a report for well-known open source projects such as ones that belong to Apache and Eclipse software foundations. For example, here is a report for curl.
To calculate security ratings for other open source projects, you can use a command-line tool. Currently, there is no installer for this tool, but you can build it from the sources:
Once it is built, you can calculate a security rating for an open source project by specifying a URL to its code repository, for example:
The environment variable “TOKEN” contains a token for accessing the GitHub API. You can create a personal token in the settings/tokens page in your profile on GitHub. It is okay to run the tool without the token, but the tool won’t be able to gather all of the information about the project.
The tool will attempt to gather information about the project. For example, it will download the source code, make several requests to GitHub and other services, fetch data from NVD and so on. Then, it prints out an overall security score, a label and sub-scores. Here is what it looks like:
If you set “--verbose” option, the tool will print out more details to help you understand how the rating was calculated. In the end, it will print out a list of recommendations for improving the rating.
Fosstars security rating helps to assess how safe it is to use a particular open source component long term. Moreover, the assessment is completely automated. You can already try it out. Fosstars offers a command-line tool and a GitHub action for calculating security ratings for projects on GitHub. Or you can check out a report for some popular open source projects. Feel free to reach out to us if you have any questions. Feedback is also much appreciated!
Why do we need security ratings?
Open source components can extend an application’s functionality, save time and reduce costs. Unfortunately, they can also introduce security risks. Like any other software, open source components may have vulnerabilities. It is quite sad but should not be surprising. What makes the difference is that open source projects can take different steps for addressing and preventing vulnerabilities. One project can use multiple tools for detecting security issues early, it can have guidelines for reporting security problems, and apply other best practices. Another project can ignore these measures. As a result, the second project brings more security risk than the first one. How can we identify projects that don’t care about security enough? This is where Fosstars security ratings comes into play.
What the security rating looks like
The open source security rating assesses how well an open source project takes care of security. It helps to answer the following question: How is it safe to use a particular open source component in the long term?
The rating procedure takes into account multiple factors:
- How well the open source project implements security testing.
- How well the project’s community is aware of best security practices and adopts them.
- How well vulnerabilities are patched in the project.
- How the community commits to supporting the project.
- How active and popular the project is.
While calculating a security rating for an open source project, the above factors get assessed separately by dedicated scoring functions. Each scoring function outputs a sub-score. Then, a top level scoring function takes all the sub-scores and calculates an overall security score. Not all sub scores contribute equally to the overall security score. The top-level scoring function weights the input sub-score which makes some of them more important than others. For example, the score for security testing is a bit more important than the scores for activity and popularity, therefore the security testing score has bigger impact on the overall security score.
The rating procedure outputs one of the following labels:
- “Good” means that the project takes care of security well enough.
- “Moderate” means that the project takes care of security relatively well.
- “Bad” means that the project doesn’t take care of security well enough.
- “Unclear” means that it was not possible to reliably assess how well the project takes care of security.
The “Unclear” label is usually returned when the rating procedure didn’t get enough data about the project. Let’s have a look at the data that are used for calculating security ratings and how the data can be gathered.
Calculating security ratings with Fosstars
Data for calculating security ratings
Fosstars gathers various publicly available information about an open source project in order to calculate a security rating. There are several main groups of data.
The first group contains data about security testing in the project:
- CodeQL scans
- CodeQL checks on pull requests
- Whether the project has a CVSS threshold for OWASP Dependency Check to fail the build
- The worst LGTM grade for the project
Whether the project uses:
- LGTM checks on pull requests
- FindSecBugs
- Dependabot
- OWASP Dependency Check
- AddressSanitizer
- MemorySanitizer
- UndefinedBehaviorSanitizer
- OWASP Enterprise Security API (ESAPI)
- OWASP Java Encoder
- OWASP Java HTML Sanitizer
- nohttp tool
- OSS-Fuzz
The second group describes security practices that are used in the project:
- Whether the project has a security policy
- Whether the project has a security team
- Whether the project has a bug bounty program
- Whether the project signs artifacts
- Whether the project uses signed commits
The third group contains data that describes the project’s community:
- Whether the project belongs to Apache Software Foundation
- Whether the project belongs to Eclipse Foundation
- Whether the project is supported by a company
- Number of commits in the last three months
- Number of contributors in the last three months
- Number of stars in the GitHub repository
- Number of watchers in the GitHub repository
The last group contains other useful information about the project:
- Information about vulnerabilities in the project
- Whether the project uses GitHub for development
- Programming languages used in the project
- Package managers used in the project
Fosstars gathers most information about open source project in an automated way without involving a user.
How are the label thresholds set?
An open source project receives one of the labels (Good, Moderate or Bad) depending on a calculated security score. If the score is above a certain threshold X, then the project gets the label “Good”. If the score is below X but still is above Y, then the project gets the label “Moderate”. Otherwise, the project receives the label “Bad”. Let’s see how X and Y can be selected.
On the one hand, the thresholds X and Y can be set by a security expert. That looks like the simplest way. The disadvantage of this approach is that the thresholds are completely based on one individual’s opinion. That introduces bias. Having multiple experts may reduce the bias.
Fosstars takes another way. The thresholds X and Y depends on security scores for multiple well-known open source projects:
- Popular projects in Apache and Eclipse foundations
- Popular Spring projects
- Popular FasterXML projects
- Several other well-known projects such as curl, Netty, OpenSSL and so on.
The thresholds are updated on every Fosstars release. Here is how it is done:
- First, Fosstars calculates security scores for the projects mentioned above and sorts them from low to high.
- Then, it assigns the label “BAD” to first 30% of the projects with the lowest scores. The highest score in this set becomes the threshold Y for "Moderate”.
- Next, it assigns "Moderate” to the next 50% of the projects. The highest score in this set becomes the threshold X for “Good”.
- Finally, the rest 20% of the projects get "Good”.
The main goal of this method is to reduce potential bias that may be introduced by those who set the thresholds. Instead of setting the thresholds directly, the experts provide a list of well-known projects and specify a desired fraction for the labels. Currently, the fraction is 20/50/30 which looks like a normal distribution.
As a result, the thresholds don't set a bar that is totally based on someone's opinion. Instead, open source projects are compared with real, well-known and trusted ones.
How to get security ratings for open source projects
If you have an open source project that stays on GitHub, you can use Fosstars GitHub action to calculate a security rating for it.
There is also a report for well-known open source projects such as ones that belong to Apache and Eclipse software foundations. For example, here is a report for curl.
To calculate security ratings for other open source projects, you can use a command-line tool. Currently, there is no installer for this tool, but you can build it from the sources:
git clone https://github.com/SAP/fosstars-rating-core
cd fosstars-rating-core
mvn package -DskipTestsOnce it is built, you can calculate a security rating for an open source project by specifying a URL to its code repository, for example:
java -jar target/fosstars-github-rating-calc.jar --url https://github.com/curl/curl --token $TOKEN The environment variable “TOKEN” contains a token for accessing the GitHub API. You can create a personal token in the settings/tokens page in your profile on GitHub. It is okay to run the tool without the token, but the tool won’t be able to gather all of the information about the project.
The tool will attempt to gather information about the project. For example, it will download the source code, make several requests to GitHub and other services, fetch data from NVD and so on. Then, it prints out an overall security score, a label and sub-scores. Here is what it looks like:
Calculating a security rating with Fosstars CLI (GIF created with Terminalizer)
If you set “--verbose” option, the tool will print out more details to help you understand how the rating was calculated. In the end, it will print out a list of recommendations for improving the rating.
Conclusion
Fosstars security rating helps to assess how safe it is to use a particular open source component long term. Moreover, the assessment is completely automated. You can already try it out. Fosstars offers a command-line tool and a GitHub action for calculating security ratings for projects on GitHub. Or you can check out a report for some popular open source projects. Feel free to reach out to us if you have any questions. Feedback is also much appreciated!
- SAP Managed Tags:
