The severity made it a high priority for us to investigate and get the word out to our customers. With commercial software, we often become aware of the problem only after a bug fix is already released. However, in this case, we got a chance to prepare in advance. We became aware of the OpenSSL advisory on March 24 and immediately started our work.
How does Arctic Security prepare its partners and customers to manage this kind of vulnerability disclosure? In practice, it is done using our Arctic Hub product and our Early Warning Service (EWS) data feed. In this article, we will open up our process, and explain how we go about getting information to our partners to help their clients.
A vulnerability in OpenSSL is particularly interesting because the library is in constant use. The initial note from the OpenSSL project didn’t provide lower bounds on which versions are affected, so we concluded that the notification does not exclude any versions before 1.1.1k as secure. OpenSSL doesn’t support these older versions anymore, so the affected parties’ solution is to upgrade to the latest OpenSSL. Alternatively, the security fix may be backported to their version by another maintainer. Since it may affect almost all versions of the library, this vulnerability is classified as a high-severity problem.
Finding the scope of the problem
One of the first steps for our analysts is knowing the scope of the problem. In our initial investigation using the Shodan search engine, we identified 3.7 million installations with OpenSSL versions 1.1.1j or older out there. That number would have potentially made it one of the most widespread vulnerabilities to date.
We then created a customized data feed in our Arctic Hub processing platform that started collecting all the relevant information from Shodan and preprocessing the data. Our platform automatically normalizes the data and correlates it with network owners and then reports all affected OpenSSL installations to our customers.
The first notifications from Arctic Security about the vulnerable OpenSSL went out by March 24. Our partners could prepare their systems to be patched and understand how many of their clients were affected. They could then inform their customers about the urgency to update in advance. That timeline puts them ahead of the vast majority of hackers seeking to exploit the vulnerabilities.
On March 25, the fixed OpenSSL version 1.1.1k was published, and we could see how severe the problem was and who was affected. Luckily it was not a catastrophic vulnerability like Heartbleed, but the two vulnerabilities fixed were severe and potentially enable a man-in-the-middle and denial-of-service (DDoS) attacks. Both of them were severe enough to merit proceeding with immediate updates for the affected systems.
Knowing the vulnerable versions also helps us to narrow down the list of affected systems. After the bug-fix release, we now know that it affects 1.1.1 versions of the OpenSSL library. In our investigation using the Shodan search engine, we identified 770,000 installations with versions 1.1.1 out there affected by CVE-2021-3449. Out of those, 162,000 are versions 1.1.1i or newer that are also affected by the x509 related second vulnerability with CVE-2021-3450.
Finally, after we have the complete information, we can refine the vulnerable service notifications and restrict them to the appropriate versions of OpenSSL in the new EWS data feed. Unfortunately, OpenSSL doesn’t provide analysis on whether the vulnerability affects 1.1.0 versions of the library, so we can’t say yet if those are affected until the security community verifies this.
Getting the word out
These vulnerabilities likely lead to further exploits down the line once malicious actors have time to productize them. We know from experience that Heartbleed vulnerable OpenSSL versions from 2014 still globally number in the tens of thousands, so this is the beginning of a long trek for us to get everyone to update.
This type of vulnerability information that has been matched to the affected asset is not something that the average IT administrator hears about promptly. The OpenSSL project is open source and does not promote the urgent need to update to the end-users. Responsibility is on software developers who use their library as part of a larger project—be that an operating system or a specific software implementation—to update the vulnerable library.
The fact that many Linux distributions backport the security patches without changing the visible version number makes it difficult for systems administrators to know whether their system is secure.
“Shodan by Arctic Security” is a data feed that we developed for promptly bringing knowledge of these vulnerabilities known by Shodan to the victims having the affected libraries. It is part of our EWS security notification service and enables you to help your clients before they get compromised. With this service, we bring essential vulnerability information to our partners and customers on time.
The OpenSSL vulnerability notifications have been part of the EWS data feed from Arctic Security since March 24. It is an excellent opportunity for proactive work with your customers having impacted versions currently in operation.