The SolarWinds case is no different, as, in the heat of the moment, everybody is focused on attributing malice and all things related. My position is that in addition, we should also pay attention to what may have led into this situation, which oftentimes simply is poor security posture due to a failure in people, processes or technology.
Even more precisely, the crux of the problem often lies in poor IT processes related to patching, access control or other “boring stuff”, such as keeping X.509 certificates from expiring.
How dare you say that out loud?
Simply, because for more than a decade I’ve been helping people in their mission to reduce incidence of suspected compromise, publicly exposed vulnerable services or open services which may be unintentionally exposed to the internet, both at a national scale and through automation.
How is this related to SolarWinds?
SolarWinds is a company whose mission is to make IT administration simpler and more secure at scale. This is a great mission and I fault the company not with what they have built or how secure their products are. The challenge seems to be in how well they have their own people, processes and technology in line with what they enable their customers to achieve through their products.
Can you show me the proof?
Yes, as the proof is out in the open for everyone to see. Look at their publicly available network assets and connect the dots between the assets and known vulnerabilities observed through Shodan. I did this with the help of our research into 86 specific known and vetted issues on Shodan and matched the data against the SolarWinds assets and the picture is quite clear and unflattering.
Why are you singling them out?
Their case is real and well-known and if a tragedy on such a scale can contribute to the common good, then we should seize the opportunity. On January 1, 2021, we reached out to SolarWinds to confirm that the network assets we used for the evaluation actually belong to them. We even offered them the findings to help them mitigate them. Unfortunately, they did not respond to our requests, and we are now publishing our results in this blog post.
Given the constraints I lay out below, they clearly have not done their due diligence and as a result we have this mess, where threat actors must have had a field day in gaining initial access to the company—no zero days needed.
I implore you to reflect that in addition to threat signatures and TTPs about the malice, we need to start looking at the security posture of organizations as well, and not just after a high-profile breach such as this one. The information I present below is public and most likely used by threat actors every single day.
Furthermore, the vulnerabilities I detail below, plague most organizations on the internet. Picking three prominent US defense contractors at random, I found most of the same issues related to their network assets as well, namely General Dynamics, Harris Corporation or Northrop Grumman.
So what’s the evidence?
As stated above, I used the 86 vetted queries from Shodan against the publicly available network asset information related to SolarWinds. In essence, I used the same approach as the cyber rating companies do, which means that they rely in large part on publicly available IP network registration information when producing ratings.
To be reproducible, the uncertainties related to the interpretation of the results must at least account for the following four things:
- That the network asset in question belongs to the right party, which in this case study should be SolarWinds
- That the service in question may be a honeypot, whose intention is to monitor for scanning and/or exploitation of the emulated service
- That the service implementation-based version matching suffers from patch backporting, which means that the detection or assessment method cannot solely rely on a simple version banner
- That the party doing the interpretation has to be able to understand the impact, as just having all the Shodan matches against a network asset alone makes interpretation difficult in practice
What are the observations?
In this case, I used evidence from the past six months mapped against the publicly available network asset attributable to SolarWinds. Mapping the 86 topics against those assets with automation, I was able to detect seven distinct vulnerabilities in their infrastructure, which are as follows:
Expired X.509
If you have ever done systems administration with any type of internet-facing services, you have battled against time and expiration of X.509 certificates, which prove the identity of your service to its users. You’re probably thinking: we should be talking about vulnerabilities, about CVEs, right? Remote code execution, the sexy stuff. My favorite retort to that is the following: the Equifax data breach which exposed the PII of 147 million people was largely caused by expired X.509 certificates in critical monitoring systems.
shodan dork: ssl.cert.expired:true
SSLv2
The arms race against crypt analysis has accelerated over the past couple of years. This means that cryptographic hashes or ciphers that were perfectly acceptable five years ago have become obsolete. SSL version 2 is a cryptographic protocol, which hasn’t met the requirement for internet-facing services for even longer. Its use was prohibited by RFC 6167 in 2011.
shodan dork: ssl.version:sslv2
Reference: https://tools.ietf.org/html/rfc6176
Exposed SNMP
Simple Network Management Protocol is very handy for gathering low-level information about networked devices and their status, especially in the switch and router space. As demonstrated by the OUSPG PROTOS project in 2002, SNMP, whatever the version, is not and will not be a protocol fit for the internet. Access to SNMP implementations must be access controlled to those devices, which are under your control, no matter which version of the protocol the device is speaking. Google “c06-snmp test suite” and assess the ramifications yourself, if you need more background on the issue. Moreover, threat actors have abused SNMP for reflected DDoS attacks for years, since sending a single spoofed UDP packet with the source address of the target will yield a huge amplification factor to their DDoS traffic.
shodan dork: port:161
shodan dork: port:161 snmp
Reference: https://en.wikipedia.org/wiki/Oulu_University_Secure_Programming_Group
Exposed Telnet
Before the advent of SSH, telnet was the de facto protocol for remote management or shell access to systems or devices. In 1995, a Finnish engineer called Tatu Ylönen invented SSH, which made telnet obsolete overnight. 25 years later, however, telnet is still going strong for some reason. There is simply no internet-facing use case for the service, period.
shodan dork: port:23
shodan dork: telnet
ref: https://en.wikipedia.org/wiki/Telnet
Exposed MySQL
Even if the MySQL database engine is the de facto component as a backend service for many an internet-facing service, exposing it directly to the internet without any further access control in front of it is inviting disaster. Recently, there was a major data breach in Finland, which allegedly resulted from the database backend being directly exposed to the internet without any further access control in front of it. Layered security is not just a fancy buzzword. In many cases the database backend doesn’t even need to be reachable over the network at all, but when it does, access to it must be locked down to only those hosts, which do actually need it.
shodan dork: product:”MySQL”
ref: https://vpsie.com/knowledge-base/securing-mysql-database-shared-hosting-environment/
CVE-2015-0204 a.k.a. FREAK
Compliance with US cryptographic export controls lies at the heart of this vulnerability. An internet-facing service simply should not support negotiation of an encrypted connection, which is possible to decipher using $100 worth of cloud computing resources. We could even rename this vulnerability ‘man-in-the-middle made easy’.
shodan dork: vuln:CVE-2015-0204
ref: https://en.wikipedia.org/wiki/FREAK
ref: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-0204
CVE-2019-10149
Exim is a popular Mail Transfer Agent (MTA) used in many (mostly) Linux-based servers. It has had a mottled history with information security and, in 2019, a serious flaw was discovered in it, which leads to remote code execution. Please note that using this dork especially, one must understand the implication of backporting security fixes and how they are reflected in the version banner.
ref: https://nvd.nist.gov/vuln/detail/CVE-2019-10149
Conclusions
I know maintaining and repairing “bicycles” is tedious. This work, however, is the thing that will keep you safer and make it more difficult for the threat actors to attack you. I’m not saying that the threat actor’s initial access in the case of SolarWinds could have been prevented by addressing the specific issues I highlighted above, but it would have made it more difficult.
Building security in and making it layered is the only foundation you can build on that gives you predictability in a much more coherent way than investing in the magic security technology X that exploits the hype of the day. Building security in means investing in people and processes as well.
One of the big challenges is that this type of information security work is not viewed as glorious or sought after. It is often given to junior people, and the career paths of the sysadmins are not made lucrative. Investing in good people will lead to them making good technology choices and a good security culture can only come from balancing usability and controls.
Understanding that you will be breached given enough time spent on the internet is the only attitude you can have. That is why using external third party information looking at your assets from the attacker’s perspective is one of the methods that clearly has not been adopted at scale. Shodan is a good service that can be used properly and can help you address the basics. Just subscribing to their notifications, however, will not suffice. You may need help to distill it into actionables, but don’t leave that work to the bad guys. All that information is out there, it is in the open and it can help you better protect yourself.
