“Life can only be understood backwards, but it must be lived forwards.”
- Soren Kierkegaard
Being successful requires being proactive and not waiting for life to come to you. It means you must be on offense, not defense. The December 2020 SolarWinds supply chain (Sunburst) hacking put thousands of organizations in a defensive and reactive mode, and billions of dollars and man hours have been lost in the aftermath.
You would think we would have been more prepared considering computer viruses have been around for decades. There are 980 million computer viruses in existence and 350,000 new viruses detected each day. There were 7 billion virus attacks in 2018 alone and viruses distributed through encrypted protocols increased 58% from 2018 to 2019.
However, Sunburst was new in that it was the “first” source code type of injection targeted at enterprises, but it will not be the last because of its success. In fact, ethical hacker Alex Birsan developed a similar attack to inject malicious code into open source developer tools. He penetrated Microsoft, Apple, PayPal and 32 other companies. He demonstrated how compromising Microsoft and Apple development tools could compromise their code and, in turn, numerous customers.
It is given that we will start seeing more instances of similar attacks in the future. Therefore, instead of reacting to what life throws at us, we should take a proactive approach, so we can anticipate and respond better when attacks happen.
According to Steven Covey, a proactive approach has three essentials elements:
- Acknowledge the exceptional situation (mistake).
- Take a corrective set of actions.
- Learn from the exceptional circumstances.
This white paper follows a similar approach for addressing the SolarWinds attack: understand what happened, identify potential gaps and risks and learn what can be done to prevent this type of attack in the future. We will focus on a defense in depth approach available to IBM Sterling B2B Collaboration customers to help mitigate risks even when IT systems are compromised by hackers.
What happened and how
The SolarWinds® Orion® Platform is a powerful, scalable infrastructure monitoring and management platform designed to simplify IT administration for on-premises, hybrid, and software as a service (SaaS) environment in a single pane of glass.
Beginning in March 2020, SolarWinds unwittingly sent out software updates to its customers that included the hacked code. The code was inserted into the software build (patch or updated version) by hackers. As the software update was released it created a backdoor to customers’ information technology systems, which hackers then used to install even more malware that helped them spy on companies and organizations.
Hackers, who appear to be associated with the nation-state sponsored hacking group Cozy Bear, were responsible for creating and spreading this virus.
This unique and novel cyberattack inserted a vulnerability (Sunburst) within the Orion Platform for versions 2019.4 HF 5 and 2020.2 HF1, by taking advantage of an Auto-Update Process “Phone Home.” When Orion Network’s customers and users downloaded a patch via that process, it turned out to include a virus named Businesslayer.dll.
Once downloaded, the virus was then sent to other systems that SolarWinds was monitoring via Network Performance Monitor (NPM),a client application installed on every server that is monitored. Once installed, the virus sits and observes the network for about two weeks. It learns if it is isolated in a sandbox, and it also learns what antivirus software the customer is using. Depending on the company’s antivirus, the virus will act differently. If it detects that it is in a sandbox environment it will not run.
The virus then creates a random DNS to see if it resolves. If so, it then sends an HTTP REST call over the Orion Improvement Program (OIP) protocol.
Once the Orion program responds to the REST call, the virus embeds commands in the response that it wants the server to execute. This can include transferring files, executing files, building a profile of the system, rebooting the machine, disabling system services, and executing jobs.
The damage begins
The virus starts to harvest credentials from the local machine. Then it enumerates users to see if they have admin permission or privileged credentials for that machine. It will eventually gain access to the global admin and even generate its own SAML token and sign certificates.
This is the key milestone that allows further spread internally. The virus uses the administrator account or SAML tokens to impersonate any of the organization’s existing users or privileged accounts. It then uses SSH/SFTP credentials to access on-premises systems and connect to trading partners and CLOUD resources (Amazon, Azure, Google).
Unfortunately, at this point, the virus knows how to use the right permissions for the right purpose. That is where the virus could start building credentials for Single-Sign-On (SSO) and build a Service Principal Name. It also capable of identifying the type of server it has accessed. For instance, if the server can transfer funds, it can initiate a funds transfer.
American businesses and government agencies have already spent hundreds of billions of dollars to remediate this risk and will continue to allocate resources to fix the damage from this nation-sponsored hack against the SolarWinds Orion Network.
Where the gaps and risks are
This type of virus attacks in depth. First, it was downloaded as a patch and attacked NPM servers. Once on those NPM clients, it then used customers’ credentials to crack nearby servers. It then spread the Businesslayer.dll to every server that it could. Because of this, we need a defense in depth approach to mitigate the risk.
Customers must start at the DMZ and provide a reverse proxy with a session break in the DMZ. It must also do virus scanning in the DMZ without letting data hit disk. Second, we must use a B2B Gateway to consolidate file transfers and route them appropriately. Third, we must not use out-of-the-box protocols that can be exploited by viruses. Instead, we must use a Whitelist approach for internal file transfers. Fourth, we must be willing to change credentials (even internal ones) more frequently.
As is evident, having an auto-update with no oversight and zero control, can have a trickle-down impact on the security of any network, and spread chaos and damage massively in a very short time. Auto-updates achieve performance efficiencies and lower costs, among other benefits. Nonetheless, there must be a balance between productivity and vetting a process with security scrutiny and/or have a maximum control in place, especially when such workflow is executed on autopilot.
There are also major risks when files, either documents or system patches, are exchanged with no proper scanning and testing performed, even in the DMZ. Practices like having authenticated the sender do not eliminate any risks. For instance, even if the trading partner’s identity has been truly confirmed, that trading partner could be sending a virus-infected file unintentionally.
It is also important to remember, it is possible for execution to occur without human intervention, such as when execution is triggered by a date or the passage of a certain amount of time. After that, the virus can spread by itself.
Best practices to mitigate this type of attack and how to remediate
Fortunately, IBM Sterling B2B Collaboration customers have tools to mitigate risk from this type of attack.
Best Practice 1:
To prevent getting these types of viruses, the first recommendation is to scan “all” files in the DMZ without letting the data hit disk. In addition, you’ll want a session break in the DMZ. This means that a trading partner does not have direct access to the secure area. The second requirement is to use Multi Factor Authentication (MFA).
IBM Sterling Secure Proxy (SSP) meets all of these requirements. SSP provides MFA and provides a Reverse Proxy. As a session comes in, a session is started from the secure area, such as from IBM Sterling File Gateway and it reaches out to SSP. This provides a session break in the DMZ. As data moves across the sessions, while in memory, SSP will communicate to the Anti-Virus software via ICAPs. This checks for viruses without letting data hit disk in the DMZ.
Best Practice 2:
To stop the spread of these viruses, Tempered Networks advises that, “Organizations need to start thinking about a security methodology that relies less on blocking specific traffic by policy and actively moving towards a zero trust. . . or whitelist” approach. This means eliminating SSH, SFTP, HTTP(s), etc.
IBM provides the market leading Whitelist file transfer approach called IBM Sterling Connect Direct (CD). CD has been in use for over 40 years and has never been breached. To prevent the spread of a virus that may have gone undetected, we recommend removing the default protocols from server installations. Then, with our unique Managed File Transfer solution, data can move from server to server with a 99.9997% success rate and data can move virus free. This means that a virus like Sunburst cannot move through your organization.
Best Practice 3:
Secure all pipeline services so they are not publicly accessible. Core servers (like ERPs) shouldn’t be given access to the public network or DMZ.
All core applications should sit behind a B2B Gateway like IBM Sterling File Gateway (SFG). SFG can handle all data types and works with SSP that provides security and anti-virus scanning in the DMZ. These two applications provide Enterprise-level security and secure file transfer management to all trading partners.
Best Practice 4:
If you’re concerned that you have been exposed to this type of virus, inventory your infrastructure assets for any presence of the virus.
- The inventory process consists of scanning each computer for the virus. If the virus is found, you need to look at the log files to see which other servers it has communicated with, because the credentials on those servers may be compromised. If compromised, the credentials on that server must be renewed.
- For any server that contains the virus, you must rebuild that server from scratch. You cannot restore from a backup as the credentials have been compromised. Rebuilding the server would mean wiping the server, laying down an operating system (without SSH, SFTP, and HTTP(s). There will be more about that in one of the following sections.
If you have suffered this type of attack and need to rebuild multiple credentials, IBM has a way to automate the creating of credentials in a Self-Service model. The system will automate the “re-onboarding” of servers and trading partners through the use of email and an automated portal.
The SolarWinds supply chain attack targeted 40 customers but infiltrated 18,000. For malicious hackers it’s a gold standard success, and for the rest of us it’s bad news, as we’ll likely start seeing more of this type of cyberattack and must prepare. It is not a question of “If it’s going to happen again” but a question of “When it will strike again”.
According to David Kennedy, founder of TrustedSec, by some estimates, cybercrime is expected to cost up to $6 trillion globally each year. Losses of this scale put incentives for business innovation at risk, and make cybercrime more profitable than the global trade of all major illegal drugs combined. Therefore, organizations must take a proactive approach to security and ensure all gaps and risks are removed entirely or at least minimized. This white paper can guide as a first step to determine where and how IBM Sterling B2B Collaboration customers can enforce security in a systematic and cost-efficient way.
As Steven Covey stated: “Being proactive is more than taking initiative. It is recognizing that we are responsible for our own choices and have the freedom to choose based on principles and values rather than on moods or condition. Proactive people are agents of change and choose not to be victims, to be reactive or to blame others.”
 Steven Covey, The Seven Habits of Effective People