Here is what we know about the compromise of Denmarks central bank: »It all seems 'too little, too late'«

According to Version2 reports on June 29, the central bank of Denmark – Danmarks Nationalbank – had a Solarwinds backdoor in their systems, from June to December 2020.

Read more: Central bank of Denmark hacked as part of 'the world’s most sophisticated hacker attack'

In an internal email, disclosed to Version2 following a freedom of information request, the central bank reports that they have not been able to find any IOCs, Indications of Compromise. In other words, nothing points towards the backdoor actually having been exploited in the seven-month period.

It turns out that the forensic analyses carried out by the bank and its subcontractors have unveiled several indications that, according to experts, could be a sign of further compromise.

The documents disclosed to Version2 show that a Zip file with the installation package for a vulnerable version of Solarwinds has shown up in a server environment.

Other documents, leaked to Version2 by a whistleblower concerned about the security in Danmarks Nationalbank, show that the affected Solarwinds deployment controls and monitors parts of the bank's production environment.

The documents also show that two core polling engines in the compromised environment start communicating on the internet the day before the backdoor is closed, and that a server with an unrecognized IP address suddenly appears on a map of the network, with unrestricted internet access. The same has been the case for one of the bank's own Solarwinds deployments as late as January 11th.

»Woah, what?!«, says Lucas Lundgren, white-hat hacker at Banshie, almost choking on his water. After catching his breath, he continues:

»That's a big one. Even if all these things turns out to be nothing, for sure something needs to be done to improve the security, it appears inadequate,« says the hacker, noting that »it's as if the reponse in general hasn't been sufficiently timely or resolute.«

»This should be the big dig. Fort Knox. And that's not my immediate impression. It all seems a little ‘too little, too late', and they risk being compromised deeper than stage1,« says Lucas Lundgren worriedly.

Moreover, the leaked documents show that the bank's subcontractor, JN Data, picks the cheaper of two mitigation strategies when working to re-secure the affected server environment.

Danmarks Nationalbank and three of its subcontractors, BEC, KMD and JN Data, have all refused an invitation for an interview with Version2, so it is impossible to tell exactly how the investigation of the mysterious events was conducted, and what was its conclusion. But the knowledge presented today causes several experts to question the bank's handling of the case. We'll get back to that later in this story.

Priority 1

To get the best understanding of the case, we need to go back to December 14th 2020. On that day, the world learns that an APT, advanced persistent threat, presumed to be controlled by the Russian state, has managed to embed backdoors in a version of Solarwinds Orion, which has been deployed in 18,000 organizations and companies across the globe, including Danmarks Nationalbank, its subcontractor BEC, many energy companies, and the Danish Agency for Governmental IT Services.

Sponseret indhold

Michael har 90 patentansøgninger under bæltet og ser AI og maskinlæring som høreapparatindustriens nye schweizerkniv

Lyd

JN Data, who provides the network monitoring for BEC's servers in the bank, reacts promptly as the systems monitoring deployed there happens to be facilitated by the infected version of Solarwinds Orion -- the compromised piece of software that has put IT departments across the world under pressure.

The leaked documents show that JN Data immediately creates a 'priority 1' case, and that JN's Cyber Defence Center (CDC) is activated.

The documents don't mention the exact contents of the servers that Solarwinds Orion has monitored and controlled on behalf of JN Data and Danmarks Nationalbank, but they do show that the system contains a production environment with more than ten rolling backups, two polling engines and several large switches.

The first big question

This leads to the first big question: Was Kronos2, the digital heart in the financial infrastructure of Denmark, fully or partly controlled and monitored by the affected Solarwinds deployment?

Kronos2 is of particular interest, because it facilitates the movement of 649 billion Danish kroner (€87 billion) daily, clearing bank-to-bank transfers and as such being the backbone of the financial infrastructure. Kronos2 indirectly facilitates all credit card payments, as well as currency and security trades.

Version2's freedom of information request with Danmarks Nationalbank shows that the Solarwinds compromise was the topic of a status meeting regarding the operational stability of Kronos2 on January 25th 2021. As a consequence, Version2 has tried to get the bank to confirm or deny the whistleblower's claim that Kronos2 in fact is handled by the compromised Solarwinds deployment. As is the case with all our inquiries, Danmarks Nationalbank has refused to comment.

Danmarks Nationalbank's hesitation when presented with Version2's questions causes some vexation with Carsten Schürmann, professor in IT security at the IT University of Denmark.

Sponseret indhold

Genskab din infrastruktur med automation og immutable backup

It-sikkerhed

»If everything has been done by the book, and everything is secure, why not just explain that?« the professor begins, in his argument that the bank should open up:

»We need more transparency so everybody can learn from this. It could well be that they have already done the right thing and if so, others can learn from them.«

»Something to help you sleep tonight :-)«

Back in December it wasn't only JN Data that was trying to piece together what had happened. Version2's freedom of information request shows that an email reaches the lead IT security officer at Danmarks Nationalbank on December 16th.

Sponseret indhold

Maj knækker koden: Hun gør bæredygtighed til forretning

Efteruddannelse

The subject is »Something to help you sleep tonight :-)« and is shown in this article.

The email mentions that KMD, who is also responsible for parts of the bank's IT setup, has run a malware check. KMD writes:

»Hi everyone. The malware checker is continously monitoring all sorts of .exe files on the servers, and look what it has reported since Saturday. Between Friday-Saturday ‘someone’ downloaded SolarWinds Orion to a production server in the MGT domain :-)«

From a subsequent email it appears that it was an installation package, that was not extracted. Consequently KMD Nationalbank concludes that it has not been installed -- and deletes it immediately. It is not clear from the freedom of information request which version of Solarwinds it was, or whether the Zip file included anything else.

Deleting possible evidence

And, intuitively, what could appear as a reasonable cause of action -- to delete something that could be dangerous for one's system -- surprises Lucas Lundgren:

»Did they delete the file? No. NOOO! If that Zip file belonged to an attacker and it disappeared suddenly, they will know for sure that they'd been caught. The bank walked right into the trap. Instead, they should have enabled monitoring on the file and copied it to a secure environment to further investigate how it behaves and what it contains,« says Lucas Lundgren.

»If I discovered a zip file that I hadn't put there myself, I'd think ‘what the hell?!’ I'd consider what is apparent in that email to be a huge Indication of Compromise,« the hacker says, who is backed by his colleague and competitor, security consultant with Improsec, Morten Munck:

»It is best practice to do a forensics assessment to figure out what's been executed on the machine, and thereby figure out exactly what has happened,« says Morten Munck who declined to speak about this case specifically, but only what is generally considered best practice in the IT security business.

As it appears from the freedom of information request, there has been made no further investigation of the file before it was deleted. Likewise, the material Version2 has obtained through whistleblower leaks doesn't mention any further action. Still, the bank has no comments.

Were the servers switched off or not?

As KMD was struggling with the zip file, per the whistleblower's documents, JN Data worked to shut down the affected Solarwinds deployment to get it patched with security updates. On December 16th 2020 Solarwinds is up and runnning again, according to documents provided by the whistleblower.

However, that is not in accordance with what BEC, who maintains contact with the bank, says. From an email in Version2's freedom of information request BEC reports that Solarwinds is down on December 18th 2020. Danmarks Nationalbank has declined to comment whether they feel adequately informed by their subcontractors.

‘This server does not belong to us’

December 18th brings about another mysterious event. A Solarwinds deployment that JN Data-employees do not recognize suddenly appears on a map of the bank's network. According to the leaked documents, that causes internal concern, which JN Data's Cyber Defence Center is presented with.

That trail is a dead end. We don't know what JN Data's CDC does then. Morten Munck declines to speak about the specific case but notes:

»When dealing with critical infrastructure, it is generally sound to keep track of what's on one's network,« says the security consultant.

Implements cheap fix

Meanwhile there is an important discussion in JN Data's Cyber Defence Center whether to re-establish all the services that Solarwinds has been controlling and monitoring from ground-up, or to make do with resetting the passwords to them?

Resetting the passwords costs just under 1 million kroner (€130,000), while a total re-establishment of all the services amounts to a little more than 3 million (€390,000), documents show.

JN Data chooses the least cost-intensive and fastest mitigation and settles for resetting the passwords, it appears from the whistleblower's documents. This worries white hat hacker Lucas Lundgren:

»It worries me that they did not re-establish these services but opted to only change the passwords. Because if the systems are really compromised, or there is a suspicion they might be, the attacker doesn't need the passwords. In other words: You can change the key for the door, but as an attacker the first thing I'd do would be to make another back door for myself. And they've had seven months to do just that in this case,« says Lucas Lundgren who again is backed by Morten Munck:

»If you've been compromised by an advanced adversary it can be hard to tell whether you've plugged all the holes. In such cases it'd be well-advised to re-establish services from a ‘known good’ backup, or to build everything from the ground-up.«

Going against the recommendations of the supplier

And in the Solarwinds-case there is certainly reason to believe we're dealing with an advanced adversary, deems professor in IT security Carsten Schürmann.

»You also have to remember who we're up against here. It is likely the Russian intelligence agency FSB. If they get to a point where they have a back door, they can cause immense damage,« says Carsten Schürmann.

The decision also goes against the hardware manufacturer Cisco's Solarwinds recommendations. According to Version2's documentation, Cisco delivers almost all networking hardware in Danmarks Nationalbank. Specifically, Cisco's security department Talos writes:

»Reset all credentials used by SolarWinds software and implement a rotation policy for these accounts. Require long and complex passwords,« says Cisco who also recommends making new credentials and not just changing the passwords for the old ones. And that makes the case even more clear for lecturer in IT security Jan Lemnitzer:

»If three million is the price for a higher degree of security against a Russian hack of the central bank, that's a cheap one,« says Jan Lemnitzer, and Lucas Lundgren adds:

»Three million is peanuts in that frame of reference, especially when there's actually several small suspicious events.«

The leaked documents also show that it takes a week before the passwords are reset, which sounds slow to Lucas Lundgren:

»It seems really weird that the central bank doesn't have any more control, and allows their subcontractors to handle these things so slowly,« says Lucas Lundgren as a direct reaction.

Version2's leaked documentation also shows that it takes a week from JN Data's Cyber Defence Center gets the choice between the two mitigation paths until it gives the green light to the least comprehensive of the two. Consequently, JN Data doesn't get started with the reset right away.

One month later: Solarwinds is taken off of the internet

A few weeks later, around January 11th, JN Data decides that the internet access for Solarwinds should be restricted, according to documents made available to Version2 by the whistleblower.

According to the documents, it would cause practical problems relating to operation and updates of the Solarwinds system, since it cannot be reached directly, but that it is possible. And that exposes another flank, Carsten Schürmann assesses:

»If you can disconnect a service from the internet, you should, and naturally before any damage occurs. It's all about making the attack surface as small as possible.«

Lucas Lundgren agrees:

»It should all have been off of the internet immediately, and remain that way, until the situation was under control. If you run systems like Kronos2 that you cannot take offline, then run a replica while doing a clean installation that can take its place as soon as possible,« says the white-hat hacker.

Servers begin talking

On January 11th more information surfaces, according to the leaked documents. Two of BEC's so-called polling engines had made network traffic at a time when they should have been silent. Specifically the servers start making network traffic on December 16th, the day before the update with the latest hotfix from Solarwinds, closing the backdoor, was scheduled.

JN Data classifies the network traffic as ‘Office 365-traffic,’ as it is described in the leak. Therefore, documents show, it is decided to not investigate this event further.

But this form of classification is not something to trust fully, since it can be manipulated in different ways, says Lucas Lundgren and the whistleblower.

»You've spent hours of our legal team's time«

Version2 has repeatedly contacted BEC, KMD, JN Data and not least Danmarks Nationalbank, who declined to do an interview with the following remark:

»Now, damn it, you've spent I don't know how many hours of our legal advisors’ time for a freedom of information request, and now you also want an interview?!,« says a communications consultant with the bank.

Since then Danmarks Nationalbank refuses having had a backdoor and has sent the following comment to Version2 by email, not responding to a single one of our questions:

»The SolarWinds attack also struck the financial infrastructure in Denmark. The relevant systems were contained and analysed as soon as the compromise of SolarWinds Orion became known. There was a timely and resolute response in a satisfying way, and according to the analyses carried out there has been no indication that the attack has had real consequences.«

Disappointment over silence

The lacking response from Danmarks Nationalbank is no surprise to Version2's sources, but it saddens them:

»The questions you've been posing are well thought out and point towards information that is in the public interest. So it is interesting that they don't say anything,« says professor in IT security Carsten Schürmann, with Lucas Lundgren adding:

»Everyone can get hacked. It's not the fact that they were compromised, but their reaction that I think is disappointing,« says Lucas Lundgren.

Danmarks Nationalbank: We have not been compromised

In a statement yesterday, Danmarks Nationalbank wrote among other things:

»Danmarks Nationalbank and the bank's suppliers use several tools and methods to monitor, contain and analyze situations where there is a suspicion of compromise. For security reasons, Danmarks Nationalbank cannot specify which tools or methods are used. In addition, in the specific situation, there has been an IT architecture that has further minimized the risk of compromise. The analyzes carried out conclude that Danmarks Nationalbank has not been compromised as a result of the vulnerabilities in SolarWind's software during the seven months the vulnerability existed. This has also been confirmed by the bank's IT subcontractors.«

For questions and comments, please contact mlo@ing.dk, Signal +45 6169 0917