Disaster Recovery - Our Client Hack Experience
Thankfully it doesn’t happen to us too often, but it can be a stark reminder when it does - last weekend we had a “cyber incident”, with one of our customers being infiltrated and infected with the .PLAY virus. In this blog, I'll share an overview of what went down, how we tackled it, and the valuable lessons we walked away with.
Discovery of the Attack
Typical of hacks, it was a Sunday lunchtime. I received a text message from a client saying they couldn’t access a server and, whilst they know it’s Sunday, “can we take a look”. I was having lunch in Wagamama’s at the time and took a quick look from my mobile. Logging into the server in question I was presented with a ‘failed xxx.exe’ program - now that’s not good! Understanding the gravity of the situation, I restarted the server and asked the client to try reconnecting whilst I started heading back to the home.
For context, the compromised server was running on Windows Server 2008 and it’s 2023… We had repeatedly warned the client that the server needed to be replaced, but all we could do now was rectify the issue for them as fast as possible.
Taking Control
Forty-five minutes later, I’m back home with Gareth, our Operations Manager on the phone and also connected. We notice that the main server had a script running on the desktop, trying to run the previously crashed file from the compromised server. Looking deeper into the server, the root was showing files with a “.PLAY” extension denoting the “PLAY” virus. Thankfully, very few files were affected on this server. The reason: our Sophos Endpoint protection with XDR (Extended Detection and Response) was doing its job. Looking at the Sophos logs, it was blocking every attempt made by the bad actors to remote onto the client’s machines, quarantining the infection.
We headed back to the first server (server 2008), found the xxx.exe file and deleted it.
However, we could now see files attempting to be run from another machine too. Investigating, we found it to be a Windows 7 workstation! To mitigate the risk, we promptly shut this machine down, planning to deal with it the following day.
Containment and Remediation
With the immediate threat addressed, we initiated several remedial actions. Password resets were conducted using our Privileged Access Management (PAM) system. All servers and devices were restarted, and we implemented a firewall lockdown for added security. We also cut off site-to-site VPNs, deactivated and shutdown any other unsupported machines, primarily Windows 7 systems.
Once the servers were back online, we monitored the Sophos logs and noticed a complete drop in activity from both machines. No more events of bad files being blocked were recorded. The infection was contained.
Aftermath and Recovery
We moved on to assess the aftermath and devised a plan for a smooth recovery. Our remote monitoring tools were disabled because of infection, the bad actors were hoping we could not gain access.
Additionally, all SQL databases had been disabled, and a backup drive connected to the infected server had been wiped. Fortunately, this was a red herring and an old backup. As of now we had implemented different and more robust backups, effectively air-gapping and concealing them from easy view. Our backup systems now included separate passwords, multi-factor authentication (MFA), and a 'pull' method for backups to prevent tracing by malicious actors.
So this has all happened in the space of an hour. We received no ransom note. We were relatively unscathed, except for original compromised server.
On Monday, we verified the backups and found a clean restore point. A relatively straightforward restoration process, (although it still took approximately six hours) brought the server back to life. By 4 PM that day, users were back in action.
Minor disruptions continued in the days following the incident, particularly with the Windows 7 machines that we then replaced. Nevertheless, the damage was limited and manageable.
Key Takeaways
At times it feels like there are things that us IT folk seem to “bang on about”, but such incidents provide valuable lessons and reinforce the importance of several of these best practices such as:
- Use supported operating systems: Ensure that all computers are supported and regularly updated to reduce vulnerability.
- Good Endpoint protection: Things change in security and not all Endpoint protection is created equal. We use Sophos XDR to ensure our clients' businesses are protected by cutting-edge security features and real-time threat detection.
- Review Backups: Often missed - regularly review and test your backup systems to ensure they are robust and effective and ultimately work.
- Separate Backups: Isolate backups from the domain, use separate passwords and enable MFA to enhance their security. Consider how the backup runs and are there ways to make it harder to find or move offsite completely? Finally, with more data being backup up to cloud services think about using “Immutable” backups so the data stored is protected from deletion.
- Leverage a SOC: A Security Operations Centre (SOC) allows for immediate response and threat mitigation, with a security team monitoring events in real-time at times when we can’t.
In conclusion, cyber incidents can and will happen to even the most prepared organisations. Our experience with the .PLAY virus underlines the importance of preparedness, vigilance, and a well-thought-out cybersecurity strategy to minimise damage and ensure a swift recovery.
Now might be a good time to check that your current IT team or IT provider has this covered…
