In the first days of 2018, published research revealed that nearly every computer chip manufactured in the last 20 years contains fundamental security flaws, the most prominent variations called Meltdown and Spectre. Fast forward to the present day and not only have our team patched vulnerabilities across all servers, but we’ve done so with the minimum amount of downtime possible. Here’s how.
Towards the back end of 2017, Google researchers discovered a deeply embedded CPU security flaw that, when later published as research, sent shockwaves throughout the IT industry.
What they'd unearthed were fundamental vulnerabilities which, if exploited, allowed attackers to gain access to data housed inside the CPU, which had been considered completely protected.
Nicknamed Meltdown and Spectre, the flaws are present in nearly every computer chip manufactured in the last 20 years and, while there’s no evidence that these vulnerabilities have ever been exploited, it could have allowed hackers to access passwords, encryption keys and other private information undetected.
While many headlines have so far focused on the impact to PCs, the real problems are how these bugs will affect servers and the cloud.
That's because Meltdown and Spectre can break through the memory walls between applications and operating system's dedicated memory, meaning that single hosts in cloud environments can share hundreds, potentially thousands of customer's data.
The flaws arose from features built into the chips that helps them run faster, called speculative execution and caching.
Speculative execution essentially involves a chip attempting to predict the future in order to work faster, whereas caching is a technique used to speed up memory access. The problem arose when caching and speculative execution started engaging with protected memory.
Protected memory is one of the foundational concepts underlying computer security. At its most basic, it ensures that no process on a computer should be able to access data unless it has permission to do so.
The Meltdown and Spectre threats stems from the fact that this wait for permission can be a (relatively) slow process, and, to get ahead of the game, speculative execution begins working with the data before it receives permission to do so.
In theory this is still secure, because if the process doesn’t pass the check the data is discarded. But, in the interim, the protected data is stored in the CPU cache that can be accessed in what is known as a side-channel attack.
In layman’s terms, a malicious program could exploit the Meltdown and Spectre vulnerabilities and get hold of potentially damaging information being held within the CPU cache – including passwords, emails, instant messages and even business-critical documents. Essentially, one carefully considered breach is all that stands in the way of a hacker and your valuable data.
The DediServe team began experimenting with different ways to patch the vulnerability as soon as we realised the severity of the threat.
First and foremost, we understood that while patching alone would be enough, to ensure the security of our customer's data further, the underlying software would need to be updated, tested and then rolled out worldwide.
As our first port of call, we began sourcing new kernels from Linux communities. Once this stage was complete, we had to apply kernel updates to hosts while avoiding any disruption, and finally we had to reset all hypervisors so that the reboot could take effect.
To keep any downtime to a minimum, we conducted the reboots in scheduled waves and throughout hours of least activity, ensuring that all customers were kept informed at all times. This also provided the team with an opportunity to advise customers to update their virtual machines to ensure their security as well.
The result: all 200+ blades are now up to date and no longer vulnerable to any of the Meltdown or Spectre vulnerabilities. And, while we certainly advise you to update your VMs to the latest kernels, with the issue patched at host-level, all VMs are unexploitable.
To learn more about the Meltdown and Spectre vulnerabilities or to discover how the DediServe team were able to fix the vulnerabilities with minimal downtime, why not get in touch? Or, to discover why DediServe is the world's largest provider of flexible, enterprise-grade OnApp hosting, start your OnApp journey today.