Is Hyper-Threading a Fundamental Security Risk?

Ever since Intel introduced Hyper-Threading (known generically as Symmetric Multi-Threading), debates about whether or not to disable the feature have almost entirely revolved around its impact on performance. Back when the feature debuted, it wasn’t unusual for programs to misinterpret what it meant for a system to have a virtual CPU core as opposed to a second physical chip (back then, it was one core to a socket, no exceptions, and programs didn’t differentiate between a physical and a logical CPU core). As software and operating systems were updated, HT settled down and it’s less common today to need to shut it off to preserve performance. But in the wake of Spectre, Meltdown, and Foreshadow, serious concerns have been raised about the security implications of Hyper-Threading.

Theo de Raadt, the founder of OpenBSD, argues that HT can no longer be trusted and should be disabled by default. wfoojjaec reached out to de Raadt to discuss the issue and why he and other developers in the open-source software community are concerned about the security risks of Hyper-Threading.

According to de Raadt, every operating system is either adding the ability to disable Symmetric Multi-Threading (Intel’s Hyper-Threading is a specific implementation of SMT) or modifying their schedulers “to avoid cotenancy on SMP cpus of different security domains.” Symmetric Multi-Processing, or SMP, refers to the modern practice of having multiple CPU cores on a single die, all with access to a combined pool of memory and devices. In contrast, Intel’s Hyper-Threading shares certain resources between the physical core and its logical counterpart, including translation lookaside buffers (TLBs), the L1 data cache, and the branch target cache (BTC) without providing any ability to differentiate between security domains and isolate data between the two CPU cores.

As we’ve previously discussed, Spectre, Meltdown, and Foreshadow are all flaws that exploit certain behavior that Intel CPUs engage in when they speculatively execute instructions. While speculative execution is a longstanding and proven technique for improving overall CPU performance, de Raadt identifies three distinct issues that have combined to create these problems. He writes:

1) Intel CPUs fetch and decode and execute instructions including their data loads without doing any security checks, then unwind them if they were wrong after the fact. CPUs from other vendors have experienced minor spectre problems, but Intel takes it to a whole new level.

2) Since they don’t do security checks up front, and tie their L1D to their TLB, Intel has a really astounding “where did a cache line come from, we don’t care” error in their L1D cache, which results in data in a cache line from a different privilege domain becoming visible to speculative instructions, which creates a further spectre problem.

3) The same speculation without security check applies to registers. Intel didn’t even check if the FPU is enabled, before accessing FPU registers. So they really don’t do *any security checks* before running an instruction. ALL decisions are made at the end. That means ALL instruction sequences have spectre side effects, and we are simply waiting for people to find worse consequences and publish them.

de Raadt also criticized Intel’s disclosure policies, noting that OpenBSD has had to study workarounds in other projects, like Xen and FreeBSD, to create their own solutions. He believes it’s likely that we’ll continue to see more security flaws related to Spectre and that there’s a possibility for black hats to combine different methods of exploiting these flaws to break security models. Foreshadow, it could be argued, is one such attack. While more complex than the first variants of Spectre, it can also be used to break Intel’s Software Guard eXtensions, or SGX — and SGX was supposed to be immune to this kind of attack. There’s even a chance that these attacks could be used to leak address information, which means Spectre and Rowhammer could be combined. It’s the gift that keeps on giving.

Thus far, OpenBSD is the first operating system to call for disabling HT altogether — Intel’s official guidance is that HT does not need to be disabled if all other fixes and patches have been deployed. But it’s incredibly difficult to practically guarantee that all necessary security contexts will be maintained and respected in the absence of hardware restrictions that prevent two different processes operating in different security domains from running at the same time. Even if you can ensure that the processes running on a CPU are compatible from a security domain perspective, as soon as the security domain of one of those processes shifts, you’d have to evict it from the CPU core it’s running on and put it somewhere else — flushing the caches and TLB in the process. Modern OS schedulers regularly move workloads across CPU cores to optimize execution, but forcing a CPU to do this in the name of security can carry a heavy performance hit. We’ve already seen some evidence of this in Spectre, although the tests that exposed it tended to be worst-case scenarios.

Is AMD Affected?

Thus far, almost all the discussion around Spectre, Meltdown, and Foreshadow has focused on Intel. There’s a practical reason for this. These attacks are believed to threaten the security of cloud and enterprise server providers and Intel dominates these markets. Prior to the launch of Epyc, Intel held 99% of the x86 server market and the overwhelming majority of servers sold per year are x86 machines. AMD has begun to chip away at Intel’s market dominance, but CEO Lisa Su has stated her company is targeting mid single-digit percentages of the market by the end of 2018. Even if AMD and Intel were equally exposed technically, Intel would be shouldering virtually the entire effective exposure.

But while this could change in the future, current evidence suggests AMD CPUs aren’t nearly as vulnerable as their Intel counterparts. AMD has released a statement indicating that it isn’t impacted by Foreshadow, which Intel calls the L1TF (L1 Terminal Fault). It recommends that its customers not implement Foreshadow protections at this time and states that its CPUs are protected by hardware paging architecture protections built into Epyc CPUs.

One additional piece of evidence in AMD’s favor is that the company’s SMT implementation is known to be different from Intel’s. We’ve previously only discussed these differences in terms of their impact on performance, but the slide above does note that the L0/L1/L2 ITLBs and L1/L2 DTLBs are shared but “SMT tagged,” which means they can only be accessed by the thread that owns them. The devil is absolutely in the details on issues like this, and we don’t want to imply that this single slide establishes the degree to which AMD’s SMT implementation is or is not secure, but AMD does appear to have implemented protections in certain areas that Intel lacks. It’s possible, for example, that future attacks could be based on cache evictions rather than speculative loads, and this type of tagging might not protect against such alternatives. In our conversation, de Raadt notes that OpenBSD has also made changes to disable SMT and CMT (that’s Bulldozer’s core-sharing technology) on AMD CPUs out of an abundance of caution, despite not knowing if the CPUs will ultimately prove to be vulnerable to this type of attack.

As of this writing, OpenBSD 6.4 (expected in October / November) is the first OS to avoid all use of Hyper-Threading and disable it by default, but other operating systems, like Red Hat, have announced that they will ship kernels with updated controls allowing users to choose whether to disable the feature or not. Whether other vendors follow Theo de Raadt’s lead may depend on what vulnerability disclosures drop next and how serious the next round of exploits is. Intel has announced that its upcoming Cascade Lake platform will contain hardware fixes for some Spectre variants, while others will still be mitigated by software solutions or a combination of the two. That’s one point everyone in the security industry seems to agree on: There will be new disclosures and security issues related to Spectre that haven’t happened yet.

Continue reading

Fermilab Experiment Hints at New Fundamental Force of Nature

The team working on Fermilab's Muon g−2 experiment has reported a tantalizing hint of a new type of physics. If confirmed, this would become the fifth fundamental force in the universe.

Affordable 5G Could Fundamentally Change How You Receive Internet Service

5G service could collapse the distinction between cellular provider and ISP, fundamentally changing how consumers buy internet service — but only if that service is kept affordable.

Intel’s Fundamentally New MESO Architecture Could Arrive in a Few Years

Intel's new MESO architecture could appear in products more quickly than anticipated, particularly in AI.

Overclocking Results Show We’re Hitting the Fundamental Limits of Silicon

CPU IPC can keep improving, but both Intel and AMD are having trouble raising clocks. It's not clear there's a solution to the problem.