We present a comprehensive analysis of all possible shadow stack mechanisms along three axes: performance, compatibility, and security. Based on our study, we propose a new shadow stack design called Shadesmar that leverages a dedicated register, resulting in low performance overhead, and minimal memory overhead. We present case studies of Shadesmar on Phoronix and Apache to demonstrate the feasibility of dedicating a general purpose register to a security monitor on modern architectures, and Shadesmar’s deployability.

Isolating the shadow stack is critical for security, and requires in process isolation of a segment of the virtual address space. We achieve this isolation by repurposing two new Intel x86 extensions for memory protection (MPX), and page table control (MPK). Building on our isolation efforts with MPX and MPK, we present the design requirements for a dedicated hardware mechanism to support intra-process memory isolation, and show how such a mechanism can empower the next wave of highly precise software security mitigations that rely on partially isolated information in a process.

source: green