The 5-hour CDN
The term “CDN” (“content delivery network“) conjures Google-scale companies managing huge racks of hardware, wrangling hundreds of gigabits per second. But CDNs are just web applications. That’s not how we tend to think of them, but that’s all they are. You can build a functional CDN on an 8-year-old laptop while you’re sitting at a coffee shop. I’m going to talk about what you might come up with if you spend the next five hours building a CDN.
It’s useful to define exactly what a CDN does. A CDN hoovers up files from a central repository (called an origin) and stores copies close to users. Back in the dark ages, the origin was a CDN’s FTP server. These days, origins are just web apps and the CDN functions as a proxy server. So that’s what we’re building: a distributed caching proxy.
SSH and User-mode IP WireGuard
For a couple hundred lines of code (not counting the entire user-mode Linux you’ll be pulling in from gVisor, HEY! Dependencies! What are you gonna do!) you can bring up a new, cryptographically authenticated network, any time you want to, in practically any program.
There really are some fun libraries out there if you want to build something crazy.
Stealing Your Private YouTube Videos, One Frame at a Time
I quickly Googled “base64 to image”, and pasted the base64 into the first decoder I found, and it displayed a thumbnail from the target Private video! It worked! I have found a working IDOR (Insecure Direct Object Reference) bug, where I could get a frame from any private video on YouTube!
Enter the Vault: Authentication Issues in HashiCorp Vault
This is tediously complex, IMO (as someone who doesn’t use the service in question), but the conclusion is worth considering.
In my experience, tricky vulnerabilities like this often exist where developers have to interact with external systems and services. A strong developer might be able to reason about all security boundaries, requirements and pitfalls of their own software, but it becomes very difficult once a complex external service comes into play. Modern cloud IAM solutions are powerful and often more secure than comparable on-premise solutions, but they come with their own security pitfalls and a high implementation complexity. As more and more companies move to the big cloud providers, familiarity with these technology stacks will become a key skill for security engineers and researchers and it is safe to assume that there will be a lot of similar issues in the next few years.
Finally, both discussed vulnerabilities demonstrate how difficult it is to write secure software. Even with memory-safe languages, strong cryptography primitives, static analysis and large fuzzing infrastructure, some issues can only be discovered by manual code review and an attacker mindset.
Ethereum is a Dark Forest
In the Ethereum mempool, these apex predators take the form of “arbitrage bots.” Arbitrage bots monitor pending transactions and attempt to exploit profitable opportunities created by them. No white hat knows more about these bots than Phil Daian, the smart contract researcher who, along with his colleagues, wrote the Flash Boys 2.0 paper and coined the term “miner extractable value” (MEV).
Phil once told me about a cosmic horror that he called a “generalized frontrunner.” Arbitrage bots typically look for specific types of transactions in the mempool (such a DEX trade or an oracle update) and try to frontrun them according to a predetermined algorithm. Generalized frontrunners look for any transaction that they could profitably frontrun by copying it and replacing addresses with their own. They can even execute the transaction and copy profitable internal transactions generated by its execution trace.
How to contact Google SRE: Dropping a shell in cloud SQL
Google Cloud SQL is a fully managed relational database service. Customers can deploy a SQL, PostgreSQL or MySQL server which is secured, monitored and updated by Google. More demanding users can easily scale, replicate or configure high-availability. By doing so users can focus on working with the database, instead of dealing with all the previously mentioned complex tasks. Cloud SQL databases are accessible by using the applicable command line utilities or from any application hosted around the world. This write-up covers vulnerabilities that we have discovered in the MySQL versions 5.6 and 5.7 of Cloud SQL.
Zero-day in Sign in with Apple
In the month of April, I found a zero-day in Sign in with Apple that affected third-party applications which were using it and didn’t implement their own additional security measures. This bug could have resulted in a full account takeover of user accounts on that third party application irrespective of a victim having a valid Apple ID or not.
AWS re:Invent 2019: Speculation & leakage: Timing side channels & multi-tenant computing
In January 2018, the world learned about Spectre and Meltdown, a new class of issues that affects virtually all modern CPUs via nearly imperceptible changes to their micro-architectural states and can result in full access to physical RAM or leaking of state between threads, processes, or guests. In this session, we examine one of these side-channel attacks in detail and explore the implications for multi-tenant computing. We discuss AWS design decisions and what AWS does to protect your instances, containers, and function invocations. Finally, we discuss what the future looks like in the presence of this new class of issue.
This is a good recap. Specific defenses starts around 42:00.
A Compendium of Container Escapes
The goal of this talk is to broaden the awareness of the how and why container escapes work, starting from a brief intro to what makes a process a container, and then spanning the gamut of escape techniques, covering exposed orchestrators, access to the Docker socket, exposed mount points, /proc, all the way down to overwriting/exploiting the kernel structures to leave the confines of the container.
My infrastructure as of 2019
The goal for my infrastructure is to run the services I need. While a lot of people in the homelab community experiment and play with software for its own sake, I actively use the stuff I host. When I stop, I kill the service (though I’m not as proficient at this as Google). These are my production systems, and when one of them is down, I do miss it.
How Tailscale works
There is one last question that comes up a lot: given that Tailscale creates a mesh “overlay” network (a VPN that parallels a company’s internal physical network), does a company have to switch to it all at once? Many BeyondCorp and zero-trust style products work that way. Or can it be deployed incrementally, starting with a small proof of concept?
Tailscale is uniquely suited to incremental deployments. Since you don’t need to install any hardware or any servers at all, you can get started in two minutes: just install the Tailscale node software onto two devices (Linux, Windows, macOS, iOS), login to both devices with the same user account or auth domain, and that’s it! They’re securely connected, no matter how the devices move around. Tailscale runs on top of your existing network, so you can safely deploy it without disrupting your existing infrastructure and security settings.
OpenBSD on DigitalOcean
They are both sort of old at this point and with OpenBSD 6.6 out I ran into a bit of a snag. The default these days is to use a GPT partition table to enable EFI booting. This is generally pretty sane but it looks to me like the FreeBSD droplet doesn’t support this. After the installer rebooted the VM failed to boot, being unable to find the bootloader.
Thankfully DigitalOcean has a recovery ISO that you can boot by simply switching to it and powering off and then on your Droplet.
dd miniroot over FreeBSD, reboot, lemonade!
Three ways to reduce the costs of your HTTP(S) API on AWS
Since we would send this five billion times per day, every byte we could shave off would save five gigabytes of outgoing data, for a saving of 25 cents per day per byte removed.
It all adds up.
Defense in depth against SSRF vulnerabilities with the EC2 Instance Metadata Service
Today, AWS is making v2 of the EC2 Instance Metadata Service (IMDSv2) available. The existing instance metadata service (IMDSv1) is fully secure, and AWS will continue to support it. But IMDSv2 adds new “belt and suspenders” protections for four types of vulnerabilities that could be used to try to access the IMDS. These new protections go well beyond other types of mitigations, while working seamlessly with existing mitigations such as restricting IAM roles and using local firewall rules to restrict access to the IMDS. AWS is also making new versions of the AWS SDKs and CLIs available that support IMDSv2.
Eh, seems this could have been better from the start, but oh well.
How I accidentally took down GitHub Actions
Commit shorthashes have a major problem: As a repository accumulates a large number of commits, eventually it will contain two commit hashes that start with the same seven characters (and have the same shorthash). After this happens, tools that use shorthashes will start to break because the commit shorthash is ambiguous (it’s no longer a pointer to a single commit). Due to the birthday problem, any repository that has at least 19291 commits is likely to have a pair of ambiguous commits somewhere. So if we waited for the actions/docker repo to have tens of thousands of commits, one of the shorthashes would eventually become ambiguous and break someone’s build.
Snap: a microkernel approach to host networking
This paper describes the networking stack, Snap, that has been running in production at Google for the last three years+. It’s been clear for a while that software designed explicitly for the data center environment will increasingly want/need to make different design trade-offs to e.g. general-purpose systems software that you might install on your own machines. But wow, I didn’t think we’d be at the point yet where we’d be abandoning TCP/IP! You need a lot of software engineers and the willingness to rewrite a lot of software to entertain that idea.
Kubernetes made my latency 10x higher
Problems often appear just because we put some pieces together in the first place.
OpenTitan - open sourcing transparent, trustworthy, and secure silicon
Today, along with our partners, we are excited to announce OpenTitan - the first open source silicon root of trust (RoT) project. OpenTitan will deliver a high-quality RoT design and integration guidelines for use in data center servers, storage, peripherals, and more. Open sourcing the silicon design makes it more transparent, trustworthy, and ultimately, secure.
OpenBSD on Google Compute Engine
This tutorial outlines a simple way to get OpenBSD working on GCE, utilizing only OpenBSD to create the image and send up into gcloud.
Defense at Scale
Last year, my colleague Chris Rohlf gave a keynote at BSidesNOLA entitled “Offense at Scale”. Offense sounds fun. Pwn all the things. And you’re always going to win! And normally I’m a big fan of being massively offensive. Unfortunately, I find myself on the defense when it comes to information security.
Here’s how you defend at scale. Can’t be done. The end. Everything’s fucked. You’re pwned.
Plenty of good points here. Also a fun read.