Implementing traceroute in Go
This tool is very useful to inspect network paths and solve problems. But aside from that, this tool is extremely interesting and its actual implementation is pretty simple.
Under the Hood of a Simple DNS Server
For this post, I will talk mostly about the details of implementing a DNS server that follows the original two RFCs that laid out the spec: 1034 and 1035.
Chromium’s impact on root DNS traffic
The root server system is, out of necessity, designed to handle very large amounts of traffic. As we have shown here, under normal operating conditions, half of the traffic originates with a single library function, on a single browser platform, whose sole purpose is to detect DNS interception. Such interception is certainly the exception rather than the norm. In almost any other scenario, this traffic would be indistinguishable from a distributed denial of service (DDoS) attack.
Certificate Transparency: a bird's-eye view
Certificate Transparency (CT) is a still-evolving technology for detecting incorrectly issued certificates on the web. It’s cool and interesting, but complicated. I’ve given talks about CT, I’ve worked on Chrome’s CT implementation, and I’m actively involved in tackling ongoing deployment challenges – even so, I still sometimes lose track of how the pieces fit together. I find it easy to forget how the system defends against particular attacks, or what the purpose of some particular mechanism is.
A Massive Leak
“Memory leaks are impossible in a garbage collected language!” is one of my favorite lies. It feels true, but it isn’t. Sure, it’s much harder to make them, and they’re usually much easier to track down, but you can still create a memory leak. Most times, it’s when you create objects, dump them into a data structure, and never empty that data structure. Usually, it’s just a matter of finding out what object references are still being held. Usually.
A few months ago, I discovered a new variation on that theme. I was working on a C# application that was leaking memory faster than bad waterway engineering in the Imperial Valley.
How CDNs Generate Certificates
Obviously, to do stuff like this, you need to generate certificates. The reasonable way to do that in 2020 is with LetsEncrypt. We do that for our users automatically, but “it just works” makes for a pretty boring writeup, so let’s see how complicated and meandering I can make this.
It’s time to talk about certificate infrastructure.
Path Building vs Path Verifying: Implementation Showdown
In my previous post, I talked about what the issue with Sectigo’s expired root was, from the perspective of the PKI graph, and talked a bit about what makes a good certificate verifier implementation. Unfortunately, despite browsers and commercial OSes mostly handling this issue, the sheer variety of open-source implementations means that there’s a number of not-so-good verifiers out there.
In this post, I’ll dig in a little deeper, looking at specific implementations, and talking about how their strategies either lead to this issue, or avoided this issue but will lead to other issues.
It’s pretty much all terrible, except the parts that are extremely terrible.
Wireless is a trap
I used to be an anti-wire crusader. I hated the clutter of cables, and my tendency to unconsciously chew on them if they got anywhere near my face. But running into bug after tricky wireless bug—mostly while trying to make my video calls work better—I’ve apostasized. The more I’ve learned about wifi, Bluetooth and related protocols, the more I’m convinced that they’re often worse, on net, than wires.
This starts off with the usual, but then there’s some real wtf.
Qt included a component which would poll for networks every 30 seconds whenever a “network access manager” was instantiated, causing pretty much any Qt app using the network to degrade your wifi for ~5 out of every 30 seconds. There were already multiple bug reports for this issue, one of which was declared “closed” by an engineer because they allowed users to use an environment variable to disable the polling.
Fixing the Breakage from the AddTrust External CA Root Expiration
A lot of stuff on the Internet is currently broken on account of a Sectigo root certificate expiring at 10:48:38 UTC today. Generally speaking, this is affecting older, non-browser clients (notably OpenSSL 1.0.x) which talk to TLS servers which serve a Sectigo certificate chain ending in the expired certificate. See also this Twitter thread by Ryan Sleevi.
Vulnerabilities! We’ve got vulnerabilities here! … See? Nobody cares.
Jurassic Park is often (mistakenly) left out of the hacker movie canon. It clearly demonstrated the risk of an insider attack on control systems (Velociraptor rampage, amongst other tragedies…) nearly a decade ahead of the Maroochy sewage incident, it’s the first film I know of with a digital troll (“ah, ah, ah, you didn’t say the magic word!”), and Samuel L. Jackson correctly assesses the possible consequence of a hard reset (namely, everyone dying), resulting in his legendary “Hold on to your butts”. The quotable mayhem is seeded early in the film, when biotech spy Lewis Dodgson gives a sack of money to InGen’s Dennis Nedry to steal some dino DNA. Dodgson’s caricatured OPSEC (complete with trilby and dark glasses) is mocked by Nedry shouting, “Dodgson! Dodgson! We’ve got Dodgson here! See, nobody cares…” Three decades later, this quote still comes to mind* whenever conventional wisdom doesn’t seem to square with observed reality, and today we’re going to apply it to the oft-maligned world of Industrial Control System (ICS) security.
Three bugs in the Go MySQL Driver
Adding to this challenge, authzd is deployed to our Kubernetes clusters, where we’ve been experiencing issues with high latencies when opening new TCP connections, something that particularly affects the pooling of connections in the Go MySQL driver. One of the most dangerous lies that programmers tell themselves is that the network is reliable, because, well, most of the time the network is reliable. But when it gets slow or spotty, that’s when things start breaking, and we get to find out the underlying issues in the libraries we take for granted.
Good walkthrough of dealing with some unfriendly bugs.
Why is This Website Port Scanning me?
Recently, I was tipped off about certain sites performing localhost port scans against visitors, presumably as part of a user fingerprinting and tracking or bot detection. This didn’t sit well with me, so I went about investigating the practice, and it seems many sites are port scanning visitors for dubious reasons.
The case of the missing DNS packets
Troubleshooting is both a science and an art. The first step is to make a hypothesis about why something is behaving in an unexpected way, and then prove whether or not the hypothesis is correct. But before you can formulate a hypothesis, you first need to clearly identify the problem, and express it with precision. If the issue is too vague, then you need to brainstorm in order to narrow down the problem—this is where the “artistic” part of the process comes in.
Bypass Facebook SSL Certificate Pinning for iOS
If you tried to intercept requests from the Facebook app on iOS using a proxy, you will be hitting by their message alert The operation couldn’t be completed. That’s because of their SSL Certificate Pinning protection which must be bypassed to be able to intercept requests and responses from their server. In this blog, We will go through their iOS app trying to bypass their SSL Pinning.
And the same for instagram: https://www.cyclon3.com/bypass-instagram-ssl-certificate-pinning-for-ios
Hunting a Linux kernel bug
Earlier last year, we identified a firewall misconfiguration which accidentally dropped most network traffic. We expected resetting the firewall configuration to fix the issue, but resetting the firewall configuration exposed a kernel bug!
A Warm Welcome to ASN.1 and DER
This document provides a gentle introduction to the data structures and formats that define the certificates used in HTTPS. It should be accessible to anyone with a little bit of computer science experience and a bit of familiarity with certificates.
The Early History of Usenet
>November 2019 is, as best I can recall, the 40th anniversary of the conception of Usenet. (What’s Usenet? The Wikipedia article is ok but not perfect.) I should have written a proper paper; instead, there will (probably) be an irregular series of blog posts.
I didn’t notice the series concluded a while back, so if you were waiting to read the whole thing, it’s done.
How Tailscale works
There is one last question that comes up a lot: given that Tailscale creates a mesh “overlay” network (a VPN that parallels a company’s internal physical network), does a company have to switch to it all at once? Many BeyondCorp and zero-trust style products work that way. Or can it be deployed incrementally, starting with a small proof of concept?
Tailscale is uniquely suited to incremental deployments. Since you don’t need to install any hardware or any servers at all, you can get started in two minutes: just install the Tailscale node software onto two devices (Linux, Windows, macOS, iOS), login to both devices with the same user account or auth domain, and that’s it! They’re securely connected, no matter how the devices move around. Tailscale runs on top of your existing network, so you can safely deploy it without disrupting your existing infrastructure and security settings.
Building a more accurate time service at Facebook scale
As Facebook’s infrastructure has grown, time precision in our systems has become more and more important. We need to know the accurate time difference between two random servers in a data center so that datastore writes don’t mix up the order of transactions. We need to sync all the servers across many data centers with sub-millisecond precision. For that we tested chrony, a modern NTP server implementation with interesting features. During testing, we found that chrony is significantly more accurate and scalable than the previously used service, ntpd, which made it an easy decision for us to replace ntpd in our infrastructure. Chrony also forms the foundation of our Facebook public NTP service, available from time.facebook.com. In this post, we will share our work to improve accuracy from 10 milliseconds to 100 microseconds and how we verified these results in our timing laboratory.
The History of the URL
On the 11th of January 1982 twenty-two computer scientists met to discuss an issue with ‘computer mail’ (now known as email). Attendees included the guy who would create Sun Microsystems, the guy who made Zork, the NTP guy, and the guy who convinced the government to pay for Unix. The problem was simple: there were 455 hosts on the ARPANET and the situation was getting out of control.