SSH and User-mode IP WireGuard
For a couple hundred lines of code (not counting the entire user-mode Linux you’ll be pulling in from gVisor, HEY! Dependencies! What are you gonna do!) you can bring up a new, cryptographically authenticated network, any time you want to, in practically any program.
There really are some fun libraries out there if you want to build something crazy.
XTerm: It's Better Than You Thought
Some useful config options showing off flexibility beyond the basics.
How to make Bash fail badly on Ubuntu 16.04 by typo'ing a command name
The simple thing to say about this is that it only happens on Ubuntu 16.04, not on 18.04 or 20.04, and it happens because Ubuntu’s normal /etc/bash.bashrc defines a command_not_found_handle function that winds up running a helper program to produce this ‘did you mean’ report. The helper program comes from the command-not-found package, which is installed because it’s Recommended by ubuntu-standard.
Introducing the In-the-Wild Series
Ok Google: please publish your DKIM secret keys
This post is about the situation with Domain Keys Identified Mail (DKIM), a harmless little spam protocol that has somehow become a monster. My request is simple and can be summarized as follows: Dear Google: would you mind rotating and publishing your DKIM secret keys on a periodic basis? This would make the entire Internet quite a bit more secure, by removing a strong incentive for criminals to steal and leak emails. The fix would cost you basically nothing, and would remove a powerful tool from hands of thieves.
Never Run ‘python’ In Your Downloads Folder
Python can execute code. Make sure it executes only the code you want it to.
Not exclusive to python either.
How CDNs Generate Certificates
Obviously, to do stuff like this, you need to generate certificates. The reasonable way to do that in 2020 is with LetsEncrypt. We do that for our users automatically, but “it just works” makes for a pretty boring writeup, so let’s see how complicated and meandering I can make this.
It’s time to talk about certificate infrastructure.
Classic ThinkPad Thermal Paste Change
Those who know me know that I am a bit fan of the oldschool Lenovo ThinkPad laptops with real 7-row keyboards. I own several *20 models from 2011 including W520, T420s and X220 ones. They still rock when it comes to ‘laptop computing’ and they are dirt cheap on any auction platform. They only got one flaw … that thermal compound on CPU (and sometimes GPU) gets older a lot faster then these laptops.
Fakecracker: NetBSD as a Function Based MicroVM
This is fun and all, but we can’t really talk about security only with chroot, and the Firecracker solution seemed about right for this matter, yet the overall NetBSD boot process was a bit too long for my taste. So how exactly can we significantly improve NetBSD‘s boot speed?
Fixing the Breakage from the AddTrust External CA Root Expiration
A lot of stuff on the Internet is currently broken on account of a Sectigo root certificate expiring at 10:48:38 UTC today. Generally speaking, this is affecting older, non-browser clients (notably OpenSSL 1.0.x) which talk to TLS servers which serve a Sectigo certificate chain ending in the expired certificate. See also this Twitter thread by Ryan Sleevi.
ZFS versus RAID: Eight Ironwolf disks, two filesystems, one winner
We exhaustively tested ZFS and RAID performance on our Storage Hot Rod server.
systemd, 10 years later: a historical and technical retrospective
10 years ago, systemd was announced and swiftly rose to become one of the most persistently controversial and polarizing pieces of software in recent history, and especially in the GNU/Linux world. The quality and nature of debate has not improved in the least from the major flame wars around 2012-2014, and systemd still remains poorly understood and understudied from both a technical and social level despite paradoxically having disproportionate levels of attention focused on it.
I am writing this essay both for my own solace, so I can finally lay it to rest, but also with the hopes that my analysis can provide some context to what has been a decade-long farce, and not, as in Benno Rice’s now famous characterization, tragedy.
Why strace doesn't work in Docker
But I wasn’t interested in fixing it, I wanted to know why it happens. So why does strace not work, and why does --cap-add=SYS_PTRACE fix it?
OpenBSD's 'spinning' CPU time category
Since this dates from early 2018, I believe it’s in everything from OpenBSD 6.4 onward. It’s definitely in OpenBSD 6.6. This new CPU time category is supported in OpenBSD’s versions of top and systat, but it is not explicitly broken out by vmstat; in fact vmstat’s ‘sy’ time is actually the sum of OpenBSD ‘system’, ‘interrupt’, and ‘spinning’. Third party tools may or may not have been updated to add this new category.
oxbar - configurable X11 status bar for OpenBSD
oxbar is a X11 status bar for OpenBSD showing various system stats. It has a configurable display and works out-of-the-box on most modern window managers in an intuitive fashion. oxbar supports FreeType font rendering and styling, true transparency & alpha blending on all UI components (including the root window), and a simple configuration format that can concisely support multiple themes.
My infrastructure as of 2019
The goal for my infrastructure is to run the services I need. While a lot of people in the homelab community experiment and play with software for its own sake, I actively use the stuff I host. When I stop, I kill the service (though I’m not as proficient at this as Google). These are my production systems, and when one of them is down, I do miss it.
Comparing Alternatives to top Written in Rust
Recently I aliased top to ytop. Then I became aware of bottom, and zenith. These are all terminal based system monitoring tools that you might use instead of top. In this post I set out to compare them.
How Tailscale works
There is one last question that comes up a lot: given that Tailscale creates a mesh “overlay” network (a VPN that parallels a company’s internal physical network), does a company have to switch to it all at once? Many BeyondCorp and zero-trust style products work that way. Or can it be deployed incrementally, starting with a small proof of concept?
Tailscale is uniquely suited to incremental deployments. Since you don’t need to install any hardware or any servers at all, you can get started in two minutes: just install the Tailscale node software onto two devices (Linux, Windows, macOS, iOS), login to both devices with the same user account or auth domain, and that’s it! They’re securely connected, no matter how the devices move around. Tailscale runs on top of your existing network, so you can safely deploy it without disrupting your existing infrastructure and security settings.
Building a more accurate time service at Facebook scale
As Facebook’s infrastructure has grown, time precision in our systems has become more and more important. We need to know the accurate time difference between two random servers in a data center so that datastore writes don’t mix up the order of transactions. We need to sync all the servers across many data centers with sub-millisecond precision. For that we tested chrony, a modern NTP server implementation with interesting features. During testing, we found that chrony is significantly more accurate and scalable than the previously used service, ntpd, which made it an easy decision for us to replace ntpd in our infrastructure. Chrony also forms the foundation of our Facebook public NTP service, available from time.facebook.com. In this post, we will share our work to improve accuracy from 10 milliseconds to 100 microseconds and how we verified these results in our timing laboratory.
Your Circuit Breaker is Misconfigured
Circuit breakers are an incredibly powerful tool for making your application resilient to service failure. But they aren’t enough. Most people don’t know that a slightly misconfigured circuit is as bad as no circuit at all! Did you know that a change in 1 or 2 parameters can take your system from running smoothly to completely failing?