Defense at Scale
> Last year, my colleague Chris Rohlf gave a keynote at BSidesNOLA entitled “Offense at Scale”. Offense sounds fun. Pwn all the things. And you’re always going to win! And normally I’m a big fan of being massively offensive. Unfortunately, I find myself on the defense when it comes to information security.
> Here’s how you defend at scale. Can’t be done. The end. Everything’s fucked. You’re pwned.
Plenty of good points here. Also a fun read.
Migrating From Cloudflare
> Okay so here’s the thing: Cloudflare isn’t just the CDN provider for the instance, it is also the domain’s nameserver. That means that it holds all the DNS records that point mastodon.technology to the various IP addresses used for HTTP requests, email, and even public DKIM keys for mail server verification. These DNS settings are really, really important. If they get messed up, everything about the instance can break.
> So I split up the migration from Cloudflare to BunnyCDN into two phases: first migrate the CDN provider, and then migrate the DNS provider. Getting this right is really important, and I mostly did okay, but hopefully you can learn from my experiences.
Building interactive SSH applications
> Writing interactive SSH applications is actually pretty easy, but it does require some knowledge of the pieces involved and a little bit of general Unix literacy
You should not run your mail server because mail is hard.
> It is therefore very important that we don’t let the myth propagate further. Our best interest is to have a WIDE variety of mail hosts and providers, small and big, commercial and not. We must not allow the number of mail hosts to shrink, they must increase so the e-mail address space out of the control of Big Mailer Corps remains significant.
One core problem with DNSSEC
> One fundamental problem of DNSSEC today is that it suffers from the false positive problem, the same one that security alerts suffer from. In practice today, for almost all people almost all of the time, a DNSSEC failure is not a genuine attack; it is a configuration mistake, and the configuration mistake is almost never on the side making the DNS query. This means that almost all of the time, DNSSEC acts by stopping you from doing something safe that you want to do and further, you can’t fix the DNSSEC problem except by turning off DNSSEC, because it’s someone else’s mistake (in configuration, in operation, or in whatever).
DragonFly kcollect(8) improvements
> DragonFly has a utility called kcollect(8), for gathering about the last day’s worth of kernel statistics. It recently gained some extra flags and details, and should work well if you want to collect stats in a low-impact way.
Killing a process and all of its descendants
> Unix-like operating systems have sophisticated process relationships. Parent-child, process groups, sessions, and session leaders. However, the details are not uniform across operating systems like Linux and macOS. POSIX compliant operating systems support sending signals to process groups with a negative PID number.
I think some of this is not entirely correct, but as noted, it’s a complicated subject.
Possibly timely items from my reliability list
Time zones, leap seconds, oh my.
I like this much more than the typical falsehoods list because it actually explains the problem and gives a hint about the solution.
OpenBSD ttyplot examples
> I said I will rewrite ttyplot examples to make them work on OpenBSD. Here they are, but a small notice before:
Couple caveats, mostly want current.
Some items from my "reliability list"
> I’ll list some of them here and some of the thinking behind them. Just about everything here has happened at some point in time, and probably has happened more than once... way more than once.
I like a lot of this. Very much.
> On the other hand, if you only need 53 bits of your 64 bit numbers, and enjoy blowing CPU on ridiculously inefficient marshaling and unmarshaling steps, hey, it’s your funeral.
OpenSSH Taking Minutes To Become Available, Booting Takes Half An Hour ... Because Your Server Waits For A Few Bytes Of Randomness
> Basically as of now the entropy file saved as /var/lib/systemd/random-seed will not - drumroll - add entropy to the random pool when played back during boot. Actually it will. It will just not be accounted for. So Linux doesn’t know. And continues blocking getrandom(). This is obviously different from SysVinit times2 when /var/lib/urandom/random-seed (that you still have lying around on updated systems) made sure the system carried enough entropy over reboot to continue working right after enough of the system was booted.
And then... it just kinda keeps getting worse. The problem is understandable, the inability to resolve it less so.
Google Groups entirely ignores SMTP time rejections
> Google Groups ignored this rejection and began sending email messages from the group/mailing list to my spamtrap address. Each of these messages was rejected at SMTP time, and each of them contained a unique MAIL FROM address (a VERP), which good mailing list software uses to notice delivery failures and unsubscribe addresses. Google Groups is, of course, not good mailing list software, since it entirely ignored the rejections. I expect that this increases the metrics of things like ‘subscribers to Google Groups’ and ‘number of active Google Groups’ and others that the department responsible for Google Groups is rewarded for. Such is the toxic nature of rewarding and requiring ‘engagement’, especially without any care for the details.
How Bash decides it's being invoked through sshd and sources your .bashrc
> In practice, a non-interactive Bash decides that it is being invoked by SSHD if either $SSH_CLIENT or $SSH2_CLIENT are defined in the environment. In a robotic sense this is perfectly correct, since OpenSSH’s sshd puts $SSH_CLIENT in the environment when you do ‘ssh host command’. In practice it is wrong, because OpenSSH sets $SSH_CLIENT all the time, including for logins. So if you use SSH to log in somewhere, $SSH_CLIENT will be set in your shell environment, and then any non-interactive Bash will decide that it should source ~/.bashrc.
This seems incredibly messy.
Building Facebook’s service encryption infrastructure
> In this post, we’ll talk about how we migrated our encryption infrastructure in data centers from the Kerberos authentication protocol to TLS. Optimizing for operability and performance, while still satisfying the right security model for each service, required navigating difficult trade-offs. By sharing our experiences, we hope to show how we think about our encryption infrastructure and help others as they think through their own implementation.
It’s the middle of the night. Do you know who your iPhone is talking to?
> On a recent Monday night, a dozen marketing companies, research firms and other personal data guzzlers got reports from my iPhone. At 11:43 p.m., a company called Amplitude learned my phone number, email and exact location. At 3:58 a.m., another called Appboy got a digital fingerprint of my phone. At 6:25 a.m., a tracker called Demdex received a way to identify my phone and sent back a list of other trackers to pair up with.
What I Learned Trying To Secure Congressional Campaigns
> I don’t believe I accomplished much, but I made so many friends along the way! And I learned a lot about the idiosyncratic world of Congressional campaigns; knowledge that I want to now hand over to you, the next person willing to take a swing at this piñata of futility.
> The candidate was hardest person to secure. They were too busy to come to the training. They didn’t want to move off their Loudong SB250 phone because it had all their favorite Flash games from the Yahoo store on it. Three different antivirus programs competed for dominion over their Windows 7 laptop.
> Ideally, there would be a billing model where the training is free, but the campaign gets charged thousands of dollars for ignoring it.
My new favorite tool for looking at TLS things is certigo
> For a long time I’ve used the OpenSSL command line tools to do things like looking at certificates and chasing certificate chains (although OpenSSL is no longer what you want to use to make self-signed certificates). This works, and is in many ways the canonical and most complete way to do this sort of stuff, but if you’ve ever used the openssl command and its many sub-options you know that it’s kind of a pain in the rear. As a result of this, for some years now I’ve been using Square’s certigo command instead.
> Put this in your .tmux.conf on both the local and the remote machines. You will have to edit it a bit if you have your keys mapped differently. Operation goes like this: press Shift-Up and you will be controlling the inner (remote) tmux, press Shift-Down to control the outer one.
Site to Site WireGuard: Part 1
> In this blogpost series I’m going to go over how I created a site to site Virtual Private Network (abbreviated as VPN) for all of my personal devices. The best way to think about what this is doing is creating a logical (or imaginary) network on top of the network infrastructure that really exists. This allows me to expose private services so that only people I trust can even know how to connect to them. For extra convenience and battery saving power, I’m going to use WireGuard as the VPN protocol.
Using an OpenBSD Router with AT&T U-Verse
> I upgraded to AT&T’s U-verse Gigabit internet service in 2017 and it came with an Arris BGW-210 as the WiFi AP and router. The BGW-210 is not a terrible device, but I already had my own Airport Extreme APs wired throughout my house and an OpenBSD router configured with various things, so I had no use for this device. It’s also a potentially-insecure device that I can’t upgrade or fully disable remote control over.
> Fully removing the BGW-210 is not possible as we’ll see later, but it is possible to remove it from the routing path. This is how I did it with OpenBSD.