node.example.com Is An IP Address
This takes a bit to get to the punchline, but man, good old duck typing for the win.
It turns out that, under certain conditions, the ipaddress module can create IPv6 addresses from raw bytes. My assumption is that it offers this behavior as a convenient way to parse IP addresses from data fresh off the wire.
Does node.example.com meet those certain conditions? You bet it does. Because we’re using Python 2 it’s just bytes and it happens to be 16 characters long.
Never Run ‘python’ In Your Downloads Folder
Python can execute code. Make sure it executes only the code you want it to.
Not exclusive to python either.
Hacking With Environment Variables
Interesting environment variables to supply to scripting language interpreters
Latency in Asynchronous Python
This week I was debugging a misbehaving Python program that makes significant use of Python’s asyncio. The program would eventually take very long periods of time to respond to network requests. My first suspicion was a CPU-heavy coroutine hogging the thread, preventing the socket coroutines from running, but an inspection with pdb showed this wasn’t the case. Instead, the program’s author had made a couple of fundamental mistakes using asyncio. Let’s discuss them using small examples.
The Hidden Number Problem
The Hidden Number Problem (HNP) is a problem that poses the question: Are the most signficant bits of a Diffie-Hellman shared key as hard to compute as the entire secret? The original problem was defined in the paper “Hardness of computing the most significant bits of secret keys in Diffie-Hellman and related schemes” by Dan Boneh and Ramarathnam Venkatesan.
In this paper Boneh and Venkatesan demonstrate that a bounded number of most signifcant bits of a shared secret are as hard to compute as the entire secret itself. They also demonstrate an efficient algorithm for recovering secrets given a significant enough bit leakage. This notebook walks through some of the paper and demonstrates some of the results.
Mercurial's Journey to and Reflections on Python 3
Speaking as a maintainer of Mercurial and an avid user of Python, I feel like the experience of making Mercurial work with Python 3 is worth sharing because there are a number of lessons to be learned.
This post is logically divided into two sections: a mostly factual recount of Mercurial’s Python 3 porting effort and a more opinionated commentary of the transition to Python 3 and the Python language ecosystem as a whole. Those who don’t care about the mechanics of porting a large Python project to Python 3 may want to skip the next section or two
snek - Python from PowerShell
Snek is a cross-platform PowerShell module for integrating with Python. It uses the Python for .NET library to load the Python runtime directly into PowerShell. Using the dynamic language runtime, it can then invoke Python scripts and modules and return the result directly to PowerShell as managed .NET objects.
PyPy's new JSON parser
In the last year or two I have worked on and off on making PyPy’s JSON faster, particularly when parsing large JSON files. In this post I am going to document those techniques and measure their performance impact.
Python is not context free
The interesting thing about Python’s syntax is, of course, its use of indentation to indicate program structure.
A nice review of interaction between lexing and parsing.
Beginner Problems With TCP & The socket Module in Python
Your operating system will deceive you and re-assemble the string you sock.recv(n) differently from the ones you sock.send(data). But here is the deceptive part. It will work sometimes, but not always. These bugs will be difficult to chase. If you have two programs communicating over TCP via the loopback device in your operating system (the virtual network device with IP 127.0.0.1), then the data does not leave your RAM, and packets are never fragmented to fit into the maximum size of an Ethernet frame or 802.11 WLAN transmission. The data arrives immediately because it’s already there, and the other side gets to read via sock.recv(n) exactly the bytestring you sent over sock.send(data). If you connect to localhost via IPv6, the maximum packet size is 64 kB, and all the packets are already there to be reassembled into a bytestream immediately! But when you try to run the same code over the real Internet, with lag and packet loss, or when you are unlucky with the multitasking/scheduling of your OS, you will either get more data than you expected, leftover data from the last sock.send(data), or incomplete data.
Not strictly a python problem, either.
Looking inside the box
This blog post talks about reverse engineering the Dropbox client, breaking its obfuscation mechanisms, de-compiling it to Python code as well as modifying the client in order to use debug features which are normally hidden from view. If you’re just interested in relevant code and notes please scroll to the end. As of this writing it is up to date with the current versions of Dropbox which are based on the CPython 3.6 interpreter.
Python Project Tooling explained
For this reason I’ve decided to create a post that lists the most important tools, when and why they are used and what problem they solve. I will try to explain with simple words how you should approach each one of these tools. If a tool is here, it means that, as a Python programmer, you’re supposed to at least know its existence. I will list only tools that can be applied to any project or workflow and that you should consider every time you start a new project. This doesn’t mean you always have to use each one of them on every single project. Too much tooling can be easily be an overkill and become hard to maintain in some cases.
Go Concurrency from the Ground Up
Sometimes the best way to learn something is to build it. This guide will walk you through how to reproduce Go’s concurrency features in another programming language.
PyPy v7.1 released
This release, coming fast on the heels of 7.0 in February, finally merges the internal refactoring of unicode representation as UTF-8. Removing the conversions from strings to unicode internally lead to a nice speed bump.
Python Decorators: Syntactic Artificial Sweetener
Python has a feature called function decorators. With a little bit of syntax, the behavior of a function or class can be modified in useful ways. Python comes with a few decorators, but most of the useful ones are found in third-party libraries.
The problem is that the Python language reference doesn’t parse an expression after @. It matches a very specific pattern that just so happens to look like a Python expression. It’s not syntactic sugar, it’s syntactic artificial sweetener!
Even a Feature That You Do Not Use Can Bite You
In this case, an obscure bit of python syntax. Not quite what I predicted, but maybe that’s the point.
The reason is that it is actually valid Python code that uses variable annotations, which is a feature introduced in Python 3.6. In the previous versions of Python, the code would have raised SyntaxError.
Oneliner-izer: An Exercise in Constrained Coding
We’ll describe the ideas and implementation behind Oneliner-izer, a “”compiler“” which can convert most Python 2 programs into one line of code. As we discuss how to construct each language feature within this unorthodox constraint, we’ll explore the boundaries of what Python permits and encounter some gems of functional programming – lambda calculus, continuations, and the Y combinator.
PyPy for low-latency systems
Recently I have merged the gc-disable branch, introducing a couple of features which are useful when you need to respond to certain events with the lowest possible latency.
Removing a recursion in Python
A recent question on Stack Overflow got me thinking about how to turn a recursive algorithm into an iterative one, and it turns out that Python is a pretty decent language for this.
Of course, the technique that I’m going to show you is not necessarily “Pythonic”. There are probably more Pythonic solutions using generators and so on. What I’d like to show here is that to remove this sort of recursion, you can do so by re-organizing the code using a series of small, careful refactorings until the program is in a form where removing the recursion is easy. Let’s see first how to get the program into that form.
And part 2: https://ericlippert.com/2018/12/17/removing-a-recursion-in-python-part-2/
Real time numbers recognition (MNIST) on an iPhone with CoreML from A to Z
Learn how to build and train a deep learning network to recognize numbers (MNIST),how to convert it in the CoreML format to then deploy it on your iPhoneX and make it recognize numbers in realtime!