The Unreasonable Effectiveness of JPEG: A Signal Processing Approach
The JPEG algorithm is rather complex and in this video, we break down the core parts of the algorithm, specifically color spaces, YCbCr, chroma subsampling, the discrete cosine transform, quantization, and lossless encoding. The majority of the focus is on the mathematical and signal processing insights that lead to advancements in image compression and the big themes in compression as a whole that we can take away from it.
Micro-Optimizing .tar.gz Archives by Changing File Order
A few weeks ago, I was doing something with a sizeable .tar.gz file, and wondered how the order of files affected the process. I’m not that knowledgable about compression, but I know that gzip uses a sliding window in which it looks for opportunities to compress repeating chunks of text. If you give it highly repetitive text, it does well, if you give it random data, it will probably give you a bigger file than when you started. So reordering files seems like it could matter.
Bzip2 And The Cve That Wasn’t
Compiling with the GCC sanitizers and then fuzzing the resulting binaries might find real bugs. But not all such bugs are security issues. When a CVE is filed there is some pressure to treat such an issue with urgency and push out a fix as soon as possible. But taking your time and making sure an issue can be replicated/exploited without the binary being instrumented by the sanitizer is often better.
I don’t think anything went wrong here, but some interesting details.
clamav: denial of service through "better zip bomb"
Recently David Fifield presented a new variant of a ZIP bomb where by using overlapping segments he was able to achieve very high compression ratios (42kb->5GB, 10MB->281TB).
However David Fifield commented in the bug report  that the fix is incomplete, by using some slight variations of his methods he could bypass the fix.
This shouldn’t be anything new, but... oops. Plus some commentary about age browsing, etc.
How to make compressed file quines, step by step
Much of the credit goes to folks much smarter than myself (they will be introduced); this tutorial is meant to curate previous work and literature as much as it is for myself to educate you. The goal here is to allow for any curious, technically-minded newcomer to make sense of all the concepts involved in creating compression quines.
A better zip bomb
This article shows how to construct a non-recursive zip bomb that achieves a high compression ratio by overlapping files inside the zip container. “Non-recursive” means that it does not rely on a decompressor’s recursively unpacking zip files nested within zip files: it expands fully after a single round of decompression. The output size increases quadratically in the input size, reaching a compression ratio of over 28 million (10 MB → 281 TB) at the limits of the zip format. Even greater expansion is possible using 64-bit extensions. The construction uses only the most common compression algorithm, DEFLATE, and is compatible with most zip parsers.
What is WofCompressedData?
The documentation for wofapi.h says merely “This header is used by Data Access and Storage.” For more information, it refers you to another web page that contains no additional information. WOF stands for Windows Overlay Filter, which is a nice name that doesn’t really tell you much about what it does or what it’s for.
Changing the native NTFS file compression would be a disk format breaking change, which is not something taken lightly. Doing it as a filter provides much more flexibility. The downside is that if you mount the volume on a system that doesn’t support the Windows Overlay Filter, all you see is an empty file. Fortunately, WOF is used only for system-installed files, and if you are mounting the volume onto another system, it’s probably for data recovery purposes, so you’re interested in user data, not system files.
Software-defined far memory in warehouse scale computers
Unraveling the JPEG
JPEG images are everywhere in our digital lives, but behind the veil of familiarity lie algorithms that remove details that are imperceptible to the human eye. This produces the highest visual quality with the smallest file size—but what does that look like? Let’s see what our eyes can’t see!
This article is about how to decode a JPEG image. In other words, it’s about what it takes to convert the compressed data stored on your computer to the image that appears on the screen. It’s worth learning about not just because it’s important to understand the technology we all use everyday, but also because, as we unravel the layers of compression, we learn a bit about perception and vision, and about what details our eyes are most sensitive to. It’s also just a lot of fun to play with images this way.
The main focus of the v1.4.0 release is the stabilization of the advanced API.
Zstd’s fastest compression level just got faster! Thanks to ideas from Intel’s igzip and @gbtucker, we’ve made level 1, zstd’s fastest strategy, 6-8% faster in most scenarios.
Modern LZ Compression
This article walks through the components of a modern LZ compressor.
LZ compression has been around for a long time, but there are still major improvements being made. The deflate algorithm is perhaps the most widely used, implemented initially in PKZIP, and these days finding broad usage in gzip. Modern incarnations are much more powerful, and use a wide array of new tricks. Two examples of the next generation are zstd and lzma. For a comprehensive overview of open source LZ compressors, check out lzbench.
Imagine you compressed a file using your favorite compression algorithm, but you realize that there was a typo in the file. You then correct it, adding a single bit to the original file.
Compress it again and you get a much larger compressed file… for just a one-bit difference! Much compression algorithms do not have this strange behavior, but for LZ’78, one of the most famous of them, this surprising scenario might well happen. This is what we will overview in this post.
A look at VDO, the new Linux compression layer
Now with Virtual Data Optimizer (VDO), all required pieces for a transparent compression/deduplication layer are available in the just-released Red Hat Enterprise Linux 7.5. With this technology, it is possible to trade CPU/RAM resources for disk space.
Advertising, but of interest.
An ode to pack: gzip’s forgotten decompressor
Another even more obscure tool it could replace was compress‘s own predecessor, pack. This rather loosely defined collection of only partially compatible formats is why compress had to use a capital Z in its extension. pack came first, and offered straight Huffman coding with a .z extension.
Faster Base64 Encoding and Decoding Using AVX2 Instructions
We estimate that billions of base64 messages are decoded every day. We are motivated to improve the efficiency of base64 encoding and decoding. Compared to state-of-the-art implementations, we multiply the speeds of both the encoding (~10x) and the decoding (~7x). We achieve these good results by using the single-instruction-multiple-data (SIMD) instructions available on recent Intel processors (AVX2).
Introducing Support for Brotli Compression
Brotli is a relatively new compression algorithm. It’s quite beneficial for web clients, with decompression performance comparable to gzip while significantly improving the compression ratio. These are powerful properties for serving static content such as fonts and html pages. Thus, if you currently use gzip or Deflate as compression for your web site (or have never used compression) you should give Brotli a try.
unarcrypto is an educational tool to depict cryptography usage in zip, rar and 7zip archives
The design, implementation and deployment of a system to transparently compress hundreds of petabytes of image files for a file storage service
The Dropbox JPEG recompressor.
Followup: design and implementation of a system to compress overly long paper titles.
Announcing Guetzli: A New Open Source JPEG Encoder
That’s why we’re excited to announce Guetzli, a new open source algorithm that creates high quality JPEG images with file sizes 35% smaller than currently available methods, enabling webmasters to create webpages that can load faster and use even less data.
Introduction to Video Coding
It says introduction, but it’s 170 text heavy slides.