A better zip bomb

Summary

This article shows how to construct a
non-recursive zip bomb
that achieves a high compression ratio by
overlapping files inside the zip container.
“Non-recursive” means that it does not rely on
a decompressor’s recursively unpacking zip files nested within zip files:
it expands fully after a single round of decompression.
The output size increases quadratically in the input size,
reaching a compression ratio of over 28 million
(10 MB → 281 TB)
at the limits of the zip format.
Even greater expansion is possible using
64-bit extensions.
The construction uses only the most common compression algorithm, DEFLATE,
and is compatible with most zip parsers.

Source code:

git clone https://www.bamsoftware.com/git/zipbomb.git

zipbomb-20190702.zip

Data and source for figures:

git clone https://www.bamsoftware.com/git/zipbomb-paper.git

Compression bombs that use the zip format
must cope with the fact that DEFLATE,
the compression algorithm most commonly supported by zip parsers,
cannot achieve a compression ratio greater than 1032.
For this reason, zip bombs typically rely on recursive decompression,
nesting zip files within zip files to get an extra factor of 1032 with each layer.
But the trick only works on implementations that
unzip recursively, and most do not.
The best-known zip bomb,
42.zip,
expands to a formidable 4.5 PB
if all six of its layers are recursively unzipped,
but a trifling 0.6 MB at the top layer.
Zip quines,
like those of Ellingsen
and Cox,
which contain a copy of themselves
and thus expand infinitely if recursively unzipped,
are likewise perfectly safe to unzip once.

This article shows how to construct a non-recursive zip bomb
whose compression ratio surpasses the DEFLATE limit of 1032.
It works by overlapping files inside the zip container,
in order to reference a “kernel” of highly compressed data
in multiple files,
without making multiple copies of it.
The zip bomb’s output size grows quadratically in the input size; i.e.,
the compression ratio gets better as the bomb gets bigger.
The construction depends on features of both zip and DEFLATE—it
is not directly portable to other file formats or compression algorithms.
It is compatible with most zip parsers,
the exceptions being “streaming” parsers that
parse in one pass without first consulting the zip file’s central directory.
We try to balance
two conflicting goals:

• Maximize the compression ratio.
  We define the compression ratio as the the sum of the sizes
  of all the files contained the in the zip file,
  divided by the size of the zip file itself.
  It does not count filenames or other filesystem metadata,
  only contents.
• Be compatible.
  Zip is a tricky format and parsers differ, especially
  around edge cases and optional features.
  Avoid taking advantage of tricks that only work with certain parsers.
  We will remark on certain ways to increase the efficiency of the zip bomb
  that come with some loss of compatibility.

Structure of a zip file

A zip file consists of
a central directory which references
files.

The central directory is at the end of the zip file.
It is a list of central directory headers.
Each central directory header contains metadata for a single file,
like its filename and CRC-32 checksum,
and a backwards pointer to a local file header.
A central directory header is 46 bytes long,
plus the length of the filename.

A file consists of a local file header
followed by compressed file data.
The local file header is 30 bytes long,
plus the length of the filename.
It contains a redundant copy
of the metadata from the central directory header,
and the compressed and uncompressed sizes of the file data
that follows.
Zip is a container format, not a compression algorithm.
Each file’s data is compressed
using an algorithm specified in the metadata—usually DEFLATE.

This description of the zip format omits many details that
are not needed for understanding the zip bomb.
For full information,
refer to
section 4.3 of APPNOTE.TXT
or The structure of a PKZip file by Florian Buchholz,
or see the source code.

The first insight: overlapping files

By compressing a long string of repeated bytes,
we can produce a kernel
of highly compressed data.
By itself, the kernel’s compression ratio cannot
exceed the DEFLATE limit of 1032,
so we want a way to reuse the kernel in many files,
without making a separate copy of it in each file.
We can do it by overlapping files:
making many central directory headers point to
a single file, whose data is the kernel.

Let’s look at an example to see how this construction affects the compression ratio.
Suppose the kernel is 1000 bytes and
decompresses to 1 MB.
Then the first MB of output “costs”
1078 bytes of input:

• 31 bytes for a local file header (including a 1-byte filename)
• 47 bytes for a central directory header (including a 1-byte filename)
• 1000 bytes for the kernel itself

But every 1 MB of output
after the first costs only 47 bytes—we don’t need another local file header or another copy of the kernel,
only an additional central directory header.
So while the first reference of the kernel has a compression ratio of
1 000 000 / 1078 ≈ 928,
each additional reference pulls the ratio closer to
1 000 000 / 47 ≈ 21 277,
A bigger kernel raises the ceiling.

The problem with this idea is a lack of compatibility.
Because many central directory headers point to a single local file header,
the metadata—specifically the filename—cannot match for every file.
Some parsers balk at that.
Info-ZIP UnZip
(the standard Unix unzip program)
extracts the files, but with warnings:

$ unzip overlap.zip
  inflating: A
B:  mismatching "local" filename (A),
         continuing with "central" filename version
  inflating: B
...

And the Python zipfile module
throws an exception:

$ python3 -m zipfile -e overlap.zip .
Traceback (most recent call last):
...
__main__.BadZipFile: File name in directory 'B' and header b'A' differ.

Next we will see how to modify the construction
for consistency of filenames,
while still retaining most of the advantage
of overlapping files.

The second insight: quoting local file headers

We need to separate the local file headers for each file,
while still reusing a single kernel.
Simply concatenating all the local file headers does not work,
because the zip parser will find a local file header
where it expects to find the beginning of a DEFLATE stream.
But the idea will work, with a minor modification.
We’ll use a feature of DEFLATE, non-compressed blocks,
to “quote” local file headers
so that they appear to be part of the same DEFLATE stream
that terminates in the kernel.
Every local file header
(except the first)
will be interpreted in two ways:
as code (part of the structure of the zip file)
and as data (part of the contents of a file).

A DEFLATE stream is a sequence of
blocks,
where each block may be compressed or non-compressed.
Compressed blocks are what we usually think of;
for example the kernel is one big compressed block.
But there are also non-compressed blocks,
which start with a
5-byte header
with a length field that means simply, “output the next n bytes verbatim.”
Decompressing a non-compressed block means only stripping the 5-byte header.
Compressed and non-compressed blocks may be intermixed freely
in a DEFLATE stream.
The output is the concatenation of the results of
decompressing all the blocks in order.
The “non-compressed” notion only has meaning at the DEFLATE layer;
the file data still counts as “compressed” at the zip layer,
no matter what kind of blocks are used.

It is easiest to understand this quoted-overlap construction from the inside out,
beginning with the last file and working backwards to the first.
Start by inserting the kernel, which will form the end of file data for every file.
Prepend a local file header LFH_N
and add a central directory header CDH_N that points to it.
Set the “compressed size” metadata field in the LFH_N and CDH_N to the compressed size of the kernel.
Now prepend a 5-byte non-compressed block header (colored green in the diagram)
whose length field is equal to the size of LFH_N.
Prepend a second local file header LFH_N−1
and add a central directory header CDH_N−1 that points to it.
Set the “compressed size” metadata field in both of the new headers to the compressed size of the kernel
plus the size of the non-compressed block header (5 bytes)
plus the size of LFH_N.

At this point the zip file contains two files, named “Y” and “Z”.
Let’s walk through what a zip parser would see while parsing it.
Suppose the compressed size of the kernel is 1000 bytes
and the size of LFH_N is 31 bytes.
We start at CDH_N−1
and follow the pointer to LFH_N−1.
The first file’s filename is “Y” and
the compressed size of its file data is 1036 bytes.
Interpreting the next 1036 bytes as a DEFLATE stream,
we first encounter the 5-byte header of a non-compressed block
that says to copy the next 31 bytes.
We write the next 31 bytes,
which are LFH_N,
which we decompress and append to file “Y”.
Moving on in the DEFLATE stream, we find a compressed block (the kernel),
which we decompress to file “Y”.
Now we have reached the end of the compressed data and are done with file “Y”.
Proceeding to the next file, we follow the pointer from CDH_N
to LFH_N and find a file named “Z”
whose compressed size is 1000 bytes.
Interpreting those 1000 bytes as a DEFLATE stream,
we immediately encounter a compressed block (the kernel again)
and decompress it to the file “Z”.
Now we have reached the end of the final file and are done.
The output file “Z” contains the decompressed kernel;
the output file “Y” is the same, but additionally prefixed by
the 31 bytes of
LFH_N.

We complete the construction by repeating the quoting procedure
until the zip file contains the desired number of files.
Each new file adds a central directory header,
a local file header,
and a non-compressed block to quote the immediately succeeding local file header.
Compressed file data is generally a chain of DEFLATE non-compressed blocks
(the quoted local file headers)
followed by the compressed kernel.
Each byte in the kernel contributes about
1032 N to the output size,
because each byte is part of all N files.
The output files are not all the same size:
those that appear earlier in the zip file
are larger than those that appear later,
because they contain more quoted local file headers.
The contents of the output files are not particularly meaningful,
but no one said they had to make sense.

This quoted-overlap construction has better compatibility
than the full-overlap construction of the previous section,
but the compatibility comes at the expense of the compression ratio.
There, each added file cost only a central directory header;
here, it costs a central directory header,
a local file header,
and another 5 bytes for the quoting header.

Optimization

Now that we have the basic zip bomb construction,
we will try to make it as efficient as possible.
We want to answer two questions:

• For a given zip file size, what is the maximum compression ratio?
• What is the maximum compression ratio, given the limits of the zip format?

Kernel compression

It pays to compress the kernel as densely as possible,
because every decompressed byte gets magnified by a factor of N.
To that end, we use a custom DEFLATE compressor
called bulk_deflate,
specialized for compressing
a string of repeated bytes.

All decent DEFLATE compressors will approach a compression ratio of 1032
when given an infinite stream of repeating bytes,
but we care more about specific finite sizes
than asymptotics.
bulk_deflate compresses more data
into the same space than the general-purpose compressors:
about 26 kB more than zlib and Info-ZIP,
and about 15 kB more than
Zopfli,
a compressor that trades speed for density.

The price of bulk_deflate’s high compression ratio is a lack of generality.
bulk_deflate can only compress strings of a single repeated byte,
and only those of specific lengths,
namely 517 + 258 k for integer k ≥ 0.
Besides compressing densely, bulk_deflate is fast,
doing essentially constant work regardless of the input size,
aside from the work of actually writing out the compressed string.

zipbomb --mode=quoted_overlap --num-files=250 --compressed-size=21179 > zbsm.zip

zipbomb --mode=quoted_overlap --num-files=65534 --max-uncompressed-size=4292788525 > zblg.zip

Efficient CRC-32 computation

Among the metadata in the central directory header and local file header
is a
CRC-32
checksum of the uncompressed file data.
This poses a problem, because directly calculating the CRC-32 of each file
requires doing work proportional to the total unzipped size,
which is large by design. (It’s a zip bomb, after all.)
We would prefer to do work that in the worst case is
proportional to the zipped size.
Two factors work in our advantage:
all files share a common suffix (the kernel),
and the uncompressed kernel is a string of repeated bytes.
We will represent CRC-32 as a matrix product—this
will allow us not only to compute the checksum of the kernel quickly,
but also to reuse computation across files.
The technique described in this section is a slight extension of the
crc32_combine
function in zlib,
which Mark Adler explains
here.

You can model CRC-32 as a state machine that updates a 32-bit state register
for each incoming bit.
The basic update operations for a 0 bit and a 1 bit are:

uint32 crc32_update_0(uint32 state) {
    // Shift out the least significant bit.
    bit b = state & 1;
    state = state >> 1;
    // If the shifted-out bit was 1, XOR with the CRC-32 constant.
    if (b == 1)
        state = state ^ 0xedb88320;
    return state;
}

uint32 crc32_update_1(uint32 state) {
    // Do as for a 0 bit, then XOR with the CRC-32 constant.
    return crc32_update_0(state) ^ 0xedb88320;
}

If you think of the state register as a 32-element binary vector,
and use XOR for addition and AND for multiplication, then
crc32_update_0 is a
linear transformation;
i.e., it can be represented as multiplication by a
32×32 binary
transformation matrix.
To see why, observe that multiplying a matrix by a vector
is just summing the columns of the matrix,
after multiplying each column by the corresponding element of the vector.
The shift operation state >> 1
is just taking each bit i of the state vector
and multiplying it by a vector that is 0 everywhere except at bit i − 1
(numbering the bits from right to left).
The conditional final XOR state ^ 0xedb88320
that only happens when bit b is 1
can instead be represented as first multiplying
b by 0xedb88320
and then XORing it into the state.

Furthermore, crc32_update_1 is just
crc32_update_0 plus (XOR) a constant.
That makes crc32_update_1 an
affine transformation:
a matrix multiplication followed by a translation (i.e., vector addition).
We can represent both the matrix multiplication and the translation
in a single step
if we enlarge the dimensions of the transformation matrix to 33×33
and append an extra element to the state vector that is always 1.
(This representation is called
homogeneous coordinates.)

Both operations crc32_update_0 and crc32_update_1
can be represented by a 33×33 transformation matrix.
The matrices M₀ and M₁ are shown.
The benefit of a matrix representation is that matrices compose.
Suppose we want to represent the state change effected by processing
the ASCII character ‘a’, whose binary representation is
01100001₂.
We can represent the cumulative CRC-32 state change of those 8 bits
in a single transformation matrix:

And we can represent the state change of a string of repeated ‘a’s
by multiplying many copies of M_a together—matrix exponentiation.
We can do matrix exponentiation quickly
using a square-and-multiply algorithm,
which allows us to compute Mⁿ
in only about log₂ n steps.
For example, the matrix representing the state change
of a string of 9 ‘a’s is

The square-and-multiply algorithm is useful
for computing M_kernel,
the matrix for the uncompressed kernel,
because the kernel is a string of repeated bytes.
To produce a CRC-32 checksum value from a matrix,
multiply the matrix by the zero vector.
(The zero vector in homogeneous coordinates, that is:
32 0’s followed by a 1.
Here we omit the minor complication of pre- and post-conditioning the checksum.)
To compute the checksum for every file, we work backwards.
Start by initializing M := M_kernel.
The checksum of the kernel is also the checksum
of the final file, file N,
so multiply M by the zero vector and store the resulting checksum in
CDH_N and LFH_N.
The file data of file N − 1 is the same as the file data of file N,
but with an added prefix of LFH_N.
So compute M_LFHN,
the state change matrix for LFH_N,
and update M := M M_LFHN.
Now M represents the cumulative state change from processing
LFH_N followed by the kernel.
Compute the checksum for file N − 1 by again multiplying M by the zero vector.
Continue the procedure, accumulating state change matrices into M,
until all the files have been processed.

Extension: Zip64

Earlier we hit a wall on expansion
due to limits of the zip format—it was impossible
to produce more than about 281 TB of output,
no matter how cleverly packed the zip file.
It is possible to surpass those limits
using Zip64,
an extension to the zip format that increases
the size of certain header fields to 64 bits.
Support for Zip64 is by no means universal,
but it is one of the more commonly implemented extensions.
As regards the compression ratio,
the effect of Zip64 is to
increase the size of a central directory header from
46 bytes to 58 bytes,
and the size of a local directory header from
30 bytes to 50 bytes.
Referring to
the formula
for optimal expansion in the simplified model,
we see that a zip bomb in Zip64 format
still grows quadratically,
but more slowly because of the larger denominator—this
is visible in
the figure below
in the Zip64 line’s
slightly lower vertical placement.
In exchange for the loss of compatibility
and slower growth,
we get the removal of all practical file size limits.

Suppose we want a zip bomb that expands to
4.5 PB,
the same size that 42.zip recursively expands to.
How big must the zip file be?
Using binary search, we find that the smallest
zip file whose unzipped size exceeds the unzipped size of 42.zip
has a zipped size of
46 MB.

zipbomb --mode=quoted_overlap --num-files=190023 --compressed-size=22982788 --zip64 > zbxl.zip

4.5 PB is roughly
the size of the data captured by the Event Horizon Telescope
to make the
first image of a black hole,
stacks and stacks of hard drives.

With Zip64, it’s no longer practically interesting to
consider the maximum compression ratio,
because we can just keep increasing the zip file size,
and the compression ratio along with it,
until even the compressed zip file is prohibitively large.
An interesting threshold, though,
is
2⁶⁴ bytes
(18 EB or
16 EiB)—that
much data
will not fit on most filesystems.
Binary search finds the smallest zip bomb that produces at least that much output:
it contains 12 million files
and has a compressed kernel of 1.5 GB.
The total size of the zip file is
2.9 GB
and it unzips to
2⁶⁴ + 11 727 895 877 bytes,
having a compression ratio of over 6.2 billion.
I didn’t make this one downloadable,
but you can generate it yourself using the source code.
It contains files so large that it uncovers
a bug
in Info-ZIP UnZip 6.0.

zipbomb --mode=quoted_overlap --num-files=12056313 --compressed-size=1482284040 --zip64 > zbxxl.zip

Extension: bzip2

DEFLATE is the most common compression algorithm
used in the zip format, but it is only one of many options.
Probably the second most common algorithm is bzip2,
while not as compatible as DEFLATE,
is probably the second most commonly supported compression algorithm.
Empirically, bzip2 has a maximum compression ratio of about 1.4 million,
which allows for denser packing of the kernel.
Ignoring the loss of compatibility,
does bzip2 enable a more efficient zip bomb?

Yes—but only for small files.
The problem is that bzip2 does not have anything like the
non-compressed blocks of DEFLATE
that we used to quote local file headers.
So it is not possible to overlap files and reuse the kernel—each file must have
its own copy, and therefore the overall compression ratio
is no better than the ratio of any single file.
In the figure we see that
no-overlap bzip2 outperforms quoted DEFLATE
only for files under about a megabyte.

There is still hope for using bzip2—an
alternative means of local file header quoting
discussed in the next section.
Additionally,
if you happen to know that a certain zip parser supports bzip2
and tolerates mismatched filenames,
then you can use the full-overlap construction,
which has no need for quoting.

Discussion

In related work,
Plötz et al.
used overlapping files to create a
near-self-replicating zip file.
Gynvael Coldwind
has previously suggested (slide 47)
overlapping files.

We have designed the quoted-overlap zip bomb construction for compatibility,
taking into consideration a number of implementation differences,
some of which are shown in the table below.
The resulting construction is compatible with zip parsers that work
in the usual back-to-front way,
first consulting the central directory
and using it as an index of files.
Among these is the example
zip parser included in Nail,
which is automatically generated from a formal grammar.
The construction is not compatible, however,
with “streaming” parsers,
those that parse the zip file from beginning to end in one pass
without first reading the central directory.
By their nature, streaming parsers
do not permit any kind of file overlapping.
The most likely outcome is that they
will extract only the first file.
They may even raise an error besides,
as is the case with sunzip,
which parses the central directory at the end and checks it for consistency
with the local file headers it has already seen.

If you need the extracted files to start with a certain prefix
other than the bytes of a local file header,
you can insert a DEFLATE block before the
non-compressed block that quotes the next header.
Not every file in the zip file has to participate in the bomb construction:
you can also include ordinary files
if needed to conform to some special format.
(The source code has a --template
option to facilitate this use case.)
Many file formats use zip as a container;
examples are Java JAR, Android APK, and LibreOffice documents.

PDF
is in many ways similar to zip.
It has a cross-reference table at the end of the file
that points to objects earlier in the file,
and it supports DEFLATE compression of objects through
the FlateDecode filter.
I haven’t tried it, but it may be possible to use
the quoted-overlap idea to make a PDF compression bomb.
It may not even be necessary to work that hard:
binaryhax0r suggests in a
blog post
that one can simply specify multiple layers of FlateDecode
on a single object, which makes a compression bomb easy.

Detecting the specific class of zip bomb we have developed in this article is easy:
just look for overlapping files.
Mark Adler has written a patch
for Info-ZIP UnZip that does just that.
In general, though, rejecting overlapping files
does not protect against all classes of zip bomb.
It is difficult to predict in advance
whether a zip file is a bomb or not,
unless you have precise knowledge of the internals of the parsers
that will be used to parse it.
Peeking into the headers and summing
the “uncompressed size” fields of all files
does not work, in general,
because the value stored in the headers
may not agree with the actual uncompressed size.
(See the “permits too-short file size” row in the compatibility table.)
Robust protection against zip bombs
involves placing time, memory, and disk space limits
on the zip parser while it operates.
Handle zip parsing,
as any complex operation on untrusted data,
with caution.

Credits

I thank
Mark Adler,
Russ Cox,
Brandon Enright,
Marek Majkowski,
Josh Wolfe,
and the USENIX WOOT 2019 reviewers
for comments on a draft of this article.
Caolán McNamara evaluated the security impact
of the zip bombs in LibreOffice.

Did you find a system that chokes on one of these zip bombs?
Did they help you demonstrate a vulnerability or
win a bug bounty?
Let me know and I’ll try to mention it here.

LibreOffice 6.1.5.2

zblg.zip renamed to zblg.odt or zblg.docx
will cause LibreOffice to
create and delete a number of ~4 GB temporary files
as it attempts to determine the file format.
It does eventually finish, and it deletes
the temporary files as it goes,
so it doesn’t fill up the disk.
Caolán McNamara replied to my bug report.

Mozilla addons-server 2019.06.06

I tried the zip bombs against a local installation of addons-server,
which is part of the software behind addons.mozilla.org.
The system handles it gracefully,
imposing a time limit
of 110 s on extraction.
The zip bomb expands as fast as the disk will let it up to the time limit,
but after that point the process is killed and the unzipped files
are eventually automatically cleaned up.

UnZip 6.0

Mark Adler wrote
a patch
for UnZip to detect this class of zip bomb.

McAfee VirusScan

Twitter user @TVqQAAMAAAAEAAA
reports

“McAfee AV on my test machine just exploded.”
I haven’t independently confirmed it, nor do I have details such as a version number.

A version of this article
will appear in the
USENIX WOOT 2019
workshop.
The source code of the paper is available.
The artifacts
prepared for submission are zipbomb-woot19.zip.

A final plea

It’s time to put an end to Facebook.
Working there is not ethically neutral:
every day that you go into work, you are doing something wrong.
If you have a Facebook account, delete it.
If you work at Facebook, quit.

And let us not forget that the National Security Agency
must be destroyed.