When speed is the entire point

A database writing its log, a message queue moving frames, a game engine pulling assets off disk — on those paths the data cannot sit and wait for a compressor to think. This library was built for exactly that moment. It is an open-source lossless compression library whose compression runs at more than 500 MB per second on a single core, and whose decompression reaches several GB per second per core, which in practice means it runs into memory bandwidth as the ceiling rather than into the algorithm.

The trade is stated plainly: compression ratio is given away, and what comes back is latency low enough to sit inside a real-time path. That is the whole design, and it explains why the library turns up in places where nobody is measuring disk savings.

Fast, with some discipline kept in reserve

There are still days when the ratio matters more than the clock. A high-compression variant spends more processor time for a better result, and dictionary compression is available both through the API and from the command line; on small files, where there is little history inside a single buffer to exploit, the dictionary is what makes the numbers respectable.

The 1.10.0 release, subtitled the multicores edition, made threading a first-class feature: it is enabled by default on Windows, the number of threads is set through an environment variable, and the dictionary API graduated from experimental to a stable interface. The format specification is public, the frame and block formats are documented in detail, and bindings exist for most languages anyone would reach for.

The invisible component

It is released under a two-clause BSD licence, so adding it to a commercial product carries no licensing homework. That freedom is why it stopped being "a compression tool" a long time ago: databases, file systems, kernel modules and archive formats embed it directly, and the library package is a familiar sight in distribution repositories. For a developer it is closer to infrastructure than to software — you call one function, the user never learns the name, and the only thing anyone notices is that the thing feels fast.

What it is not deserves the same clarity. It will not produce the smallest file; when the goal is saving bandwidth or disk, a heavier algorithm wins and should be used. It is not a backup format, not an archive manager, and not something a person interacts with. Its job is to disappear inside somebody else's program.

Reading a benchmark honestly

Numbers like these invite misuse, so it is worth saying what they mean. Throughput measured on a single core describes the algorithm, not your application: real pipelines add copies, locks, network hops and serialisation, and the moment data crosses a socket the compressor's advantage shrinks. The honest way to evaluate it is to measure the path you actually run, and the honest way to choose it is to ask whether the bottleneck is time or space.

How it gets chosen in a real project

The decision is rarely a competition between libraries. It usually arrives as a sentence in a design discussion: the log does not need to be small, it needs to be written before the next transaction commits. Once that is the requirement, the argument is over, because the candidates that produce smaller files are the ones that cannot finish in the time available.

Integration is correspondingly modest. A buffer in, a buffer out, and a frame format that other tools already understand — which is a quiet advantage of a public specification, since a file written today can be read by a different program written in a different language. When something goes wrong, the failure tends to be a size or a version mismatch rather than a mystery, and that is worth a great deal at three in the morning.

Where the ceiling actually sits

It is worth repeating that the ceiling is the machine, not the algorithm. Storage and memory have become fast enough that a compressor working at this rate spends much of its time waiting for bytes to arrive, which is why pushing further would buy less than it looks: the remaining headroom is in the surrounding program, not in this library.

That also explains the shape of the project over time. The interesting work has moved from making the core faster to making it easier to use correctly — threading, a stable dictionary interface, formats documented well enough that a second implementation can exist. For infrastructure, being boring and being reliable are the same compliment.

The trade in compression has always been speed against ratio, and this library pushes the balance all the way to the speed end, then defends that position with a permissive licence and more than a decade of steady maintenance. When the need is to save bandwidth or disk it is not the best answer; when the need is to save time, it is almost always the first candidate.