OpenZL
摘要
OpenZL 是一个压缩库及配套工具，能根据用户提供的数据描述自动生成针对特定格式优化的专用压缩器，所有专用压缩器均兼容同一个通用解压器。文章以 Silesia 压缩语料库中的 SAO 文件为例，对比了 OpenZL 与 zstd -3、xz -9 的压缩比、压缩速度和解压速度：OpenZL 压缩比达到 2.06，压缩速度 203 MB/s，解压速度 822 MB/s，在保持数据中心所需高速的同时，实现了远超通用压缩器的压缩比。网站还提供了快速入门指南，供用户几分钟内上手试用。
荐读理由
你可以为你的AI数据集生成专用压缩器，获得类似SAO上2.06压缩比和203 MB/s压缩速度的收益，直接加速数据处理管线。
原文
OpenZL
OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors.
OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format. Learn how it works →
OpenZL consists of a core library and tools to generate specialized compressors — all compatible with a single universal decompressor. It is designed for engineers that deal with large quantities of specialized datasets (like AI workloads for example) and require high speed for their processing pipelines.
Here are some examples:
- SAO, part of the Silesia Compression Corpus:
| SAO | zstd -3 | xz -9 | OpenZL |
|---|---|---|---|
| Ratio | x1.31 | x1.64 | x2.06 |
| Compression Speed | 115 MB/s | 3.1 MB/s | 203 MB/s |
| Decompression Speed | 890 MB/s | 30 MB/s | 822 MB/s |
The result: much stronger ratios than generic compressors, at the speeds required by datacenter workloads.
Want to try it yourself? Get started in minutes with the Quick start guide
这条对你有帮助吗？