Facebook open sources data compression and storage tools
Thu 1 Sep 2016
Facebook engineers have this week released a collection of compression and storage tools to the open source community.
Sharing the source codes, the social network hopes to encourage other firms to build and scale platforms to reach larger and more diverse user bases while reducing storage needs. The open code will also support the discovery and patching of existing vulnerabilities, and therefore improve the technology over time.
The open source software includes the Zstandard lossless data compression algorithm. Facebook claims that the technology is significantly faster and more efficient than standard compression models. It explained that the technology is currently deployed daily within its network to compress and decompress very large amounts of data across multiple formats and use cases.
Facebook noted that it may face resistance introducing the new standard, with traditional Zip, gzip, and zlib formats having been the common currency for over twenty years. ‘The reigning data compression standard is Deflate, the core algorithm inside Zip, gzip, and zlib. For two decades, it has provided an impressive balance between speed and space, and, as a result, it is used in almost every modern electronic device (and, not coincidentally, used to transmit every byte of the very blog post you are reading). Over the years, other algorithms have offered either better compression or faster compression, but rarely both.’
In addition, Facebook published the code for its next-generation MySQL storage engine, MyRocks. Facebook engineer Yoshinori Matsunobu explained in an official post that after deploying MyRocks technology in one of its data centre regions, it was able to use 50% less storage for the same amount of data compared with compressed InnoDB.
The open source releases were announced at the Facebook @Scale conference in San Jose, which draws developers working on larger-scale engineering challenges, and promotes the growth of open source technologies.