How much free space can Lepton, a lossless jpeg compression tool made by Dropbox, secure?

Hello.
I'm Mandai, in charge of Wild on the development team.

I tried various things to see how much image size can actually be compressed using lepton, a lossless compression tool for JPEG files developed by Dropbox.

The OS used this time is CentOS7, created on VirtualBox.
Please see the capture below for Windows system information and VirtualBox allocation.

If you would like to easily create the same environment and try it out, this article and build a virtual environment.

Since the compilation environment is selected, please note that there is a high possibility that it will not compile properly on CentOS 6 or lower.
(Actually, I couldn't compile it on a CentOS 6.7 VM, even though I tried various things.)

 

Installing lepton

Installing lepton is very easy.

As described on github, run the included autogen.sh to download the dependent modules used for build, and then use autoreconf to build the build environment.
If you have trouble using autogen.sh, try installing automake and autoconf using yum.

git clone https://github.com/dropbox/lepton.git sh autogen.sh ./configure make -j4 sudo make install

 It is much faster to make with multi-core, so it is better to add the "-j" option when making.
The above command is made on a PC with a 4-core CPU.

The number of CPU cores can be obtained using the nproc command, so you can also do something like "make -j$(nproc)" (that's wild!).

 

How much can be reduced?

Now, it's time for the long-awaited conversion time.

Let's cook this one first

File size is 5816548 bytes. It's just over 5MB. As a JPEG image, I think it's quite large.

The command to convert this to lepton format is as follows.

time lepton IMGP2319.JPG compressed.lep lepton v1.0-1.2.1-35-ge230990 14816784 bytes needed to decompress this file 4349164 5816548 74.77% 15946404 bytes needed to decompress this file 4349164 5816548 74.7 7% real 0m1.102s user 0m2.676s sys 0m0.053s

 I only added the time command because I wanted to know the execution time, so originally just "lepton IMGP2319.JPG compressed.lep" would be sufficient.
The result is that it was compressed to 74.77% of the size.

The result of ls is as below.

ls -al -rw-rw-r--. 1 vagrant vagrant 5816548 Oct 11 09:33 IMGP2319.JPG -rw-------. 1 vagrant vagrant 4349164 Oct 11 09:49 compressed.lep

 
I think it was processed without any problems.

Now let's go back to the beginning.

time lepton compressed.lep decompressed.jpg lepton v1.0-1.2.1-35-ge230990 15946404 bytes needed to decompress this file 4349164 5816548 74.77% real 0m0.370s user 0m1.235s sys 0m0.022s ls -al -rw-rw -r--. 1 vagrant vagrant 5816548 October 11 09:33 IMGP2319.JPG -rw-------. 1 vagrant vagrant 4349164 October 11 09:52 compressed.lep -rw------- . 1 vagrant vagrant 5816548 October 11 09:54 decompressed.jpg

 
The original image is IMGP2319.JPG, after converting to lepton format, the result of reconverting to jpeg is "decompressed.jpg".

The file size is the same, and even if you do a diff, the files are exactly the same.

However, be careful when permissions change.

 

Try converting a large number of images (jpeg → lep format)

We tried converting a large number of images and investigated how much space can be saved on average.

First, convert from jpeg to lep format.

The number of files handled by lepton was 159 files, and the total number of bytes was 807MB.
The time it took to convert this to lepton was 569.686 seconds in total, with an average of 3.582931 seconds.

A jpeg file with a total file size of 826,028 bytes is converted to a lep format file with a total file size of 627,912 bytes.
The average compression ratio is 23.98%, which is nominally 22%, so the result is as advertised.
The file with the highest compression rate was successfully compressed by 29.58%.
On the other hand, the file with the worst compression ratio is 21.23%.
The median value is 23.82%, so I think it's safe to assume that the average compression is over 22% (right?)

By the way, the CPU load seems to be distributed to each core, so there was no situation where only one CPU was being used at a high rate.

 

Try converting a large number of images (lep format → jpeg)

Next, I will try converting from lep format to jpeg.

Let's convert the lep format file that we converted earlier back to jpeg.
This conversion took a total of 232.158 seconds, with an average of 1.460113 seconds.

This also varies depending on the file, but some take a maximum of 1.992 seconds, while others
take a minimum of 1.246 seconds.
Median execution time was 1.436 seconds.

 

Tighten

If you use a service that stores a lot of jpeg images, you can save money by converting them when saving them to AWS S3, etc.
However, the machine power required for compression and decompression is also relatively reasonable.
The time it takes to convert depends not only on CPU performance but also on memory usage, so you need to adjust it to suit your machine.
Memory, processes, and threads can be controlled with options.

However, as a feature that cannot be overlooked, it seems that it can also be used to start to retrieve files via a Unix domain socket or by specifying a listening port.
I wonder if you are thinking of using it to set up a separate lepton conversion server and do conversions all the time. Depending on your idea, it might shine.

lepton is already in use within Dropbox and seems to be producing results.
If you are having trouble with the file size of jpeg images, please take this opportunity to try it out.

 
That's it.