In the early 1970s, American writer Michael Hart managed to
gain unlimited access to the Xerox Sigma 5 computer installed at the University of Illinois. To adequately use the resources of the machine, he decided to create the first e-book, reprinting the US Declaration of Independence.
Today, digital literature is widespread, largely due to the development of portable devices (smartphones, readers, laptops). This led to the emergence of a large number of e-book formats. Let's try to understand their features and tell the story of the most popular of them - let's start with the format DjVu.
/ Flickr / lane pearman / CCAppearance format
DjVu was developed in 1996 by AT & T Labs with one goal - to provide web developers with a tool for distributing high-resolution images via the Internet.
The fact is that at that time 90% of all information was still
stored on paper, and many of the important documents had color images and photographs. In order to preserve the readability of the text and the quality of images, it was necessary to make scans in high resolution.
Classic web formats — JPEG, GIF, and PNG — made it possible to work with such images, but at the cost of volume. In the case of JPEG, so that the text
was readable on the monitor screen, one had to scan a document with a resolution of 300 dpi. The color page of the magazine at the same time occupied about 500 KB. Downloading files of this size from the Internet at that time was quite a laborious process.
The alternative was to digitize paper documents using text recognition technologies, but 20 years ago their accuracy was far from ideal — after processing, the final result had to be seriously edited by hand. At the same time, the graphics and images remained “overboard”. And even if it was possible to embed a scanned image into a text document, some visual details were lost, for example, the color of the paper, its texture, and these are important components of historical documents.
In order to solve these problems, AT & T developed DjVu. He allowed to compress scanned color documents with a resolution of 300 dpi to 40–60 Kbytes, with an original size of 25 MB. The size of black and white pages DjVu reduced to 10-30 KB.
How DjVu Compresses Documents
DjVu can work with paper scanned documents, as well as with other digital formats, such as PDF. DjVu
is based
on technology that divides an image into three components: foreground, background, and a black and white (bit) mask.
The mask is saved with the resolution of the original file and
contains an image of the text and other clear details - thin lines and diagrams - as well as contrasting pictures.
It has a resolution of 300 dpi, so that the thin lines and contours of the letters remain clear, and is compressed using the JB2 algorithm, which is a variation of the JBIG2 algorithm proposed by AT & T for faxing. A feature of JB2
is that it searches for duplicate characters on a page and saves their image only once. Thus, in multi-page documents every several consecutive pages use a common “dictionary”.
The background contains the page texture and illustrations, and its resolution is less than that of the mask. The lossless background for perception is saved with a resolution of 100 dpi.
The foreground
stores the color information about the mask, and its resolution usually decreases even more, since in most cases the text color is black and the same for one printed mark.
Wavelet compression is used to compress the foreground and background.
The final step in creating a DjVu document becomes entropy coding when an adaptive arithmetic encoder turns sequences of identical characters into a binary value.
Advantages of format
The task of DjVu was
to preserve the “properties” of a paper document in digital form, allowing even weak computers to work with such documents. Therefore, software for viewing DjVu-files has the ability to "quickly render." Thanks to it, only that piece of DjVu page that should be displayed on the screen is
loaded into memory.
It also makes it possible to view the “under-downloaded” files, that is, individual pages of a multi-page DjVu document. At the same time, progressive drawing of image details is used, when the components seem to “appear” as the file is downloaded (as in JPEG).
20 years ago, when this format was introduced, the page was loaded in three stages: first, the text component was loaded, after a couple of seconds, the first versions of the images and the background were loaded. Already after the whole page of the book was “displayed”.
The presence of a three-level structure also allows you to search for scanned books (as there is a special text layer). This turned out to be convenient when working with technical literature and reference books, so DjVu became the basis for several libraries of scientific books. For example, in 2002, it was chosen by the
Internet Archive as one of the formats (along with TIFF and PDF) for a project to save scanned books from open sources.
Format flaws
However, like all technologies, DjVu has its drawbacks. For example, when encoding scans of books in the DjVu format, some characters in the document may be replaced by others that are similar in appearance. Most often this happens with the letters “i” and “n”, which is why this problem
is called the “yin problem”. It does not depend on the language of the text and affects, in particular, the numbers and other small repeating signs.
Its cause is character classification errors in the JB2 encoder. It “crushes” scans into groups of 10–20 pieces and forms a dictionary of common symbols for each group. The dictionary contains samples of common letters and numbers with pages and coordinates of their appearance. When you look through the DjVu-book, the characters from the dictionary are substituted in the right places.
This allows you to reduce the size of the DjVu file, however, if the display of two letters is visually similar, the encoder may either confuse them or take them for the same. Sometimes this leads to a deterioration of the formulas in the technical document. To solve this problem, you can opt out of the compression algorithms, but this will increase the size of the digital copy of the book.
Another disadvantage of the format is that it is not supported by default in many modern OS (including mobile). Therefore, to work with it, you need to install third-party
programs , such as DjVuReader, WinDjView, Evince, etc. However, here I would like to note that some electronic readers (for example, ONYX BOOX) support the DjVu format out of the box - since the necessary applications there already installed.
By the way, about what else can applications for readers based on Android, we told in one of the previous materials .
Reader ONYX BOOX ChronosAnother format issue manifests itself when working with DjVu documents on small screens of mobile devices - smartphones, tablets, readers. Sometimes DjVu-files are presented as a scan of a book reversal, and professional literature and working documents are often A4 format, so you have to “move” the image in search of information.
However, we note that this problem is also solvable. The easiest way, of course, is to search for a document in a different format - but if this option is not possible (for example, you need to work with a lot of technical literature in DjVu), then you can use electronic readers with a large diagonal from 9.7 to 13.3 inches, which specially "sharpened" under work with similar documents.
For example, in the ONYX BOOX lineup such devices are
Chronos and
MAX 2 (by the way, we prepared an overview of this reader model, and will soon publish it in our blog), as well as the
Note , which has an E Ink Mobius Carta screen with a diagonal of 10.3 inches and higher resolution. Such devices allow you to safely see all the details of the illustrations in their original size and will suit those who often have to read educational or technical literature. To view DjVu and PDF files
, NEO Reader is used, which allows you to adjust the contrast and thickness of digitized fonts.
Despite the format’s flaws, today DjVu remains one of the most popular formats for “preserving” literary works. This is largely due to the fact that it
is open, and modern technology and development today allow some of the technological limitations to circumvent it.
In the following materials we will continue the story of the emergence of e-book formats and the features of their work.
PS Several tools of ONYX BOOX readers: