Google Neural Machine Translation

The report was written in December 2017.

This is not the best algorithm that wins. It's who has the most data. The winner is not the one who has the best algorithm, but the one who has more data. Andrew Ng, Coursera Machine Tutor.

If you scroll up, you can learn finer distinctions or more complex features. ... These models are typically taken a lot more context. Jeff Dean, an engineer helping to lead the research at Google. If you increase the size of the model and give it more data for training, it will begin to distinguish between more subtle and complex features. ... These models usually take a broader context. Jeff Dean, a research management engineer at Google.

I tested Google Translate on the same texts in March and December 2011, January 2016 and December 2017. I took the same passages in English, Russian, German, French, Ukrainian and Polish and translated each into the other five languages from the sample. In addition, in December 2017, she additionally took new texts and tested them in all directions of translation. The results of cross-verification generally coincided with trends in the original sample. The result was a cut of the Google translator's work for 2011–2017, and on the basis of these materials, one can draw conclusions about the evolution of the service and comment on the company's marketing statements (quotations are planned to be published separately).

Chronicle of events

In the period up to 2011 (and, possibly, later), Google actually stated that the statistical translator is self-taught (see the relevant staff quotes in my article “Language Compatibility”, [1]) and that as parallel texts accumulate in each language To a pair, product quality is constantly striving for the human level solely thanks to an innovative statistical approach. In 2012, Google launched a self-learning neural network [6] and in September 2016 they announced the transfer of their machine translator to in-depth training, which again promises a steady increase in product quality [3, 4, 5]. Since March 2017, the neural network has been used to translate into Russian.

Let's look at what exactly Google translator managed over the years, what are its strengths (no one denies that this is one of the best machine translators).

Retrospective

Year 2011

(According to the materials of my 2012 article, [1].)

Translation in many language pairs comes through an intermediate translation into English with the effect of a “broken phone”

When translating in these directions, English acts as an "intermediary": the text is first translated into English and only then into the selected language of translation. A distorted English version is translated into the target language, with inaccurate fragments inevitable with machine translation. So on the "primary deformation" superimposed second layer. As a result, the same text gets more distortion when translated into German, French and other languages than into English.

We take the received English phrase [translation from Russian] and translate it into German and French. The result is 100% identical to the translation in the third and fourth lines, allegedly from Russian. Errors of the German and French translations correlate with inaccuracies and the structure of the English phrase, but not the Russian original.

When the order of words in the English text is disrupted, the forms of the words and their sequence are not a reliable indicator of the role of these words in the sentence [with further translation into the target languages].

Statistical translation is optimal between related languages

The most intuitively correct of all those reviewed, closest to the finished product, has less distortion of the meaning or rules of the language, less literal translation.

Proper “conversion” of grammatical constructions of one language by means of another is the limit for statistical machine translation. This restriction is not abolished even on the basis of related languages and creates the more “noise” in interpretation, the less the grammatical similarity of languages in a pair.

Google can continue to analyze web documents and replenish the database of matches, but they will not improve the translation only by “optimizing search results”. You can not create a translation database for all possible phrases. This means that a certain significant percentage of sentences, phrases must be composed independently of the machine, and not taken ready, which means that the developer needs to teach the machine to grammar in other, "non-statistical" ways.

English - “core” language in Google Translate

Translation from English and into English into Google Translate - direct, without the mediation of another language. This provides a good quality, in which the advantages of the service work are especially noticeable: often correct translation of names and names, terms, phraseological units, use of lively turns of speech as opposed to literal translation, often the right choice of lexical meaning depending on the context.

Translated in other language directions, “Anglocentricity” is the biggest disadvantage of Google Translate. The translation is not into English or not from English so far not the most successful: the text undergoes double distortion due to the fact that the original is translated into the target language not directly, but from an “intermediate”, broken translation into English. One of the ways to improve the quality of translation in the service can be “unloading” of English and creating “nests” around other key languages: one of the Slavic, Turkic, Romance, etc.

Significant improvement in the quality of translations does not occur over time.

Translations of the same text at different times (March, October, December 2011) demonstrated the pattern of development of the statistical translation of Google. In later translations, a greater variety of vocabulary was noticeable, but in general, in terms of accuracy and clarity, they were not much better, even worse in some places.

Year 2016

Google removes repetitions from the translation; a better built, coherent sentence, sometimes a better choice of words; sometimes a rollback to a less successful translation (“stone tool” instead of “stone tool” in 2011); sometimes a less successful interpretation of the role of a member of a sentence is sometimes more. Total: better in some places, worse in some places than translations of 2011, but in general the level and ceiling is the same.

Year 2017

English as an intermediate language retains its role, but gives up a little

There are more variations, deviations from the English intermediate translation. Often these experiments are unsuccessful, that is, if the translation into the target language still blindly followed English, the result would be better. However, at the same time, “possession” of the grammar of the target language improved: if the English version of the text is decoded adequately, then you can be 90% sure that the correct endings will be put down in the translations into other languages, suitable lexical means are selected, the optimal word order is built. If in English “porridge” ... No, there is already no porridge in the results of 2017, and this is a great achievement. If the English translation is a small failure, then the translation into the target languages, according to the law of a broken phone, misunderstanding increases. However, the distortion (wrong choice of the word) in the target languages is also found in the perfect English translation.

Compared with the translations of 2011–2016, the nature of deviations from the English translation in 2017 is such that it seems that 1) screwed the randomizer, 2) the translator processes the text in several stages and can distort individual pieces in the process or refine them the value of the source, not the English intermediary.

Nevertheless, the structure of sentences and the choice of vocabulary in the target languages are still largely determined by the English translation, and the translations themselves into languages using the Latin script sometimes contain pieces in English that were not in the original.

The tendency to generate text in the target language in accordance with the laws of its grammar
Correlations between translations of one text into different languages are less than before. The service does not translate literally, the result has become more free: adequate rephrasing, rearrangement of words, rearrangement of words from the beginning to the end of a sentence, if required by the rules of the language (in German it is implemented perfectly). Unlike the previous level (phrase-based translation — finding single matches of individual words and phrases), the neural translator transforms sentences to some extent, analyzes them as a single whole and establishes “end-to-end” correspondences in several stages (end- to-end mapping - end-to-end conversion, full cycle, continuous transformation of data manifold from input to output).

More accurate analysis of sentence and word structure

The main achievement in the 2017 translation results is a more solid, confident recognition of the structure of a sentence and the transfer of grammatical meanings in target languages. In English, endings do not play such an important role in the transmission of grammatical meanings, as in Russian, German, Polish and Ukrainian. However, when “running” through a neural network, grammatical connections began to “get lost” less often than with statistical translation. Also, rarely used multi-root words were recognized: the translator copes well with the articulation of not only sentences, but also words.
However, the "skill" of analysis depends largely on the language. It is better and more consistently implemented in German and Polish than in Russian (but not bad either). In translations from Ukrainian, it works, then it is frankly buggy (in such fragments the level is worse than in translations of previous years).

The quality of translation over the last year has grown significantly

In 2011–2016, translations of complex phrases into English had only the appearance of coherence: the translated words and phrases were strung together in a slightly adjusted order, but there was no “in-depth understanding” of the structure, and sometimes the translation looked smooth only because in English it was often not endings are needed, and the absence of official words in some styles is permissible. But this “misunderstanding” has always manifested itself in further translations into target languages. In the translations for December 2017, the structure of the English sentence is better verified - and is better interpreted into other languages. The quality in these languages has increased commensurately: a little lower than English, but much higher than the previous plus there are sporadic omissions of words and deviations from English (in most cases unsuccessful).

Some positions on lexical accuracy are lost in comparison with translations of 2011 and 2016, but the general clarity of the final text is more important than the fact that the translator flaunts knowledge of certain terms and expressions. In 2011, against the background of other machine translators, high-quality work with vocabulary and phraseology was an achievement. Only the best managed to find such exact correspondences of stable phrases, proper names and terms. However, separate correspondences with a general incoherence were not enough. Required to tighten the "knowledge of grammar." For five years, the “self-study” of a statistical translator (from 2011 to 2016) did not improve connectivity. A qualitative leap occurred after integration with the neural network (or so it coincided). Now, on the samples that I took on December 3, 2017, I can confirm that the most important task has become more achievable: the “computer” (rather, a huge computer network) can recognize the text without laboriously prescribing the rules manually. (But sometimes it is mistaken. Therefore, it is better to give it texts simpler, without florid sentences five lines long.)

If in 2011 and 2016, the share of “dark places” (incoherent word set) in the samples of translation into all languages I took was 1 2 fragments per text 65–90 words long, in 2017 there are no “dark places”. (I took for translation not puns and other abstruse expressions, but ordinary texts. Wrong and even comical translation of certain words and phrases still takes place, but does not lead to the creation of "dark places.") Reading the translation, you understand something about speech, even if it is “clumsy”. Moreover, the quality of translation in English is higher than in other target languages.

If in 2011 the main feature of Google translator was to find the correspondences between languages (lexical, phrase level) that were ideal in this context, then in 2017, having lost some lexical accuracy, the translator gained momentum in the syntactic analysis of sentences and the transfer of grammatical links.

In 2011, the service sometimes perceived pieces of a complex sentence as isolated and simply stringed their translation one after another into a chain. In 2017, having solved this problem, it is also better to insulate truly foreign pieces, so that they do not create a “noise”. These are specks of words in another language and typos. This brings the car to the level of a person: if we do not hear a few words in a sentence, as a rule, this does not prevent us from catching the general meaning.

Translation into Ukrainian "untied" from the Russian intermediary language

Previously (up to the penultimate “measurement” in January 2016), transfers into Ukrainian and Russian coincided by 99.9%, and if this lowered the quality of the translation into Ukrainian, it was insignificant, despite the fact that it was separated from the original translation first by English, then into Russian (“third water on kissel”).

Now between the translations of one text into Russian and Ukrainian appeared scatter. Instead of blindly following the Russian translation, Ukrainian now goes its own way. Sometimes this means that it simply contains more incorrect translations and forms of the word. Sometimes - that there are no errors in it where there is in Russian.

Previously, if there was a wrong translation, then immediately in all languages: the same mistake in the same place. This was due to the "hitch" in the English translation. Now, errors also appear sporadically: in one language or another, when everything is OK in English and other target languages. In Ukrainian, this is happening more often than in other languages from the sample. In addition, the translation of three different texts from Ukrainian into German, French and Polish contains many absurd distortions, which are not translated into English. Also, when paired with the Ukrainian, about a third of the names are distorted, although the exact transfer of names is a traditional feature of Google from “immemorial” times. Examples: Bloodd instead Bloodd, Daphne du Morley instead Daphne de Maurier, Racine instead Rachel; elsewhere Rachel was written correctly only in English, but Racch appeared in German, French and Polish. I assumed that such distortions are not a glossary error, but a “situational” system failure, and in another text the same name can be correctly conveyed. The hypothesis was confirmed, except in the case of Daphne du "Morley".

Neural translator does not operate with meanings

The statistical translator worked well with the recognition of terms, names, phrases, and often successfully selected word meanings in the context of a sentence. Problems began when it was impossible to correctly interpret the relationship between words, their grammatical role. In 2017 translations, a significant improvement in this direction is noticeable, that is, there are fewer unrecognized places. Is this translation adequate? Not really. The neural translator is linked to the sentence structure. It transforms language constructions quite well from one language to another. The service takes the meaning not of a sentence or a paragraph, but of smaller segments (which it began to define much better) and “sticks together” pieces in accordance with the grammar rules of the target language.

Such products are still an aid to understanding the original, but not a high-quality text (although some of the sentences can be translated perfectly).

What is missing?

Professional translators are taught to convey meaning in their own words, without being attached to the structure of the source code sentence. Adequate translation should strive from word-by-word and phrasal transmission to semantic transformations.

How many intermediate phases from the current stage of development of machine translation to literary, implying the possession of style norms and transfer of figurativeness? As much as to create an artificial intelligence capable of reasoning.
Sometimes it seems to me that language norms will change earlier. Due to the fact that we read a lot of “under-translations” from English, our native language can become more tolerant of Anglicisms - not only individual phrases, but also the foreign structure of phrases. The languages that are formed on the basis of the language of the colonizer and the language of the local population are called pidgin. They take neologisms from the colonizer; grammar leave their own. A vivid example: the speech of our former fellow citizens in Brighton Beach (and in other places of residence abroad) or office runglish: Russian endings are attached to English words. To deny, zapapruvit, share, etc.

What happens in machine translation is the opposite process: Russian words with Russian endings form a little awkward, but understandable phrases. We don't say that. But you can understand. This is like a speech of a foreigner who speaks Russian well: the accent seems to be gone, the vocabulary is chic, and the form of the words is correct, but sometimes “I take a shower”, “I take a taxi”, “honoring this principle requires us to try a new method of persuasion ". This level is still the limit for a neural translator. Needless to say that this is a very high bar? Moreover, the service “owns” at this level more than a hundred languages and adds new ones relatively easily. "Translator with an English accent."

What specifically is missing to the level of literary translation

To make it clearer what the translators grumble about, in conclusion I will give examples of machine and non-machine translation from English to Russian.

There have been fewer such moments.
Google: No one who reflected on his life, never dreamed that there were fewer such moments.
Human: Looking at the life you have lived, no one will say that there were too many such moments.

Is civilization just a veneer over a violent core?
Google: Is civilization just a veneer over a furious core?
Human: Is man a cruel beast under a thin layer of civilization? // Is a person under a thin layer of civilization - a cruel beast?