Machine Translation Jan Odijk Utrecht. March 7, 2011 1.
-
Upload
lizette-odom -
Category
Documents
-
view
218 -
download
2
Transcript of Machine Translation Jan Odijk Utrecht. March 7, 2011 1.
Machine Translation
Jan Odijk
Utrecht. March 7, 2011
1
Overview
• Lexicons• Statistical MT• MT: What is (perhaps) possible• Conclusions
2
Lexicons
• “Wat helemaal niet moeilijk is– Grote woordenboeken met veel moeilijke
woorden en vaktermen”– (Steven Krauwer, vorige college)
• I disagree
3
Lexicons
• True if you know the words and terms in advance
• But new words and terms (usually with different translations) are created all the time in science, technology and industry
• So you must have techniques to find (identify, extract) such new words/terms and their translations as automatically as possible
– To tune the lexicons to specific domains– to continuously extend them
4
Lexicons
• Many terms are multiword expressions – With some internal variation– Not always contiguous– This requires special treatment in the lexicon
and in the grammar• House* of representatives (Chambre* des représentants)• Patatas* fritas* (French fries*)• Chômeur* (Unemployed person*)
5
Lexicons
• Modern formal grammars depend highly on lexical properties
• They have very general rule schemata, which are filled in by properties of lexical items– e.g. a word of category X and its complements
form a XPhrase– E.g. mass nouns can occur without article in
singular; – count nouns can occur with een in singular 6
Lexicons
• Properties of lexical items– E.g. which complements a verb takes
• E.g. a direct object noun phrase, also an indirect object, predicate, prepositional complement, etc
• E.g. an infinitival complement, with or with te, with or without om, with or without a subject, etc.
– With which preposition it can be combined• Kijken naar, zorgen voor, houden van
– Nouns: mass or count?
7
Lexicons
• Traditional dictionaries do not contain such information (or very rarely)
• And what is available is not represented in a formal manner
• So computers cannot use this information directly
8
Lexicons
• It is very difficult to assign such properties correctly in a systematic manner– It requires very good knowledge of syntax– Often the phenomena are not understood well
enough– Words often have multiple options with
different meanings and translations– Try it yourself for lopen; innemen– Count/Mass: vis; wijn; bestek; meubilair
9
Lexicons
• It is very difficult to assign such properties correctly in a systematic manner (Cont.)– Lexicographers are not trained to assign such
properties– It must be done for many words– Consistency within one person is hard to
achieve– Consistency among multiple people is evebn
harder
10
Lexicon: Semantics
• Selection restrictions with type system to approach modeling of world knowledge– Requires sophisticated syntactic analysis
• Boek: info (legible)• Uur: time unit duration• Vergadering: event duration• Lezen: subject=human; object=info (legible)• Durational adjunct must be a duration phrase
11
Lexicon: Semantics
• Selection restrictions– Pak (1) (suit): cloths– Pak (2) (package): entity– Dragen (1) (wear): subj=animate; object=cloths– Dragen (2) (carry): subj=animate; object= entity – Schoen: cloths– Entity > cloths– Identity preferred over subsumption– Homogeneous object preferred over heterogeneous one
12
Lexicon: Semantics
• Selection restrictions– Hij draagt een bruin pak
• He wears a brown suit (1: cloths=cloths) • He carries a brown package (1: entity=entity)• He carries a brown suit (2: entity > cloth)• *He wears a brown package (cloth ¬> entity)
– Hij draagt een bruin pak en zwarte schoenen• He wears a brown suit and black shoes (1: homogeneous and
cloths=cloths)• He carries a brown suit and black shoes (2: homogeneous but
entity > cloths)• He carries a brown package and black shoes(2:
inhomogeneous but entity=entity)• *He wears a brown package and black shoes (cloths ¬> entity)
13
Statistical MT
• Statistical MT • Derives MT-system automatically
– From statistics taken from• Aligned parallel corpora ( translation model)• Monolingual target language corpora ( language
model)• Being worked since early 90’s
14
Statistical MT
• Plus:– No or very limited grammar development– Includes language and world knowledge automatically
(but implicitly)– Based on actually occurring data– Currently many experimental and commercial systems
• Minus:– Requires large aligned parallel corpora– Unclear how much linguistics will be needed anyway– Probably restricted to very limited domains only
15
Statistical MT
• Google Translate (statistical MT)• Hij draagt een pak. √He wears a suit.• Hij draagt schoenen. √ He wears shoes.• Hij draagt bruine schoenen en een pak.
• √ He wears a suit and brown shoes. (!!)• Hij draagt het pakket √ He carries the package• Hij heeft een pak aan. *He has a suit.• Voert uw bedrijf sloten uit?
– *Does your company locks out?
16
Hybrid MT
• Euromatrix esp. “the Euromatrix”, and– Successor project EuromatrixPlus– …– Efficient inclusion of linguistic knowledge into statistical
machine translation– The development and testing of hybrid architectures for the
integration of rule-based and statistical approaches
17
Hybrid MT
• META-NET 2010-2013 (EU-funding)– Building a community with shared vision and strategic
research agenda– Building META-SHARE, an open resource exchange
facility– Building bridges to neighbouring technology fields
• Bringing more Semantics into Translation• Optimising the Division of Labour in Hybrid MT• Exploiting the Context for Translation• Empirical Base for Machine Translation
18
Hybrid MT
• PACO-MT 2008-2011• Investigates hybrid approach to MT
– Rule-based and statistical– Uses existing parser for source language
analysis– Uses statistical n-gram language models for
generation– Uses statistical approach to transfer
19
MT: What is (perhaps) possible
• Cross-Language Information Retrieval• Low Quality MT for Gist extraction• MT and Speech Technology• Controlled Language• Limited Domain• Interaction with author• Combinations of the above• Computer-aided translation
20
MT: What is (perhaps) possible
• Cross-Language Information Retrieval (CLIR)– Input query: in own language– Input query translated into target languages– Search in target language documents– Results in target language
• Translation of individual words only• Growing need (growing multilingual Web)• No perfect translation required
21
MT: What is (perhaps) possible
22
MT: What is (perhaps) possible
• Low quality MT for Gist extraction• Low quality but still useful• If interesting high quality human translation
can be requested (has to be paid for)
23
MT: What is (perhaps) possible
24
MT: What is (perhaps) possible
25
MT: What is (perhaps) possible
• CLIR– Fills a growing need in the market– Is technically feasible– Creates need for translation of found
documents• Solved partially by low quality MT• Potentially creates need for more human translation• Stimulates (funds) research into more sophisticated
MT
26
MT: What is (perhaps) possible
• Combine MT (statistical or rule-based) with OCR technology– Make a picture of a text with your phone– Text is OCR-ed– Text is translated– (usually a short and simple text)
• Linguatec Shoot & Translate• Word Lens
27
MT: What is (perhaps) possible
• Combine MT (statistical or rule-based) with Speech technology– Complicates the problem on the one hand but– Speech technology (ASR) is currently limited to very
limited domains (makes MT simpler)– Many useful applications for speech technology
currently in the market• Directory assistance Tourist Information• Tourist communication Call Centers• Navigation Hotel reservations
– Some will profit from in-built automatic translation
28
MT: What is (perhaps) possible
• Large EC FP6 project TC-STAR (2004-)– (http://www.tc-star.org/)– Research into improved speech technology
(ASR and TTS)– Research into statistical MT– Research in combining both (speech-to-speech
translation)– In a few selected limited domains
29
MT: What is (perhaps) possible
• Commercial Speech2Speech Translation• Jibbigo
– http://www.jibbigo.com• Speech-to-speech translation (iPhone, Android) • http://www.phonedog.com/2009/10/30/iphone-app-j
ibbigo-speech-translator
• Talk to Me (Android phones)
30
MT: What is (perhaps) possible
• Controlled Language– Authoring System limits vocabulary and syntax
of document authors– Often desirable in companies to get consistent
documentation (e.g. aircraft maintenance manuals)• AECMA Simplified English• GIFAS Rationalized French
– Makes MT easier (language well-defined)
31
MT: What is (perhaps) possible
• Limited Domain– Translation of
• Weather reports (TAUM-Meteo, Canada)• Avalanche warnings (Switzerland)
– Fast adaptation to domain/company-specific vocabulary and terminology
32
MT: What is (perhaps) possible
• Interaction with author– No fully automatic translation– Document author resolves
• Ambiguities unresolved by the system• In a dialogue between the author and the system in
the source language• Approach taken in Rosetta project (Philips)• Will only work if the
– #unresolved ambiguities is low– Questions to resolve ambiguity are clear
33
MT: What is (perhaps) possible
• Hij droeg een bruin pak– Wat bedoelt u met “pak”
• (1) kostuum• (2) pakket
• Hij droeg een bruin pak– Wat bedoelt u met “dragen (droeg)”
• (1) aan of op hebben (kleding)• (2) bij zich hebben (bijv. in de hand)
34
MT: What is (perhaps) possible
• Combinations of the above
35
MT: What is (perhaps) possible
• Computer-aided translation– For end-users– For professional translators/localization industry
• Limited functionality– Specific terminology
• Bootstrap translation automatically– Human revision and correction (Post-edit)
• Only if– MT Quality is such that it reduces effort– The system is fully integrated in the workflow system
36
Conclusions
• MT is really very difficult!• Even making a lexicon for an MT system is very
difficult (and a lot of work) • Statistical MT yields practical relatively quick to
produce systems (but low-quality)– Provided you have huge amounts of data
• Focus of research is on hybrid systems (mixed statistically based/knowledge based) (PACO-MT, META-NET,…)
37
Conclusions
• Several constrained versions do yield usable technology with state-of-the-art MT
• In some cases: even potentially creates additional needs for MT and human translation
38
– Try it yourself for lopen; innemen– Count/Mass: vis; wijn; bestek; meubilair
• http://www.vandale.nl/
39
Do not go beyond this slide
40
MT Evaluation
• Evaluation depends on purpose of MT and how it is used– application, domain, controlled language
• Many aspects can be evaluated– functionality, efficiency, usability, reliability,
maintainability, portability– translation quality– embedding in work flow
• post-editing options/tools
41
MT Evaluation
• Focus here:– does the system yield good translations
according to human judgement– in the context of developing a system
• Again, many aspects:– fidelity (how close), correctness, adequacy,
informativeness, intelligibility, fluency– and many ways to measure these aspects
42
MT Evaluation• Test suite
– Reference = • list of (carefully selected) sentences• with their translations (ordered by score)
– translations judged correct by human (usually developer)– upon every update of the system output of the new system is compared to the
reference• if different: system has to be adapted, or reference has to be adapted
• Advantages– focus on specific translation problems possible– excellent for regression testing– Manual judgement needed only once for each new output
• –other comparisons are automatic• Disadvantages
– not really independent– particularly suited for pure rule-based systems– human judgement needed if output differs from reference
43
MT Evaluation
• Comparison against– translation corpus– independently created by human translators– possibly multiple equivalently correct translations of a sentence
• Advantages– truely independent– also suited for data-driven systems
• Disadvantage– requires human judgement (every time there is a system update)
• high effort by highly skilled people, high costs, requires a lot of time– human judgement is not easy (unless there is a perfect match)
• Useful – for a one-time evaluation of a stable system– not for evaluation during development
44
MT Evaluation
• Edit-Distance (Word Accuracy)– metric to determine closeness of translations
automatically– the least number of edit operations to turn the
translated sentence into the reference sentence– Alshawi et al. 1998
45
MT Evaluation
• WA = 1- ((d+s+i)/max(r,c))• d= number of deletions• s = number of substitutions• i = number of insertions• r = reference sentence length• c = candidate sentence length• easy to calculate using Levenshtein distance
algorithm (dynamic programming)• various extensions have been proposed
46
MT Evaluation
• Advantages– fully automatic given a reference set
• Disadvantages– penalizes candidates if a synonym is used– penalizes swaps of words and block of words
too much
47
MT Evaluation
• BLEU (method to automate MT Evaluation)– the closer a machine translation is to a
professional human translation, the better it is– BiLingual Evaluation Understudy
• Required:– corpus of good quality human reference
translations– a “closeness” metric 48
MT Evaluation
• Two candidate translations from Chinese source– C1: It is a guide to action which ensures that
the military always obeys the commands of the party
– C2: It is to insure the troops forever hearing the activity guidebook that party direct
• Intuitively: C1 is better than C2
49
MT Evaluation
• Three reference translations– R1: It is a guide to action that ensures that the
military will forever heed Party commands– R2: It is the guiding principle which guarantees
the military forces always being under the command of the Party
– R3: It is the practical guide for the army always to heed the directions of the party
50
MT Evaluation
• Basic idea:– a good candidate translation shares many words
and phrases with reference translations– comparing n-gram matches can be used to
rank candidate translations• n-gram: a sequence of n word occurrences
– in BLEU n=1,2,3,4- 1-grams give a measure of adequacy- longer n-grams give a measure of fluency
51
MT Evaluation
• For unigrams:– count the number of matching unigrams
• in all references
– divide by the total number of unigrams (in the candidate sentence)
52
MT Evaluation
• Problem– C1: the the the the the the the (=7/7=1)– R1: the cat is on the mat
• Solution:– clip matching count (7) by maximum reference
count (2) 2 (CountClip)
– modified unigram precision = 2/7=0.29
53
MT Evaluation
• Example (unigrams)– C1: It is a guide to action which ensures that the
military always obeys the commands of the party (17/18=0.94)
– R1: It is a guide to action that ensures that the military will forever heed Party commands
– R2: It is the guiding principle which guarantees the military forces always being under the command of the Party
– R3: It is the practical guide for the army always to heed the directions of the party
54
MT Evaluation
• Example (unigrams)– C2: It is to insure the troops forever hearing the activity
guidebook that party direct (8/14=0.57)– R1: It is a guide to action that ensures that the military
will forever heed Party commands– R2: It is the guiding principle which guarantees the
military forces always being under the command of the Party
– R3: It is the practical guide for the army always to heed the directions of the party
55
MT Evaluation
• Example (bigrams)– C1: It is a guide to action which ensures that the
military always obeys the commands of the party (10/17=0.59)
– R1: It is a guide to action that ensures that the military will forever heed Party commands
– R2: It is the guiding principle which guarantees the military forces always being under the command of the Party
– R3: It is the practical guide for the army always to heed the directions of the party
56
MT Evaluation
• Example (bigrams)– C2: It is to insure the troops forever hearing the activity
guidebook that party direct (1/13=0.08)– R1: It is a guide to action that ensures that the military
will forever heed Party commands– R2: It is the guiding principle which guarantees the
military forces always being under the command of the Party
– R3: It is the practical guide for the army always to heed the directions of the party
57
MT Evaluation
• Extend to a full multi-sentence corpus• compute n-gram matches sentence by sentence• sum the clipped n-gram counts for all candidates• divide by the number of n-grams in the text corpus• pn =
– ∑C ∈ {Candidates}∑n-gram ∈ C Countclip(n-gram)– divided by– ∑C’ ∈ {Candidates}∑n-gram’ ∈ C’ Count(n-gram’)
58
MT Evaluation
• Combining n-gram precision scores• weighted linear average works reasonable
– ∑Nn=1 wn pn
• but: n-gram decisions decays exponentially with n (so log to compensate for this)– exp (∑N
n=1 wn log pn)
• weights in BLEU: wn = 1/N
59
MT Evaluation
• BLEU is a precision measure– #(C ∩ R) / #C
• Recall is difficult to define because of multiple reference translations– e.g. #(C ∩ Rs) / # Rs
• where Rs = Ui Ri
– will not work
60
MT Evaluation
• C1: I always invariably perpetually do• C2: I always do• R1: I always do• R2: I invariably do• R3: I perpetually do• Recall of C1 over R1-3 is better than C2• but C2 is a better translation
61
MT Evaluation
• But without Recall:– C1: of the– compared with R1-3 as before– modified unigram precision = 2/2– modified bigram precision = 1/1– which is the wrong result
62
MT Evaluation
• Length– n-gram precision penalizes translations longer
than the reference– but not translations shorter than the reference– Add Brevity Penalty (BP)
63
MT Evaluation
• bi= best match length = reference sentence length closest to candidate sentence i‘s length (e.g. r:12, 15, 17, c: 12 12)
• r = test corpus effective reference length = ∑i bi
• c = total length of candidate translation corpus
64
MT Evaluation
• BP = – computed over the corpus– not sentence by sentence and averaged– 1 if c > r– e(1-r/c) if c <= r
• BLEU = BP • exp (∑Nn=1 wn log pn)
65
MT Evaluation
• BLEU:– claim: BLEU closely matches human judgement
• when averaged over a test corpus• not necessarily on individual sentences• shown extensively in Papineni et al. 2001
– multiple reference translations are desirable• to cancel out translation styles of individual translators• (e.g. East Asian economy v. economy of East Asia)
66
MT Evaluation
• Variants on BLEU– NIST
• http://www.nist.gov/speech/tests/mt/doc/ngram-study.pdf
• different weights• different BP
– ROUGE (Lin and Hovy 2003) • for text summarization• Recall-Oriented Understudy for Gisting Evaluation
67
MT Evaluation
• Main Advantage of BLEU– automatic evaluation
• good for use during development• particularly useful for data-based systems
• Disadvantage– defined for a whole test corpus– not for individual sentences– just measures difference with reference
68