+ h/nRt^RIt^RIHt^RIHtHtHtHt^RI H t !RR]4t !RR] 4t R#) a Lexical translation model that considers word order. IBM Model 2 improves on Model 1 by accounting for word order. An alignment probability is introduced, a(i | j,l,m), which predicts a source word position, given its aligned target word's position. The EM algorithm used in Model 2 is: :E step: In the training data, collect counts, weighted by prior probabilities. - (a) count how many times a source language word is translated into a target language word - (b) count how many times a particular position in the source sentence is aligned to a particular position in the target sentence :M step: Estimate new probabilities based on the counts from the E step Notations --------- :i: Position in the source sentence Valid values are 0 (for NULL), 1, 2, ..., length of source sentence :j: Position in the target sentence Valid values are 1, 2, ..., length of target sentence :l: Number of words in the source sentence, excluding NULL :m: Number of words in the target sentence :s: A word in the source language :t: A word in the target language References ---------- Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. N defaultdict) AlignedSent AlignmentIBMModel IBMModel1)Countscjaa]tRt^;toRtR V3RlltRtRtRtRt Rt Rt R t R t R tVtV;t#) IBMModel2u Lexical translation model that considers word order >>> bitext = [] >>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus', 'ist', 'ja', 'groß'], ['the', 'house', 'is', 'big'])) >>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house'])) >>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book'])) >>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book'])) >>> ibm2 = IBMModel2(bitext, 5) >>> print(round(ibm2.translation_table['buch']['book'], 3)) 1.0 >>> print(round(ibm2.translation_table['das']['book'], 3)) 0.0 >>> print(round(ibm2.translation_table['buch'][None], 3)) 0.0 >>> print(round(ibm2.translation_table['ja'][None], 3)) 0.0 >>> print(round(ibm2.alignment_table[1][1][2][2], 3)) 0.939 >>> print(round(ibm2.alignment_table[1][2][2][2], 3)) 0.0 >>> print(round(ibm2.alignment_table[2][2][4][5], 3)) 1.0 >>> test_sentence = bitext[2] >>> test_sentence.words ['das', 'buch', 'ist', 'ja', 'klein'] >>> test_sentence.mots ['the', 'book', 'is', 'small'] >>> test_sentence.alignment Alignment([(0, 0), (1, 1), (2, 2), (3, 2), (4, 3)]) c <<\SV`V4Vf7\V^V,4pVPVnVP V4MVR,VnVR,Vn\ ^V4FpVPV4K VPV4R#)a Train on ``sentence_aligned_corpus`` and create a lexical translation model and an alignment model. Translation direction is from ``AlignedSent.mots`` to ``AlignedSent.words``. :param sentence_aligned_corpus: Sentence-aligned parallel corpus :type sentence_aligned_corpus: list(AlignedSent) :param iterations: Number of iterations to run training algorithm :type iterations: int :param probability_tables: Optional. Use this to pass in custom probability values. If not specified, probabilities will be set to a uniform distribution, or some other sensible value. If specified, all the following entries must be present: ``translation_table``, ``alignment_table``. See ``IBMModel`` for the type and purpose of these tables. :type probability_tables: dict[str]: object Ntranslation_tablealignment_table) super__init__rr set_uniform_probabilitiesr rangetrain align_all)selfsentence_aligned_corpus iterationsprobability_tablesibm1n __class__s&&&& 5/usr/lib/python3/dist-packages/nltk/translate/ibm2.pyrIBMModel2.__init__cs, 01  %4a*nED%)%;%;D "  * *+B C&88K%LD "#56G#HD q*%A JJ. /& ./c \4pVFp\VP4p\VP4pWE3V9gK7VP WE34^V^,, pV\ P 8d.\P!R\V4,R,4\^V^,4F@p\^V^,4F&pW`PV,V,V,V&K( KB K R#)zA source sentence is too long (z& words). Results may be less accurate.N) setlenmotswordsaddrMIN_PROBwarningswarnstrrr ) rrl_m_combinationsaligned_sentencelm initial_probijs && rr#IBMModel2.set_uniform_probabilitiess5 7 $))*A$**+Av-- $$aV, AE{ ("3"33MM9a&!BC q!a%A"1a!e_;G,,Q/215a8-)!8rc J\4pVFpR.VP,pR.VP,p\VP4p\VP4pVP WE4p\ ^V^,4Fsp WY,p \ ^V^,4FQp WK,p VP WWE4p WV ,, pVPWV 4VPWWV4KS Ku K VPV4VPV4R#)NUNUSED) Model2Countsr"r#r!prob_all_alignmentsrprob_alignment_pointupdate_lexical_translationupdate_alignment*maximize_lexical_translation_probabilities maximize_alignment_probabilities)rparallel_corpuscountsr* src_sentence trg_sentencer+r, total_countr/tr.scountnormalized_counts&& rrIBMModel2.trains /  6$4$9$99L$:(8(>(>>L$))*A$**+A22<NK1a!e_ Oq!a%A$A 55aLWE',1~'=$556F1M++,<qI )%!0* 77? --f5rc\PpVPP4Fwr4VP4FwrVVP4FwrxVF~p VPV,V,V,V ,VPV,V,V ,, p \ W4VP V,V,V,V &K K K K R#N)rr% alignmentitemsalignment_for_any_imaxr ) rr;r%r.j_sr/src_sentence_lengthsr+trg_sentence_lengthsr,estimates && rr9*IBMModel2.maximize_alignment_probabilitiess$$&&,,.FA+.99;'/C/I/I/K+A1",,Q/215a8$88;A>qAB!**1q51!a%8;A>?* *I+,( =  ! !1"; <#:&#,N"; r)r r rE)__name__ __module__ __qualname____firstlineno____doc__rrrr9r4r5r[rr^__static_attributes____classdictcell__ __classcell__r __classdict__s@@rr r ;sA%N'0RH(64 S$. O,*&&<&KModel2Counts.__init__......3s K...3s 4N(Orrrxrrry'Model2Counts.__init__..3s K OPrc\R4#)c \\4#rErwrxrrryr{6s E(:rrrxrrryr|6s K :;rN)rrrrFrH)rrs&rrModel2Counts.__init__0s/ $ P $/ ;$  rcVPV,V;;,V, uu&VPV;;,V, uu&R#rE) t_given_s any_t_given_s)rrAr@r?s&&&&rr6'Model2Counts.update_lexical_translation9s1 q!% 1&rcVPV,V,V,V;;,V, uu&VPV,V,V;;,V, uu&R#rErFrH)rrAr.r/r+r,s&&&&&&rr7Model2Counts.update_alignment=sE q!Q"e+"   #A&q)U2)rr) rhrirjrkrlrr6r7rmrnrorps@@rr3r3*s  '33rr3) rlr& collectionsrnltk.translaterrrrnltk.translate.ibm_modelrr r3rxrrrs7*X#FF+l<l<^363r