+ h Rt^RIt^RIt^RIHtHt^RIHt^RIHt^RI H t ^RI H t ^RI Ht!RR ]R 7tR tR tRR lt!RR]R 7tR#)zLanguage Model Interface.N)ABCMetaabstractmethod)bisect) accumulate) NgramCounter) log_base2) VocabularycJa]tRt^toRtRt]R4t]R4tRt Vt R#) SmoothingzNgram Smoothing Interface Implements Chen & Goodman 1995's idea that all smoothing algorithms have certain features in common. This should ideally allow smoothing algorithms to work both with Backoff and Interpolation. c WnW nR#)z :param vocabulary: The Ngram vocabulary object. :type vocabulary: nltk.lm.vocab.Vocabulary :param counter: The counts of the vocabulary items. :type counter: nltk.lm.counter.NgramCounter N)vocabcounts)self vocabularycounters&&&-/usr/lib/python3/dist-packages/nltk/lm/api.py__init__Smoothing.__init__s  c\4hNNotImplementedError)rwords&&r unigram_scoreSmoothing.unigram_score& !##rc\4hrrrrcontexts&&&r alpha_gammaSmoothing.alpha_gamma*rr)r r N) __name__ __module__ __qualname____firstlineno____doc__rrrr __static_attributes____classdictcell__ __classdict__s@rr r s7$$$$rr ) metaclassc8\V4\V4, #)z0Return average (aka mean) for sequence of items.)sumlen)itemss&r_meanr0/s u:E ""rcr\V\P4'dV#\P!V4#r) isinstancerandomRandom)seed_or_generators&r_random_generatorr64s*#V]]33  ==* ++rcV'g \R4h\V4\V48wd \R4h\\V44pVR,pVP 4pV\ W4V,4,#)zXLike random.choice, but with weights. Heavily inspired by python 3.6 `random.choices`. z"Can't choose from empty populationz3The number of weights does not match the population) ValueErrorr.listrr3r) populationweightsrandom_generator cum_weightstotal thresholds&&& r_weighted_choicerA:sj =>> :#g,&NOOz'*+K OE '')I f[)*;< ==rc|a]tRt^ItoRtR RltRRltRRlt]RRl4t RRlt Rt R t R t RR ltR tVtR#) LanguageModelzCABC for Language Models. Cannot be directly instantiated itself. Nc WnV'dH\V\4'g2\P!RVP P : R2^R7Vf \4MTVnVf\4Vn R#TVn R#)aCreates new LanguageModel. :param vocabulary: If provided, this vocabulary will be used instead of creating a new one when training. :type vocabulary: `nltk.lm.Vocabulary` or None :param counter: If provided, use this object to count ngrams. :type counter: `nltk.lm.NgramCounter` or None :param ngrams_fn: If given, defines how sentences in training text are turned to ngram sequences. :type ngrams_fn: function or None :param pad_fn: If given, defines how sentences in training text are padded. :type pad_fn: function or None z$The `vocabulary` argument passed to z- must be an instance of `nltk.lm.Vocabulary`.) stacklevelN) orderr2rwarningswarn __class__r"r rr )rrFrrs&&&&rrLanguageModel.__init__Psi jZ@@ MM6t~~7N7N6QR??  &0%7Z\Z (/ln W rc aSP'g+Vf \R4hSPPV4SPPV3RlV44R#)zUTrains the model on a text. :param text: Training text as a sequence of sentences. Nz:Cannot fit without a vocabulary or text to create it from.c3Z<"TF pSPPV4xK" R#5ir)r lookup).0sentrs& r $LanguageModel.fit..ts#Dtt4::,,T22ts(+)r r9updater )rtextvocabulary_textsf&&rfitLanguageModel.fithsP zzz& P JJ  o . DtDDrc TPVPPV4V'd VPPV44#R4#)zMasks out of vocab (OOV) words and computes their model score. For model-specific logic of calculating scores, see the `unmasked_score` method. N)unmasked_scorer rMrs&&&rscoreLanguageModel.scorevsI "" JJ  d #7TZZ%6%6w%?  PT  rc \4h)aScore a word given some optional context. Concrete models are expected to provide an implementation. Note that this method does not mask its arguments with the OOV label. Use the `score` method for that. :param str word: Word for which we want the score :param tuple(str) context: Context the word is in. If `None`, compute unigram score. :param context: tuple(str) or None :rtype: float rrs&&&rrXLanguageModel.unmasked_scores "##rc 6\VPW44#)zwEvaluate the log score of this word in this context. The arguments are the same as for `score` and `unmasked_score`. )rrYrs&&&rlogscoreLanguageModel.logscores D233rc V'd+VP\V4^,,V,#VPP#)zHelper method for retrieving counts for a given context. Assumes context has been checked and oov words in it masked. :type context: tuple(str) or None )r r.unigrams)rrs&&rcontext_countsLanguageModel.context_countss77>DKKG q( )' 2 CG;;CWCW rc  R\VUu.uFq PVR,VRR4NK up4,#uupi)aCalculate cross-entropy of model for given evaluation text. This implementation is based on the Shannon-McMillan-Breiman theorem, as used and referenced by Dan Jurafsky and Jordan Boyd-Graber. :param Iterable(tuple(str)) text_ngrams: A sequence of ngram tuples. :rtype: float Nr8)r0r^)r text_ngramsngrams&& rentropyLanguageModel.entropys@E?J K{e]]59eCRj 1{ K   Ks$< c 8\RVPV44#)z~Calculates the perplexity of the given text. This is simply 2 ** cross-entropy for the text, so the arguments are the same. g@)powrg)rres&&r perplexityLanguageModel.perplexitys 3 [122rc  aaVf.M \V4p\V4pV^8XEd\V4SP8dVSP)^,RMToSP SP P S44pS'dPV'gH\S4^8d SR,M.oSP SP P S44pKW\V4p\T\;QJd.VV3RlV4FNK 5V4#!VV3RlV44V4#.p\V4F,pVPSP^W&,VR74K. V#)ahGenerate words from the model. :param int num_words: How many words to generate. By default 1. :param text_seed: Generation can be conditioned on preceding context. :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. :return: One (str) word or a list of words generated from model. Examples: >>> from nltk.lm import MLE >>> lm = MLE(2) >>> lm.fit([[("a", "b"), ("b", "c")]], vocabulary_text=['a', 'b', 'c']) >>> lm.fit([[("a",), ("b",), ("c",)]]) >>> lm.generate(random_seed=3) 'a' >>> lm.generate(text_seed=['a']) 'b' N:NNc3H<"TFpSPVS4xK R#5ir)rY)rNwrrs& rrP)LanguageModel.generate..s>gdjjG,,gs") num_words text_seed random_seed) r:r6r.rFrbr rMsortedrAtuplerangeappendgenerate) rrrrsrtr=samples generated_rs f&&& @rryLanguageModel.generatesD*$+Bi ,[9 >y>TZZ/4::+/+,  ))$***;*;G*DEG'),W)9'"+r--djj.?.?.HIWoG#>g>  >g>>    y!A    '3 0 "r)r rFr )NNr)rnNN)r"r#r$r%r&rrUrYrrXr^rbrgrkryr'r(r)s@rrCrCIsM E0 E  $ $4    355rrCr)r&r3rGabcrrr itertoolsrnltk.lm.counterr nltk.lm.utilrnltk.lm.vocabularyrr r0r6rArCrrrsK ' (")$'$6# , >eger