+ hn`Rt^RIt^RIt^RIt^RIHtHtHt^RIH t ^RI H t ^RI H t ^RIHt^RIHt^RIHtHt^R IHt^R IHt^R IHt^R IHtHtHt]!R .RO4t!RR4t !RR4t!!RR4t"!RR4t#!RR]#4t$Rt%]&R8Xd]%!4.ROt'R#)a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation cut_string tokenwrapConcordanceLinecfa]tRt^'toRt]R4tRRR3RltRtRt R Rlt R R lt R t Vt R#) ContextIndexz A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. c V^8wdW^, ,P4MRpV\V4^, 8wdW^,,P4MRpW#3#)z;One left token and one right token, normalized to lowercase*START**END*)lowerlen)tokensileftrights&& +/usr/lib/python3/dist-packages/nltk/text.py_default_contextContextIndex._default_context/sM)*Qv!e}""$I)*c&kAo)=1u ##%7}NcV#Nxs&rContextIndex.6sQr!cTaaVSnSSnV'd VSnMSPSnV'd"SUu.uFqS!V4'gKVNK upo\ VV3Rl\ S444Sn\ VV3Rl\ S444SnR#uupi)c3n<"TF*wrSPV4SPSV43xK, R#5ir#)_key _context_func.0rwselfrs& r (ContextIndex.__init__..?s1% FWdaTYYq\4--fa8 9FW25c3n<"TF*wrSPSV4SPV43xK, R#5ir#)r,r+r-s& rr1r2Bs1% FWdaT   *DIIaL 9FWr3N)r+_tokensr,rCFD enumerate_word_to_contexts_context_to_words)r0r context_funcfilterkeytsff&&& r__init__ContextIndex.__init__6s  !-D !%!6!6D  !'5A6!9aa5F!$% FOPVFW% " "%% FOPVFW% "  6s B%B%c VP#)zW :rtype: list(str) :return: The document that this context index was created from. r5r0s&rrContextIndex.tokensF ||r!c VPV4p\VPV,4p/pVPP4FwrE\ V\V44W4&K V#)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r+setr8itemsr )r0word word_contextsscoresr/ w_contextss&& rword_similarity_dict!ContextIndex.word_similarity_dictNs_ yyD22489 !3399;MA!-ZAFI< r!c\\4pVPVPV4,FppVPV,FVpWQ8wgK W5;;,VPV,V,VPV,V,,, uu&KX Kr \ W3P RR7RV#)T)r<reverseN)rintr8r+r9sortedget)r0rHnrJcr/s&&& r similar_wordsContextIndex.similar_words]sS!'' $88A++A..9I..q1$7$:P:PQR:STU:VVI/9 f**d;BQ??r!c aaVUu.uFpSPV4NK ppVUu.uFp\SPV,4NK! pp\\ V44Uu.uFqTV,'dKW,NK pp\ \P V4oV'd$V'd\RRPV44hS'g \4#\VV3RlV44pV#uupiuupiuupi)ag Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. z%The following word(s) were not found: c3l<"TF)pSPV,Fq"S9gK VxK K+ R#5ir#)r8)r.r/rTcommonr0s& rr1/ContextIndex.common_contexts..|s1 a$*@*@*C*CQF{*C5s44) r+rFr8rangerr intersection ValueErrorjoinr ) r0wordsfail_on_unknownr/contextsremptyfdrZs f&& @rcommon_contextsContextIndex.common_contextsgs(--u!1u-rrLrUre__static_attributes____classdictcell__ __classdict__s@rrr'sF -1;  @r!rcXa]tRt^toRtR3RltRtRtRtR Rlt R Rlt R t Vt R #) ConcordanceIndexzg An index that can be used to look up the offset locations at which a given word occurs in a document. cV#r#r$r%s&rr'ConcordanceIndex.sQr!c WnW n\\4Vn\ V4F8wr4VPV4pVPV,P V4K: R#)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r5r+rlist_offsetsr7append)r0rr<indexrHs&&& rr>ConcordanceIndex.__init__s\   D#D) L$V,KE99T?D MM$  & &u --r!c VP#)z[ :rtype: list(str) :return: The document that this concordance index was created from. rArBs&rrConcordanceIndex.tokensrDr!c JVPV4pVPV,#)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r+ryr0rHs&&roffsetsConcordanceIndex.offsetss yy}}T""r!cdR\VP4\VP43,#)z+)rr5ryrBs&r__repr__ConcordanceIndex.__repr__s-<    @   r!c  \V\4'dTpMV.pRPV4p\RV44pW%, ^, ^,pV^,p.pVP V^,4p \ VR,4FLwrVP V4U u0uFqV , ^, kK p p \ V PV 44p KN V 'dV Fp RPVPW\V4,4p VP\^W, 4V pVPV \V4,W,p\RPV4V)4PV4p\RPV4V4pRPVV V.4p\VV VV VVV4pVPV4K V#uup i)zs Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. rXc3b"TF%p\P!V4'dK!^xK' R#5i)N) unicodedata combining)r.chars& rr14ConcordanceIndex.find_concordance..sUzt9N9Nt9Tzs/ /:rNN) isinstancerxr_sumrr7rQr]r5rmaxrrjustrrz)r0rHwidthphrase phrase_str phrase_len half_widthcontextconcordance_listrroffset word_offsets query_word left_context right_context left_print right_print line_printconcordance_lines&&& rfind_concordance!ConcordanceIndex.find_concordances dD ! !FVFXXf% UzUU (1,2 1*,,vay) ,GA9=d9KL9KvQJNN9KLL\66w?@G-  XXdll13v;&GH #||C1;,?!D $ QV_q{ K '(> LRR )-)@*M  XXz:{&KL #2 !$ !''(89-. 5Ms#Gc VPWR7pV'g\R4R#\V\V44p\RV R\V4 R24\ VRV4FwrV\VP 4K R#)aa Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )rz no matchesz Displaying z of z matches:N)rprintminrr7line)r0rHrlinesrrrs&&&& rprint_concordance"ConcordanceIndex.print_concordances} 000C , s#345E Kwd3/?+@*AK L'01A&51I'J#&++,(Kr!)r+ryr5N)P)r) rirjrkrlrmr>rrrrrrorprqs@rrtrts3 $/.4# . `--r!rtc0a]tRtRtoRtRtRtRtVtR#) TokenSearcheria A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. c@RPRV44VnR#)c3B"TFpRV,R,xK R#5i)<>Nr$)r.r/s& rr1)TokenSearcher.__init__.. s:6aC!GcMM6sN)r__raw)r0rs&&rr>TokenSearcher.__init__ sGG:6:: r!c \P!RRV4p\P!RRV4p\P!RRV4p\P!RRV4p\P!WP4pVF>pVP R4'dKVP R4'gK5\ R 4h VUu.uFq3^R PR 4NK ppV#uupi) ak Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srrz(?:<(?:rz)>)z (?]z$Bad regexp for TokenSearcher.findallz><)resubfindallr startswithendswithr^splitr0regexphitshs&& rrTokenSearcher.findalls0r6*i0eV, ff5zz&)),A<<$$C !GHH .22T!B d#T2 3s C))rN) rirjrkrlrmr>rrorprqs@rrrs;''r!rca]tRtRtoRtRtRRltRtRtRRlt RR lt RR lt RR lt R t R tRtRRltRRltRtR RltR!RltRtRtRt]P2!R4tRtRtRtRtVtR#)"Texti9a A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNc RVP'd \V4pWnV'd W nR#RVR,9d;VR,P R4pRP RV^V44VnR#RP RVR,44R,VnR#) zV Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]:NrhNrXc38"TFp\V4xK R#5ir#strr.toks& rr1 Text.__init__..bs C]cS]c38"TFp\V4xK R#5ir#rrs& rr1rds @ZcSZr:NNz...N) _COPY_TOKENSrxrnamer{r_)r0rrends&&& rr> Text.__init__Ss    &\F I F3K +##C(C CVAc] CCDI @VBZ @@5HDIr!c(VPV,#r#)r)r0rs&&r __getitem__Text.__getitem__js{{1~r!c,\VP4#r#)rrrBs&r__len__ Text.__len__ms4;;r!c RVP9d\VPRR7VnVPP WV4#)a| Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexc"VP4#r#rss&rr'"Text.concordance.. 1779r!r<)__dict__rtrrrr0rHrrs&&&&r concordanceText.concordancetsB t}} 4&6 !4'D #&&88eLLr!c RVP9d\VPRR7VnVPP W4RV#)a~ Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rc"VP4#r#rrs&rr''Text.concordance_list..rr!rN)rrtrrrrs&&&&rrText.concordance_listsG t}} 4&6 !4'D #&&77DVeLLr!c aRVP9d#VPV8XdVPV8XgYnW n^RIHpVP R4o\ P!VPV4pVP^4VPV3Rl4\4p\VPVPV44VnVP#)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocations) stopwordsenglishcV<\V4^8;'gVP4S9#)rr)r/ ignored_wordss&rr''Text.collocation_list..s#s1vz/W/WQWWY-=W/Wr!)r_num _window_size nltk.corpusrr`r from_wordsrapply_freq_filterapply_word_filterr rxnbestlikelihood_ratior)r0num window_sizerfinderbigram_measuresrs&&& @rcollocation_listText.collocation_lists t}} , S !![0I +  .%OOI6M,77 [QF  $ $Q '  $ $%W X13O!% _==sC"D !!!r!c VPW4UUu.uFwr4VR,V,NK ppp\\VRR74R#uuppi)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE United States; fellow citizens; years ago; four years; Federal Government; General Government; Vice President; American people; God bless; Chief Justice; one another; fellow Americans; Old World; Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian tribes; public debt; foreign nations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int rX; ) separatorN)rrr)r0rrw1w2collocation_stringss&&& r collocationsText.collocationssO()-(=(=c(O (OfbBHrMM(O   i+t<= sA c 8VPPV4#)z: Count the number of times this word appears in the text. )rcountrs&&rr  Text.count{{  &&r!c 8VPPV4#)zA Find the index of the first occurrence of the word in the text. )rr{rs&&rr{ Text.indexr r!c\hr#)NotImplementedError)r0methods&&r readabilityText.readabilitys!!r!c aaaRVP9d \VPRRR7VnSP 4oVPP oSSP 49dr\SS,4o\VVV3RlSP 444pVPV4UUu.uFwrEVNK ppp\\V44R#\R4R#uuppi)a> Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexc"VP4#r#)isalphar%s&rr'Text.similar..s aiikr!c"VP4#r#rrs&rr'rs r!)r;r<c3l<"TF)pSV,FpVS9gK VS8XdKVxK K+ R#5ir#r$)r.r/rTrbwcirHs& rr1Text.similar..sB)AQA=*+d)s 444z No matchesN) rrrrrr8 conditionsrFr most_commonrr) r0rHrrdr/_r`rbrs &f& @@rsimilar Text.similars ! 5'3 $9?R(D $ zz|&&88 3>># #3t9~H)B $&>>##67#641Q#6E7 )E" # , 8s: C-c RVP9d\VPRR7VnVPP VR4pV'g\ R4R#VP V4UUu.uFwrEVNK ppp\ \RV444R#uuppi \dp\ T4Rp?R#Rp?ii;i)a Find contexts where the specified words appear; list most frequent common contexts first. :param words: The words used to seed the similarity search :type words: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.common_contexts() rc"VP4#r#rrs&rr'&Text.common_contexts..rr!rTzNo common contexts were foundc3F"TFwrVR,V,xK R#5i)r Nr$)r.rrs& rr1'Text.common_contexts..!sLO&"S2 Os!N) rrrrrerrrr^)r0r`rrdr/r ranked_contextses&&& rreText.common_contexts s ! 5'3 !4(D $ ))99%FB56131D"E1D11D"EiLOLLM#F  !HH s5#B( B(!B(5 B"B("B(( C 3 CC c "^RIHpV!W4R#)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() )dispersion_plotN) nltk.drawr,)r0r`r,s&& rr,Text.dispersion_plot&s .$r!cZ\W!4wr4\VR7pVPW44V#))order)r rfit)r0tokenized_sentsrS train_data padded_sentsmodels&&& r_train_default_ngram_lmText._train_default_ngram_lm3s)#\ R\PR7VPVP^R7Vn .pV^8gQR4h\V4V8d^\VPPWVR74F(wrgVR8XdKVR 8XdMVPV4K* V^, pKmV'dRPV4R,MR pV\VR V4,p \ V 4V #uupi) a Print random text, generated using a trigram language model. See also `help(nltk.lm)`. :param length: The length of text to generate (default=100) :type length: int :param text_seed: Generation can be conditioned on preceding context. :type text_seed: list(str) :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. (default=42) :type random_seed: int rX_trigram_modelzBuilding ngram index...)file)rSz!The `length` must be more than 0.) text_seed random_seedzzrN)rr_rr_tokenized_sentshasattrrsysstderrr6r9rr7generaterzr) r0lengthr;r<sentgenerated_tokensidxtokenprefix output_strs &&&& rrA Text.generate9sC")6chht{{6K(L! (LJJsO(L! t-.. +#** ="&">">%%#?#D z>>>z"#f,'##,,[-  E>F? ''. 1 K.7)$s*Ri(8&(ABB  j9! sEc >VP4P!V!#)zK See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r0argss&*rrL Text.plotgs zz|  $''r!c \RVP9d\V4VnVP#)z :seealso: nltk.prob.FreqDist _vocab)rr rPrBs&rrK Text.vocabns% 4== ("4.DK{{r!c RVP9d\V4VnVPPV4pVUu.uFpRP V4NK pp\ \ VR44R#uupi)aC Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searcherrXrN)rrrSrr_rrrs&& rr Text.findallwsc. DMM 1#0#6D ##++F3%)*T T* id#$+sA4z \w+|[\.\!\?]c xV^, pV^8d3VPPW,4'g V^,pK9V^8wd W,MRpV^,pV\V48d3VPPW,4'g V^, pKBV\V48wd W,MRpWE3#)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rr) _CONTEXT_REmatchr)r0rrjrrs&&& r_context Text._contexts E1fT--33FI>> FAFvy  E#f+od&6&6&<&rrBs&r__str__ Text.__str__dii''r!c(RVP,#r\r]rBs&rr Text.__repr__r`r!) rrrrSr=r9rPrrrrr#)Or)rhrgr)dN*) rirjrkrlrmrr>rrrrrrr r{rr!rer,r6rArLrKrrcompilerVrYr^rrorprqs@rrr9s.LI. M*M(!"F>0' ' "  D8 % ,\(%D**_-K0(((r!rc<a]tRtRtoRtRtRtRtRtRt Vt R#) TextCollectioniaA collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> from nltk.book import text1, text2, text3 >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. c\VR4'd.VP4Uu.uFq!PV4NK ppWn\P V\ V44/VnR#uupi)r`N)r>fileidsr`_textsrr>r _idf_cache)r0sourcefs&& rr>TextCollection.__init__sX 67 # #/5~~/?@/?!ll1o/?F@  d-f56 AsA.c DVPV4\V4, #)z"The frequency of the term in text.)r rr0termtexts&&&rtfTextCollection.tfszz$#d)++r!c lVPPV4pVf\VPUu.uF q1V9gK RNK up4p\VP4^8Xd \ R4hV'd&\ \VP4V, 4MRpW PV&V#uupi)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Tz+IDF undefined for empty document collectiong)rmrRrrlr^r)r0rsidfrtmatchess&& rrxTextCollection.idfs oo!!$' ;DKKHKD4<44KHIG4;;1$ !NOO5<#c$++&01#C$'OOD ! Is B1B1cPVPW4VPV4,#r#)rurxrrs&&&rtf_idfTextCollection.tf_idfswwt"TXXd^33r!)rmrlN) rirjrkrlrmr>rurxr|rorprqs@rriris# , 44r!ric^RIHp\VPRR74p\ V4\ 4\ R4VP R4\ 4\ R4VP R4\ 4\ R4VP4\ 4\ R4VP.RO4\ 4\ R4VP^24\ 4\ R 4\ R V^,4\ R VR ,4\ R VP4R,4R#))brownnews) categoriesz Concordance:zDistributionally similar words:z Collocations:zDispersion plot:zVocabulary plot:z Indexing:ztext[3]:z text[3:5]::rNztext.vocab()['news']:N)rreportsaid announced) rrrr`rrr!rr,rLrK)rrts rdemors!  v . /D $K G .V G +,LL G / G @A G IIbM G + *d1g ,S " !4::<#78r!__main__)rqueryrrrrr)rrtrrri)(rmrr?r collectionsrrr functoolsrmathrnltk.collocationsrnltk.lmrnltk.lm.preprocessingr nltk.metricsr r nltk.probabilityr r6r nltk.tokenizer nltk.utilrrrrrrtrrrirri__all__r$r!rrs 885;77%'>>M XXv|-|-~55p~(~(D +4T+4\9< zF r!