+ h$l^RIt^RIt^RIHtHtHt^RIHt^RIH t !RR4t !RR]4t R#) N)IteratorListTuple) TokenizerI) align_tokensc2]tRt^tRt.ROtRR.tRR.tRtR#) MacIntyreContractionszA List of contractions adapted from Robert MacIntyre's tokenizer. z(?i) ('t)(?#X)(is)\bz(?i) ('t)(?#X)(was)\bz(?i)\b(whad)(dd)(ya)\bz(?i)\b(wha)(t)(cha)\bN)z(?i)\b(can)(?#X)(not)\bz(?i)\b(d)(?#X)('ye)\bz(?i)\b(gim)(?#X)(me)\bz(?i)\b(gon)(?#X)(na)\bz(?i)\b(got)(?#X)(ta)\bz(?i)\b(lem)(?#X)(me)\bz(?i)\b(more)(?#X)('n)\bz(?i)\b(wan)(?#X)(na)(?=\s)) __name__ __module__ __qualname____firstlineno____doc__ CONTRACTIONS2 CONTRACTIONS3 CONTRACTIONS4__static_attributes__r ;/usr/lib/python3/dist-packages/nltk/tokenize/destructive.pyr r s& M-.FGM.0HIMrr c a]tRt^%toRt]P !R]P4R3]P !R4R3]P !R4R3]P !R4R3]P !R ]P4R 3.t]P !R ]P4R3]P !R 4R 3]P !R4R 3]P !R4R3]P !R4R3]P !R4R3.t ]P !R]P4R3]P !R4R3]P !R4R3]P !R]P4R3]P !R4R3]P !R4R3]P !R4R3]P !R4R 3]P !R!]P4R3. t ]P !R"4R3t ]P !R#4R$3]P !R%4R&3]P !R'4R(3]P !R)4R*3]P !R+4R,3]P !R-4R.3.t ]P !R/4R03t ]!4t]!]!]P ]P$44t]!]!]P ]P&44tR7V3R1lR2lltV3R3lR4ltR5tVtR6#)8NLTKWordTokenizera! The NLTK tokenizer that has improved upon the TreebankWordTokenizer. This is the method that is invoked by ``word_tokenize()``. It assumes that the text has already been segmented into sentences, e.g. using ``sent_tokenize()``. The tokenizer is "destructive" such that the regexes applied will munge the input string to a state beyond re-construction. It is possible to apply `TreebankWordDetokenizer.detokenize` to the tokenized outputs of `NLTKDestructiveWordTokenizer.tokenize` but there's no guarantees to revert to the original string. u([«“‘„]|[`]+)z \1 z^\"``z(``)z([ \(\[{<])(\"|\'{2})z\1 `` z$(?i)(\')(?!re|ve|ll|m|t|s|d|n)(\w)\bz\1 \2u ([»”’])''z '' "z\s+ z([^' ])('[sS]|'[mM]|'[dD]|') z\1 \2 z)([^' ])('ll|'LL|'re|'RE|'ve|'VE|n't|N'T) u&([^\.])(\.)([\]\)}>"\'»”’ ]*)\s*$z \1 \2 \3 z ([:,])([^\d])z \1 \2z([:,])$z\.{2,}z \g<0> z[;@#$%&]z([^\.])(\.)([\]\)}>"\']*)\s*$z\1 \2\3 z[?!]z([^'])' z\1 ' z[*]z[\]\[\(\)\{\}\<\>]z\(z-LRB-z\)z-RRB-z\[z-LSB-z\]z-RSB-z\{z-LCB-z\}z-RCB-z--z -- c B<V^8dQhRS[RS[RS[RS[S[,/#)textconvert_parentheses return_strreturn)strboolr)format __classdict__s"r __annotate__NLTKWordTokenizer.__annotate__ys7FFF.2FHLF cFrc V'd\P!R\^R7VPFwrEVP WQ4pK VP FwrEVP WQ4pK VP wrEVP WQ4pV'd'VPFwrEVP WQ4pK VPwrEVP WQ4pRV,R,pVPFwrEVP WQ4pK VPFpVP RV4pK VPFpVP RV4pK VP4#)aJReturn a tokenized copy of `text`. >>> from nltk.tokenize import NLTKWordTokenizer >>> s = '''Good muffins cost $3.88 (roughly 3,36 euros)\nin New York. Please buy me\ntwo of them.\nThanks.''' >>> NLTKWordTokenizer().tokenize(s) # doctest: +NORMALIZE_WHITESPACE ['Good', 'muffins', 'cost', '$', '3.88', '(', 'roughly', '3,36', 'euros', ')', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks', '.'] >>> NLTKWordTokenizer().tokenize(s, convert_parentheses=True) # doctest: +NORMALIZE_WHITESPACE ['Good', 'muffins', 'cost', '$', '3.88', '-LRB-', 'roughly', '3,36', 'euros', '-RRB-', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks', '.'] :param text: A string with a sentence or sentences. :type text: str :param convert_parentheses: if True, replace parentheses to PTB symbols, e.g. `(` to `-LRB-`. Defaults to False. :type convert_parentheses: bool, optional :param return_str: If True, return tokens as space-separated string, defaults to False. :type return_str: bool, optional :return: List of tokens from `text`. :rtype: List[str] zHParameter 'return_str' has been deprecated and should no longer be used.)category stacklevelrz \1 \2 ) warningswarnDeprecationWarningSTARTING_QUOTESsub PUNCTUATIONPARENS_BRACKETSCONVERT_PARENTHESES DOUBLE_DASHES ENDING_QUOTESrrsplit)selfrrr regexp substitutions&&&& rtokenizeNLTKWordTokenizer.tokenizeysF8  MM"+  %)$8$8 F::l1D%9%)$4$4 F::l1D%5 $33zz,- (,(@(@$zz,5)A $11zz,-TzC$($6$6 F::l1D%7((F::j$/D)((F::j$/D)zz|rcL<V^8dQhRS[RS[S[S[S[3,,/#)rrr!)r"rrint)r$r%s"rr&r's().).#).(5c?*C).rc# H"VPV4pRV9gRV9d^\P!RV4Uu.uFq3P4NK ppVUu.uFpVR9dVP ^4MTNK ppMTp\ Wa4RjxL R#uupiuupiL5i)a Returns the spans of the tokens in ``text``. Uses the post-hoc nltk.tokens.align_tokens to return the offset spans. >>> from nltk.tokenize import NLTKWordTokenizer >>> s = '''Good muffins cost $3.88\nin New (York). Please (buy) me\ntwo of them.\n(Thanks).''' >>> expected = [(0, 4), (5, 12), (13, 17), (18, 19), (19, 23), ... (24, 26), (27, 30), (31, 32), (32, 36), (36, 37), (37, 38), ... (40, 46), (47, 48), (48, 51), (51, 52), (53, 55), (56, 59), ... (60, 62), (63, 68), (69, 70), (70, 76), (76, 77), (77, 78)] >>> list(NLTKWordTokenizer().span_tokenize(s)) == expected True >>> expected = ['Good', 'muffins', 'cost', '$', '3.88', 'in', ... 'New', '(', 'York', ')', '.', 'Please', '(', 'buy', ')', ... 'me', 'two', 'of', 'them.', '(', 'Thanks', ')', '.'] >>> [s[start:end] for start, end in NLTKWordTokenizer().span_tokenize(s)] == expected True :param text: A string with a sentence or sentences. :type text: str :yield: Tuple[int, int] rrz ``|'{2}|\"N)rrr)r9refinditergrouppopr)r6r raw_tokensmmatchedtoktokenss&& r span_tokenizeNLTKWordTokenizer.span_tokenizes.]]4( 4KTT\*,++mT*JK*JQwwy*JGK &%C#&):": AC% F  F---L .s(9B"BB"#B<B"B B"r N)FF)r r r rrr>compileUr.r4r0r1r2r3r _contractionslistmaprrr9rGr__classdictcell__)r%s@rrr%s  *BDD 17; F U# G g& , -y9 ;RTT BHM O NBDD )73 E F# D 6" F S! 4 5yA @ A9M M( Dbdd K\Z $ %y1 J ) JJy"$$ '   K *- JJ7 8   G j) K (+ JJvrtt $  K,zz"78*EO E G$ E G$ E G$ E G$ E G$ E G$ ZZ&0M*+MRZZ)D)DEFMRZZ)D)DEFMFFP).).rr) r>r+typingrrrnltk.tokenize.apirnltk.tokenize.utilrr rr rrrRs3 (((+JJ&E. E.r