+ h?~Rt^RIt^RIt^RIHtHtHt^RIHt^RI H t ^RI H t !RR]4t !RR ]4tR#) a  Penn Treebank Tokenizer The Treebank tokenizer uses regular expressions to tokenize text as in Penn Treebank. This implementation is a port of the tokenizer sed script written by Robert McIntyre and available at http://www.cis.upenn.edu/~treebank/tokenizer.sed. N)IteratorListTuple) TokenizerI)MacIntyreContractions) align_tokensc >a]tRt^toRt]P !R4R3]P !R4R3]P !R4R3.t]P !R4R 3]P !R 4R3]P !R 4R 3]P !R 4R3]P !R4R3]P !R4R3]P !R4R3.t]P !R4R3t ]P !R4R3]P !R4R3]P !R4R3]P !R4R3]P !R4R3]P !R4R 3.t ]P !R!4R"3t ]P !R#4R$3]P !R%4R$3]P !R&4R'3]P !R(4R'3.t ] !4t]!]!]P ]P"44t]!]!]P ]P$44tR/V3R)lR*lltV3R+lR,ltR-tVtR.#)0TreebankWordTokenizera The Treebank tokenizer uses regular expressions to tokenize text as in Penn Treebank. This tokenizer performs the following steps: - split standard contractions, e.g. ``don't`` -> ``do n't`` and ``they'll`` -> ``they 'll`` - treat most punctuation characters as separate tokens - split off commas and single quotes, when followed by whitespace - separate periods that appear at the end of line >>> from nltk.tokenize import TreebankWordTokenizer >>> s = '''Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\nThanks.''' >>> TreebankWordTokenizer().tokenize(s) ['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks', '.'] >>> s = "They'll save and invest more." >>> TreebankWordTokenizer().tokenize(s) ['They', "'ll", 'save', 'and', 'invest', 'more', '.'] >>> s = "hi, my name can't hello," >>> TreebankWordTokenizer().tokenize(s) ['hi', ',', 'my', 'name', 'ca', "n't", 'hello', ','] z^\"``z(``)z \1 z([ \(\[{<])(\"|\'{2})z\1 `` z ([:,])([^\d])z \1 \2z([:,])$z\.\.\.z ... z[;@#$%&]z \g<0> z([^\.])(\.)([\]\)}>"\']*)\s*$z\1 \2\3 z[?!]z([^'])' z\1 ' z[\]\[\(\)\{\}\<\>]z\(-LRB-z\)-RRB-z\[-LSB-z\]-RSB-z\{-LCB-z\}-RCB--- -- ''z '' "z([^' ])('[sS]|'[mM]|'[dD]|') z\1 \2 z)([^' ])('ll|'LL|'re|'RE|'ve|'VE|n't|N'T) c B<V^8dQhRS[RS[RS[RS[S[,/#)textconvert_parentheses return_strreturn)strboolr)format __classdict__s"8/usr/lib/python3/dist-packages/nltk/tokenize/treebank.py __annotate__"TreebankWordTokenizer.__annotate__es7EEE.2EHLE cEc VRJd\P!R\^R7VPFwrEVP WQ4pK VP FwrEVP WQ4pK VP wrEVP WQ4pV'd'VPFwrEVP WQ4pK VPwrEVP WQ4pRV,R,pVPFwrEVP WQ4pK VPFpVP RV4pK VPFpVP RV4pK VP4#)aUReturn a tokenized copy of `text`. >>> from nltk.tokenize import TreebankWordTokenizer >>> s = '''Good muffins cost $3.88 (roughly 3,36 euros)\nin New York. Please buy me\ntwo of them.\nThanks.''' >>> TreebankWordTokenizer().tokenize(s) # doctest: +NORMALIZE_WHITESPACE ['Good', 'muffins', 'cost', '$', '3.88', '(', 'roughly', '3,36', 'euros', ')', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks', '.'] >>> TreebankWordTokenizer().tokenize(s, convert_parentheses=True) # doctest: +NORMALIZE_WHITESPACE ['Good', 'muffins', 'cost', '$', '3.88', '-LRB-', 'roughly', '3,36', 'euros', '-RRB-', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks', '.'] :param text: A string with a sentence or sentences. :type text: str :param convert_parentheses: if True, replace parentheses to PTB symbols, e.g. `(` to `-LRB-`. Defaults to False. :type convert_parentheses: bool, optional :param return_str: If True, return tokens as space-separated string, defaults to False. :type return_str: bool, optional :return: List of tokens from `text`. :rtype: List[str] FzHParameter 'return_str' has been deprecated and should no longer be used.)category stacklevel z \1 \2 ) warningswarnDeprecationWarningSTARTING_QUOTESsub PUNCTUATIONPARENS_BRACKETSCONVERT_PARENTHESES DOUBLE_DASHES ENDING_QUOTES CONTRACTIONS2 CONTRACTIONS3split)selfrrrregexp substitutions&&&& rtokenizeTreebankWordTokenizer.tokenizeesK6 U " MM"+  %)$8$8 F::l1D%9%)$4$4 F::l1D%5 $33zz,- (,(@(@$zz,5)A $11zz,-TzC$($6$6 F::l1D%7((F::j$/D)((F::j$/D)zz|r"cL<V^8dQhRS[RS[S[S[S[3,,/#)rrr)rrrint)rrs"rr r!s().).#).(5c?*C).r"c# H"VPV4pRV9gRV9d^\P!RV4Uu.uFq3P4NK ppVUu.uFpVR9dVP ^4MTNK ppMTp\ Wa4RjxL R#uupiuupiL5i)a Returns the spans of the tokens in ``text``. Uses the post-hoc nltk.tokens.align_tokens to return the offset spans. >>> from nltk.tokenize import TreebankWordTokenizer >>> s = '''Good muffins cost $3.88\nin New (York). Please (buy) me\ntwo of them.\n(Thanks).''' >>> expected = [(0, 4), (5, 12), (13, 17), (18, 19), (19, 23), ... (24, 26), (27, 30), (31, 32), (32, 36), (36, 37), (37, 38), ... (40, 46), (47, 48), (48, 51), (51, 52), (53, 55), (56, 59), ... (60, 62), (63, 68), (69, 70), (70, 76), (76, 77), (77, 78)] >>> list(TreebankWordTokenizer().span_tokenize(s)) == expected True >>> expected = ['Good', 'muffins', 'cost', '$', '3.88', 'in', ... 'New', '(', 'York', ')', '.', 'Please', '(', 'buy', ')', ... 'me', 'two', 'of', 'them.', '(', 'Thanks', ')', '.'] >>> [s[start:end] for start, end in TreebankWordTokenizer().span_tokenize(s)] == expected True :param text: A string with a sentence or sentences. :type text: str :yield: Tuple[int, int] rrz ``|'{2}|\"N)rr r)r7refinditergrouppopr)r4r raw_tokensmmatchedtoktokenss&& r span_tokenize#TreebankWordTokenizer.span_tokenizes.]]4( 4KTT\*,++mT*JK*JQwwy*JGK &%C#&):": AC% F  F---L .s(9B"BB"#B<B"B B"N)FF)__name__ __module__ __qualname____firstlineno____doc__r<compiler*r,r-r.r/r0r _contractionslistmapr1r2r7rE__static_attributes____classdictcell__)rs@rr r s0 F U# G g& , -y9O $ %y1 J ) I ) K *- JJ7 8   G j) K (+ Kzz"78*EO E G$ E G$ E G$ E G$ E G$ E G$ ZZ&0M E F# D 6" 4 5yA @ A9M M*+MRZZ)D)DEFMRZZ)D)DEFMEEN).).r"r c >a]tRt^toRt]!4t]PUu.uF)p\P!VPRR44NK+ upt]PUu.uF)p\P!VPRR44NK+ upt ]P!R4R3]P!R4R3]P!R4R3]P!R 4R3]P!R 4R 3.t ]P!R 4R 3t ]P!R4R3]P!R4R3]P!R4R3]P!R4R3]P!R4R3]P!R4R3.t]P!R4R3]P!R4R3]P!R4R3.t]P!R4R3]P!R 4R3]P!R!4R"3]P!R#4R3]P!R$4R3]P!R%4R&3]P!R'4R(3.t]P!R)4R*3]P!R+4R(3]P!R,4R 3.tR3V3R-lR.lltR3V3R/lR0lltR1tVtR2#uupiuupi)4TreebankWordDetokenizera The Treebank detokenizer uses the reverse regex operations corresponding to the Treebank tokenizer's regexes. Note: - There're additional assumption mades when undoing the padding of ``[;@#$%&]`` punctuation symbols that isn't presupposed in the TreebankTokenizer. - There're additional regexes added in reversing the parentheses tokenization, such as the ``r'([\]\)\}\>])\s([:;,.])'``, which removes the additional right padding added to the closing parentheses precedding ``[:;,.]``. - It's not possible to return the original whitespaces as they were because there wasn't explicit records of where `'\n'`, `'\t'` or `'\s'` were removed at the text.split() operation. >>> from nltk.tokenize.treebank import TreebankWordTokenizer, TreebankWordDetokenizer >>> s = '''Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\nThanks.''' >>> d = TreebankWordDetokenizer() >>> t = TreebankWordTokenizer() >>> toks = t.tokenize(s) >>> d.detokenize(toks) 'Good muffins cost $3.88 in New York. Please buy me two of them. Thanks.' The MXPOST parentheses substitution can be undone using the ``convert_parentheses`` parameter: >>> s = '''Good muffins cost $3.88\nin New (York). Please (buy) me\ntwo of them.\n(Thanks).''' >>> expected_tokens = ['Good', 'muffins', 'cost', '$', '3.88', 'in', ... 'New', '-LRB-', 'York', '-RRB-', '.', 'Please', '-LRB-', 'buy', ... '-RRB-', 'me', 'two', 'of', 'them.', '-LRB-', 'Thanks', '-RRB-', '.'] >>> expected_tokens == t.tokenize(s, convert_parentheses=True) True >>> expected_detoken = 'Good muffins cost $3.88 in New (York). Please (buy) me two of them. (Thanks).' >>> expected_detoken == d.detokenize(t.tokenize(s, convert_parentheses=True), convert_parentheses=True) True During tokenization it's safe to add more spaces but during detokenization, simply undoing the padding doesn't really help. - During tokenization, left and right pad is added to ``[!?]``, when detokenizing, only left shift the ``[!?]`` is needed. Thus ``(re.compile(r'\s([?!])'), r'\g<1>')``. - During tokenization ``[:,]`` are left and right padded but when detokenizing, only left shift is necessary and we keep right pad after comma/colon if the string after is a non-digit. Thus ``(re.compile(r'\s([:,])\s([^\d])'), r'\1 \2')``. >>> from nltk.tokenize.treebank import TreebankWordDetokenizer >>> toks = ['hello', ',', 'i', 'ca', "n't", 'feel', 'my', 'feet', '!', 'Help', '!', '!'] >>> twd = TreebankWordDetokenizer() >>> twd.detokenize(toks) "hello, i can't feel my feet! Help!!" >>> toks = ['hello', ',', 'i', "can't", 'feel', ';', 'my', 'feet', '!', ... 'Help', '!', '!', 'He', 'said', ':', 'Help', ',', 'help', '?', '!'] >>> twd.detokenize(toks) "hello, i can't feel; my feet! Help!! He said: Help, help?!" z(?#X)z\sz+([^' ])\s('ll|'LL|'re|'RE|'ve|'VE|n't|N'T) z\1\2 z([^' ])\s('[sS]|'[mM]|'[dD]|') z (\S)\s(\'\')\1\2z(\'\')\s([.,:)\]>};%])rrrrr (r )r [r]r{r}z([\[\(\{\<])\sz\g<1>z\s([\]\)\}\>])z([\]\)\}\>])\s([:;,.])z ([^'])\s'\sz\1' z\s([?!])z([^\.])\s(\.)([\]\)}>"\']*)\s*$z\1\2\3z([#$])\sz\s([;%])z \s\.\.\.\sz...z\s([:,])z\1z([ (\[{<])\s``z\1``z(``)\sr c<<V^8dQhRS[S[,RS[RS[/#rrDrrrrr)rrs"rr $TreebankWordDetokenizer.__annotate__[s'33tCy3t3PS3r"c RPV4pRV,R,pVPFpVPRV4pK VPFpVPRV4pK VPFwrEVPWS4pK VP 4pVP wrEVPWS4pV'd'VPFwrEVPWS4pK VPFwrEVPWS4pK VPFwrEVPWS4pK VPFwrEVPWS4pK VP 4#)a] Treebank detokenizer, created by undoing the regexes from the TreebankWordTokenizer.tokenize. :param tokens: A list of strings, i.e. tokenized text. :type tokens: List[str] :param convert_parentheses: if True, replace PTB symbols with parentheses, e.g. `-LRB-` to `(`. Defaults to False. :type convert_parentheses: bool, optional :return: str r&rU) joinr2r+r1r0stripr/r.r-r,r*)r4rDrrr5r6s&&& rr7 TreebankWordDetokenizer.tokenize[sNxxTzC((F::gt,D)((F::gt,D)%)$6$6 F::l1D%7zz| $11zz,- (,(@(@$zz,5)A%)$8$8 F::l1D%9%)$4$4 F::l1D%5%)$8$8 F::l1D%9zz|r"c<<V^8dQhRS[S[,RS[RS[/#r]r^)rrs"rr r_s'::c::RU:r"c $VPW4#)z&Duck-typing the abstract *tokenize()*.)r7)r4rDrs&&&r detokenize"TreebankWordDetokenizer.detokenizes}}V99r"rGN)F)rHrIrJrKrLrrNr1r<rMreplacer2r0r/r.r-r,r*r7rfrQrR)patternrs0@rrTrTs:x*+M%222G 7??7E232M %222G 7??7E232M B CXN 6 7B O $g. JJ0 1   E C  MZZ(%0M G c" G c" G c" G c" G c" G c"  % &1 % &1 - .8O N #W- K (+ 6 7C K (+ K (+ M "F+ JJ{ #  K, % &0 I & E D!O 33j:::us /J!/JrT)rLr<r'typingrrrnltk.tokenize.apirnltk.tokenize.destructivernltk.tokenize.utilrr rTrGr"rrns> (((;+x.Jx.vz:jz:r"