+ h<'`^RIHt^RIHtHtRtRtRt!RR4tRt R t R t R t R #) )finditer)escapeunescapec# "\V4^8Xd \R4h^pVPW4pV^8wdW#3xT\T4,pK3 \d$T\T48wdT\T43xR#i;i5i)aw Return the offsets of the tokens in *s*, as a sequence of ``(start, end)`` tuples, by splitting the string at each occurrence of *sep*. >>> from nltk.tokenize.util import string_span_tokenize >>> s = '''Good muffins cost $3.88\nin New York. Please buy me ... two of them.\n\nThanks.''' >>> list(string_span_tokenize(s, " ")) # doctest: +NORMALIZE_WHITESPACE [(0, 4), (5, 12), (13, 17), (18, 26), (27, 30), (31, 36), (37, 37), (38, 44), (45, 48), (49, 55), (56, 58), (59, 73)] :param s: the string to be tokenized :type s: str :param sep: the token separator :type sep: str :rtype: iter(tuple(int, int)) z!Token delimiter must not be emptyN)len ValueErrorindex)ssepleftrights&& 4/usr/lib/python3/dist-packages/nltk/tokenize/util.pystring_span_tokenizer s$ 3x1}<== D  GGC&Ezk! s3x  s1v~CFl"  s'BAB*B=BBBc#"^p\W4F"pVP4wrEWB8wdW$3xTpK$ V\V43xR#5i)a Return the offsets of the tokens in *s*, as a sequence of ``(start, end)`` tuples, by splitting the string at each successive match of *regexp*. >>> from nltk.tokenize.util import regexp_span_tokenize >>> s = '''Good muffins cost $3.88\nin New York. Please buy me ... two of them.\n\nThanks.''' >>> list(regexp_span_tokenize(s, r'\s')) # doctest: +NORMALIZE_WHITESPACE [(0, 4), (5, 12), (13, 17), (18, 23), (24, 26), (27, 30), (31, 36), (38, 44), (45, 48), (49, 51), (52, 55), (56, 58), (59, 64), (66, 73)] :param s: the string to be tokenized :type s: str :param regexp: regular expression that matches token separators (must not be empty) :type regexp: str :rtype: iter(tuple(int, int)) N)rspanr)r regexpr mr nexts&& rregexp_span_tokenizer.sJ$ D f ffh  =+  ! A,sAAc#P"^pVFwr#W!, W2, 3xTpK R#5i)a{ Return a sequence of relative spans, given a sequence of spans. >>> from nltk.tokenize import WhitespaceTokenizer >>> from nltk.tokenize.util import spans_to_relative >>> s = '''Good muffins cost $3.88\nin New York. Please buy me ... two of them.\n\nThanks.''' >>> list(spans_to_relative(WhitespaceTokenizer().span_tokenize(s))) # doctest: +NORMALIZE_WHITESPACE [(0, 4), (1, 7), (1, 4), (1, 5), (1, 2), (1, 3), (1, 5), (2, 6), (1, 3), (1, 2), (1, 3), (1, 2), (1, 5), (2, 7)] :param spans: a sequence of (start, end) offsets of the tokens :type spans: iter(tuple(int, int)) :rtype: iter(tuple(int, int)) N)spansprevr r s& rspans_to_relativerIs, D k5<''s$&cN]tRt^_tRtRtRtRtRtRt R t R t R t ]]]]] ] ] ] .t RtR#) CJKCharsaJ An object that enumerates the code points of the CJK characters as listed on https://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Basic_Multilingual_Plane This is a Python port of the CJK code point enumerations of Moses tokenizer: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/detokenizer.perl#L309 rNiii.iϤi@iiiiii0iOieiii)__name__ __module__ __qualname____firstlineno____doc__ Hangul_Jamo CJK_RadicalsPhags_PaHangul_SyllablesCJK_Compatibility_IdeographsCJK_Compatibility_FormsKatakana_Hangul_HalfwidthSupplementary_Ideographic_Planeranges__static_attributes__rrrr_seK*"LH&$2 -!/'# $!' Fr4rc \RUUu.uF$wrV\V4u;8*;'dV8*MuNK& upp4#uuppi)u` Python port of Moses' code to check for CJK character. >>> CJKChars().ranges [(4352, 4607), (11904, 42191), (43072, 43135), (44032, 55215), (63744, 64255), (65072, 65103), (65381, 65500), (131072, 196607)] >>> is_cjk(u'㏾') True >>> is_cjk(u'﹟') False :param character: The character that needs to be checked. :type character: char :return: bool )rrrr r!r"r#r$)anyord) characterstartends& ris_cjkr;sJ      S^ * *s *    s*= c 0\VRRRRRRRRR R /R 7#) aP This function transforms the input text into an "escaped" version suitable for well-formed XML formatting. Note that the default xml.sax.saxutils.escape() function don't escape some characters that Moses does so we have to manually add them to the entities dictionary. >>> input_str = ''')| & < > ' " ] [''' >>> expected_output = ''')| & < > ' " ] [''' >>> escape(input_str) == expected_output True >>> xml_escape(input_str) ')| & < > ' " ] [' :param text: The text that needs to be escaped. :type text: str :rtype: str ''""||[[]]entities)rtexts&r xml_escaperKs2(  ) ) ) ( (    r4c 0\VRRRRRRRRR R /R 7#) a2 This function transforms the "escaped" version suitable for well-formed XML formatting into humanly-readable string. Note that the default xml.sax.saxutils.unescape() function don't unescape some characters that Moses does so we have to manually add them to the entities dictionary. >>> from xml.sax.saxutils import unescape >>> s = ')| & < > ' " ] [' >>> expected = ''')| & < > ' " ] [''' >>> xml_unescape(s) == expected True :param text: The text that needs to be unescaped. :type text: str :rtype: str r>r=r@r?rBrArDrCrFrErG)rrIs&r xml_unescaperMs2&  t t t d d    r4c ^p.pVF9pVPWB4pT\T4,pTPYR34K; V# \dp\RT RT R24ThRp?ii;i)a] This module attempt to find the offsets of the tokens in *s*, as a sequence of ``(start, end)`` tuples, given the tokens and also the source string. >>> from nltk.tokenize import TreebankWordTokenizer >>> from nltk.tokenize.util import align_tokens >>> s = str("The plane, bound for St Petersburg, crashed in Egypt's " ... "Sinai desert just 23 minutes after take-off from Sharm el-Sheikh " ... "on Saturday.") >>> tokens = TreebankWordTokenizer().tokenize(s) >>> expected = [(0, 3), (4, 9), (9, 10), (11, 16), (17, 20), (21, 23), ... (24, 34), (34, 35), (36, 43), (44, 46), (47, 52), (52, 54), ... (55, 60), (61, 67), (68, 72), (73, 75), (76, 83), (84, 89), ... (90, 98), (99, 103), (104, 109), (110, 119), (120, 122), ... (123, 131), (131, 132)] >>> output = list(align_tokens(tokens, s)) >>> len(tokens) == len(expected) == len(output) # Check that length of tokens and tuples are the same. True >>> expected == list(align_tokens(tokens, s)) # Check that the output is as expected. True >>> tokens == [s[start:end] for start, end in output] # Check that the slices of the string corresponds to the tokens. True :param tokens: The list of strings that are the result of tokenization :type tokens: list(str) :param sentence: The original string :type sentence: str :rtype: list(tuple(int,int)) z substring "z" not found in "r?N)r rrappend)tokenssentencepointoffsetstokenr9es&& r align_tokensrVs< EG VNN50EE "~&  N  V{5'1A(1MNTU U VsA A)A$$A)N) rerxml.sax.saxutilsrrrrrrr;rKrMrVrr4rrYs>- D6,??D@@>'r4