+ h= Rt^RIt^RIHt^RIt^RIHtRtRtRt Rt ]]RR R R R R ] R3 t ] ^,] .] R,O5t ]P!R4t]P!]]P]P ,]P",4t]P!R4t]P!R4tRRltRRlt!RR]4tRtRtRRltR#)a Twitter-aware tokenizer, designed to be flexible and easy to adapt to new domains and tasks. The basic logic is this: 1. The tuple REGEXPS defines a list of regular expression strings. 2. The REGEXPS strings are put, in order, into a compiled regular expression object called WORD_RE, under the TweetTokenizer class. 3. The tokenization is done by WORD_RE.findall(s), where s is the user-supplied string, inside the tokenize() method of the class TweetTokenizer. 4. When instantiating Tokenizer objects, there are several options: * preserve_case. By default, it is set to True. If it is set to False, then the tokenizer will downcase everything except for emoticons. * reduce_len. By default, it is set to False. It specifies whether to replace repeated character sequences of length 3 or greater with sequences of length 3. * strip_handles. By default, it is set to False. It specifies whether to remove Twitter handles of text used in the `tokenize` method. * match_phone_numbers. By default, it is set to True. It indicates whether the `tokenize` method should look for phone numbers. N)List) TokenizerIac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | {}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]uR.(?: [🏻-🏿]?(?:‍.[🏻-🏿]?)+ | [🏻-🏿] )a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. :NNz([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);zZ(?>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b'Price: £100') 'Price: \xa3100' >>> print(_replace_html_entities(b'Price: £100')) Price: £100 >>> c><VP^4pVP^4'd`VP^4'd\V^4pM \V^ 4p^Tu;8:d^8:dMM]\V34PR4#MAVS9dVP^4#\ P PPV4pVe \V4#S'dR#VP^4# \dRpL:i;i \\3dLAi;i)cp1252N) groupintrr ValueErrorhtmlentitiesname2codepointgetchr OverflowError)match entity_bodynumberkeepremove_illegals& r _convert_entity/_replace_html_entities.._convert_entityskk!n ;;q>> ;;q>> b1F b1F 6)T) &+228<<* d"{{1~%]]1155kBF   6{"$r7Q7   .  s/C5*C5-C5 D5 DDDD)ENT_REsubr)r r!r"r r#s&ff& r _replace_html_entitiesr's 888 ::ot'F GGrca]tRtRtoRtRtRtR RltV3RlRlt] V3RlRl4t ] V3R lR l4t R t Vt R#) TweetTokenizeri2a Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) # doctest: +NORMALIZE_WHITESPACE ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] Examples using `strip_handles` and `reduce_len parameters`: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = '@remy: This is waaaaayyyy too much for you!!!!!!' >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] Nc 6WnW nW0nW@nR#)a Create a `TweetTokenizer` instance with settings for use in the `tokenize` method. :param preserve_case: Flag indicating whether to preserve the casing (capitalisation) of text used in the `tokenize` method. Defaults to True. :type preserve_case: bool :param reduce_len: Flag indicating whether to replace repeated character sequences of length 3 or greater with sequences of length 3. Defaults to False. :type reduce_len: bool :param strip_handles: Flag indicating whether to remove Twitter handles of text used in the `tokenize` method. Defaults to False. :type strip_handles: bool :param match_phone_numbers: Flag indicating whether the `tokenize` method should look for phone numbers. Defaults to True. :type match_phone_numbers: bool N preserve_case reduce_len strip_handlesmatch_phone_numbers)selfr,r-r.r/s&&&&&r __init__TweetTokenizer.__init__Ls.+$*#6 rc6<V^8dQhRS[RS[S[,/#)r return)strr)format __classdict__s"r __annotate__TweetTokenizer.__annotate__hsST#Yrc \V4pVP'd \V4pVP'd \ V4p\ P RV4pVP'dVPPV4pMVPPV4pVP'g\\RV44pV#)zTokenize the input text. :param text: str :rtype: list(str) :return: a tokenized list of strings; joining this list returns the original string if `preserve_case=False`. \1\1\1c\\PV4'dV#VP4#)N) EMOTICON_REsearchlower)xs&r )TweetTokenizer.tokenize..s"K$6$6q$9$9qHqwwyHr)r'r.remove_handlesr-reduce_lengtheningHANG_REr&r/ PHONE_WORD_REfindallWORD_REr,listmap)r0r safe_textwordss&& r tokenizeTweetTokenizer.tokenizehs&d+    !$'D ???%d+DKK 40  # # #&&..y9ELL((3E!!!H5QE rc<V^8dQhRR/#r4r5z regex.Pattern)r7r8s"r r9r:s###rc J\V4P'gu\P!RRP \ 4 R2\P \P,\P,4\V4n\V4P#)zCore TweetTokenizer regex(|)) type_WORD_REregexcompilejoinREGEXPSVERBOSEIUNICODEr0s&r rITweetTokenizer.WORD_REsjDz""""'--CHHW%&a( '%--7#DJ Dz"""rc<V^8dQhRR/#rQrR)r7r8s"r r9r:s)))rc J\V4P'gu\P!RRP \ 4 R2\P \P,\P,4\V4n\V4P#)z#Secondary core TweetTokenizer regexrTrUrV) rW_PHONE_WORD_RErYrZr[ REGEXPS_PHONEr]r^r_r`s&r rGTweetTokenizer.PHONE_WORD_REsjDz((((- CHH]+,A. '%--7)DJ %Dz(((r)r/r,r-r.TFFT)__name__ __module__ __qualname____firstlineno____doc__rXrdr1rNpropertyrIrG__static_attributes____classdictcell__)r8s@r r)r)2sM(HN78<##))rr)cR\P!R4pVPRV4#)zY Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}r<)rYrZr&)r patterns& r rErEs# mmL)G ;;y$ ''rc.\PRV4#)z, Remove Twitter username handles from text.  ) HANDLES_REr&)r s&r rDrDs >>#t $$rc>\VVVVR7PV4#)z2 Convenience function for wrapping the tokenizer. r+)r)rN)r r,r-r.r/s&&&&&r casual_tokenizervs( ##/   htn r)Nstrict)rRTrrg)rlrtypingrrYnltk.tokenize.apir EMOTICONSURLSFLAGS PHONE_REGEXr\rerZrFr]r^r_r>r%rtrr'r)rErDrvrRrr r~s@  ($ $)f   $  (.   /" J[772;7  --/ 0mmIu}}uww'>'NO  . /]]H 8H|h)Zh)`(%r