+ h>Rt^RIt^RIHt^RIHt^RI5^RIHt^RI H t ^RI H t !RR ]4t !R R ]4tR#) z Corpus reader for corpora whose documents are xml files. (note -- not named 'xml' to avoid conflicting w/ standard xml package) N) ElementTree) CorpusReader)*)SeekableUnicodeStreamReader)ElementWrapper)WordPunctTokenizercBa]tRt^toRtRRltRRltRRltRtVt R#) XMLCorpusReaderz Corpus reader for corpora whose documents are xml files. Note that the ``XMLCorpusReader`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. See the XML specs for more info. c@W0n\P!WV4R#N) _wrap_etreer__init__)selfrootfileids wrap_etrees&&&&c$--0A5]]1%F&#&&FG G \\& ! & & ( (B##B'//1C)     %C ) (s :%C C c zVPV4pVPV4p\4pVP4p.pTF^pTP pTfK\ T\4'dTPT4pTPT4p TPT 4K` T# TP 4pL|;i)a Returns all of the words and punctuation symbols in the specified file that were in text nodes -- ie, tags are ignored. Like the xml() method, fileid can only specify one file. :return: the given file's text nodes as a list of words and punctuation symbols :rtype: list(str) ) r"encodingr getiteratoritertextrbytesdecodetokenizeextend) rrr!r%word_tokenizeriteratoroutnoder(tokss && rwordsXMLCorpusReader.words4shhv==(+- "(HD99DdE**;;x0D%..t4 4   "xxzHs B&&B:)r )Fr ) __name__ __module__ __qualname____firstlineno____doc__r r"r2__static_attributes____classdictcell__ __classdict__s@rr r s3 rr c(a]tRt^QtoRtRtRtRRltRtRt ] PR] P] P,4t] PR 4t] PR ] P] P,4tR tRR ltR tVtR#) XMLCorpusViewa A corpus view that selects out specified elements from an XML file, and provides a flat list-like interface for accessing them. (Note: ``XMLCorpusView`` is not used by ``XMLCorpusReader`` itself, but may be used by subclasses of ``XMLCorpusReader``.) Every XML corpus view has a "tag specification", indicating what XML elements should be included in the view; and each (non-nested) element that matches this specification corresponds to one item in the view. Tag specifications are regular expressions over tag paths, where a tag path is a list of element tag names, separated by '/', indicating the ancestry of the element. Some examples: - ``'foo'``: A top-level element whose tag is ``foo``. - ``'foo/bar'``: An element whose tag is ``bar`` and whose parent is a top-level element whose tag is ``foo``. - ``'.*/foo'``: An element whose tag is ``foo``, appearing anywhere in the xml tree. - ``'.*/(foo|bar)'``: An wlement whose tag is ``foo`` or ``bar``, appearing anywhere in the xml tree. The view items are generated from the selected XML elements via the method ``handle_elt()``. By default, this method returns the element as-is (i.e., as an ElementTree object); but it can be overridden, either via subclassing or via the ``elt_handler`` constructor parameter. FiNc V'dW0n\PVR,4Vn^R/VnVP V4p\ PWVR7R#)a Create a new corpus view based on a specified XML file. Note that the ``XMLCorpusView`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. :type tagspec: str :param tagspec: A tag specification, indicating what XML elements should be included in the view. Each non-nested element that matches this specification corresponds to one item in the view. :param elt_handler: A function used to transform each element to a value for the view. If no handler is specified, then ``self.handle_elt()`` is called, which returns the element as an ElementTree object. The signature of elt_handler is:: elt_handler(elt, tagspec) -> value z\Z)r%N) handle_eltrecompile_tagspec _tag_context_detect_encodingStreamBackedCorpusViewr )rrtagspec elt_handlerr%s&&&& rr XMLCorpusView.__init__usY* )O 7U?3 9G  ((0''x'Hrc\V\4'd3VP4pVP4pVP 4M-\VR4;_uu_4pVP4pRRR4XP \ P4'dR#VP \ P4'dR#VP \ P4'dR#VP \ P4'dR#VP \ P4'dR#\PRV4pV'd VP^4P4#\PRV4pV'd VP^4P4#R# XP 4i;i +'giELe;i) rbNz utf-16-bez utf-16-lez utf-32-bez utf-32-lezutf-8s!\s*<\?xml\b.*\bencoding="([^"]+)"s!\s*<\?xml\b.*\bencoding='([^']+)')r PathPointerrreadlineclose startswithcodecs BOM_UTF16_BE BOM_UTF16_LE BOM_UTF32_BE BOM_UTF32_LEBOM_UTF8rBmatchgroupr*)rrinfilesms&& rrFXMLCorpusView._detect_encodings: fk * * OO% fd##vOO%$ <<++ , , <<++ , , <<++ , , <<++ , , << ( ( HH:A > 771:$$& & HH:A > 771:$$& &+ ###s F5G 5G G c V#)a Convert an element into an appropriate value for inclusion in the view. Unless overridden by a subclass or by the ``elt_handler`` constructor argument, this method simply returns ``elt``. :return: The view value corresponding to ``elt``. :type elt: ElementTree :param elt: The element that should be converted. :type context: str :param context: A string composed of element tags separated by forward slashes, indicating the XML context of the given element. For example, the string ``'foo/bar/baz'`` indicates that the element is a ``baz`` element whose parent is a ``bar`` element and whose grandparent is a top-level ``foo`` element. r@)rr!contexts&&&rrAXMLCorpusView.handle_elts ( ra; [^<]* ( (() | # comment () | # doctype decl (<[^!>][^>]*>)) # tag or PI [^<]*)* \Zz<\s*(?:/\s*)?([^\s>]+)a6 # Include these so we can skip them: (?P )| (?P )| (?P <\?.*?\?> )| (?P ]*(\[[^\]]*])?\s*>)| # These are the ones we actually care about: (?P <\s*[^>/\?!\s][^>]*/\s*> )| (?P <\s*[^>/\?!\s][^>]*> )| (?P <\s*/[^>/\?!\s][^>]*> )c TRp\V\4'dVP4pVPVP4pW$, pVP P V4'dV#\PRV4P^4R8Xd\VP4\V4\PRV4P4, , p\RV,4hV'g \R4hVPR4pV^8gKVP P VRV4'gEK \V\4'd$VPX4VPV4M#VP\V4V, )^4VRV#)aC Read a string from the given stream that does not contain any un-closed tags. In particular, this function first reads a block from the stream of size ``self._BLOCK_SIZE``. It then checks if that block contains an un-closed tag. If it does, then this function either backtracks to the last '<', or reads another block. z[<>]>zUnexpected ">" near char %sz&Unexpected end of file: tag not closed!DEE !IJJ !)s 3  1$%%++H5G6G,HII!&*EFF H-001BC c(m6G&G$H!L#$6%677rc  Vf VPpVf VPp\VPP VP 444pVfQh.pRpRpRpV.8XgVEe\ V\4'dVP 4p VPV4p V 'gVfEM\R4hVPPV 4EFp VP'd<\RPRPV4RRV P!444V P!R4'dVP"P%V P!44P!^4p VP'V 4VfK\(P%VRPV44'dV P+4p\-V4pKKEKV P!R4'dVP"P%V P!44P!^4p V'g\RV ,4hWR,8wd\R VR, R V R 24hVePV\-V48Xd@WWkP/4, pVP'VRPV434R;rgRpVP14EKV P!R 4'gEKVP"P%V P!44P!^4p VeEKP\(P%VRPV4R,V ,4'gEKVP'V P!4RPV4R,V ,34EK VfEKMV.8XdWVR, p^pEKdVP'd \R4\ V\4'd$VP3X 4VP5V4M#VP3\-V 4V, )^4VRV^, pR;rgRpEKVP 4p WP9d%\7V4VPV ,8XgQhM\7V4VPV &VUUu.uF2wrV!\8P:!VP=R R44V4NK4 upp#uuppi)z Read from ``stream`` until we find at least one element that matches ``tagspec``, and return the result of applying ``elt_handler`` to each element found. NrazUnexpected end of filez {:>25} {}/ START_TAGEND_TAGzUnmatched tag zUnmatched tag ...>!;;=226:L $$%=>>11,?;;;+,,SXXg->st-DekkmTU;;{++--33EKKMBHHKDNN4( (88GSXXg->??(- I(+G I@) [[++--33EKKMBHHKD"()>)EFFr{*(?72;-vdVST)UVV ,c'l1J YY[$II Xsxx/@$AB044 #%KKM[[11--33EKKMBHHKD (88GSXXg->-Dt-KLL KK8IC8ORV8V(WXE@H$2:YZ 88H !I{{{67!&*EFF H-00; c,&7)&C$DaH%o A 6G,00I!Hkkm ## #>T%6%6s%;; ;;%*7^D  c "#'  #' &&szz';N'OP #'    s38S/)rErDrAr )NN)r4r5r6r7r8rrfr rFrArBrCDOTALLVERBOSErgrrrtrr9r:r;s@rr>r>Qs<FK"IH:0JJ  BJJ MJJ89M  E BJJ J,8bk k rr>)r8rQ xml.etreernltk.corpus.reader.apirnltk.corpus.reader.util nltk.datarnltk.internalsr nltk.tokenizerr rGr>r@rrrs= !/%1),6l6r| *| r