+ hF:^RIt^RIt^RIt^RIt^RIHt^RIHtHtH t ^RI H t H t ^RI HtHtRtRtRtR tR tR tR tR tRtRtRtRtRRltRtRRlt]!RR.4t]!.RO4t Rt!]"R8Xd ]!4R#R#)N)treebank)BrillTaggerTrainer RegexpTagger UnigramTagger)PosWord)Template error_listc\4R#)zx Run a demo with defaults. See source comments for details, or docstrings of any of the more specific demo_* functions. Npostag//usr/lib/python3/dist-packages/nltk/tbl/demo.pydemors  Hrc\RR7R#)F Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) repr ruleformatNr rrrdemo_repr_rule_formatrs  frc\RR7R#)rstrrNr rrrdemo_str_rule_formatr$s  erc\RR7R#)z" Exemplify Rule.format("verbose") verboserNr rrrdemo_verbose_rule_formatr+s  i rcH\\\.RO44.R7R#)a The feature/s of a template takes a list of positions relative to the current word where the feature should be looked for, conceptually joined by logical OR. For instance, Pos([-1, 1]), given a value V, will hold whenever V is found one step to the left and/or one step to the right. For contiguous ranges, a 2-arg form giving inclusive end points can also be used: Pos(-3, -1) is the same as the arg below.  templatesN))r r rrrrdemo_multiposition_featurer$2s hs<0123rc ^\\\^.4\RR.44.R7R#)z0 Templates can have more than a single feature. rNr"r#)r r rrrrrdemo_multifeature_templater&As$ htQCy#r2h-89:rc \RRR7R#)aP Show aggregate statistics per template. Little used templates are candidates for deletion, much used templates may possibly be refined. Deleting unused templates is mostly about saving time and/or space: training is basically O(T) in the number of templates T (also in terms of memory usage, which often will be the limiting factor). T)incremental_statstemplate_statsNr rrrdemo_template_statisticsr*Hs T$7rc \P!.RO^^.RR7p\P!.R O^^.RR7p\\P!W.R R74p\ RP \V444\VRRR7R#) z Template.expand and Feature.expand are class methods facilitating generating large amounts of templates. See their documentation for details. Note: training with 500 templates can easily fill all available even on relatively small corpora F) excludezeroT) combinationsz8Generated {} templates for transformation-based learning)r r(r)N)r#r)r"r#rr.)r.) rexpandrlistr printformatlenr )wordtplstagtplsr s rdemo_generated_templatesr7Tss{{:1v5AHjj!QTBGX__h%8vNOI BII  N   Y$tLrc"\RRRR7R#)zr Plot a learning curve -- the contribution on tagging accuracy of the individual rules. Note: requires matplotlib Tzlearningcurve.png)r(separate_baseline_datalearning_curve_outputNr rrrdemo_learning_curver;hs  #1rc\RR7R#)zO Writes a file with context for each erroneous word after tagging testing data z errors.txt) error_outputNr rrrdemo_error_analysisr>us  %rc\RR7R#)za Serializes the learned tagger to a file in pickle format; reloads it and validates the process. z tagger.pcl)serialize_outputNr rrrdemo_serialize_taggerrA|s  L)rc"\RR^ R7R#)z Discard rules with low accuracy. This may hurt performance a bit, but will often produce rules which are more interesting read to a human. i gQ?) num_sentsmin_acc min_scoreNr rrrdemo_high_accuracy_rulesrFs  T426rc ` T;'g\pVf^RIHpHpV!4p\ WW(V4wppppV'd\ P PV4'g\\VVR7p\VR4;_uu_4p\P!VV4RRR4\RPV44\V4;_uu_4p\P!V4p\RV 24RRR4M\VVR7p\R4V'd*\RPXPV444\ P !4p\#XWV R 7p\R 4VP%VW4V4p\R \ P !4V, R R 24V'd"\RVPV4,4V^8XdQ\R4\'VP)4^4F'wpp\VR RVPV 4R 24K) V 'd\R4VP+VV4wp p!\R4V'g \R4VP-4p"V 'dVP/V!4V'd\1VV!V"VR7\RV 24M4\R4VP3V4p V 'dVP/4V e\V R4;_uu_4p#V#P5RV ,4V#P5RP7\9VV 44P;R4R,4RRR4\RV 24V eVP3V4p \V R4;_uu_4p\P!VV4RRR4\RV 24\V 4;_uu_4p\P!V4p$RRR4\RV 24VP3V4p%V V%8Xd\R4R#\R4R#R# +'giEL;i +'giEL\;i +'giEL;i +'giL;i +'giL;i) a Brill Tagger Demonstration :param templates: how many sentences of training and testing data to use :type templates: list of Template :param tagged_data: maximum number of rule instances to create :type tagged_data: C{int} :param num_sents: how many sentences of training and testing data to use :type num_sents: C{int} :param max_rules: maximum number of rule instances to create :type max_rules: C{int} :param min_score: the minimum score for a rule in order for it to be considered :type min_score: C{int} :param min_acc: the minimum score for a rule in order for it to be considered :type min_acc: C{float} :param train: the fraction of the the corpus to be used for training (1=all) :type train: C{float} :param trace: the level of diagnostic tracing output to produce (0-4) :type trace: C{int} :param randomize: whether the training data should be a random subset of the corpus :type randomize: C{bool} :param ruleformat: rule output format, one of "str", "repr", "verbose" :type ruleformat: C{str} :param incremental_stats: if true, will tag incrementally and collect stats for each rule (rather slow) :type incremental_stats: C{bool} :param template_stats: if true, will print per-template statistics collected in training and (optionally) testing :type template_stats: C{bool} :param error_output: the file where errors will be saved :type error_output: C{string} :param serialize_output: the file where the learned tbl tagger will be saved :type serialize_output: C{string} :param learning_curve_output: filename of plot of learning curve(s) (train and also test, if available) :type learning_curve_output: C{string} :param learning_curve_take: how many rules plotted :type learning_curve_take: C{int} :param baseline_backoff_tagger: the file where rules will be saved :type baseline_backoff_tagger: tagger :param separate_baseline_data: use a fraction of the training data exclusively for training baseline :type separate_baseline_data: C{bool} :param cache_baseline_tagger: cache baseline tagger to this file (only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions) :type cache_baseline_tagger: C{string} Note on separate_baseline_data: if True, reuse training data both for baseline and rule learner. This is fast and fine for a demo, but is likely to generalize worse on unseen data. Also cannot be sensibly used for learning curves on training data (the baseline will be artificially high). N)brill24describe_template_sets)backoffwz)Trained baseline tagger, pickled it to {}zReloaded pickled tagger from zTrained baseline taggerz! Accuracy on test set: {:0.4f}rzTraining tbl tagger...zTrained tbl tagger in z0.2fz secondsz Accuracy on test set: %.4fz Learned rules: 4d szJIncrementally tagging the test data, collecting individual rule statisticsz Rule statistics collectedzbWARNING: train_stats asked for separate_baseline_data=True; the baseline will be artificially high)takez Wrote plot of learning curve to zTagging the test datazErrors for Brill Tagger %r  zutf-8z)Wrote tagger errors including context to zWrote pickled tagger to z4Reloaded tagger tried on test set, results identicalz;PROBLEM: Reloaded tagger gave different results on test set) REGEXP_TAGGERnltk.tag.brillrHrI_demo_prepare_dataospathexistsropenpickledumpr2r3loadaccuracytimertrain enumeraterulesbatch_tag_incremental train_statsprint_template_statistics _demo_plot tag_sentswritejoinr encode)&r tagged_datarC max_rulesrErDr]trace randomizerr(r)r=r@r:learning_curve_takebaseline_backoff_taggerr9cache_baseline_taggerrHrI training_data baseline_data gold_data testing_databaseline_tagger print_rulestbrilltrainer brill_taggerrulenorule taggedtest teststats trainstatsfbrill_tagger_reloadedtaggedtest_reloadeds&&&&&&&&&&&&&&&&&&&& rr r sp6FFB I >PI2H?;]M9lww~~344+'>O+S11[ O[92 ;BB)  ' ( (K$kk+6O 12G1HI J) (( ?VW '( / 6 6((3  YY[F jG "#== gNL "499;#7"=X FG .1F1Fy1QQR z !"%l&8&8&:A>LFD VBKqZ!8 ;< =?  X #/"D"D )# Y -.% , "--/   2 29 = %y*CV  45J4KL M %&!++L9   2 2 4 , $ $ GG47GG H GGDIIjJ?@GGPSWW X% 9,HI#!++L9 "C ( (K KK k 2) ()9(:;< " # #{$*KK $< !$ -.>-?@A*44\B , , H I O P$U211) ( (z% $ $) ($ #s=Q%Q"AQ6,R -R Q " Q3 6 R R  R- c Vf!\R4\P!4pVe\V4V8:d \V4pV'd6\P !\V44\P !V4\W!,4pVRVpWVpVUU u.uFqU u.uF q^,NK up NK p pp V'gTp M\V4^,p VRV WlRrk\V4wr\V 4wpp\V 4wpp\RVR RVR R24\RV R RVR R24\RPTTV'dRMR 44WkWz3#uup iuup pi) Nz%Loading tagged data from treebank... zRead testing data (dz sents/z wds)zRead training data (z-Read baseline data ({:d} sents/{:d} wds) {:s}z[reused the training set]) r2r tagged_sentsr4randomseedshuffleint corpus_sizer3)rhr]rCrkr9cutoffrorqsenttrrrp bl_cutoff trainseqs traintokenstestseqs testtokens bltrainseqs bltraintokenss&&&&& rrSrSQss  56++- C , 9 $  C $%{# " #F(M9-I5>?YT4(4aqTT4(YL? !%  &!+ *9 % * %& +=9Y(6Xz#.}#= [- |7:a. FG 1 W[O5 IJ 7>>  (B.I  ) BB+)?s E((E# 9E(#E(cPVR,.pVR,F"pVPVR,V, 4K$ VRVUu.uFp^WaR,, , NK ppVR,.pVR,F"pVPVR,V, 4K$ VRVUu.uFp^WbR,, , NK pp^RIHp\\ \ V444p VP WW4VP.RO4VPV4R#uupiuupi) initialerrors rulescoresN tokencountr#)NNNg?) appendmatplotlib.pyplotpyplotr1ranger4plotaxissavefig) r:r{r|rO testcurve rulescorex traincurvepltrs &&&& rrcrcys?+,I|,, 223-:CET:JK:JQQ<0000:JIK_-.J -- *R.945. "#AHHQ1)HH $%KK%&L Os D"D#c<\V4\RV443#)c38"TFp\V4xK R#5i)N)r4).0rs& r corpus_size..s04a3q664s)r4sum)seqss&rrrs Is0400 11r__main__)NNi,r/Ng?r/FrFFNNNrNFN)NNz^-?[0-9]+(\.[0-9]+)?$CDz.*NN) r)z(The|the|A|a|An|an)$AT)z.*able$JJ)z.*ness$r)z.*ly$RB)z.*s$NNS)z.*ing$VBG)z.*ed$VBDr)#rTrXrr\ nltk.corpusrnltk.tagrrrrRrrnltk.tblr r rrrrr$r&r*r7r;r>rArFr rSrc NN_CD_TAGGERrQr__name__rrrrs DD$) ! 4; 8M( &*7BQJ%CP'&=}MN    2 zr