+ hRt^RIt^RIt^RIt^RIHt^RIHt^RI H t H t H t ^RI HtHtHt^RIHtHtHt^RIHt^RIHt^R IHtR t!R R ]4t]t!R R4t!RR]4t!RR]4t !RR] 4t!!RR] 4t"!RR]4t#R'Rlt$Rt%Rt&R'Rlt'Rt(Rt)R(Rlt*!R R!]4t+R"t,R)R#lt-R$t.R%t/]0R&8Xd ]/!4R#R# ]dLi;i)*a A classifier model based on maximum entropy modeling framework. This framework considers all of the probability distributions that are empirically consistent with the training data; and chooses the distribution with the highest entropy. A probability distribution is "empirically consistent" with a set of training data if its estimated frequency with which a class and a feature vector value co-occur is equal to the actual frequency in the data. Terminology: 'feature' ====================== The term *feature* is usually used to refer to some property of an unlabeled token. For example, when performing word sense disambiguation, we might define a ``'prevword'`` feature whose value is the word preceding the target word. However, in the context of maxent modeling, the term *feature* is typically used to refer to a property of a "labeled" token. In order to prevent confusion, we will introduce two distinct terms to disambiguate these two different concepts: - An "input-feature" is a property of an unlabeled token. - A "joint-feature" is a property of a labeled token. In the rest of the ``nltk.classify`` module, the term "features" is used to refer to what we will call "input-features" in this module. In literature that describes and discusses maximum entropy models, input-features are typically called "contexts", and joint-features are simply referred to as "features". Converting Input-Features to Joint-Features ------------------------------------------- In maximum entropy models, joint-features are required to have numeric values. Typically, each input-feature ``input_feat`` is mapped to a set of joint-features of the form: | joint_feat(token, label) = { 1 if input_feat(token) == feat_val | { and label == some_label | { | { 0 otherwise For all values of ``feat_val`` and ``some_label``. This mapping is performed by classes that implement the ``MaxentFeatureEncodingI`` interface. N) defaultdict) ClassifierI) call_megamparse_megam_weightswrite_megam_file) call_tadmparse_tadm_weightswrite_tadm_file) CutoffCheckeraccuracylog_likelihood)gzip_open_unicode)DictionaryProbDist) OrderedDictz epytext enca]tRt^NtoRtRRltRtRtRtRt Rt RRlt RR lt RR lt R t.ROt]RR l4tRtVtR #)MaxentClassifiera A maximum entropy classifier (also known as a "conditional exponential classifier"). This classifier is parameterized by a set of "weights", which are used to combine the joint-features that are generated from a featureset by an "encoding". In particular, the encoding maps each ``(featureset, label)`` pair to a vector. The probability of each label is then computed using the following equation:: dotprod(weights, encode(fs,label)) prob(fs|label) = --------------------------------------------------- sum(dotprod(weights, encode(fs,l)) for l in labels) Where ``dotprod`` is the dot product:: dotprod(a,b) = sum(x*y for (x,y) in zip(a,b)) c jWnW nW0nVP4\ V48XgQhR#)a Construct a new maxent classifier model. Typically, new classifier models are created using the ``train()`` method. :type encoding: MaxentFeatureEncodingI :param encoding: An encoding that is used to convert the featuresets that are given to the ``classify`` method into joint-feature vectors, which are used by the maxent classifier model. :type weights: list of float :param weights: The feature weight vector for this classifier. :type logarithmic: bool :param logarithmic: If false, then use non-logarithmic weights. N) _encoding_weights _logarithmiclengthlen)selfencodingweights logarithmics&&&&6/usr/lib/python3/dist-packages/nltk/classify/maxent.py__init__MaxentClassifier.__init__as-"" ' CL000c6VPP4#N)rlabelsrs&rr"MaxentClassifier.labelsxs~~$$&&rc fWnVPP4\V48XgQhR#)z Set the feature weight vector for this classifier. :param new_weights: The new feature weight vector. :type new_weights: list of float N)rrrr)r new_weightss&&r set_weightsMaxentClassifier.set_weights{s) $ ~~$$&#k*::::rc VP#)zO :return: The feature weight vector for this classifier. :rtype: list of float )rr#s&rrMaxentClassifier.weightss }}rc@VPV4P4#r!) prob_classifymax)r featuresets&&rclassifyMaxentClassifier.classifys!!*-1133rc/pVPP4FpVPPW4pVP'd4RpVF%wrgWPPV,V,, pK' WRV&KcRpVF%wrgWPV,V,,pK' WV&K \ W PRR7#)g?T)log normalize)rr"encoderrr) rr. prob_dictlabelfeature_vectortotalf_idf_valprods && rr,MaxentClassifier.prob_classifys ^^**,E!^^22:EN   #1KD]]40588E$2#(% #1KDMM$/588D$2#'% - ")1B1BdSSrc  aa a^2pR\V^, 4,R,pSPV4o \S P4S PRR7pVRVp\ RP V4RPRV44,4\ R R V^, ^\V4,,,,4\\4o\V4EF$wrgSPPW4pVPV3R lRR7VFwrSP'dSP V ,V ,p MSP V ,V ,p SPP#V 4p V P%R 4^,p V R V ,, p \V 4^/8dV R,R,p \ WLV^,R,V 3,4SV;;,V , uu&K EK' \ R R V^, ^\V4,,,,4\ RP V4RPV3RlV44,4\ RP V4RPV 3RlV44,4R#)z Print a table showing the effect of each of the features in the given feature set, and how they combine to determine the probabilities of each label for that featureset. z %-zs%s%8.3fTkeyreverseNz Featurec3P"TFpRRV,R,,xK R#5i)z%8s%s:NNN).0ls& r +MaxentClassifier.explain..s ?1eq"~..s$&z -cJ<\SPV^,,4#)absr)fid__rs&r*MaxentClassifier.explain..s#dmmE!H&=">r and label is z (%s):N,Nz... z TOTAL:c3D<"TFpRSV,,xK R#5iz%8.3fNrF)rGrHsumss& rrIrJs3Vv!Gd1g4E4Evs z PROBS:c3T<"TFpRSPV4,xK R#5irW)prob)rGrHpdists& rrIrJs!>v!g 1 --vs%()strr,sortedsamplesrZprintljustjoinrrint enumeraterr5sortrrdescribesplit)rr.columns descr_widthTEMPLATEr"ir7r8r:r;scoredescrr[rXsf&& @@rexplainMaxentClassifier.explains%  C a00:="":. UZZF!   k *gg??? @  dSK!Oa#f+o=>>?3!&)HA!^^22:EN   >   . $$$ MM$/%7E MM$/58E//5 $45a85(u:?!#J.EhQe!<<=U u$  . *" dSK!Oa#f+o=>>?   [ )BGG3Vv3V,V V     [ )gg>v>> ? rc  a\SR4'dSPRV#\\\ \ SP 444V3RlRR7SnSPRV#)zG Generates the ranked list of informative features from most to least. _most_informative_featuresNc<<\SPV,4#r!rN)fidrs&rrQ.sDMM#$6 7rTr?)hasattrrpr]listrangerr)rnsf&rmost_informative_features*MaxentClassifier.most_informative_featuressd 45 6 6222A6 6.4U3t}}-./7/D + 222A6 6rc VPR4pVR8Xd,VUu.uFq@PV,^8gKVNK ppM1VR8Xd+VUu.uFq@PV,^8gKVNK ppVRVF>p\VPV,R RVPP V4 24K@ R#uupiuupi)z :param show: all, neg, or pos (for negative-only or positive-only) :type show: str :param n: The no. of top features :type n: int Nposnegz8.3frU)rxrr_rre)rrwshowfidsrrs&&& rshow_most_informative_features/MaxentClassifier.show_most_informative_featuress--d3 5=#'B4C==+=+ACC4DBD U]#'B4C==+=+ACC4DB8C T]]3'-Qt~~/F/Fs/K.LM NCBsB>B>C-CcR\VPP44VPP43,#)z:)rrr"rr#s&r__repr__MaxentClassifier.__repr__s:K %%' ( NN ! ! #O   rNc pVfRpVFpVR 9gK \RV,4h VP4pVR8Xd\WWE3/VB#VR8Xd\WWE3/VB#VR8Xd\ WWEV3/VB#VR8Xd*Tp W9R&WIR&WYR&WiR &\ P !V3/V B#\R V,4h) aV Train a new maxent classifier based on the given corpus of training samples. This classifier will have its weights chosen to maximize entropy while remaining empirically consistent with the training corpus. :rtype: MaxentClassifier :return: The new maxent classifier :type train_toks: list :param train_toks: Training data, represented as a list of pairs, the first member of which is a featureset, and the second of which is a classification label. :type algorithm: str :param algorithm: A case-insensitive string, specifying which algorithm should be used to train the classifier. The following algorithms are currently available. - Iterative Scaling Methods: Generalized Iterative Scaling (``'GIS'``), Improved Iterative Scaling (``'IIS'``) - External Libraries (requiring megam): LM-BFGS algorithm, with training performed by Megam (``'megam'``) The default algorithm is ``'IIS'``. :type trace: int :param trace: The level of diagnostic tracing output to produce. Higher values produce more verbose output. :type encoding: MaxentFeatureEncodingI :param encoding: A feature encoding, used to convert featuresets into feature vectors. If none is specified, then a ``BinaryMaxentFeatureEncoding`` will be built based on the features that are attested in the training corpus. :type labels: list(str) :param labels: The set of possible labels. If none is given, then the set of all labels attested in the training data will be used instead. :param gaussian_prior_sigma: The sigma value for a gaussian prior on model weights. Currently, this is supported by ``megam``. For other algorithms, its value is ignored. :param cutoffs: Arguments specifying various conditions under which the training should be halted. (Some of the cutoff conditions are not supported by some algorithms.) - ``max_iter=v``: Terminate after ``v`` iterations. - ``min_ll=v``: Terminate after the negative average log-likelihood drops under ``v``. - ``min_lldelta=v``: Terminate if a single iteration improves log likelihood by less than ``v``. iiszUnexpected keyword arg %rgismegamtadmtracerr"gaussian_prior_sigmazUnknown algorithm %s) max_itermin_ll min_lldeltamax_acc min_accdelta count_cutoffnormexplicit bernoulli) TypeErrorlower train_maxent_classifier_with_iis train_maxent_classifier_with_gis"train_maxent_classifier_with_megamTadmMaxentClassifiertrain ValueError) cls train_toks algorithmrrr"rcutoffsr@kwargss &&&&&&&, rrMaxentClassifier.trains |  IC   ;c ABBOO%  387> % 387> ' !585IMT & F#7O!): %8 -A) *'--jCFC C3i?@ @r)rrrpr)T)) )rall)GISIISMEGAMTADM)NNNrM)__name__ __module__ __qualname____firstlineno____doc__rr"r'rr/r,rmrxrr ALGORITHMS classmethodr__static_attributes____classdictcell__ __classdict__s@rrrNs\$1.';4T(* X 7O  1JaAaArrcBa]tRtRtoRtRtRtRtRtRt Rt Vt R #) MaxentFeatureEncodingIidap A mapping that converts a set of input-feature values to a vector of joint-feature values, given a label. This conversion is necessary to translate featuresets into a format that can be used by maximum entropy models. The set of joint-features used by a given encoding is fixed, and each index in the generated joint-feature vectors corresponds to a single joint-feature. The length of the generated joint-feature vectors is therefore constant (for a given encoding). Because the joint-feature vectors generated by ``MaxentFeatureEncodingI`` are typically very sparse, they are represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. Feature encodings are generally created using the ``train()`` method, which generates an appropriate encoding based on the input-feature values and labels that are present in a given corpus. c \4h)a  Given a (featureset, label) pair, return the corresponding vector of joint-feature values. This vector is represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. :type featureset: dict :rtype: list(tuple(int, int)) NotImplementedErrorrr.r7s&&&rr5MaxentFeatureEncodingI.encode{ "##rc \4h)zr :return: The size of the fixed-length joint-feature vectors that are generated by this encoding. :rtype: int rr#s&rrMaxentFeatureEncodingI.length "##rc \4h)z :return: A list of the "known labels" -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. :rtype: list rr#s&rr"MaxentFeatureEncodingI.labelss "##rc \4h)z :return: A string describing the value of the joint-feature whose index in the generated feature vectors is ``fid``. :rtype: str rrrrs&&rreMaxentFeatureEncodingI.describerrc \4h)a7 Construct and return new feature encoding, based on a given training corpus ``train_toks``. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. r)rrs&&rrMaxentFeatureEncodingI.trainrrrFN) rrrrrr5rr"rerrrrs@rrrds(, $$$$ $ $rrcBa]tRtRtoRtRtRtRtRtRt Rt Vt R #) #FunctionBackedMaxentFeatureEncodingiz A feature encoding that calls a user-supplied function to map a given featureset/label pair to a sparse joint-feature vector. c *W nWnW0nR#)a Construct a new feature encoding based on the given function. :type func: (callable) :param func: A function that takes two arguments, a featureset and a label, and returns the sparse joint feature vector that encodes them:: func(featureset, label) -> feature_vector This sparse joint feature vector (``feature_vector``) is a list of ``(index,value)`` tuples. :type length: int :param length: The size of the fixed-length joint-feature vectors that are generated by this encoding. :type labels: list :param labels: A list of the "known labels" for this encoding -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. N)_length_func_labels)rfuncrr"s&&&&rr,FunctionBackedMaxentFeatureEncoding.__init__s0   rc$VPW4#r!)rrs&&&rr5*FunctionBackedMaxentFeatureEncoding.encodeszz*,,rcVP#r!rr#s&rr*FunctionBackedMaxentFeatureEncoding.length ||rcVP#r!rr#s&rr"*FunctionBackedMaxentFeatureEncoding.labelsrrcR#)zno description availablerFrs&&rre,FunctionBackedMaxentFeatureEncoding.describes)r)rrrN) rrrrrrr5rr"rerrrs@rrrs( 8-**rrcZa]tRtRtoRtR RltRtRtRtRt ] R R l4t R t Vt R#) BinaryMaxentFeatureEncodingia A feature encoding that generates vectors containing a binary joint-features of the form: | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. This method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. The ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form:: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. c \VP44\\\V4448wd \ R4h\ V4VnW n\V4VnRVn RVn V'db\V4UUu/uFwrVWeVP,bK uppVn V;P\VP4, unV'dqVUUUu0uFwrxqgkK p ppp\V 4UUu/uFwrWWuVP,bK uppVn V;P\V 4, unR#R#uuppiuupppiuuppi)a[ :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode(..., fname:fval, ..., label)[id]`` is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. HMapping values must be exactly the set of integers from 0...len(mapping)N setvaluesrvrrrur_mappingr _alwayson_unseenrc rr"mappingunseen_featuresalwayson_featuresrjr7fnamefvalfnamess &&&&& rr$BinaryMaxentFeatureEncoding.__init__*" w~~ Cc'l(;$< <8  F| ( 97| <, , :CF:K:KJQ4<<'':KDN LLC/ /L 8?@ 4eeF@FOPVFWXFW Et||#33FWXDL LLCK 'L   AXE!5 E'E.c$.pVP4FwrEWEV3VP9d)VPVPWEV3,^34K@VP'gKTVPFpWEV3VP9gKK{ W@P9gKVPVPV,^34K VP 'd5W P 9d%VPVP V,^34V#)itemsrappendrrrrr.r7rrrlabel2s&&& rr5"BinaryMaxentFeatureEncoding.encode6s&++-KEU#t}}4uE/A!BA FG"llFV, =+  , e)>>e~~5 OOT^^E2A6 7rcN\V\4'g \R4hVPV\ VP 48d#VPV,wrEpV RV: RV: 2#VP'dUWPP49d7VPP4FwrgW8XgK RV,u# R#VP'dUWPP49d7VPP4FwrGW8XgK RV,u# R#\R4h \dXR.\ TP 4,TnTP P4Fwr#Y PT&K ELti;izdescribe() expected an intz==rSz label is %rz %s is unseenzBad feature idN isinstancerbr _inv_mappingAttributeErrorrrrrrrrrr:inforjrrr7f_id2s&& rre$BinaryMaxentFeatureEncoding.describeQP$$$89 9 ,    #dmm$ $#'#4#4T#: U%WBthnUI> > ^^^(=(=(? ? $ 4 4 6 =(500!7\\\dll&9&9&;; $ 2 2 4 =)E11!5-. .# ,!#s4=='9 9D ==..0'+!!!$1 , EAF$#F$cVP#r!rr#s&rr""BinaryMaxentFeatureEncoding.labelsj ||rcVP#r!rr#s&rr"BinaryMaxentFeatureEncoding.lengthnrrNc /p\4p\\4pVFwrV'dW9d\RV ,4hVP V 4VP 4FFwrWzV 3;;,^, uu&WzV 3,V8gK,WV 3V9gK7\ V4WZW3&KH K VfTpV!W53/VB#)a Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``BinaryMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. Unexpected label %s)rrrbraddrr rrrr"optionsr seen_labelscounttokr7rrs &&&&, rr!BinaryMaxentFeatureEncoding.trainrs8e C $JC%- !6!>?? OOE " #yy{ Tk"a'"%5U+7:69'lt 23 + % > F6.g..rrrrrrrFFrMNrrrrrrr5rer"rrrrrrs@rrrs9&P/(b6/20/0/rrcPa]tRtRtoRtR Rlt]R4tRtRt Rt R t Vt R#) GISEncodingia A binary feature encoding which adds one new joint-feature to the joint-features defined by ``BinaryMaxentFeatureEncoding``: a correction feature, whose value is chosen to ensure that the sparse vector always sums to a constant non-negative number. This new feature is used to ensure two preconditions for the GIS training algorithm: - At least one feature vector index must be nonzero for every token. - The feature vector must sum to a constant non-negative number for every token. Nc  \PWW#V4Vf)\VUUUu0uFwrgqkK uppp4^,pWPnR#uupppi)z :param C: The correction constant. The value of the correction feature is based on this value. In particular, its value is ``C - sum([v for (f,v) in encoding])``. :seealso: ``BinaryMaxentFeatureEncoding.__init__`` N)rrr_C) rr"rrrCrrr7s &&&&&& rrGISEncoding.__init__sO $,, '4E  9w?w3UUw?@1DA@s A c VP#)zGThe non-negative constant that all encoded feature vectors will sum to.rr#s&rr GISEncoding.Cs wwrc\PWV4p\PV4p\RV44pWPP8d \ R4hVP W@PV, 34V#)c3*"TF wrVxK R#5ir!rF)rGfvs& rrI%GISEncoding.encode..s-H&1AHz&Correction feature is not high enough!)rr5rsumrrr)rr.r7r base_lengthr9s&&& rr5GISEncoding.encodesj.55dN188> -H-- GG EF Fggo67rc:\PV4^,#r)rrr#s&rrGISEncoding.lengths*11$7!;;rcV\PV48XdRVP,#\PW4#)zCorrection feature (%s))rrrre)rr:s&&rreGISEncoding.describes6 .55d; ;,tww6 6.77C Crr)FFN) rrrrrrpropertyrr5rrerrrs@rrrs9   <DDrrcVa]tRtRtoR RltRtRtRtRt] R Rl4t R t Vt R#) TadmEventMaxentFeatureEncodingic\V4Vn\4Vn\P WVPW44R#r!)rr_label_mappingrr)rr"rrrs&&&&&rr'TadmEventMaxentFeatureEncoding.__init__s1#G, )m#,, $-- rc.pVP4FwrEWB3VP9d$\VP4VPWB3&WPP9dH\ V\ 4'g$\VP4VPV&MWPPV&VP VPWB3,VPV,34K V#r!)rrrr/rrbr)rr.r7rfeaturevalues&&& rr5%TadmEventMaxentFeatureEncoding.encodes(..0NGt}}425dmm2D w.////!%--14T5H5H1ID''.16''. OO/0$2E2Ee2LM 1rcVP#r!rr#s&rr"%TadmEventMaxentFeatureEncoding.labelsrrclVPF#wr#VPW#3,V8XgK W#3u# R#r!)r)rrrr2r7s&& rre'TadmEventMaxentFeatureEncoding.describes/"mmNG}}g-.#5'',rc,\VP4#r!)rrr#s&rr%TadmEventMaxentFeatureEncoding.lengths4==!!rNc \4pV'g.p\V4pVFwrgWs9gK VPV4K VF0wrgVF%pVFpW3V9gK \V4WXV3&K K' K2 V!W53/VB#r!)rrurr) rrrr"r rr.r7r2s &&&&, rr$TadmEventMaxentFeatureEncoding.trains-F*% !+ J" e$",", J)G'w647L% 01 * ", 6.g..r)r/rrr) rrrrrr5r"rerrrrrrs@rr-r-s2  ( "//rr-cZa]tRtRtoRtR RltRtRtRtRt ] R R l4t R t Vt R#) TypedMaxentFeatureEncodingia A feature encoding that generates vectors containing integer, float and binary joint-features of the form: Binary (for string and boolean features): | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Value (for integer and float features): | joint_feat(fs, l) = { fval if (fs[fname] == type(fval)) | { and (l == label) | { | { not encoded otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. For string and boolean features [type(fval) not in (int, float)] this method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. For integer and float features [type(fval) in (int, float)] this method will create one feature for each combination of ``fname`` and ``label`` that occurs at least once in the training corpus. For binary features the ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. c \VP44\\\V4448wd \ R4h\ V4VnW n\V4VnRVn RVn V'db\V4UUu/uFwrVWeVP,bK uppVn V;P\VP4, unV'dqVUUUu0uFwrxqgkK p ppp\V 4UUu/uFwrWWuVP,bK uppVn V;P\V 4, unR#R#uuppiuupppiuuppi)a] :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode({..., fname:fval, ...``, label)[id]} is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. rNrrs &&&&& rr#TypedMaxentFeatureEncoding.__init__Trrc.pVP4EF!wrE\V\\34'dRV\ V4V3VP 9d3VP VP V\ V4V3,V34KqKsWEV3VP 9d)VP VP WEV3,^34KVP'gKVPFpWEV3VP 9gKK W@P9gKVP VPV,^34EK$ VP'd5W P9d%VP VPV,^34V#r) rrrbfloattyperrrrrrs&&& rr5!TypedMaxentFeatureEncoding.encodes(&++-KE$e --4:u->OOT]]5$t*e3K%Ld$ST?'4==8OOT]]53E%F$JK\\\"&,,!0DMMA!#/ !LL0$OOT\\%-@!,DE'., >>>e~~5 OOT^^E2A6 7rcN\V\4'g \R4hVPV\ VP 48d#VPV,wrEpV RV: RV: 2#VP'dUWPP49d7VPP4FwrgW8XgK RV,u# R#VP'dUWPP49d7VPP4FwrGW8XgK RV,u# R#\R4h \dXR.\ TP 4,TnTP P4Fwr#Y PT&K ELti;irrrs&& rre#TypedMaxentFeatureEncoding.describerrcVP#r!rr#s&rr"!TypedMaxentFeatureEncoding.labelsrrcVP#r!rr#s&rr!TypedMaxentFeatureEncoding.lengthrrNc /p\4p\\4pVFwrV'dW9d\RV ,4hVP V 4VP 4Fkwr\ V 4\\39d \ V 4p WzV 3;;,^, uu&WzV 3,V8gKQWV 3V9gK\\V4WZW3&Km K VfTpV!W53/VB#)aq Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``TypedMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. Note: recognized feature values types are (int, float), over types are interpreted as regular binary features. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. r ) rrrbrr rrCrBrr s &&&&, rr TypedMaxentFeatureEncoding.trains>e C $JC%- !6!>?? OOE " #yy{ :#u-:DTk"a'"%5U+7:69'lt 23 + %" > F6.g..rrrrrrs@rr>r>s:6p/(b@/25/5/rr>c VPR^d4\V4pVf\PWR7p\ VR4'g \ R4hRVP , p\W4p\\P!V^8H4^,4p\P!\V4R4p VFp \PW&K \W)4p \P!V4p ?V^8d\!RVR,,4V^8d!\!4\!R4\!R 4V^8d_VP";'g \%W4p VP&;'g \)W4pVP*p\!R WV3,4\-WV4pVFp VV ;;,^, uu&K \P!V4p?V P/4p WV, V,, p V P1V 4VP3W4'gKT^8d*\%Y4p \)Y4p\!R T R RTR 24T # \4d\!R 4LJh;i)aj Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Generalized Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rr"rzJThe GIS algorithm requires an encoding that defines C (e.g., GISEncoding).r2d ==> Training (%d iterations)- Iteration Log Likelihood Accuracy- --------------------------------------- %9d %14.5f %9.3f* Training stopped: keyboard interrupt Final 14.5f 9.3f) setdefaultr rrrtrrcalculate_empirical_fcountrnumpynonzerozerosrNINF ConditionalExponentialClassifierlog2r_llr accr itercalculate_estimated_fcountrr'checkKeyboardInterrupt)rrrr"r cutoffcheckerCinvempirical_fcount unattestedrrr classifierlog_empirical_fcountrarbiternumestimated_fcountlog_estimated_fcounts&&&&, rrrsZ z3'!'*M$$Z$? 8S ! ! -    D2*GU]]#3q#89!<=Jkk#./5Gzz 1(DJ!::&67 qy .1DDE qy  => => qy"%%OO )O#''KK8J+K',,3wC6HHI :   " %*%"#(::.>#?  !((*G /CCtK KG  " "7 +""::: qy J 3z. "2e*DT ;<  < :; sI+I+;B8I++JJc\P!VP4R4pVF3wr4VPW44FwrVW%;;,V, uu&K K5 V#rO)r[r]rr5)rrfcountrr7indexvals&& rrZrZfsM [[*C 0F "//#5JE MS M6! Mrc@\P!VP4R4pVFrwrEVPV4pVP 4FHpVP V4pVP WE4FwrW8;;,Wy,, uu&K! KJ Kt V#rq)r[r]rr,r^rZr5) rkrrrrrr7r[rZrrrs &&& rrdrdps} [[*C 0F ((-]]_E::e$D%__S8  t{* 9%! Mrc VPR^d4\V4pVf\PWR7p\ W4\ V4, p\ W4p\P!\WwPR7R4p\P!V\ V4^34p \\P!V^8H4^,4p \P!\ V4R4p V Fp \PW&K \!W+4p V^8d\#RVR,,4V^8d!\#4\#R4\#R4V^8d_VP$;'g \'W4pVP(;'g \+W4pVP,p\#RVW3,4\/VV V VVVV V4pV P14p V V, p V P3V 4VP5W4'gKT^8d*\'Y4p\+Y4p\#R TR R TR 24T # \6d\#R 4LJh;i)af Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Improved Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rrN)r@rOrPrQrRrSrTrUrVrWrX)rYr rrrZrcalculate_nfmapr[arrayr] __getitem__reshaperr\r]r^r_r_rar rbr rccalculate_deltasrr'rerf)rrrr"rrgempirical_ffreqnfmapnfarray nftransposerjrrrrkrarbrmdeltass&&&&, rrrs6 z3'!'*M.44Z4O1FZXO J 1Ekk&,=,=>DG--#g,):;KU]]?a#78; => qy"%%OO )O#''KK8J+K',,3w6HHI& F!((*G v G  " "7 +""::: qy J 3z. "2e*DT ;<  < :; s"I=IA int) c3*"TF wrVxK R#5ir!rFrGidrts& rrI"calculate_nfmap..sK/J)2#/Jr#)rr"r r$r5rc)rrnfsetr_r7rjnfs&& rrwrwsm* EE__&E IIcKxs/JKK L'"+5!1 2!1gqBE!1 22 2s+ A<c  RpRp \P!VP4R4p \P!\ V4VP43R4p VFwrVP V 4pVP 4Fep VPW4p\RV44pVF9wppWV,V3;;,VPV 4V,, uu&K; Kg K V \ V4,p \V 4Fp\P!WZ4p^V,pVV,p\P!VV ,^R7p\P!VV ,^R7pVFpVV;;,^, uu&K WV, V), ,p \P!\VV, 44\P!\V 44, pVV8gKV u# V #)aW Calculate the update values for the classifier weights for this iteration of IIS. These update weights are the value of ``delta`` that solves the equation:: ffreq_empirical[i] = SUM[fs,l] (classifier.prob_classify(fs).prob(l) * feature_vector(fs,l)[i] * exp(delta[i] * nf(feature_vector(fs,l)))) Where: - *(fs,l)* is a (featureset, label) tuple from ``train_toks`` - *feature_vector(fs,l)* = ``encoding.encode(fs,l)`` - *nf(vector)* = ``sum([val for (id,val) in vector])`` This method uses Newton's method to solve this equation for *delta[i]*. In particular, it starts with a guess of ``delta[i]`` = 1; and iteratively updates ``delta`` with: | delta[i] -= (ffreq_empirical[i] - sum1[i])/(-sum2[i]) until convergence, where *sum1* and *sum2* are defined as: | sum1[i](delta) = SUM[fs,l] f[i](fs,l,delta) | sum2[i](delta) = SUM[fs,l] (f[i](fs,l,delta).nf(feature_vector(fs,l))) | f[i](fs,l,delta) = (classifier.prob_classify(fs).prob(l) . | feature_vector(fs,l)[i] . | exp(delta[i] . nf(feature_vector(fs,l)))) Note that *sum1* and *sum2* depend on ``delta``; so they need to be re-computed each iteration. The variables ``nfmap``, ``nfarray``, and ``nftranspose`` are used to generate a dense encoding for *nf(ltext)*. This allows ``_deltas`` to calculate *sum1* and *sum2* using matrices, which yields a significant performance improvement. :param train_toks: The set of training tokens. :type train_toks: list(tuple(dict, str)) :param classifier: The current classifier. :type classifier: ClassifierI :param ffreq_empirical: An array containing the empirical frequency for each feature. The *i*\ th element of this array is the empirical frequency for feature *i*. :type ffreq_empirical: sequence of float :param unattested: An array that is 1 for features that are not attested in the training data; and 0 for features that are attested. In other words, ``unattested[i]==0`` iff ``ffreq_empirical[i]==0``. :type unattested: sequence of int :param nfmap: A map that can be used to compress ``nf`` to a dense vector. :type nfmap: dict(int -> int) :param nfarray: An array that can be used to uncompress ``nf`` from a dense vector. :type nfarray: array(float) :param nftranspose: The transpose of ``nfarray`` :type nftranspose: array(float) g-q=i,rOc3*"TF wrVxK R#5ir!rFrs& rrI#calculate_deltas..Ts9.YbS.r#)axis) r[onesrr]rr,r"r5r$rZrvouterrO)rrkrjffreq_empiricalr}r~rrNEWTON_CONVERGE MAX_NEWTONrArr7distr8rrrtrangenumnf_delta exp_nf_deltanf_exp_nf_deltasum1sum2rrn_errors&&&&&&&& rr{r{sROJ ZZ)3 /F  SZ!23S9A '',__&E%__S8N9.99B)C)R- DIIe$4s$:: * '!ZA*%;;w/({ % 4yy)2yy1,15C INI T)dU22))C$ 678599S[;QQ _ $M#&& Mrc RpRpRV9d VR,pRV9d VR,pVf,VPR^4p\PWVRR7pMVe \R4h\P !RR 7wr\ V R 4;_uu_4p \WWVR 7RRR4\P!V 4.p T .RO, p T'd T R ., p T'g T R., p T'dRT^,, pM^pT RRT,R., p T^8d T R., p RT9dT RRTR,,., p RT9d#T RR\TR,4,., p \TR4'd T R., p T RT ., p \T 4p\P!T 4\!YP#4T4pT\$P&!\$P(4,p\+TT4# +'giELz;i \\3dp \R T ,4T hRp ? ii;i \dp \RT RT 24Rp ? LRp ? ii;i)a{ Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the external ``megam`` library. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. :see: ``nltk.classify.megam`` TrrNr)r"rz$Specify encoding or labels, not bothznltk-prefixw)rrz,Error while creating megam training file: %sz -explicitz-fvalsr2z-lambdaz%.2fz-tunez-quietrz-maxirDll_deltaz-dppcostz -multilabel multiclasszWarning: unable to delete z: )z-nobiasz-repeat10)getrrrtempfilemkstempopenroscloseOSErrorrOrtrremover_rrr[r`er)rrrr"rrrrrfdtrainfile_name trainfilerr  inv_variancestdoutrs&&&&&, rrrsiHIV*%f;' zz.!4 .44 Vt5   ?@@T%--W= .# & &) ii '  G ++GK=  H:1144   6L0'::G qyH:VGTF:$6677V FD3vj'9#::;;x  M?" n--G  FB .! "&//*;XFG uzz%''""G Hg ..c' & & Z TG!KLRSSTD B *>*:"QC@AABsH$-H%H H%*I H"  H%%I6I  I I7I22I7c0a]tRtRto]R4tRtVtR#)ric :VPRR4pVPR^4pVPRR4pVPRR4pVPR^4pVPR^4pVPR4p VPR 4p V'g\PWVR 7p\P!R R R 7wr\P!RR7wr\ V R4p\ WV4VP4.pVPR.4VPRV.4V'd"VPRRV^,,.4V 'dVPRRV ,.4V 'd$VPRR\V 4,.4VPRV .4VPRV.4V^8dVPR.4MVPR.4\V4\V4;_uu_4p\V4pRRR4\P!V 4\P!V4X\P !\P"4,pV!VV4# +'giLp;i)rtao_lmvmrrNr"rrrrrNznltk-tadm-events-z.gz)rsuffixznltk-tadm-weights-rrz-monitorz-methodz-l2z%.6fz-max_itz%dz-fatolz -events_inz -params_outz2>&1z-summary)rr-rrrr r rextendrOrrrrrr[r`r)rrrrrrr"sigmarrr trainfile_fdr weightfile_fdweightfile_namerr  weightfilers&&, rrTadmMaxentClassifier.trainsJJ{J7  7A&::j$/Hd+ 115zz.!4 ::j)::m,5;;<H(0'7'7&u( $ *2)9)9AU)V& %nc:  i8 |$ 9-.  NNE6E1H#45 6  NNIth7 8  NNHfs8}&<= > n56 78 19 NNF8 $ NNJ< (' / " "j(4G# .! /" 5::egg&&8W%%# "s  J  J rFN)rrrrrrrrrs@rrrs5&5&rrc ^RIp^RIHpV!4p\V R24;_uu_4pVP!\ \ VPVPV4444pRRR4\V R24;_uu_4pVPV4pRRR4\V R24;_uu_4pVPV4pRRR4\V R24;_uu_4pVPV4pRRR4XXXX3# +'giL;i +'giL;i +'giLl;i +'giLO;i)rMN) MaxentDecoder /weights.txt /mapping.tab /labels.txt /alwayson.tab) r[ nltk.tabdatarrrxrumapfloat64txt2list tupkey2dict tab2ivdict) tab_dirr[rmdecr wgtmpglabaons & rload_maxent_paramsrs* ?D  & ' '1kk$s5==$--2BCDE (  & ' '1q! (  % & &!mmA '  ' ( (Aooa  ) S#  ( ' ( ' ' & ) (s/?DD5D+%D> D  D( + D; > E c Z^RIHp^RIHp^RIHpV!4pV!V4'g V!V4\ RV 24\V R2R4;_uu_4p V PVP\\VP444 4RRR4\V R2R4;_uu_4p V PVPV4 4RRR4\V R 2R4;_uu_4p V PVPV4 4RRR4\V R 2R4;_uu_4p V PVPV4 4RRR4R# +'giL;i +'giL;i +'giLy;i +'giR#;i) rM)mkdir)isdir) MaxentEncoderzSaving Maxent parameters in rrNrrr)rros.pathrrrr_rwritelist2txtrreprtolist tupdict2tab ivdict2tab) rrrrrrrrmencr s &&&&& rsave_maxent_paramsr2s1* ?D >> g ( 23  & , , 4==T3::E 2"E33"F4"F E0 3 F  F  F* c^RIHp^RIHpV!R4p\ V4wr4rV\ \ WTVR7V4pV!VR7#)rM)find)ClassifierBasedPOSTaggerz.taggers/maxent_treebank_pos_tagger_tab/english)r)rk) nltk.datarnltk.tag.sequentialrrrr)rrrrrrrmcs rmaxent_pos_taggerrIsE<CDG+G4Cc #CDc B $r 22rc>^RIHpV!\P4pR#)rM) names_demoN)nltk.classify.utilrrr)rrks rdemorXs-,223Jr__main__)rNN)rNNrM)z/tmp)1rr[ ImportErrorrr collectionsrnltk.classify.apirnltk.classify.megamrrrnltk.classify.tadmrrr rr r r rr nltk.probabilityr nltk.utilr __docformat__rr_rrrrr-r>rrZrdrrwr{rrrrrrrrFrrrs),Z  #)QQMMFF'/! JA{JA\$4 F$F$R,**@,*^G/"8G/T:D-:Dz5/%@5/pa/!7a/R_D $Yx38{NT/x7&+7&~,+. 34  zFM1  sC""C,+C,