+ h>Rt^RIt^RIHtHt^RIHtRt]Pt Rt Rt ^RI Ht^tRtRt!R R ]R 7t!R R ]4t!RR]4t!RR]4t!RR4tR# ]dRtLNi;i)z Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. N)ABCMetaabstractmethodreducec.\P!V4#N)_mathlog2)xs&:/usr/lib/python3/dist-packages/nltk/metrics/association.pyr s %**Q-c\RV4#)cW,#r)r ys&&r r ..sr r)ss&r r r s V.2r g#B ;) fisher_exactc\hrNotImplementedError)_args_kwargss*,r rrs!!r ca]tRt^-toRt^t]]R44t]]R44t ] R4t ]R4t ] R4t ] R4t]R4t] R 4t] R 4t] R 4t] R 4tR tVtR#)NgramAssocMeasuresa An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. c \R4h)z>Calculates values of a contingency table from marginal values.?The contingency table is not availablein the general ngram caser marginalss*r _contingencyNgramAssocMeasures._contingencyB" P  r c \R4h)ACalculates values of contingency table marginals from its values.rr) contingencys*r _marginalsNgramAssocMeasures._marginalsJr"r c# 0aaa"\S4p\SP4Uu.uF p^V,NK pp\\S44F;o\ VVV3RlV44VSP^, ,, xK= R#uupi5i)3Calculates expected values for a contingency table.c3<a"TF6o\VVV3Rl\^SP,444xK8 R#5i)c3b<"TF$qS,SS,8XgKSV,xK& R#5irr).0r contijs& r @NgramAssocMeasures._expected_values...]s(P)9A!eQ=OQ)9s//N)sumrange_n)r,r/clsr-r.s&@r r06NgramAssocMeasures._expected_values..\s3!Pq#&&y)9PPP!s>AN)r2r3r4len_product)r5r-n_allr.bitssff ` r _expected_values#NgramAssocMeasures._expected_valuesRsD  %cff . 1Q .s4y!A!SVVaZ( * "/s#BBABc @V\,V\,, #)z Scores ngrams by their frequency)NGRAMTOTALrs*r raw_freqNgramAssocMeasures.raw_freqcs)E"222r c V\,\V\,4V\,VP^, ,, , V\,\ ,R,, #)zqScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. g?)r>r8UNIGRAMSr?r4_SMALLr5rs&*r student_tNgramAssocMeasures.student_thsR e y*+y/?CFFQJ/OP Q u  &3 ./ /r c zVP!V!pVPV4p\R\W#444#)zJScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. c3h"TF(wrW, ^,V\,, xK* R#5i)N)rDr,obsexps& r r0,NgramAssocMeasures.chi_sq..ys%U_CI!#sV|44_s02)r r;r2zip)r5rr-expss&* r chi_sqNgramAssocMeasures.chi_sqrs9 +##D)US_UUUr c V\,VPR^4,\V\,4, #)zScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. power)r>getr8rC)rkwargss*,r mi_likeNgramAssocMeasures.mi_like{s5 6::gq#99H h =   r c \V\,V\,VP^, ,,4\\ V\ ,44, #)zNScores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. )_log2r>r?r4r8rCrEs&*r pmiNgramAssocMeasures.pmisG Yu% %(8SVVaZ(HHIE Yx( )M   r c  VP!V!p^\R\W PV4444,#)zFScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4.c3"TF5wrV\W\,, \,4,xK7 R#5ir)_lnrDrKs& r r06NgramAssocMeasures.likelihood_ratio..s2 A #c6\*V34 4 4As=?)r r2rOr;r5rr-s&* r likelihood_ratio#NgramAssocMeasures.likelihood_ratiosE+3 &:&:4&@A    r c \V\,4V\,VP^, ,, pV\,\ V\,V, 4^, ,#)z1Scores ngrams using the Poisson-Stirling measure.)r8rCr?r4r>rZ)r5rrMs&* r poisson_stirling#NgramAssocMeasures.poisson_stirlingsNy*+y/?CFFQJ/OP55)9C)?#@1#DEEr c ZVP!V!pV^,\VRR4, #)z&Scores ngrams using the Jaccard index.N)r r2ras&* r jaccardNgramAssocMeasures.jaccards-+AwT#2Y''r rN)__name__ __module__ __qualname____firstlineno____doc__r4 staticmethodrr r& classmethodr;r@rFrQrWr[rbreri__static_attributes____classdictcell__ __classdict__s@r rr-s$ B     33//VV      FF ((r r) metaclassca]tRt^toRt^t]R4t]R4t]R4t ] R4t ] R4t ] R4t ]R4tR tVtR #) BigramAssocMeasuresa8 A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_ii counts ``(w1, w2)``, i.e. the bigram being scored - n_ix counts ``(w1, *)`` - n_xi counts ``(*, w2)`` - n_xx counts ``(*, *)``, i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx c \Vwr4W@, pW0, pWWbV, V, V, 3#)zECalculates values of a bigram contingency table from marginal values.r)n_ii n_ix_xi_tuplen_xxn_ixn_xin_oin_ios&&& r r BigramAssocMeasures._contingencys2% {{D+"4t";<VwrEW0PWV3V4,#)zsScores bigrams using chi-square, i.e. phi-sq multiplied by the number of bigrams, as in Manning and Schutze 5.3.3. )r)r5rzr{r|r}r~s&&&& r rQBigramAssocMeasures.chi_sqs# % jjd|T:::r c RVP!V!wr#rE\W#.WE..RR7wrgV#)zScores bigrams using Fisher's Exact Test (Pedersen 1996). Less sensitive to small counts than PMI or Chi Sq, but also more expensive to compute. Requires scipy. less) alternative)r r)r5rrzrrroddspvalues&* r fisherBigramAssocMeasures.fishers6"%!1!19!=D%|d\&BPVW r c 6Vwr4^V,W4,, #)z(Scores bigrams using Dice's coefficient.r)rzr{r|r}r~s&&& r diceBigramAssocMeasures.dices% 4x4;''r rN)rkrlrmrnror4rpr r&r;rqrrQrrrrrsrts@r rxrxs6 B==MMKK  ;;  ((r rxcHa]tRt^toRt^t]R4t]R4tRt Vt R#)TrigramAssocMeasuresa A collection of trigram association measures. Each association measure is provided as a function with four arguments:: trigram_score_fn(n_iii, (n_iix, n_ixi, n_xii), (n_ixx, n_xix, n_xxi), n_xxx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iii counts ``(w1, w2, w3)``, i.e. the trigram being scored - n_ixx counts ``(w1, *, *)`` - n_xxx counts ``(*, *, *)``, i.e. any trigram c <VwrEpVwrxp W`, p WP, p W@, p W, V , V , p W, V , V , pWp, V , V , pW0, V , V , V , V , V, V, pW WWVV3#)zCalculates values of a trigram contingency table (or cube) from marginal values. >>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) r)n_iii n_iix_tuple n_ixx_tuplen_xxxn_iixn_ixin_xiin_ixxn_xixn_xxin_oiin_ioin_iion_ooin_oion_ioon_ooos&&&& r r !TrigramAssocMeasures._contingencys!,u +u    %- %- %- %-5=EMeE%GGr c Vwrr4rVrxVW,W,W,3W,V,V,W,V,V,W,V,V,3\V43#)zCalculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) r2) r%rrrrrrrrs * r r&TrigramAssocMeasures._marginals&sm BM>eE%  ]EM5= 9 %- %- %-     r rN rkrlrmrnror4rpr r&rrrsrts@r rrs9& BHH$  r rcHa]tRtRtoRt^t]R4t]R4tRt Vt R#)QuadgramAssocMeasuresi9a  A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iiii counts ``(w1, w2, w3, w4)``, i.e. the quadgram being scored - n_ixxi counts ``(w1, *, *, w4)`` - n_xxxx counts ``(*, *, *, *)``, i.e. any quadgram c VwrVrxVwrrrVwppppW, pWp, pW`, pW, V, V, pW, V, V, pW, V, V, pVV, V, V, V, V, V, V, pWP, pW, V, V, pW, V, V, pVV, V, V, V, V, V, V, pW, V, V, pVV, V, V, V, V, V, V, pW, V, V, V, V, V, V, p VV, V, V, V, V, V, V, V, V, V, V, V, V, V, V , p!VVVVVVVVVVVVVVV V!3#)zHCalculates values of a quadgram contingency table from marginal values. r)"n_iiii n_iiix_tuple n_iixx_tuple n_ixxx_tuplen_xxxxn_iiixn_iixin_ixiin_xiiin_iixxn_ixixn_ixxin_xixin_xxiin_xiixn_ixxxn_xixxn_xxixn_xxxin_oiiin_ioiin_iioin_ooiin_oioin_iooin_oooin_iiion_oiion_ioion_ooion_iioon_oioon_iooon_oooos"&&&&& r r "QuadgramAssocMeasures._contingencyPs ,8(;G8+7(6)F26)F26)F2&6)F2V;fDvMPVV6)F26)F2&6)F2V;fDvMPVV6)F2&6)F2V;fDvMPVV6)F2V;fDvMPVV                      (                !  r c @Vwppppppppp p p p p pppW,pW,pW,pW,pW,V ,V ,pW,V ,V ,pW,V,V,pW,V,V,pW,V,V,pW,V ,V ,pW,V,V ,V,V ,V ,V,pW,V,V ,V,V ,V ,V,pW,V,V ,V,V ,V ,V ,pW,V,V,V,V,V,V,p\V4pVVVVV3VVVVVV3VVVV3V3#)aCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) r) r%rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr9s * r r& QuadgramAssocMeasures._marginalss. #                6)F26)F26)F26)F26)F26)F26)F2V;fDvMPVV6)F2V;fDvMPVV6)F2V;fDvMPVV6)F2V;fDvMPVVK   VVV , VVVVV < VVV ,    r rNrrts@r rr9s8( B9 9 v1 1 r rc:a]tRtRtoRtRt]R4tRtVt R#)ContingencyMeasuresizWraps NgramAssocMeasures classes such that the arguments of association measures are contingency table values rather than marginals. c 2RVPP,VPn\V4F[pVPR4'dK\ W4pVPR4'gVP W4p\ WV4K] R#)zAConstructs a ContingencyMeasures given a NgramAssocMeasures class Contingency___N) __class__rkdir startswithgetattr_make_contingency_fnsetattr)selfmeasureskvs&& r __init__ContingencyMeasures.__init__sr"/(2D2D2M2M"MXA||D!!$A<<$$--h: DQ  r c \aaVV3RlpSPVnSPVnV#)zwFrom an association measure function, produces a new function which accepts contingency table values as its arguments. c,<S!SP!V!!#r)r&)r%rold_fns*r res5ContingencyMeasures._make_contingency_fn..ress8.. <= =r )rork)rrrsff r r(ContingencyMeasures._make_contingency_fns%  >nn   r rN) rkrlrmrnrorrprrrrsrts@r rrs#    r rrh)romathrabcrr functoolsrrZlogr_r8rD scipy.statsr ImportErrorr>rCr?rrxrrrrr r rs ' ii 2 "( ) 7 9t(7t(nV(,V(r9 -9 xE .E PM """sA22 A?>A?