U bG@s<ddlmZddlmZmZddlmZmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZGdddZGdd d eZGd d d eZGd d d eZGdddeZGdddeZGdddeZGdddeZGdddeZ ee!ee!e"dddZ#eddd#e!e$e"e$dd d!Z%d"S)$) lru_cache)ListOptional)COMMON_SAFE_ASCII_CHARACTERSUNICODE_SECONDARY_RANGE_KEYWORD)is_accentuatedis_asciiis_case_variableis_cjk is_emoticon is_hangul is_hiragana is_katakanais_latinis_punctuation is_separator is_symbolis_thai remove_accent unicode_rangec@sPeZdZdZeedddZeddddZddd d Ze e dd d Z dS) MessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods.  characterreturncCstdS)z@ Determine if given character should be fed in. NNotImplementedErrorselfrr9/tmp/pip-unpacked-wheel-2ta4nrol/charset_normalizer/md.pyeligibleszMessDetectorPlugin.eligibleNcCstdS)z The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. Nrrrrr feed$szMessDetectorPlugin.feedrcCstdS)zB Permit to reset the plugin to the initial state. Nrrrrr reset+szMessDetectorPlugin.resetcCstdS)z Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. Nrr$rrr ratio1szMessDetectorPlugin.ratio) __name__ __module__ __qualname____doc__strboolr!r"r%propertyfloatr&rrrr rs rc@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS) TooManySymbolOrPunctuationPluginNr#cCs"d|_d|_d|_d|_d|_dS)NrF)_punctuation_count _symbol_count_character_count_last_printable_charZ_frenzy_symbol_in_wordr$rrr __init__;s z)TooManySymbolOrPunctuationPlugin.__init__rcCs|SN isprintablerrrr r!Csz)TooManySymbolOrPunctuationPlugin.eligiblecCsp|jd7_||jkrf|tkrft|r8|jd7_n.|dkrft|rft|dkrf|jd7_||_dS)NrF) r2r3rrr0isdigitrr r1rrrr r"Fs  z%TooManySymbolOrPunctuationPlugin.feedcCsd|_d|_d|_dSNr)r0r2r1r$rrr r%Xsz&TooManySymbolOrPunctuationPlugin.resetcCs0|jdkrdS|j|j|j}|dkr,|SdS)Nrg333333?)r2r0r1)rZratio_of_punctuationrrr r&]s   z&TooManySymbolOrPunctuationPlugin.ratio r'r(r)r4r+r,r!r"r%r-r.r&rrrr r/:s r/c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)TooManyAccentuatedPluginNr#cCsd|_d|_dSr:r2_accentuated_countr$rrr r4jsz!TooManyAccentuatedPlugin.__init__rcCs|Sr5)isalpharrrr r!nsz!TooManyAccentuatedPlugin.eligiblecCs(|jd7_t|r$|jd7_dSNr)r2rr?rrrr r"qszTooManyAccentuatedPlugin.feedcCsd|_d|_dSr:r>r$rrr r%wszTooManyAccentuatedPlugin.resetcCs*|jdkrdS|j|j}|dkr&|SdS)Nrr;gffffff?r>)rZratio_of_accentuationrrr r&{s   zTooManyAccentuatedPlugin.ratior<rrrr r=is r=c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)UnprintablePluginNr#cCsd|_d|_dSr:)_unprintable_countr2r$rrr r4szUnprintablePlugin.__init__rcCsdSNTrrrrr r!szUnprintablePlugin.eligiblecCs@|dkr.|dkr.|dkr.|jd7_|jd7_dS)NFr)isspacer7rCr2rrrr r"s  zUnprintablePlugin.feedcCs d|_dSr:)rCr$rrr r%szUnprintablePlugin.resetcCs|jdkrdS|jd|jS)Nrr;)r2rCr$rrr r&s zUnprintablePlugin.ratior<rrrr rBs  rBc@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)SuspiciousDuplicateAccentPluginNr#cCsd|_d|_d|_dSr:_successive_countr2_last_latin_characterr$rrr r4sz(SuspiciousDuplicateAccentPlugin.__init__rcCs|ot|Sr5)r@rrrrr r!sz(SuspiciousDuplicateAccentPlugin.eligiblecCst|jd7_|jdk rjt|rjt|jrj|rJ|jrJ|jd7_t|t|jkrj|jd7_||_dSrA)r2rKrisupperrJrrrrr r"sz$SuspiciousDuplicateAccentPlugin.feedcCsd|_d|_d|_dSr:rIr$rrr r%sz%SuspiciousDuplicateAccentPlugin.resetcCs|jdkrdS|jd|jS)Nrr;r8)r2rJr$rrr r&s z%SuspiciousDuplicateAccentPlugin.ratior<rrrr rHs rHc@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)SuspiciousRangeNr#cCsd|_d|_d|_dSr:)"_suspicious_successive_range_countr2_last_printable_seenr$rrr r4szSuspiciousRange.__init__rcCs|Sr5r6rrrr r!szSuspiciousRange.eligiblecCsx|jd7_|s&t|s&|tkr0d|_dS|jdkrD||_dSt|j}t|}t||rn|jd7_||_dSrA)r2rFrrrOr is_suspiciously_successive_rangerN)rrunicode_range_aunicode_range_brrr r"s&  zSuspiciousRange.feedcCsd|_d|_d|_dSr:)r2rNrOr$rrr r%szSuspiciousRange.resetcCs.|jdkrdS|jd|j}|dkr*dS|S)Nrr;r8g?)r2rN)rZratio_of_suspicious_range_usagerrr r&s zSuspiciousRange.ratior<rrrr rMs rMc@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)SuperWeirdWordPluginNr#cCs:d|_d|_d|_d|_d|_d|_d|_d|_d|_dS)NrF) _word_count_bad_word_count_foreign_long_count_is_current_word_bad_foreign_long_watchr2_bad_character_count_buffer_buffer_accent_countr$rrr r4szSuperWeirdWordPlugin.__init__rcCsdSrDrrrrr r! szSuperWeirdWordPlugin.eligiblecCs|rd|j|g|_t|r0|jd7_|jdkrt|dksNt|rt|dkrt|dkrt |dkrt |dkrt |dkrd|_dS|jsdS| st |st|r|jr|jd7_t|j}|j|7_|dkr:|j|dkrd|_t|jdr:|jdr:|jd7_d|_|dkr`|jr`|jd7_d|_|jr|jd7_|jt|j7_d|_d|_d|_d |_n6|d kr|dkrt|rd|_|j|7_dS) NrTrFTg(\?r>_=><|~-)r@joinr[rr\rYrr r rrrrFrrrUlenr2rXrLrWrVrZr9r)rrZ buffer_lengthrrr r"sv           zSuperWeirdWordPlugin.feedcCs4d|_d|_d|_d|_d|_d|_d|_d|_dS)NrTFr)r[rXrYrVrUr2rZrWr$rrr r%DszSuperWeirdWordPlugin.resetcCs$|jdkr|jdkrdS|j|jS)N rr;)rUrWrZr2r$rrr r&NszSuperWeirdWordPlugin.ratior<rrrr rSs 6 rSc@s^eZdZdZddddZeedddZeddd d Zddd d Z e e dd dZ dS)CjkInvalidStopPluginu GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and can be easily detected. Searching for the overuse of '丅' and '丄'. Nr#cCsd|_d|_dSr:_wrong_stop_count_cjk_character_countr$rrr r4\szCjkInvalidStopPlugin.__init__rcCsdSrDrrrrr r!`szCjkInvalidStopPlugin.eligiblecCs4|dkr|jd7_dSt|r0|jd7_dS)N>丄丅r)rlr rmrrrr r"cs zCjkInvalidStopPlugin.feedcCsd|_d|_dSr:rkr$rrr r%jszCjkInvalidStopPlugin.resetcCs|jdkrdS|j|jS)Nr;)rmrlr$rrr r&ns zCjkInvalidStopPlugin.ratio) r'r(r)r*r4r+r,r!r"r%r-r.r&rrrr rjVsrjc@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)ArchaicUpperLowerPluginNr#cCs.d|_d|_d|_d|_d|_d|_d|_dS)NFrT)_buf_character_count_since_last_sep_successive_upper_lower_count#_successive_upper_lower_count_finalr2_last_alpha_seen_current_ascii_onlyr$rrr r4vsz ArchaicUpperLowerPlugin.__init__rcCsdSrDrrrrr r!sz ArchaicUpperLowerPlugin.eligiblecCs$|ot|}|dk}|r|jdkr|jdkrV|dkrV|jdkrV|j|j7_d|_d|_d|_d|_|j d7_ d|_dS|jdkrt |dkrd|_|jdk r| r|j s| r|j r|jdkr|jd7_d|_qd|_nd|_|j d7_ |jd7_||_dS)NFr@rTr8) r@r rsr9rwrurtrvrrr2r rLislower)rrZ is_concernedZ chunk_seprrr r"sF   zArchaicUpperLowerPlugin.feedcCs.d|_d|_d|_d|_d|_d|_d|_dS)NrFT)r2rsrtrurvrrrwr$rrr r%szArchaicUpperLowerPlugin.resetcCs|jdkrdS|j|jS)Nrr;)r2rur$rrr r&s zArchaicUpperLowerPlugin.ratior<rrrr rqus  * rq)rQrRrcCs||dks|dkrdS||kr dSd|kr4d|kr4dSd|ksDd|krHdSd|ksXd|krld|kshd|krldS|d|d}}|D]}|tkrq||krdSq|dk|dk}}|s|rd |ksd |krdS|r|rdSd |ksd |kr d |ksd |krdS|d ks|d kr dSd |ksHd |ksH|d krx|d krxd |ks\d |kr`dSd|kstd|krxdSdS)za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFZLatinZ EmoticonsZ Combining )HiraganaKatakanaCJKZHangulz Basic Latin)r|r{Z PunctuationZForms)splitr)rQrRZkeywords_range_aZkeywords_range_belZrange_a_jp_charsZrange_b_jp_charsrrr rPshrPi)maxsize皙?F)decoded_sequencemaximum_thresholddebugrc CsddtD}t|d}d}|dkr0d}n|dkr>d}nd }t|d t|D]d\}}|D]} | |r`| |q`|d kr||d ks||dkrTtd d |D}||krTqqT|r|D]} t| j | j qt |dS)zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. cSsg|] }|qSrr).0Zmd_classrrr szmess_ratio..rr;i irx rcss|] }|jVqdSr5)r&)rdtrrr &szmess_ratio..) r__subclasses__rhzipranger!r"sumprint __class__r&round) rrrZ detectorslengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcrindexdetectorrrrr mess_ratios6     rN)rF)& functoolsrtypingrrZconstantrrutilsrr r r r r rrrrrrrrrrr/r=rBrHrMrSrjrqr+r,rPr.rrrrr s2 D"/%6ZM F