is UdZddlmZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZddlmZmZmZddlmZddlmZd Zej0Gd d e ZGd d e Zej6ej8ej:ej<ej>ej@ejBejDejFejHejJejLd Z'de(d<Gdde Z) d dZ*dZ+ d!dZ,d"d#dZ- d" d$dZ. d" d$dZ/ d%dZ0 d" d#dZ1e-Z d" d#dZ2 d& d'dZ3d(dZ4d)dZ5d*dZ6y)+z{ ftfy: fixes text for you This is a module for making text less broken. See the `fix_text` function for more information. ) annotationsN)Iterator)AnyBinaryIOCallableLiteral NamedTupleTextIOcast) bad_codecschardatafixes)is_bad) display_ljustz6.3.1c.eZdZUdZded<ded<ddZy)ExplanationStepa A step in an ExplainedText, explaining how to decode text. The possible actions are: - "encode": take in a string and encode it as bytes, with the given encoding - "decode": take in bytes and decode them as a string, with the given encoding - "transcode": convert bytes to bytes with a particular named function - "apply": convert str to str with a particular named function The `parameter` is the name of the encoding or function to use. If it's a function, it must appear in the FIXERS dictionary. straction parameterc*tt|S)z Get the string representation of an ExplanationStep. We output the representation of the equivalent tuple, for simplicity. )reprtuple)selfs i/home/developers/rajanand/mypropertyqr-fmb-refixing-v2/venv/lib/python3.12/site-packages/ftfy/__init__.py__repr__zExplanationStep.__repr__5s E$K  N)returnr)__name__ __module__ __qualname____doc____annotations__rrrrr#s  KN!rrc&eZdZUdZded<ded<y) ExplainedTexta The return type from ftfy's functions that provide an "explanation" of which steps it applied to fix the text, such as :func:`fix_and_explain()`. When the 'explain' option is disabled, these functions return the same type, but the `explanation` will be None. rtextlist[ExplanationStep] | None explanationN)rrr r!r"r#rrr%r%=s I--rr%) unescape_htmlremove_terminal_escapesrestore_byte_a0replace_lossy_sequencesdecode_inconsistent_utf8fix_c1_controlsfix_latin_ligaturesfix_character_width uncurl_quotesfix_line_breaksfix_surrogatesremove_control_charszdict[str, Callable]FIXERSceZdZUdZdZded<dZded<dZded<dZded <dZ ded <dZ ded <dZ ded <dZ ded <dZ ded<dZded<dZded<dZded<dZded<dZded<dZded<dZded<y)TextFixerConfigu[ A TextFixerConfig object stores configuration options for ftfy. It's implemented as a namedtuple with defaults, so you can instantiate it by providing the values to change from their defaults as keyword arguments. For example, to disable 'unescape_html' and keep the rest of the defaults:: TextFixerConfig(unescape_html=False) Here are the options and their default values: - `unescape_html`: "auto" Configures whether to replace HTML entities such as & with the character they represent. "auto" says to do this by default, but disable it when a literal < character appears, indicating that the input is actual HTML and entities should be preserved. The value can be True, to always enable this fixer, or False, to always disable it. - `remove_terminal_escapes`: True Removes "ANSI" terminal escapes, such as for changing the color of text in a terminal window. - `fix_encoding`: True Detect mojibake and attempt to fix it by decoding the text in a different encoding standard. The following four options affect `fix_encoding` works, and do nothing if `fix_encoding` is False: - `restore_byte_a0`: True Allow a literal space (U+20) to be interpreted as a non-breaking space (U+A0) when that would make it part of a fixable mojibake string. Because spaces are very common characters, this could lead to false positives, but we try to apply it only when there's strong evidence for mojibake. Disabling `restore_byte_a0` is safer from false positives, but creates false negatives. - `replace_lossy_sequences`: True Detect mojibake that has been partially replaced by the characters '�' or '?'. If the mojibake could be decoded otherwise, replace the detected sequence with '�'. - `decode_inconsistent_utf8`: True When we see sequences that distinctly look like UTF-8 mojibake, but there's no consistent way to reinterpret the string in a new encoding, replace the mojibake with the appropriate UTF-8 characters anyway. This helps to decode strings that are concatenated from different encodings. - `fix_c1_controls`: True Replace C1 control characters (the useless characters U+80 - U+9B that come from Latin-1) with their Windows-1252 equivalents, like HTML5 does, even if the whole string doesn't decode as Latin-1. - `fix_latin_ligatures`: True Replace common Latin-alphabet ligatures, such as ``fi``, with the letters they're made of. - `fix_character_width`: True Replace fullwidth Latin characters and halfwidth Katakana with their more standard widths. - `uncurl_quotes`: True Replace curly quotes with straight quotes. - `fix_line_breaks`: True Replace various forms of line breaks with the standard Unix line break, ``\n``. - `fix_surrogates`: True Replace sequences of UTF-16 surrogate codepoints with the character they were meant to encode. This fixes text that was decoded with the obsolete UCS-2 standard, and allows it to support high-numbered codepoints such as emoji. - `remove_control_chars`: True Remove certain control characters that have no displayed effect on text. - `normalization`: "NFC" Choose what kind of Unicode normalization is applied. Usually, we apply NFC normalization, so that letters followed by combining characters become single combined characters. Changing this to "NFKC" applies more compatibility conversions, such as replacing the 'micro sign' with a standard Greek lowercase mu, which looks identical. However, some NFKC normalizations change the meaning of text, such as converting "10³" to "103". `normalization` can be None, to apply no normalization. - `max_decode_length`: 1_000_000 The maximum size of "segment" that ftfy will try to fix all at once. - `explain`: True Whether to compute 'explanations', lists describing what ftfy changed. When this is False, the explanation will be None, and the code that builds the explanation will be skipped, possibly saving time. Functions that accept TextFixerConfig and don't return an explanation will automatically set `explain` to False. autoz str | boolr)Tboolr* fix_encodingr+r,r-r.r/r0r1r2r3r4NFCz,Literal['NFC', 'NFD', 'NFKC', 'NFKD'] | None normalizationi@Bintmax_decode_lengthexplainN)rrr r!r)r"r*r:r+r,r-r.r/r0r1r2r3r4r<r>r?r#rrr7r7[svp!'M:&$(T(L$ OT $(T(%)d) OT $$ $$M4 OT ND!%$%BGM?G$s$GTrr7cd|vr7tjdtd|j}|d|d<|d=|jdi|}|S)z Handle parameters provided as keyword arguments to ftfy's top-level functions, converting them into a TextFixerConfig. fix_entitiesz2`fix_entities` has been renamed to `unescape_html`) stacklevelr)r#)warningswarnDeprecationWarningcopy_replace)configkwargss r_config_from_kwargsrKs] @  "("8 > " V__ &v &F MraHey wait, this isn't Unicode. ftfy is designed to fix problems with text. Treating bytes like they're interchangeable with Unicode text is usually something that introduces problems with text. You should first decode these bytes from the encoding you think they're in. If you're not sure what encoding they're in: - First, try to find out. 'utf-8' is a good assumption. - If the encoding is simply unknowable, try running your bytes through ftfy.guess_bytes. As the name implies, this may not always be accurate. For more information on the distinction between bytes and text, read the Python Unicode HOWTO: http://docs.python.org/3/howto/unicode.html ct||rCt|}||}| ||k7r|jtd|t t |S|S)z A helper function used across several 'fixer' steps, deciding whether to apply the fix and whether to record the fix in `steps`. apply)getattrr5appendrr r) fixer_namer&rIstepsfixerfixeds r_try_fixrTsRvz"z"d   $ LL*= >C Krc | td}t||}t|trt t g}d}|t |kr|jd|dz}|dk(r t |}||z |jkDr||jz}|||}|jdk(rd|vr|jd}t||\}}|j||}|t |krd j|S) u Given Unicode text as input, fix inconsistencies and glitches in it, such as mojibake (text that was decoded in the wrong encoding). Let's start with some examples: >>> fix_text('✔ No problems') '✔ No problems' >>> print(fix_text("¯\\_(ã\x83\x84)_/¯")) ¯\_(ツ)_/¯ >>> fix_text('Broken text… it’s flubberific!') "Broken text... it's flubberific!" >>> fix_text('ＬＯＵＤ　ＮＯＩＳＥＳ') 'LOUD NOISES' ftfy applies a number of different fixes to the text, and can accept configuration to select which fixes to apply. The configuration takes the form of a :class:`TextFixerConfig` object, and you can see a description of the options in that class's docstring or in the full documentation at ftfy.readthedocs.org. For convenience and backward compatibility, the configuration can also take the form of keyword arguments, which will set the equivalently-named fields of the TextFixerConfig object. For example, here are two ways to fix text but skip the "uncurl_quotes" step:: fix_text(text, TextFixerConfig(uncurl_quotes=False)) fix_text(text, uncurl_quotes=False) This function fixes text in independent segments, which are usually lines of text, or arbitrarily broken up every 1 million codepoints (configurable with `config.max_decode_length`) if there aren't enough line breaks. The bound on segment lengths helps to avoid unbounded slowdowns. ftfy can also provide an 'explanation', a list of transformations it applied to the text that would fix more text like it. This function doesn't provide explanations (because there may be different fixes for different segments of text). To get an explanation, use the :func:`fix_and_explain()` function, which fixes the string in one segment and explains what it fixed. Fr?r r8<r))r7rK isinstancebytes UnicodeErrorBYTES_ERROR_TEXTlenfindr>r)rHfix_and_explainrOjoin) r&rIrJoutpos textbreaksegment fixed_segment_s rfix_textrj"sd~ /  0F$+,, C C D /IIdC(1, >D I Ov77 7f666Is9%   6 )cWn__5_9F*7F; q =! D / 773<rc x| t}t|trttt ||}|j dk(rd|vr|jd}|jrg}nd} |}td|||}|jr0| t|}n"t||\}}||j|dD]}t||||}|jNtj|j|}|*||k7r%|j!t#d|j|}||k(r t%||S) z Fix text as a single segment, returning the fixed text and an explanation of what was fixed. The explanation is a list of steps that can be applied with :func:`apply_plan`, or if config.explain is False, it will be None. Nr8rYFrZr))r.r/r0r1r2r3r*r4 normalize)r7r\r]r^r_rKr)rHr?rTr:fix_encoding_and_explainextendr< unicodedatarlrOrr%)r&rIrJrQorigtextencoding_stepsrRrSs rrbrblsK~ "$+,,  0F v%#+u5 ~~.0 vu=   }#D)'?f'M$n!-LL0  8EE47D 8    +))&*>*>EE Ud] _[&:N:NOPD 8  u- -A rc | t}t|trttt ||}|j s t|gSg} |}t||\}}||j|||k(r t||S6)u Apply the steps of ftfy that detect mojibake and fix it. Returns the fixed text and a list explaining what was fixed. This includes fixing text by encoding and decoding it in different encodings, as well as the subordinate fixes `restore_byte_a0`, `replace_lossy_sequences`, `decode_inconsistent_utf8`, and `fix_c1_controls`. Examples:: >>> fix_encoding_and_explain("só") ExplainedText(text='só', explanation=[('encode', 'latin-1'), ('decode', 'utf-8')]) >>> result = fix_encoding_and_explain("voilà le travail") >>> result.text 'voilà le travail' >>> result.explanation [('encode', 'latin-1'), ('transcode', 'restore_byte_a0'), ('decode', 'utf-8')] ) r7r\r]r^r_rKr:r%"_fix_encoding_one_step_and_explainrn)r&rIrJ plan_so_farprevtextplans rrmrms.~ "$+,,  0F   T2&&)+K 7fE d     t $ 8  {3 3 rc| t}t|dk(r t|gStj|ds t |s t|gSg}tj D]E}tj||s|j||j|}td|}g} d}|jr[|dk7rVtjj|r7tj|}||k7r|jtdd|}|jrH|jdr7tj|}||k7r|jtdd |}d |vsd |vrd }td |} |g|z| gz} |j!|} t| | cS|j$rRtj&j|r3tddg} tj$|} | |k7r t| | Sd|vr[d|vr t|gS |jdj!d} | |k7r$tddtd dg} t| | S |j(rMtj*j|r.tddg} tj(|} t| | St|gS#t"$rYwwxYw#t"$rYwxYw)z: Perform one step of fixing the encoding of text. rasciiencodeutf-8macroman transcoder+sloppyr,utf-8-variantsdecoderMr-zlatin-1z windows-1252r.)r7r`r%r possible_encodingrCHARMAP_ENCODINGSrOryrr+ALTERED_UTF8_REsearchrr, startswithrUnicodeDecodeErrorr-UTF8_DETECTOR_REr. C1_CONTROL_RE) r&rIpossible_1byte_encodingsencoding encoded_bytes encode_steptranscode_stepsdecodingreplaced_bytes decode_steprQrSs rrsrss ~ " 4yA~T2&&!!$0t T2&& " ...  % %dH 5 $ + +H 5 KK1M)(H=K O% "** J. 0077 F%*%:%:=%IN%6'..+K9JK)7 11h6I6I(6S%*%B%B=%QN%6'..+K9RS)7 =(DM,A/H-hA $ 7;-G%,,X6$UE22W.b&&8+D+D+K+KD+Q *DEF..t4 D= . . ,, 5 5!r* *  I.55nED=')<'.AE)66 !("8"8"?"?"E .?@A%%d+UE** r ""[&  <&  s&2C9K"AK! KK! K-,K-c X| td}t||}t||\}}|S)u Apply just the encoding-fixing steps of ftfy to this text. Returns the fixed text, discarding the explanation. >>> fix_encoding("ó") 'ó' >>> fix_encoding("&ATILDE;&SUP3;") '&ATILDE;&SUP3;' FrV)r7rKrmr&rIrJrS_explans rr:r:Js4~ /  0F-dF;NE7 Lrc X| td}t||}t||\}}|S)z Fix text as a single segment, with a consistent sequence of steps that are applied to fix the text. Discard the explanation. FrV)r7rKrbrs rfix_text_segmentras4~ /  0F$T62NE7 Lrc+K| t}t||}|D]l}t|tr"|t |\}}n|j |}|j dk(rd|vr|jd}t||\}}|nyw)a Fix text that is found in a file. If the file is being read as Unicode text, use that. If it's being read as bytes, then we hope an encoding was supplied. If not, unfortunately, we have to guess what encoding it is. We'll try a few common encodings, but we make no promises. See the `guess_bytes` function for how this is done. The output is a stream of fixed lines of text. Nr8rYFrZ) r7rKr\r] guess_bytesrr)rHrb) input_filerrIrJline fixed_liners rfix_fileros ~ "  0F  dE "!,T!2h{{8,   6 )cTk__5_9F-dF; G sB B ct|tr td|jds|jdr|j ddfSt |} d|vsd|vr|j ddfS|j ddfS#t $rYnwxYwd |vrd |vr|j d d fS|j d d fS) a? NOTE: Using `guess_bytes` is not the recommended way of using ftfy. ftfy is not designed to be an encoding detector. In the unfortunate situation that you have some bytes in an unknown encoding, ftfy can guess a reasonable strategy for decoding them, by trying a few common encodings that can be distinguished from each other. Unlike the rest of ftfy, this may not be accurate, and it may *create* Unicode problems instead of solving them! The encodings we try here are: - UTF-16 with a byte order mark, because a UTF-16 byte order mark looks like nothing else - UTF-8, because it's the global standard, which has been used by a majority of the Web since 2008 - "utf-8-variants", or buggy implementations of UTF-8 - MacRoman, because Microsoft Office thinks it's still a thing, and it can be distinguished by its line breaks. (If there are no line breaks in the string, though, you're out of luck.) - "sloppy-windows-1252", the Latin-1-like encoding that is the most common single-byte encoding. z^This string was already decoded as Unicode. You should pass bytes to guess_bytes, not Unicode.sszutf-16r~rrrz r{zsloppy-windows-1252)r\rr^rrsetr)bstringbytesets rrrs2'3 1  +&'*<*<[*I~~h'11'lG 7?dgo,>>"235EE E>>'*G3 3     w4w.~~j):55 >>/ 02G GGsB 8B BBc|}|D]i\}}|dk(r|j|}|dk(r|j|}4|dvr%|tvrt||}Otd|td||S)u( Apply a plan for fixing the encoding of text. The plan is a list of tuples of the form (operation, arg). `operation` is one of: - `'encode'`: convert a string to bytes, using `arg` as the encoding - `'decode'`: convert bytes to a string, using `arg` as the encoding - `'transcode'`: convert bytes to bytes, using the function named `arg` - `'apply'`: convert a string to a string, using the function named `arg` The functions that can be applied by 'transcode' and 'apply' are specifically those that appear in the dictionary named `FIXERS`. They can also can be imported from the `ftfy.fixes` module. Example:: >>> mojibake = "schön" >>> text, plan = fix_and_explain(mojibake) >>> apply_plan(mojibake, plan) 'schön' ryr)r|rMzUnknown function to apply: zUnknown plan step: )ryrr5 ValueError)r&rvobj operationrs r apply_planrs0 C# @ 8  **X&C ( "**X&C 0 06!X&s+ #>xj!IJJ29+>? ? @ Jrc (|D]}|jr|}n |jdjd}tdj t |dt |tj|tj|dy)u A utility method that's useful for debugging mysterious Unicode. It breaks down a string, showing you for each codepoint its number in hexadecimal, its glyph, its category in the Unicode standard, and its name in the Unicode standard. >>> explain_unicode('(╯°□°)╯︵ ┻━┻') U+0028 ( [Ps] LEFT PARENTHESIS U+256F ╯ [So] BOX DRAWINGS LIGHT ARC UP AND LEFT U+00B0 ° [So] DEGREE SIGN U+25A1 □ [So] WHITE SQUARE U+00B0 ° [So] DEGREE SIGN U+0029 ) [Pe] RIGHT PARENTHESIS U+256F ╯ [So] BOX DRAWINGS LIGHT ARC UP AND LEFT U+FE35 ︵ [Ps] PRESENTATION FORM FOR VERTICAL LEFT PARENTHESIS U+0020 [Zs] SPACE U+253B ┻ [So] BOX DRAWINGS HEAVY UP AND HORIZONTAL U+2501 ━ [So] BOX DRAWINGS HEAVY HORIZONTAL U+253B ┻ [So] BOX DRAWINGS HEAVY UP AND HORIZONTAL zunicode-escaperxz+U+{code:04X} {display} [{category}] {name}z )displaycodecategorynameN) isprintableryrprintformatrordrorr)r&charrs rexplain_unicoders,      Gkk"23::7CG 9 @ @%gq1Y$--d3 %%dK8 A   r)rIr7rJzdict[str, Any]rr7) rPrr&rrIr7rQr'rr)N)r&rrITextFixerConfig | NonerJrrr)r&rrIrrJrrr%)r&rrIr7rr%)NN) rzTextIO | BinaryIOrz str | NonerIrrJrrz Iterator[str])rr]rztuple[str, str])r&rrvzlist[tuple[str, str]]rr)r&rrNone)7r! __future__rrorDcollections.abcrtypingrrrrr r r ftfyr r r ftfy.badnessrftfy.formattingr __version__okrr%r)r*r+r,r-r.r/r0r1r2r3r4r5r"r7rKr_rTrjrbrmrsr:rrrrrr#rrrsl#$-,)  !j!4 .J .(($<<,,$<< % > >,, 44 44((,,**!66   HjHV %3((   (   (GV159. 9.-9.@C9.9.z15)4 )4-)4@C)4)4Xs# s#&s#s#n15 -@C(15   - @C    %)! #   BDHN&R" r