i9GdZddlZddlZddlZddlZddlmZddlmZddlZddl m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZdedefd Zdedefd Zded eeeefdefd Zd eedefdZdedefdZej>dZ dedefdZ!dedefdZ"dedefdZ#dedefdZ$dedefdZ%ej>dZ&ej>dZ'd eedefdZ(dedefdZ)dedefdZ*dedefdZ+ej>dejXejZzZ.dedefdZ/ej>dZ0de1de1fd Z2de1de1fd!Z3dedefd"Z4d eedefd#Z5dedefd$Z6y)%a The `ftfy.fixes` module contains the individual fixes that :func:`ftfy.fix_text` can perform, and provides the functions that are named in "explanations" such as the output of :func:`ftfy.fix_and_explain`. Two of these functions are particularly useful on their own, as more robust versions of functions in the Python standard library: - :func:`ftfy.fixes.decode_escapes` - :func:`ftfy.fixes.unescape_html` N)Match)Any)is_bad) ALTERED_UTF8_RE C1_CONTROL_RE CONTROL_CHARSDOUBLE_QUOTE_RE HTML_ENTITIESHTML_ENTITY_RE LIGATURES LOSSY_UTF8_RESINGLE_QUOTE_REUTF8_DETECTOR_RE WIDTH_MAPtextreturncdtjdtdtj|S)z? Deprecated copy of `ftfy.fix_encoding_and_explain()`. zB`fix_encoding_and_explain()` has moved to the main module of ftfy. stacklevel)warningswarnDeprecationWarningftfyfix_encoding_and_explainrs f/home/developers/rajanand/mypropertyqr-fmb-refixing-v2/venv/lib/python3.12/site-packages/ftfy/fixes.pyrr%s, MML  ( ( ..cdtjdtdtj|S)z3 Deprecated copy of `ftfy.fix_encoding()`. z6`fix_encoding()` has moved to the main module of ftfy.rr)rrrr fix_encodingrs rr r 1s, MM@   T ""rplancftjdtdtj||S)z1 Deprecated copy of `ftfy.apply_plan()`. z4`apply_plan()` has moved to the main module of ftfy.rr)rrrr apply_plan)rr!s rr#r#=s, MM> ??4 &&rmatchc|jd}|tvr t|S|jdrtj|}d|vr|S|S|S)z\ Replace one matched HTML entity with the character it represents, if possible. rz&#;)groupr startswithhtmlunescape)r$r unescapeds r_unescape_fixupr,IsX ;;q>D }T""  t,  ) K  rc6tjt|S)uQ Decode HTML entities and character references, including some nonstandard ones written in all-caps. Python has a built-in called `html.unescape` that can decode HTML escapes, including a bunch of messy edge cases such as decoding escapes without semicolons such as "&". If you know you've got HTML-escaped text, applying `html.unescape` is the right way to convert it to plain text. But in ambiguous situations, that would create false positives. For example, the informally written text "this¬ that" should not automatically be decoded as "this¬ that". In this function, we decode the escape sequences that appear in the `html.entities.html5` dictionary, as long as they are the unambiguous ones that end in semicolons. We also decode all-caps versions of Latin letters and common symbols. If a database contains the name 'P&EACUTE;REZ', we can read that and intuit that it was supposed to say 'PÉREZ'. This is limited to a smaller set of entities, because there are many instances where entity names are case-sensitive in complicated ways. >>> unescape_html('<tag>') '' >>> unescape_html('𝒥ohn ℋancock') '𝒥ohn ℋancock' >>> unescape_html('✓') '✓' >>> unescape_html('Pérez') 'Pérez' >>> unescape_html('P&EACUTE;REZ') 'PÉREZ' >>> unescape_html('BUNDESSTRA&SZLIG;E') 'BUNDESSTRASSE' >>> unescape_html('ñ Ñ &NTILDE; &nTILDE;') 'ñ Ñ Ñ &nTILDE;' )r subr,rs r unescape_htmlr/^sZ   ot 44rz\[((?:\d|;)*)([a-zA-Z])c.tjd|S)a Strip out "ANSI" terminal escape sequences, such as those that produce colored text on Unix. >>> print(remove_terminal_escapes( ... "\033[36;44mI'm blue, da ba dee da ba doo...\033[0m" ... )) I'm blue, da ba dee da ba doo... )ANSI_REr.rs rremove_terminal_escapesr3s ;;r4  rcVtjdtjd|S)z Replace curly quotation marks with straight equivalents. >>> print(uncurl_quotes('\u201chere\u2019s a test\u201d')) "here's a test" '")rr.r rs r uncurl_quotesr7s$   sO$7$7T$B CCrc,|jtS)uh Replace single-character ligatures of Latin letters, such as 'fi', with the characters that they contain, as in 'fi'. Latin ligatures are usually not intended in text strings (though they're lovely in *rendered* text). If you have such a ligature in your string, it is probably a result of a copy-and-paste glitch. We leave ligatures in other scripts alone to be safe. They may be intended, and removing them may lose information. If you want to take apart nearly all ligatures, use NFKC normalization. >>> print(fix_latin_ligatures("fluffiest")) fluffiest ) translater rs rfix_latin_ligaturesr:s >>) $$rc,|jtS)uU The ASCII characters, katakana, and Hangul characters have alternate "halfwidth" or "fullwidth" forms that help text line up in a grid. If you don't need these width properties, you probably want to replace these characters with their standard form, which is what this function does. Note that this replaces the ideographic space, U+3000, with the ASCII space, U+20. >>> print(fix_character_width("ＬＯＵＤ　ＮＯＩＳＥＳ")) LOUD NOISES >>> print(fix_character_width("Ｕﾀｰﾝ")) # this means "U-turn" Uターン )r9rrs rfix_character_widthr<s" >>) $$rc|jddjddjddjddjddS)a8 Convert all line breaks to Unix style. This will convert the following sequences into the standard \\n line break: - CRLF (\\r\\n), used on Windows and in some communication protocols - CR (\\r), once used on Mac OS Classic, and now kept alive by misguided software such as Microsoft Office for Mac - LINE SEPARATOR (\\u2028) and PARAGRAPH SEPARATOR (\\u2029), defined by Unicode and used to sow confusion and discord - NEXT LINE (\\x85), a C1 control character that is certainly not what you meant The NEXT LINE character is a bit of an odd case, because it usually won't show up if `fix_encoding` is also being run. \\x85 is very common mojibake for \\u2026, HORIZONTAL ELLIPSIS. >>> print(fix_line_breaks( ... "This string is made of two things:\u2029" ... "1. Unicode\u2028" ... "2. Spite" ... )) This string is made of two things: 1. Unicode 2. Spite For further testing and examples, let's define a function to make sure we can see the control characters in their escaped form: >>> def eprint(text): ... print(text.encode('unicode-escape').decode('ascii')) >>> eprint(fix_line_breaks("Content-type: text/plain\r\n\r\nHi.")) Content-type: text/plain\n\nHi. >>> eprint(fix_line_breaks("This is how Microsoft \r trolls Mac users")) This is how Microsoft \n trolls Mac users >>> eprint(fix_line_breaks("What is this \x85 I don't even")) What is this \n I don't even z   u
u
…)replacers rfix_line_breaksrBsHX VT" t  4 4 4 ru [-]u[-][-]c|jd}dt|ddz dzzt|ddz z}t|S)z Convert a surrogate pair to the single codepoint it represents. This implements the formula described at: http://en.wikipedia.org/wiki/Universal_Character_Set_characters#Surrogates riiii)r'ordchr)r$paircodepts rconvert_surrogate_pairrIsI ;;q>D DG v-6 6#d1g,:O PF v;rctj|r0tjt|}tjd|}|S)uf Replace 16-bit surrogate codepoints with the characters they represent (when properly paired), or with � otherwise. >>> high_surrogate = chr(0xd83d) >>> low_surrogate = chr(0xdca9) >>> print(fix_surrogates(high_surrogate + low_surrogate)) 💩 >>> print(fix_surrogates(low_surrogate + high_surrogate)) �� The above doctest had to be very carefully written, because even putting the Unicode escapes of the surrogates in the docstring was causing various tools to fail, which I think just goes to show why this fixer is necessary. �) SURROGATE_REsearchSURROGATE_PAIR_REr.rIrs rfix_surrogatesrOs<"4  $$%;TB$/ Krc,|jtS)a Remove various control characters that you probably didn't intend to be in your text. Many of these characters appear in the table of "Characters not suitable for use with markup" at http://www.unicode.org/reports/tr20/tr20-9.html. This includes: - ASCII control characters, except for the important whitespace characters (U+00 to U+08, U+0B, U+0E to U+1F, U+7F) - Deprecated Arabic control characters (U+206A to U+206F) - Interlinear annotation characters (U+FFF9 to U+FFFB) - The Object Replacement Character (U+FFFC) - The byte order mark (U+FEFF) However, these similar characters are left alone: - Control characters that produce whitespace (U+09, U+0A, U+0C, U+0D, U+2028, and U+2029) - C1 control characters (U+80 to U+9F) -- even though they are basically never used intentionally, they are important clues about what mojibake has happened - Control characters that affect glyph rendering, such as joiners and right-to-left marks (U+200C to U+200F, U+202A to U+202E) - Musical notation control characters (U+1D173 to U+1D17A) because wow if you're using those you probably have a good reason - Tag characters, because they are now used in emoji sequences such as "Flag of Wales" )r9rrs rremove_control_charsrQ)s< >>- ((rc6|jtdS)z Remove a byte-order mark that was accidentally decoded as if it were part of the text. >>> print(remove_bom(chr(0xfeff) + "Where do you want to go today?")) Where do you want to go today? i)lstriprFrs r remove_bomrTJs ;;s6{ ##ra ( \\U........ # 8-digit hex escapes | \\u.... # 4-digit hex escapes | \\x.. # 2-digit hex escapes | \\[0-7]{1,3} # Octal escapes | \\N\{[^}]+\} # Unicode characters by name | \\[\\'"abfnrtv] # Single-character escapes )c\dttdtfd}tj||S)u[ Decode backslashed escape sequences, including \\x, \\u, and \\U character references, even in the presence of other Unicode. This function has to be called specifically. It's not run automatically by ftfy, because escaped text is not necessarily a mistake, and there is no way to distinguish when it is. This is what Python's "string-escape" and "unicode-escape" codecs were meant to do, but in contrast, this actually works. It will decode the string exactly the same way that the Python interpreter decodes its string literals. >>> factoid = '\\u20a1 is the currency symbol for the colón.' >>> print(factoid[1:]) u20a1 is the currency symbol for the colón. >>> print(decode_escapes(factoid)) ₡ is the currency symbol for the colón. Even though Python itself can read string literals with a combination of escapes and literal Unicode -- you're looking at one right now -- the "unicode-escape" codec doesn't work on literal Unicode. (See http://stackoverflow.com/a/24519338/773754 for more details.) Instead, this function searches for just the parts of a string that represent escape sequences, and decodes them, leaving the rest alone. All valid escape sequences are made of ASCII characters, and this allows "unicode-escape" to work correctly. r$rcLtj|jddS)z.decode_matchs}}U[[^-=>>r)rstrESCAPE_SEQUENCE_REr.)rrZs rdecode_escapesr]cs.>?E#J?3?  ! !, 55rs (?! |quele|quela|quilo|s )bytsctjd|}dttdtfd}t j||S)a] Some mojibake has been additionally altered by a process that said "hmm, byte A0, that's basically a space!" and replaced it with an ASCII space. When the A0 is part of a sequence that we intend to decode as UTF-8, changing byte A0 to 20 would make it fail to decode. This process finds sequences that would convincingly decode as UTF-8 if byte 20 were changed to A0, and puts back the A0. For the purpose of deciding whether this is a good idea, this step gets a cost of twice the number of bytes that are changed. This is used as a step within `fix_encoding`. sà r$rcD|jdjddS)z.The function to apply when this regex matches.r )r'rArYs r replacementz$restore_byte_a0..replacements{{1~%%gw77r)A_GRAVE_WORD_REr.rbytesr)r^rcs rrestore_byte_a0rfsB   |T 2D85<8E8   {D 11rcJtjdj|S)u This function identifies sequences where information has been lost in a "sloppy" codec, indicated by byte 1A, and if they would otherwise look like a UTF-8 sequence, it replaces them with the UTF-8 sequence for U+FFFD. A further explanation: ftfy can now fix text in a few cases that it would previously fix incompletely, because of the fact that it can't successfully apply the fix to the entire string. A very common case of this is when characters have been erroneously decoded as windows-1252, but instead of the "sloppy" windows-1252 that passes through unassigned bytes, the unassigned bytes get turned into U+FFFD (�), so we can't tell what they were. This most commonly happens with curly quotation marks that appear ``“ like this â€�``. We can do better by building on ftfy's "sloppy codecs" to let them handle less-sloppy but more-lossy text. When they encounter the character ``�``, instead of refusing to encode it, they encode it as byte 1A -- an ASCII control code called SUBSTITUTE that once was meant for about the same purpose. We can then apply a fixer that looks for UTF-8 sequences where some continuation bytes have been replaced by byte 1A, and decode the whole sequence as �; if that doesn't work, it'll just turn the byte back into � itself. As a result, the above text ``“ like this â€�`` will decode as ``“ like this �``. If U+1A was actually in the original string, then the sloppy codecs will not be used, and this function will not be run, so your weird control character will be left alone but wacky fixes like this won't be possible. This is used as a transcoder within `fix_encoding`. rK)r r.encode)r^s rreplace_lossy_sequencesrisH   X__. 55rcbdttdtffd }tj|S)z Sometimes, text from one encoding ends up embedded within text from a different one. This is common enough that we need to be able to fix it. This is used as a transcoder within `fix_encoding`. r$rc|jd}t|tkr t|rtj|S|S)Nr)r'lenrrr )r$substrrs rfix_embedded_mojibakez7decode_inconsistent_utf8..fix_embedded_mojibakes>Q v;T "vf~$$V, ,Mr)rr[rr.)rrns` rdecode_inconsistent_utf8ros0U3ZC    5t <  + 2 23H IIrc6tjt|S)z If text still contains C1 control characters, treat them as their Windows-1252 equivalents. This matches what Web browsers do. )rr.rqrs rfix_c1_controlsrss   Y --r)7__doc__rWr)rerrtypingrr ftfy.badnessr ftfy.chardatarrrr r r r r rrrr[rr listtupler#r,r/compiler2r3r7r:r<rBrLrNrIrOrQrTUNICODEVERBOSEr\r]rdrerfrirorqrsrrrsT        /3 /3 / #s #s # 'S 'U38_ 5 '# '5:#*-5-5-5` "**4 5 !# !# !DDD%c%c%$%c%c%(1#1#1hrzz+, BJJ?@ %*  .)s)s)B$S$S$ RZZ JJ #6#6#6z"**?@2%2E2.$6%$6E$6N=3=3=(JU3ZJCJ.#.#.r