Repository navigation
euc_kr char '0x3164' decode('ksx1001') cause UnicodeDecodeError #101863
Description
Bug report
char '0x3164' can be encode('ksx1001'), but can not decode('ksx1001')
def main():
code_point = 0x3164
c = chr(code_point)
raw = c.encode('ksx1001')
c2 = raw.decode('ksx1001') # <--- this cause error
print(f'{c} {c2}')
if __name__ == '__main__':
main()Traceback (most recent call last):
File "/Users/takwolf/Develop/FontDev/fusion-pixel-font/build.py", line 11, in <module>
main()
File "/Users/takwolf/Develop/FontDev/fusion-pixel-font/build.py", line 6, in main
c2 = raw.decode('ksx1001')
^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'euc_kr' codec can't decode bytes in position 0-1: incomplete multibyte sequence
The char is Hangul Compatibility Jamo -> Hangul Filler
https://unicode-table.com/en/3164/
The following code is get the zone in ks-x-1001:
def main():
code_point = 0x3164
c = chr(code_point)
raw = c.encode('ksx1001')
block_offset = 0xA0
zone_1 = raw[0] - block_offset
zone_2 = raw[1] - block_offset
print(f'{zone_1} {zone_2}')
if __name__ == '__main__':
main()zone_1 = 4
zone_2 = 52
https://en.wikipedia.org/wiki/KS_X_1001#Hangul_Filler
other chars in ksx1001 encode an decode is ok, but only this.
Your environment
- CPython versions tested on: Python 3.11.1
- Operating system and architecture: macOS 13.0
Activity
arhadthedev commentedon Feb 13, 2023
@ezio-melotti (as an active Unicode expert)
corona10 commentedon Feb 17, 2023
I will take a look at this issue as the internal OSS sprint issue.
I will delegate this issue to my mentees.
b8choi commentedon Feb 20, 2023
I will try to resolve this issue.
TakWolf commentedon Feb 23, 2023
I'm not a python expert.
str.encode()andstr.decode()is a builtin fun?I found the coding logic maybe here:
https://github.com/python/cpython/tree/main/Lib/encodings
https://github.com/python/cpython/blob/main/Lib/encodings/euc_kr.py#L7-L10
that register a codec named 'euc_kr' to provide
encode('euc_kr')and
import _codecs_kris a c source?https://github.com/python/cpython/blob/main/Modules/cjkcodecs/_codecs_kr.c#L7-L9
and
mappings_kr.hhttps://github.com/python/cpython/blob/main/Modules/cjkcodecs/mappings_kr.h
It seems that some arrays are provided here to encode/decode query.
and
mappings_kr.his Generate by https://github.com/python/cpython/blob/main/Tools/unicode/genmap_korean.pyhttps://github.com/python/cpython/blob/main/Tools/unicode/genmap_korean.py#L12-L58
I don't understand why the
cp949encmapsize not equals theksx1001decmapsize?Is
euc_krbi-directional in rules? That if can be encoded, it must can be decoded?Is CP949 equals to ksx1001?
corona10 commentedon Feb 23, 2023
Not equals
corona10 commentedon Feb 23, 2023
Because CP949 is a kind of extended version of KSX1001
TakWolf commentedon Feb 23, 2023
iterate unicode :
iterate ksx1001 :
TakWolf commentedon Feb 23, 2023
not_2_bytes_alphabet:
b8choi commentedon Mar 2, 2023
From my investigation so far, there are some unclear aspects of the decoding process related to Korean characters encoded in EUC-KR.
FYI, the KS X 1001 standard defines the Korean character set, while EUC-KR is an encoding method for this set. In Python, "ksx1001" is an alias of "euc_kr".
Korean alphabet, also known as Hangul, expresses one syllable consisting of consonants and vowels as one letter. The KS X 1001 standard defines a precomposed set of characters that correspond to one syllable rather than individual consonants and vowels. However, to express a character not included in this set, a method of composing consonants and vowels is defined. The "Hangul Filler" character is used for this purpose, and it, along with the following three consonants and vowels, are arranged to express one character in a total of 8 bytes.
The KS X 1001 standard does not provide any specific guidelines other than the method for expressing one character using the "Hangul Filler".
The main issue is whether to allow an independent "Hangul Filler" that is not followed by valid consonants and vowels to compose a single character. Currently, the EUC-KR encoder does not validate strings containing "Hangul Filler" and just convert "Hangul Filler" of KS X 1001 to the corresponding Unicode "Hangul Filler".
I think there are three possible solutions to this issue, which need to be discussed:
I have tried the third method and confirmed that no errors have occurred.For more information, please refer to the KS X 1001 specification and a detailed description of "Hangul Filler" character.
corona10 commentedon Mar 2, 2023
I prefer supporting the round-trip approach since we already support encoding.
But I would need to listen to the advice from the other core devs :)
cc @ezio-melotti @vstinner
corona10 commentedon Mar 2, 2023
@b8choi
Would you like to submit the patch anyway? We may need to analyze the side effect when we support the decoding for the character.
But please submit the patch with a nice test code.
gh-101863: Fix wrong comments in EUC-KR codec (gh-102417)
pythongh-101863: Fix wrong comments in EUC-KR codec (pythongh-102417)
Merge branch 'main' into nodeaddict
Merge branch 'main' into inlinecomp2
corona10 commentedon Mar 14, 2023
Since the decoding implementation for this case is undefined and difficult to support decoding for this edge case due to streaming issues.
I close this issue as not a bug.
If someone think that this is a bug please let me know.
TakWolf commentedon Mar 14, 2023
So far, I still haven't got an explanation that
chr(0x3164).encode('ksx1001').decode('ksx1001')will get a exception is or not a expected behavior (Why python do like this? What is the document or implements reference?).At least in my understanding, the encoder should be idempotent (That if some char can be encoded, it should be decoded).
For example there are some control characters. They have special purposes in the display layer, but this does not affect that they can be encoded and decoded.
I also deal with other encoding standards ( gb2312, shitf-jis, big5), that have not the similar behaviors (They are all idempotent).
And this problem does not exist in Java. (I will test later in Rust , Go and node-Javascript)
EUC was used to solve computer coding in the early stage, if they can't convert with unicode(utf-8) equivalent, it seems incredible to me.
TakWolf commentedon Mar 14, 2023
My opinion, encoding is a storage structure, data should not be changed through coders but keep it as it is.
So for b8choi's proposal 2 and 3, they are not suitable.
If the char is in the scope, coder should not throw a exception. And coder should not changed the data.
(Imagine that you just browse the ksx file in a utf8 system, the data is changed, this is not reasonable)
Ksx can be fully mapped to unicode, convert the hangul is meaningless. There is also a Hangul Filler in Unicode. Should we convert all the Hangul Filler combin chars to hangul in utf-8 data layer?
An evidence that may support my opinion, there is a Emoji Combinations in Unicode.
For example: 👩❤️💋👨 = 👩+❤️+💋+👨
This just a display, on data they are equal.
Hangul Filler ’s role should be similar to Ideographic Description Characters only effect on display layer.
If the text engine support Hangul Filler, it will display hangul only on display. If not display as char is.
pythongh-107862: Reference the existing issues for two of the codec e…
pganssle commentedon Oct 22, 2026
Following up on my discussion with @corona10 at the core dev sprint, I've taken a look at how other languages handle this by comparing how they render the filler character by itself and a full valid 8-byte sequence (
A4 D4 A4 B6 A4 D0 A4 D4)A4 D4EUC-KRA4 D4ㅤㅆㅠㅤEUC-KRA4 D4ㅤㅆㅠㅤencoding_rs0.8.28,EUC_KRA4 D4ㅤㅆㅠㅤTextDecoder("euc-kr")TextEncoderㅤㅆㅠㅤiconv,EUC-KRA4 D4ㅤㅆㅠㅤuconv,EUC-KRA4 D4ㅤㅆㅠㅤgolang.org/x/text/encoding/korean.EUCKR(source inspection)A4 D4ㅤㅆㅠㅤeuc_krA4 D4쓔Looks like they all accept
A4 D4as U+3164, though none of them implements Python's eight-byte Hangul composition behavior, which actually seems more correct than the four separate character behavior, which makes me think that complete compatibility is not something we should aim for.The executed implementations also accepted
A4 D4 41as U+3164 followed by ASCIIA, andA4 D4 A4 B6as U+3164 followed by U+3146. They treat each two-byte pair independently, even when the input would be a prefix of a composition sequence in Python. The Java probes usedCharsetDecoderandCharsetEncoderwith their default error-reporting behavior, the Rust decoder used its no-replacement API, and the JavaScript decoder usedfatal: true.I think this supports accepting a standalone filler, but preserving Python's existing handling of complete composition sequences would still differ from every other implementation in this survey. There is also an ambiguity that a fallback for incomplete sequences cannot resolve: the four-character string
U+3164 U+3146 U+3160 U+3164encodes to exactly the same bytes as U+C4D4. Accepting incomplete prefixes as literal characters would fix the reported example, but cannot restore all string round trips while retaining both the existing encoder mappings and composition behavior.I don't really know how these things are used in practice, so maybe the fact that we are very weird in doing this lossy one-way composition translation thing is actually very annoying to people and we should actually get rid of it.
I'm inclined to say that we should start with allowing standalone filler characters as a pareto improvement (makes us both more like other languages and the behavior less surprising), but I don't have strong opinions about the trade-offs involved in stopping the composition behavior.
pythongh-107862: Reference the existing issues for two of the codec e…