Repository navigation
codecs.StreamReader doesn't pass final=1 to the UTF-8 codec #56717
Description
| BPO | 12508 |
|---|---|
| Nosy | @malemburg, @vstinner, @ezio-melotti |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-07-06.22:45:47.044>
labels = ['3.10', 'type-bug', '3.8', '3.9', 'expert-unicode']
title = "codecs.StreamReader doesn't pass final=1 to the UTF-8 codec"
updated_at = <Date 2020-11-11.19:26:15.934>
user = 'https://bugs.python.org/spatz123'bugs.python.org fields:
activity = <Date 2020-11-11.19:26:15.934>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2011-07-06.22:45:47.044>
creator = 'spatz123'
dependencies = []
files = ['22597', '26013']
hgrepos = []
issue_num = 12508
keywords = []
message_count = 6.0
messages = ['139954', '139956', '139957', '143473', '144373', '144376']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'spatz123']
pr_nums = []
priority = 'high'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12508'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
spatz123 commentedon Jul 6, 2011
The attached script produces the output
'A\ufffdBC\ufffd'
'A\ufffdBC'
although it seems to me that both lines should be the same. The first line is correct, I think, since the <F4> at the end is a maximal subpart of an ill-formed subsequence, according to the definition in the Unicode standard.
vstinner commentedon Jul 6, 2011
I confirm, there is a bug in codecs.StreamReader.
vstinner commentedon Jul 6, 2011
You should use the io module, it doesn't have the bug :)
ezio-melotti commentedon Sep 4, 2011
IIUC this happens because StreamReader calls codecs.utf_8_decode without passing final=1 [0], so when the decoder finds the trailing F4 it doesn't decode it yet because it waits from the other 3 bytes (F4 is the start byte of a 4-bytes UTF-8 sequence):
While passing 1 makes the attached script work as expected, it breaks several other test in test_codecs (apparently not all the decoders accept the 'final' argument).
Also passing 1 should be done only for the last call: read can be called several times with a specific size, and it shouldn't use final=1 until the last call to avoid errors mid-stream.
[0]: see Lib/codecs.py:482
malemburg commentedon Sep 21, 2011
The final parameter is an extension to the decoder API signature,
so it's not surprising that not all codecs implement it.
The ones that do should use it for all calls, since that way
the actual consumed number of bytes is correctly reported
back to the StreamReader instance.
Note: The parameter name "final" is a bit misleading. What happens
is that the number of bytes consumed by the decoder were previously
always reported as len(buffer), since the C API for decoders did
not provide a way to report back the number of bytes consumed.
This was changed when stateful decoders were added to the C API,
since these do allow reporting back the consumed bytes. A more
appropriate name for the parameter would have been
"report_bytes_consumed".
ezio-melotti commentedon Sep 21, 2011
AFAIU final means:
[-]Codecs Anomaly[/-][+]codecs.StreamReader doesn't pass final=1 to the UTF-8 codec[/+]pythongh-107862: Reference the existing issues for two of the codec e…
pythongh-107862: Reference the existing issues for two of the codec e…