Repository navigation
Expose Utf8LossyChunksIter #54
Description
Proposal
Problem statement
When Utf8Error::valid_up_to and Utf8Error::error_len are used, their results will almost always be used to get substrings of the original string. However, since Utf8Error does not have a reference to the original string, it cannot have methods to return the substrings.
Utf8LossyChunksIter is also much easier to use.
Motivation, use-cases
This is useful when creating a custom byte string formatter. UTF-8 portions are usually output using the Display implementation for str or str::escape_debug, but invalid portions might require custom formatting.
Example
Code using str::from_utf8 (requires unsafe):
while !string.is_empty() {
let (valid, invalid) = match str::from_utf8(string) {
Ok(string) => (string, &[][..]),
Err(error) => {
let valid_len = error.valid_up_to();
let valid = unsafe { str::from_utf8_unchecked(&string[..valid_len]) };
let mut invalid = &string[valid_len..];
if let Some(invalid_len) = error.error_len() {
invalid = &invalid[..invalid_len];
}
(valid, invalid)
}
};
// formatting for `valid` and `invalid`
string = &string[valid.len() + invalid.len()..];
}Code using the new API:
for chunk in Utf8Chunks::new(string) {
let valid = chunk.valid();
let invalid = chunk.invalid();
// formatting for `valid` and `invalid`
}Solution sketches
Make the following changes, and change the feature for these structs from str_internals to utf8_chunks.
- Remove
Utf8Lossy. - Rename
Utf8LossyChunksItertoUtf8Chunks. - Rename
Utf8LossyChunktoUtf8Chunk. - Rename
Utf8LossyChunk::brokentoinvalid. - Change fields of
Utf8Chunksinto getter methods (i.e.,validandinvalid). - Add:
impl<'a> Utf8Chunks<'a> { fn new(bytes: &'a [u8]) -> Self; }
Links and related work
What happens now?
This issue is part of the libs-api team API change proposal process. Once this issue is filed the libs-api team will review open proposals in its weekly meeting. You should receive feedback within a week or two.
Activity
m-ou-se commentedon Jun 20, 2022
The
bstrcrate has a similar API, which also exposes another field/method on the chunks:incomplete, to check whether the incomplete sequence might still become valid if more bytes are added: https://docs.rs/bstr/latest/bstr/struct.Utf8Chunk.htmlm-ou-se commentedon Jun 20, 2022
Regardless of that minor detail, I'm in favor of working towards exposing this as a public API!
dylni commentedon Jun 20, 2022
Great! I agree that basing the API off bstr would be sensible, so I have updated the issue. The changes were actually surprisingly close already, but providing methods instead of fields will be better for future compatibility.
I'm leaving out the
incompletemethod for now, since that can be added later and I don't personally need it. However, I can add it back if it seems useful.Also, it might be worth discussing if the constructor should be
<[u8]>::utf8_chunksorUtfChunks::new. If I understand the process correctly, that discussion can happen on the tracking issue.dylni commentedon Jul 17, 2022
@m-ou-se Is this proposal ready for a PR? It still has the "initial-comment-period" label, so I'm not sure if it's been agreed that this should be added.
m-ou-se commentedon Jul 20, 2022
@dylni Sorry! We don't have this processs automated yet. Yes, feel free to go ahead and open a tracking issue for this new unstable feature and send a PR for its implementation.
dylni commentedon Jul 21, 2022
Thanks!
Tracking issue: rust-lang/rust#99543
Rollup merge of rust-lang#99544 - dylni:expose-utf8lossy, r=Mark-Simu…
dylni commentedon Aug 20, 2022
This has been merged: rust-lang/rust#99544
Rollup merge of #99544 - dylni:expose-utf8lossy, r=Mark-Simulacrum