Skip to content

compiler panic: "byte index 10 is not a char boundary" #62524

Closed
#66429
#66054
Closed
compiler panic: "byte index 10 is not a char boundary"#62524
#66429
#66054
Labels
A-UnicodeArea: UnicodeA-parserArea: The lexing & parsing of Rust source code to an ASTC-bugCategory: This is a bug.I-ICEIssue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.glacierICE tracked in rust-lang/glacier.regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.

Description

@dwrensha
dwrensha
opened on Jul 9, 2019
$ rustc -Vv
rustc 1.36.0 (a53f9df32 2019-07-03)
binary: rustc
commit-hash: a53f9df32fbb0b5f4382caaad8f1a46f36ea887c
commit-date: 2019-07-03
host: x86_64-apple-darwin
release: 1.36.0
LLVM version: 8.0

$ echo Zm4gbWFpbigo2Lw= | base64 -D > main.rs

$ rustc main.rs
error: this file contains an un-closed delimiter
 --> main.rs:1:11
  |
1 | fn main((ؼ
  |        -- ^
  |        ||
  |        |un-closed delimiter
  |        un-closed delimiter

thread 'rustc' panicked at 'byte index 10 is not a char boundary; it is inside 'ؼ' (bytes 9..11) of `fn main((ؼ`', src/libcore/str/mod.rs:2034:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: aborting due to previous error


error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.36.0 (a53f9df32 2019-07-03) running on x86_64-apple-darwin

Found with the help of libfuzzer-sys.

Activity

  1. ExpHP commented on Jul 9, 2019

    @ExpHP
    ExpHP
    on Jul 9, 2019
    Contributor
      12: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}
  13: syntax::source_map::SourceMap::find_width_of_character_at_span
  14: syntax::source_map::SourceMap::next_point
  15: syntax::parse::diagnostics::<impl syntax::parse::parser::Parser>::unexpected_try_recover
  16: syntax::parse::parser::Parser::parse_fn_args::{{closure}}
  17: syntax::parse::parser::Parser::parse_fn_args
  18: syntax::parse::parser::Parser::parse_fn_decl
  19: syntax::parse::parser::Parser::parse_item_fn
  20: syntax::parse::parser::Parser::parse_item_implementation
  21: syntax::parse::parser::Parser::parse_item_

    There's two slicing operations in find_width_of_character_at_span. I'm trying to figure out how a non-boundary index could be appearing in there... maybe unexpected_try_recover is calling next_point with a span that begins at index 10? (where does this come from? Parser.prev_span?) Building a debug compiler to check those log statements...

    Edit: Building the debug compiler was a bust. Embarassingly, things seem to have changed and I don't know how to get those debug macros to fire in the current compiler.

  2. jonas-schievink
    added
    A-parserArea: The lexing & parsing of Rust source code to an AST
    C-bugCategory: This is a bug.
    I-ICEIssue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️
    I-nominated
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    on Jul 10, 2019

  3. pnkfelix commented on Jul 11, 2019

    @pnkfelix
    pnkfelix
    on Jul 11, 2019
    Contributor

    @ExpHP do you mean you were using RUST_LOG and not seeing debug output?

    If so, that is because the environment variable name under rustc was changed to RUSTC_LOG. (I make this mistake pretty much every day due to muscle memory.)

    As an example, here is the tail of my log output:

    % RUSTC_LOG=syntax::parse,syntax::source_map ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc  /tmp/off_index.rs
DEBUG 2019-07-11T12:58:22Z: syntax::parse::attr: parse_outer_attributes: self.token=Token { kind: OpenDelim(Paren), spa\
n: Span { lo: BytePos(8), hi: BytePos(9), ctxt: #0 } }
DEBUG 2019-07-11T12:58:22Z: syntax::parse::parser: parse_arg_general parse_pat (is_name_required:true)
DEBUG 2019-07-11T12:58:22Z: syntax::source_map: find_width_of_character_at_span: local_begin=`SourceFileAndBytePos { sf\
: SourceFile(/tmp/off_index.rs), pos: BytePos(10) }`, local_end=`SourceFileAndBytePos { sf: SourceFile(/tmp/off_index.r\
s), pos: BytePos(11) }`
DEBUG 2019-07-11T12:58:22Z: syntax::source_map: find_width_of_character_at_span: start_index=`10`, end_index=`11`
DEBUG 2019-07-11T12:58:22Z: syntax::source_map: find_width_of_character_at_span: source_len=`11`
thread 'rustc' panicked at 'byte index 10 is not a char boundary; it is inside 'ؼ' (bytes 9..11) of `fn main((ؼ`', src/\
libcore/str/mod.rs:2039:5

  4. pnkfelix commented on Jul 11, 2019

    @pnkfelix
    pnkfelix
    on Jul 11, 2019
    Contributor

    triage: P-medium. Removing nomination.

  5. pnkfelix
    added
    P-mediumMedium priority
    and removed
    I-nominated
    on Jul 11, 2019

  6. pnkfelix commented on Jul 11, 2019

    @pnkfelix
    pnkfelix
    on Jul 11, 2019
    Contributor

    This was broken between 1.15 and 1.16, but those are so old and the nature of the ICE itself has changed from 1.28 to 1.29, so I do not think bisection would be worthwhile.

  7. pnkfelix
    added
    regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.
    on Jul 11, 2019

  8. ExpHP commented on Jul 11, 2019

    @ExpHP
    ExpHP
    on Jul 11, 2019
    Contributor
    Offtopic

    @pnkfelix : Thanks. Unfortunately, I did guess the name of RUSTC_LOG after RUST_LOG didn't work, and all it showed me was this:

    $ RUSTC_LOG=trace rustc +stage1 src/main.rs
 INFO 2019-07-11T13:33:39Z: jobserver::imp: created a jobserver: Client { read: File { fd: 3, path: "pipe:[2538076]", read: true, write: false }, write: File { fd: 4, path: "pipe:[2538076]", read: false, write: true } }
 INFO 2019-07-11T13:33:39Z: rustc_interface::util: codegen backend candidate: /home/lampam/asd/clone/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends
 INFO 2019-07-11T13:33:39Z: rustc_interface::util: probing /home/lampam/asd/clone/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends for a codegen backend
error: this file contains an un-closed delimiter
 --> src/main.rs:1:11
  |
1 | fn main((ؼ
  |        -- ^
  |        ||
  |        |un-closed delimiter
  |        un-closed delimiter

thread 'rustc' panicked at 'byte index 10 is not a char boundary; it is inside 'ؼ' (bytes 9..11) of `fn main((ؼ`', src/libcore/str/mod.rs:2039:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: aborting due to previous error

    I set debug = true in my config.toml before building, and didn't see anything else in there. (I recall that debug! and trace! used to be explicitly mentioned in the comment for debug-assertions but it is no longer, so I did not enable it.)

  9. ExpHP commented on Jul 11, 2019

    @ExpHP
    ExpHP
    on Jul 11, 2019
    Contributor

    Okay, I did the following evil, evil thing:

    Pure evil, do not open 
    diff --git a/src/libsyntax_pos/span_encoding.rs b/src/libsyntax_pos/span_encoding.rs
index 525ec13623..d7da2206ab 100644
--- a/src/libsyntax_pos/span_encoding.rs
+++ b/src/libsyntax_pos/span_encoding.rs
@@ -74,6 +74,12 @@ pub const DUMMY_SP: Span = Span { base_or_index: 0, len_or_tag: 0, ctxt_or_zero:
 impl Span {
     #[inline]
     pub fn new(mut lo: BytePos, mut hi: BytePos, ctxt: SyntaxContext) -> Self {
+        if lo == BytePos(10) || hi == BytePos(10) {
+            eprintln!("=========================");
+            eprintln!("{:?}", (lo, hi));
+            eprintln!();
+            let _ = std::panic::catch_unwind(|| panic!()); // potentially print backtrace
+        }
         if lo > hi {
             std::mem::swap(&mut lo, &mut hi);
         }

    And acquired the following backtrace of when the span was first constructed:

    =========================
(BytePos(10), BytePos(11))

  15: syntax_pos::span_encoding::Span::new at /home/lampam/asd/clone/rust/src/libsyntax_pos/span_encoding.rs:81
  16: syntax_pos::SpanData::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:218
  17: syntax_pos::<impl syntax_pos::span_encoding::Span>::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:268
  18: syntax::tokenstream::TokenTree::close_tt at src/libsyntax/tokenstream.rs:141
  19: syntax::parse::parser::TokenCursor::next at src/libsyntax/parse/parser.rs:315
  20: syntax::parse::parser::Parser::next_tok at src/libsyntax/parse/parser.rs:528
  21: syntax::parse::parser::Parser::bump at src/libsyntax/parse/parser.rs:1031
  22: syntax::parse::parser::Parser::parse_ident_common at src/libsyntax/parse/parser.rs:632
  23: syntax::parse::parser::Parser::parse_ident at src/libsyntax/parse/parser.rs:617
  24: syntax::parse::parser::Parser::parse_pat_ident at src/libsyntax/parse/parser.rs:4132
  25: syntax::parse::parser::Parser::parse_pat_with_range_pat at src/libsyntax/parse/parser.rs:3987
  26: syntax::parse::parser::Parser::parse_pat at src/libsyntax/parse/parser.rs:3908
  27: syntax::parse::parser::Parser::parse_pat_list at src/libsyntax/parse/parser.rs:3582
  28: syntax::parse::parser::Parser::parse_parenthesized_pat_list at src/libsyntax/parse/parser.rs:3547
  29: syntax::parse::parser::Parser::parse_pat_with_range_pat at src/libsyntax/parse/parser.rs:3938
  30: syntax::parse::parser::Parser::parse_pat at src/libsyntax/parse/parser.rs:3908
  31: syntax::parse::parser::Parser::parse_arg_general at src/libsyntax/parse/parser.rs:1510
  32: syntax::parse::parser::Parser::parse_fn_args::{{closure}} at src/libsyntax/parse/parser.rs:5380
  33: syntax::parse::parser::Parser::parse_seq_to_before_tokens at src/libsyntax/parse/parser.rs:983
  34: syntax::parse::parser::Parser::parse_seq_to_before_end at src/libsyntax/parse/parser.rs:916
  35: syntax::parse::parser::Parser::parse_fn_args at src/libsyntax/parse/parser.rs:5368
  36: syntax::parse::parser::Parser::parse_fn_decl at src/libsyntax/parse/parser.rs:5429
  37: syntax::parse::parser::Parser::parse_item_fn at src/libsyntax/parse/parser.rs:5660

    It also gets constructed a second time:

    Second backtrace 
    =========================
(BytePos(10), BytePos(11))

  15: syntax_pos::span_encoding::Span::new at /home/lampam/asd/clone/rust/src/libsyntax_pos/span_encoding.rs:81
  16: syntax_pos::SpanData::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:218
  17: syntax_pos::<impl syntax_pos::span_encoding::Span>::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:268
  18: syntax::tokenstream::TokenTree::close_tt at src/libsyntax/tokenstream.rs:141
  19: syntax::parse::parser::TokenCursor::next at src/libsyntax/parse/parser.rs:315
  20: syntax::parse::parser::Parser::next_tok at src/libsyntax/parse/parser.rs:528
  21: syntax::parse::parser::Parser::bump at src/libsyntax/parse/parser.rs:1031
  22: syntax::parse::parser::Parser::expect_one_of at src/libsyntax/parse/parser.rs:590
  23: syntax::parse::parser::Parser::expect at src/libsyntax/parse/parser.rs:577
  24: syntax::parse::parser::Parser::parse_parenthesized_pat_list at src/libsyntax/parse/parser.rs:3555
  25: syntax::parse::parser::Parser::parse_pat_with_range_pat at src/libsyntax/parse/parser.rs:3938
  26: syntax::parse::parser::Parser::parse_pat at src/libsyntax/parse/parser.rs:3908
  27: syntax::parse::parser::Parser::parse_arg_general at src/libsyntax/parse/parser.rs:1510
  28: syntax::parse::parser::Parser::parse_fn_args::{{closure}} at src/libsyntax/parse/parser.rs:5380
  29: syntax::parse::parser::Parser::parse_seq_to_before_tokens at src/libsyntax/parse/parser.rs:983
  30: syntax::parse::parser::Parser::parse_seq_to_before_end at src/libsyntax/parse/parser.rs:916
  31: syntax::parse::parser::Parser::parse_fn_args at src/libsyntax/parse/parser.rs:5368
  32: syntax::parse::parser::Parser::parse_fn_decl at src/libsyntax/parse/parser.rs:5429
  33: syntax::parse::parser::Parser::parse_item_fn at src/libsyntax/parse/parser.rs:5660

    (the region of the backtrace that differs between the two is items 22-29 in the first one, or 22-25 in the second)

    Some interesting spots from here:

    This is clearly where the value of 10 is created:

    rust/src/libsyntax/tokenstream.rs

    Lines 136 to 144 in 4bb6b4a

    /// Returns the closing delimiter as a token tree.
    pub fn close_tt(span: Span, delim: DelimToken) -> TokenTree {
    let close_span = if span.is_dummy() {
    span
    } else {
    span.with_lo(span.hi() - BytePos(delim.len() as u32))
    };
    TokenTree::token(token::CloseDelim(delim), close_span)
    }

    TokenCursor::next calls this if there are no remaining tokens and it still hasn't seen the closing delimiter (it seems to have as an unstated precondition that the closing delimiter must exist).

    Here's the innermost bit that's specific to ident parsing. Doesn't look that odd...

    rust/src/libsyntax/parse/parser.rs

    Lines 622 to 634 in 4bb6b4a

    token::Ident(name, _) => {
    if self.token.is_reserved_ident() {
    let mut err = self.expected_ident_found();
    if recover {
    err.emit();
    } else {
    return Err(err);
    }
    }
    let span = self.token.span;
    self.bump();
    Ok(Ident::new(name, span))
    }

  10. pnkfelix commented on Jul 11, 2019

    @pnkfelix
    pnkfelix
    on Jul 11, 2019
    Contributor

    @ExpHP I’m not familiar with RUSTC_LOG=trace

    I tend to list (comma separated) module paths in my own use of RUSTC_LOG; did you try that? You can see an example in the transcript I gave in my comment above

  11. 5 remaining items

  12. rust-lang-glacier-bot
    added
    glacierICE tracked in rust-lang/glacier.
    on Oct 15, 2019
  13. Centril
    added 2 commits that reference this issue on Nov 5, 2019

    Rollup merge of rust-lang#66054 - petrochenkov:delspan, r=estebank

    de02742

    Rollup merge of rust-lang#66054 - petrochenkov:delspan, r=estebank

    e5da1a1
  14. bors
    closed this as completedin e5da1a1on Nov 6, 2019

  15. JohnTitor commented on Nov 13, 2019

    @JohnTitor
    JohnTitor
    on Nov 13, 2019
    Member

    Hm, the example from #62524 (comment) hasn't been fixed?

  16. Alexendoo commented on Nov 13, 2019

    @Alexendoo
    Alexendoo
    on Nov 13, 2019
    Member

    Seems so, on the current nightly:

    error: this file contains an un-closed delimiter
 --> 62524-2.rs:2:3
  |
1 | y![
  |   - un-closed delimiter
2 | Ϥ,
  |   ^

error: macros that expand to items must be delimited with braces or followed by a semicolon
 --> 62524-2.rs:1:3
  |
1 |   y![
  |  ___^
2 | | Ϥ,
  | |__^
  |
thread 'rustc' panicked at 'byte index 1 is not a char boundary; it is inside 'Ϥ' (bytes 0..2) of `Ϥ,`', src\libcore\str\mod.rs:2069:5
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: <std::io::IoSliceMut as core::fmt::Debug>::fmt
   3: std::panicking::take_hook
   4: std::panicking::take_hook
   5: rustc_driver::report_ice
   6: std::panicking::rust_panic_with_hook
   7: std::panicking::begin_panic_fmt
   8: rust_begin_unwind
   9: core::panicking::panic_fmt
  10: core::str::slice_error_fail
  11: <rustc_driver::args::Error as core::fmt::Debug>::fmt
  12: <rustc_errors::lock::acquire_global_lock::Handle as core::ops::drop::Drop>::drop
  13: rustc_errors::annotate_snippet_emitter_writer::AnnotateSnippetEmitterWriter::ui_testing
  14: <rustc_errors::emitter::EmitterWriter as rustc_errors::emitter::Emitter>::emit_diagnostic
  15: rustc_errors::HandlerInner::emit_diagnostic
  16: rustc_errors::diagnostic_builder::DiagnosticBuilder::emit
  17: rustc_parse::parser::item::<impl rustc_parse::parser::Parser>::parse_foreign_item
  18: rustc_parse::parser::item::<impl rustc_parse::parser::Parser>::parse_item
  19: rustc_parse::parser::item::<impl rustc_parse::parser::Parser>::parse_item
  20: rustc_parse::parser::item::<impl rustc_parse::parser::Parser>::parse_item
  21: rustc_parse::parser::module::<impl rustc_parse::parser::Parser>::parse_crate_mod
  22: rustc_parse::parser::module::<impl rustc_parse::parser::Parser>::parse_crate_mod
  23: rustc_parse::parse_crate_from_file
  24: <rustc_interface::proc_macro_decls::Finder as rustc::hir::itemlikevisit::ItemLikeVisitor>::visit_item
  25: <rustc_interface::proc_macro_decls::Finder as rustc::hir::itemlikevisit::ItemLikeVisitor>::visit_item
  26: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::compile
  27: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::parse
  28: <syntax_pos::symbol::SymbolStr as core::fmt::Display>::fmt
  29: <syntax_pos::symbol::SymbolStr as core::fmt::Display>::fmt
  30: <syntax_pos::symbol::SymbolStr as core::fmt::Display>::fmt
  31: <syntax_pos::symbol::SymbolStr as core::fmt::Display>::fmt
  32: _rust_maybe_catch_panic
  33: <syntax_pos::symbol::SymbolStr as core::fmt::Display>::fmt
  34: ZN244_$LT$std..error..$LT$impl$u20$core..convert..From$LT$alloc..string..String$GT$$u20$for$u20$alloc..boxed..Box$LT$dyn$u20$std..error..Error$u2b$core..marker..Send$u2b$core..marker..Sync$GT$$GT$..from..StringError$u20$as$u20$core..fmt..Display$GT$3fmt17
  35: std::sys::windows::thread::Thread::new
  36: BaseThreadInitThunk
  37: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.40.0-nightly (4f03f4a98 2019-11-12) running on x86_64-pc-windows-msvc

query stack during panic:
end of query stack
error: aborting due to previous error
  17. Alexendoo
    reopened this on Nov 13, 2019

  18. guanqun commented on Nov 13, 2019

    @guanqun
    guanqun
    on Nov 13, 2019

    @Alexendoo I guess my fix #66264 would help. I'll get some time to test it out.

  19. guanqun commented on Nov 15, 2019

    @guanqun
    guanqun
    on Nov 15, 2019

    Confirmed that my fix fixed this issue, I also add this unit test in this PR: #66429

  20. Centril
    added 2 commits that reference this issue on Nov 15, 2019

    add a unit test for rust-lang#62524

    ed9c5da

    Rollup merge of rust-lang#66429 - guanqun:add-unit-test, r=Centril

    0c04c8e
  21. tmandry
    added a commit that references this issue on Nov 15, 2019

    Rollup merge of rust-lang#66429 - guanqun:add-unit-test, r=Centril

    c6cdbe9
  22. bors
    added a commit that references this issue on Nov 15, 2019

    Auto merge of #66449 - tmandry:rollup-3p1t0sb, r=tmandry

    1bd30ce
  23. bors
    closed this as completedin c6cdbe9on Nov 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-UnicodeArea: UnicodeA-parserArea: The lexing & parsing of Rust source code to an ASTC-bugCategory: This is a bug.I-ICEIssue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.glacierICE tracked in rust-lang/glacier.regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions