Skip to content

to_string causes crazy memory usage problems #16415

Closed

Description

@thestinger

5u32.to_string().byte_capacity() -> 128
"foo".to_string().byte_capacity() -> 128

I think the right solution to this is adding size hints to the underlying formatting system.

Activity

  1. liigo commented on Aug 11, 2014

    @liigo
    Contributor

    String is a buffer, later appends will benefits from it. If don't need a
    buffer, why not use &str?

  2. huonw commented on Aug 11, 2014

    @huonw
    Contributor

    @liigo it needs to be dynamic constructed (i.e. &str is completely inappropriate for this), and 128 bytes is way way way too much when there is only 1 or 3 bytes actually used (like in the two examples).

  3. pnkfelix commented on Aug 11, 2014

    @pnkfelix
    Contributor

    Just so it does not get lost in the shuffle, here is my current half-baked idea to address this:

    • a separate trait ToStringSupport : fmt::Show { fn size_hint(&self) -> Option<uint> { None } }, and then,
    • change the generic impl to impl<T: ToStringSupport> ToString for T (and then use write! into a buffer of appropriate capacity instead of format!)

    see related bit of discussion here: https://botbot.me/mozilla/rust-internals/2014-08-11/?msg=19511520&page=2

    It would put a minor road-block before you could call to_string() on an arbitrary type implementing Show, but it seems like a pretty simple way to get most of what is desired here.

  4. eddyb commented on Aug 12, 2014

    @eddyb
    Contributor

    @pnkfelix adding the hint method to Show as a default method would break no existing code, I would prefer that.

  5. pczarn commented on Aug 15, 2014

    @pczarn
    Contributor

    Good ideas. Should we use the upper bound for all integers? What about different bases? I think that integers shouldn't implement size_hint, because leading_zeros (Int trait) and size_of are sufficient for determining the number of digits in all radixes.

  6. huonw commented on Aug 15, 2014

    @huonw
    Contributor

    The size_hint could take a &fmt::Formatter as well as self to be able to answer the question more precisely.

    I think that integers shouldn't implement size_hint, because leading_zeros (Int trait) and size_of are sufficient for determining the number of digits in all radixes.

    How will they convey the computed number of digits if they don't implement that trait and method?

  7. pczarn commented on Aug 15, 2014

    @pczarn
    Contributor

    I've just found out that Show is implemented for RadixFmt<T, Radix>, not directly on integers. This resolves my doubts.

    How will they convey the computed number of digits if they don't implement that trait and method?

    By using either size_hint or leading_zeros. Looks like it was a bad idea. Or by implementing size_hint for T: Show + Int.

  8. huonw commented on Aug 15, 2014

    @huonw
    Contributor

    not directly on integers

    It is.

  9. pczarn commented on Aug 15, 2014

    @pczarn
    Contributor

    Then it's as simple as extending these macros. I can do that.

  10. barosl commented on Dec 2, 2014

    @barosl
    Contributor

    It seems that this issue can be closed now.

    5u32.to_string().capacity() -> 1
    "foo".to_string().capacity() -> 4
    "hello".to_string().capacity() -> 8

    Now the buffer size is incremented to fit the minimum 2^n.

  11. thestinger commented on Dec 2, 2014

    @thestinger
    ContributorAuthor

    That makes it seem like it's just pushing them without using with_capacity at all...

  12. barosl commented on Dec 2, 2014

    @barosl
    Contributor

    @thestinger Uhm, yeah, you're right. Now I'm worried that not using with_capacity() could impact the performance...

  13. thestinger commented on Dec 2, 2014

    @thestinger
    ContributorAuthor

    Rust was just picking 128 as an arbitrary amount to reserve up front before. Not using a size hint will have a severe impact on performance, but clamping to 128 was worse.

  14. added a commit that references this issue on Jan 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    I-compiletimeIssue: Problems and improvements with respect to compile times.I-slowIssue: Problems and improvements with respect to performance of generated code.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions