Skip to content

Sparse-registry fetching total size keeps increasing #10820

Closed

Description

@pickfire

Problem

From original thread https://internals.rust-lang.org/t/call-for-testing-cargo-sparse-registry/16862/20?u=pickfire

One thing I noticed is that the fetching need to go through several loops now, previously it just fetch until 100% then done, but now when it reached 100%, then it had to fetch another time again, and the total stuff needed to fetch which keeps increasing (1 -> 10 -> 50 -> 100 -> 130) resulting in the progress decreasing, feels weird.

I feel like the total should be - until there is a definite answer to the total number that needs to fetch then only show it rather than keep increasing it.

Steps

  1. rustup update nightly
  2. cargo +nightly -Z sparse-registry update

Possible Solution(s)

A way to improve that in the registry would be to on publish, provide an optional expected_dep_forest_size from the number of transitive dependencies used at publish time. This number could be used to provide a reasonable progress estimate quickly.

Notes

No response

Version

No response

Activity

  1. arlosi commented on Jul 6, 2022

    @arlosi
    Contributor

    This is expected behavior, but I understand it's confusing to have progress go backwards.

    @rustbot modify labels: -C-bug +C-enhancement

  2. rustbot commented on Jul 6, 2022

    @rustbot
    Collaborator

    Error: The feature relabel is not enabled in this repository.
    To enable it add its section in the triagebot.toml in the root of the repository.

    Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

  3. arlosi commented on Jul 6, 2022

    @arlosi
    Contributor

    Some additional ideas on what we could do.

    Remove the progress bar

    • But keep the X/Y status. Y will still change as deps are discovered.
    Fetching (35/38)
    

    Include an estimate of the dependency graph size when publishing

    • Dependency graph size can change depending on features, platform, etc.
    • Would require running a batch job to generate for all existing crate versions
    • Increases metadata size

    Attempt to run the resolver using cached metadata to get an estimate

    • Only works if the local cache is fresh
    • Wastes CPU time

    Use the lockfile to get an estimate of the total number of dependencies

    • Only works if a lockfile is present.
  4. pickfire commented on Jul 7, 2022

    @pickfire
    ContributorAuthor

    But keep the X/Y status. Y will still change as deps are discovered.

    That sounds pretty much the same issue to me, it's basically a progress bar that never ends, I rather it be X/- or X/∞ if we don't know it yet.

  5. arlosi commented on Jul 7, 2022

    @arlosi
    Contributor

    Maybe instead of 35/38 something like:

    Fetching (35 completed; 3 downloading)
    
  6. pickfire commented on Jul 8, 2022

    @pickfire
    ContributorAuthor

    Or maybe X/N+3 then until it is confirmed, X/Y

  7. arlosi commented on Jul 8, 2022

    @arlosi
    Contributor

    What do you mean by confirmed? Each dependency that's fetched could potentially require additional dependencies that aren't fetched yet.

  8. pickfire commented on Jul 9, 2022

    @pickfire
    ContributorAuthor

    Confirmed as in cargo knows that there won't be any additional dependencies.

  9. added a commit that references this issue on Sep 9, 2022
    bd99c04
  10. added a commit that references this issue on Dec 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions