cargo package --no-verify fails if a package's version req is too high for the registry (but works locally)

Problem

#13447 dd698ff has some unintentional consequences:

cargo package after that generates a lockfile also when packaging libraries.
However, if the crate being packaged depends on some unpublished dependencies. When generating the lockfile Cargo then can't find unpublished deps on registry index, so it bailed out.

Steps

This could be simplly reproduced in rust-lang/cargo as of today (2025-01-13)

Checkout to 54df3c7
Bump cargo-test-support to an unpublished version such as 99.99.99
cargo +1.83.0 package -p cargo-test-support --no-verify — succeeded
cargo +1.84.0 package -p cargo-test-support --no-verify — failed

Possible Solution(s)

Assume if we have passed the local registry unconditionally, we shouldn't have the issue today.

cargo/src/cargo/ops/cargo_package/mod.rs

Lines 219 to 225 in f15df8f

    
           let mut local_reg = if ws.gctx().cli_unstable().package_workspace { 
        
               let reg_dir = ws.target_dir().join("package").join("tmp-registry"); 
        
               sid.map(|sid| TmpRegistry::new(ws.gctx(), reg_dir, sid)) 
        
                   .transpose()? 
        
           } else { 
        
               None 
        
           };

Version

cargo 1.84.0 (66221abde 2024-11-19)
cargo 1.86.0-nightly (088d49608 2025-01-10)

So this is a problem that existed before 54df3c7 but only for some crates. Now all see it, right?

Yeah I believe it was only bugging for packages having [[bin]] crates, but now every package.

@Eh2406

@Eh2406 our resolve call is

cargo/src/cargo/ops/cargo_package/mod.rs

Lines 655 to 664 in 54df3c7

    
           let mut new_resolve = ops::resolve_with_previous( 
        
               &mut tmp_reg, 
        
               &tmp_ws, 
        
               &CliFeatures::new_all(true), 
        
               HasDevUnits::Yes, 
        
               orig_resolve.as_ref(), 
        
               None, 
        
               &[], 
        
               true, 
        
           )?;

Is there a way to say "prefer existing but don't lock them? Would keep = |_| false do that? Seems like what we'd want (have to double check yanked) and then cargo package already has logic to warn in these situations which imo is good because its easy to accidentally release a package and not realize you forgot to release a dependency.

In terms of a swift fix: How stable is TmpRegistry?
It might be helpful if we could use it internally without stabilizing the entire -Zpackage-workspace.

I'm not sure how that would fix it. We wouldn't be putting the package into TmpRegistry to be able to be used unless the user requests it at which point it will work anyways.

More like Cargo automatically publishes unpublished dependencies crates when needed to TmpRegistry before building the lock?

I suspect getting it right to sometimes publish to TmpRegistry can cause us other problems and would get in the way of stabilizing workspace packaging that I feel like that less likely to be our best option for a swift fix.

if we feel the need to backport to beta, a revert is likely the best route. If not, we should at least understand what it would take on the resolver side.

We are running into same issue. Can this is be reverted?

I cannot reproduce this if I bump the package's version but don't bump the version requirement. I also missed the detail that --no-verify is required..

This means that cargo is able to adjust the lockfile correctly as needed and the problem is that we moved "dependency exists" checks from the verify step to the packaging step, forcing it to always happen.

The documentation for cargo publish --no-verify just says

Don’t verify the contents by building them.

I think this still leaves room for other verification. Historically, we've done some registry and dependency verification and we recently added "is this published?" (#14448).

cargo/src/cargo/ops/registry/publish.rs

Lines 96 to 139 in 54df3c7

    
           let just_pkgs: Vec<_> = pkgs.iter().map(|p| p.0).collect(); 
        
           let reg_or_index = match opts.reg_or_index.clone() { 
        
               Some(r) => { 
        
                   validate_registry(&just_pkgs, Some(&r))?; 
        
                   Some(r) 
        
               } 
        
               None => { 
        
                   let reg = super::infer_registry(&just_pkgs)?; 
        
                   validate_registry(&just_pkgs, reg.as_ref())?; 
        
                   if let Some(RegistryOrIndex::Registry(ref registry)) = &reg { 
        
                       if registry != CRATES_IO_REGISTRY { 
        
                           // Don't warn for crates.io. 
        
                           opts.gctx.shell().note(&format!( 
        
                               "found `{}` as only allowed registry. Publishing to it automatically.", 
        
                               registry 
        
                           ))?; 
        
                       } 
        
                   } 
        
                   reg 
        
               } 
        
           }; 
        
           // This is only used to confirm that we can create a token before we build the package. 
        
           // This causes the credential provider to be called an extra time, but keeps the same order of errors. 
        
           let source_ids = super::get_source_id(opts.gctx, reg_or_index.as_ref())?; 
        
           let (mut registry, mut source) = super::registry( 
        
               opts.gctx, 
        
               &source_ids, 
        
               opts.token.as_ref().map(Secret::as_deref), 
        
               reg_or_index.as_ref(), 
        
               true, 
        
               Some(Operation::Read).filter(|_| !opts.dry_run), 
        
           )?; 
        
           { 
        
               let _lock = opts 
        
                   .gctx 
        
                   .acquire_package_cache_lock(CacheLockMode::DownloadExclusive)?; 
        
               for (pkg, _) in &pkgs { 
        
                   verify_unpublished(pkg, &mut source, &source_ids, opts.dry_run, opts.gctx)?; 
        
                   verify_dependencies(pkg, &registry, source_ids.original)?; 
        
               } 
        
           }

Its also not clear to me why you would be publishing packages without dependencies present. Is this a workaround for #4242? We already have a method for this for some cases (leaving off versions for dev-dependencies). If there is a case that doesn't cover, it would be good to make sure its reported in #4242.

Note that this is about cargo package not publish.

There are workflows packaging source code into .crate files before any publishment. Not going to judge in this comment whether those workflows should be supported by Cargo, but the behavior did change in 1.84.

This Cargo-flavored test works in 1.83 but fails in 1.84.

#[cargo_test]
fn unpublished_dependency() {
    Package::new("dep", "0.1.0").publish();

    let p = project()
        .file(
            "Cargo.toml",
            r#"
            [package]
            name = "main"
            version = "0.0.1"
            edition = "2015"

            [dependencies]
            dep = { path = "dep", version = "0.1.1" }
        "#,
        )
        .file("src/lib.rs", "")
        .file(
            "dep/Cargo.toml",
            r#"
            [package]
            name = "dep"
            version = "0.1.1"
            edition = "2015"
        "#,
        )
        .file("dep/src/lib.rs", "")
        .build();

    p.cargo("package --package main --no-verify --no-metadata")
        .with_status(101)
        .with_stderr_data(str![[r#"
[PACKAGING] main v0.0.1 ([ROOT]/foo)
[UPDATING] `dummy-registry` index
[ERROR] failed to prepare local package for uploading

Caused by:
  failed to select a version for the requirement `dep = "^0.1.1"`
  candidate versions found which didn't match: 0.1.0
  location searched: `dummy-registry` index (which is replacing registry `crates-io`)
  required by package `main v0.0.1 ([ROOT]/foo)`
  perhaps a crate was updated and forgotten to be re-vendored?

"#]])
        .run();
}

@morenol

@morenol could you expand on your use case?

We have a binary that generates a rust project. Normally, the project generated uses crates.io packages, but while we are developing unstable features we pack the crates and embed them in the binary so the code generator can generate code the references these non released packages

Could those uses be switched to cargo -Zpackage-workspace package -p a -p b? That was at least the intent behind my suggestion of -Zpackage-workspace.

That could be one solution, yes. However, it then becomes a user problem of my tool, which they need to intervene their broken builds (not too hard but still a churn).

@morenol

I still get same error

@morenol, did you mean it still failed with -Zpackage-workspace on? Could you show the redacted version of the Cargo command your tool invoked?

This is call in a build.rs,

    let status = Command::new("cargo")
        .arg("-Zpackage-workspace")
        .arg("package")
        .arg("-p")
        .arg(name)
        .arg("--target")
        .arg(WASI_TARGET)
        .arg("--all-features")
        .arg("--no-verify")
        .arg("--allow-dirty")
        .arg("--target-dir")
        .arg(out_dir)
        .output()

@morenol

@morenol
While within build script you can do whatever you like, I truly sympathize you need to do such workflow in build.rs. Can you try packaging everything in one cargo package call, or using --workspace instead of -p for packaging the entire workspace?

To summarize, cargo package --no-verify at least fails in three different cases after #13447:

An unpublished package depending on itself as a dev-dependency (cyclic self-referential dev-dependencies).
- Can be resolved by removing the version field from the affected dev-dependency.
- -Zpackage-workspace has the same behavior as without the flag, so doesn't help fix the failure.
Existing cargo package has --package <pkg> specifying some but not all unpublished packages.
- Can be resolved by specifying all unpublished packages in one cargo call.
- -Zpackage-workspace requires all dependency versions available in the target registry or specified through the -p flag when packaging. You still need to specify all unpublished when using -Zpackage-workspace.
cargo package --no-verify has been used as a kind of “plumbing commands” to create tarballs without considering dependency orders. The use cases include:
- Preparing tarballs for other package managers.
- Integrating into custom develop workflows for unpublished/internal crates.
- Constructing custom/private registries.

One question continues popping up: why it wasn't a problem for [[bin]] until cargo started generating lockfile for [lib].

Here is my answer:

A lot of packages on crates.io and in the wild are lib only crates. People learn to use Cargo workspaces to separate bins from lib packages for a better dependency management (you can't have bin-only dependency in one package). So [[bin]] is less a problem, but it still a problem. People could work around it, like removing [[bin]] from Cargo.toml if there is no interest. And most packagers doing cargo package --no-verify cares more about libraries. If it were for binaries usually they just build them won't “package” the source.

I think upon this moment, as -Zpackage-workspace might not fit every need here, especially for packaging circular dev-dependencies, I would personally favor the proposal in #15159.

Existing cargo package has --package specifying certain unpublished packages.

-Zpackage-workspace also requires all dependency versions available in the target registry when calling, so doesn't help.

I think some context to this summary item is missing because I'm not following this.

And most packagers doing cargo package --no-verify cares more about libraries. If it were for binaries usually they just build them won't “package” the source.

If we're talking doing weird things for first-party code, then all of this is being done for the sake of releasing a [[bin]].

If the process involves common crates.io dependencies, then at least clap will run into this because it as a bin for testing.

-Zpackage-workspace also requires all dependency versions available in the target registry when calling, so doesn't help.

I think some context to this summary item is missing because I'm not following this.

Sorry for that. The comment just got updated. Let me know if it is still not clear.

If we're talking doing weird things for first-party code, then all of this is being done for the sake of releasing a [[bin]].

If the process involves common crates.io dependencies, then at least clap will run into this because it as a bin for testing.

The process can be more complicated when using custom registries or other approaches for sharing code through tarballs. [[bin]] does not need to be there in the first place when packaging libraries. It could be a separate bin package for final production. And packages on crates.io may have a different way to consume than first-party code/crates. People chimed in this thread has already provided the oddity of their environments.

Can be resolved by specifying all unpublished packages in one cargo call.

How would that be done? Do you have an example?

How would that be done? Do you have an example?

It depends on whether you know what you're going to publish beforehand. For example, if you have a names slice of all packages you're going to package, and they are in the same workspace, you shall be able to do something like

let cmd = Command::new("cargo");
cmd.arg("package")
    .arg("-Zpackage-workspace")
    .arg("--target")
    .arg(WASI_TARGET);
for name in names {
    cmd.arg("-p").arg(name);
}
let output = cmd.output()

ok, that seems to work now for me using the +nightly channel.

feat(package): add --exclude-lockfile flag (#15234)

Navigation Menu

Repository navigation

`cargo package --no-verify` fails if a package's version req is too high for the registry (but works locally) #15059

Problem

Steps

Possible Solution(s)

Version

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	let mut local_reg = if ws.gctx().cli_unstable().package_workspace {
	let reg_dir = ws.target_dir().join("package").join("tmp-registry");
	sid.map(\|sid\| TmpRegistry::new(ws.gctx(), reg_dir, sid))
	.transpose()?
	} else {
	None
	};

	let mut new_resolve = ops::resolve_with_previous(
	&mut tmp_reg,
	&tmp_ws,
	&CliFeatures::new_all(true),
	HasDevUnits::Yes,
	orig_resolve.as_ref(),
	None,
	&[],
	true,
	)?;

	let just_pkgs: Vec<_> = pkgs.iter().map(\|p\| p.0).collect();
	let reg_or_index = match opts.reg_or_index.clone() {
	Some(r) => {
	validate_registry(&just_pkgs, Some(&r))?;
	Some(r)
	}
	None => {
	let reg = super::infer_registry(&just_pkgs)?;
	validate_registry(&just_pkgs, reg.as_ref())?;
	if let Some(RegistryOrIndex::Registry(ref registry)) = &reg {
	if registry != CRATES_IO_REGISTRY {
	// Don't warn for crates.io.
	opts.gctx.shell().note(&format!(
	"found `{}` as only allowed registry. Publishing to it automatically.",
	registry
	))?;
	}
	}
	reg
	}
	};

	// This is only used to confirm that we can create a token before we build the package.
	// This causes the credential provider to be called an extra time, but keeps the same order of errors.
	let source_ids = super::get_source_id(opts.gctx, reg_or_index.as_ref())?;
	let (mut registry, mut source) = super::registry(
	opts.gctx,
	&source_ids,
	opts.token.as_ref().map(Secret::as_deref),
	reg_or_index.as_ref(),
	true,
	Some(Operation::Read).filter(\|_\| !opts.dry_run),
	)?;

	{
	let _lock = opts
	.gctx
	.acquire_package_cache_lock(CacheLockMode::DownloadExclusive)?;

	for (pkg, _) in &pkgs {
	verify_unpublished(pkg, &mut source, &source_ids, opts.dry_run, opts.gctx)?;
	verify_dependencies(pkg, &registry, source_ids.original)?;
	}
	}

cargo package --no-verify fails if a package's version req is too high for the registry (but works locally) #15059

Description

Problem

Steps

Possible Solution(s)

Version

Activity

epage commented on Jan 13, 2025

weihanglo commented on Jan 13, 2025

epage commented on Jan 13, 2025

weihanglo commented on Jan 13, 2025

epage commented on Jan 13, 2025

weihanglo commented on Jan 13, 2025

epage commented on Jan 14, 2025

sehz commented on Jan 14, 2025

epage commented on Jan 14, 2025

weihanglo commented on Jan 14, 2025

13 remaining items

epage commented on Feb 18, 2025

morenol commented on Feb 18, 2025

weihanglo commented on Feb 18, 2025

weihanglo commented on Feb 18, 2025

morenol commented on Feb 18, 2025

weihanglo commented on Feb 25, 2025

weihanglo commented on Feb 25, 2025

weihanglo commented on Feb 25, 2025

epage commented on Feb 25, 2025

weihanglo commented on Feb 25, 2025

morenol commented on Feb 25, 2025

weihanglo commented on Feb 25, 2025

morenol commented on Feb 25, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`cargo package --no-verify` fails if a package's version req is too high for the registry (but works locally) #15059