Dependencies cache key has to be specific to OS version

Description:

A cache key looks like this setup-python-Linux-python-3.11.0-beta.3-pip-60d374041db12689c3d346e149aa43744f06aa781ab1c8cbbe70e0b09b98dd70.
It is specific to a Python version and OS type. It has to be specific to a version of an operating system too.

Justification:

PyPI provides source distributions of packages (archives with code) and built distributions (wheels). When a wheel suitable for an environment is not found on PyPI, a source distribution is usually downloaded and a wheel is built locally. The source may contain some code that needs to be compiled (C, Rust, etc.)

On Linux, a compiled wheel may depend on a specific version of a shared library. Different Ubuntu releases used on GitHub Actions may provide different versions of shared libraries. Therefore, a wheel compiled on Ubuntu 20.04 may be incompatible with Ubuntu 22.04. This makes the cache feature unusable when a workflow runs jobs on multiple Ubuntu versions or when an Ubuntu version is changed in a workflow.

https://github.com/urllib3/urllib3/runs/6817232938 is an example of a failed run.
The error is caused by a cffi wheel that was compiled and cached in Ubuntu 20.04 previously where it depended on libffi.so.7.
Ubuntu 22.04 (that provides libffi.so.8) restored that cache archive because it composed the same cache key, and cffi become unusable there (urllib3/urllib3#2626 (comment) provides a little bit more info.)

BTW, #188 may be a related bug report issue.

Are you willing to submit a PR?

No. Although, I guess it is a matter of changing the following lines of code to include a version of an OS in the key.

setup-python/src/cache-distributions/pip-cache.ts

Lines 60 to 61 in 813f9b1

    
           const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`; 
        
           const restoreKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}`;

@illia-v

Hi @illia-v, we will take a look at this, thank you.

@e-korolevskii

@e-korolevskii implementing this feature should solve this issue and this

@illia-v

Hi @illia-v, we merged PR https://github.com/actions/setup-python/pull/467 that is solving this issue.

For now, I will close this issue, if there are any questions or other issues please feel free to continue the conversation here, reopen the issue, or create a new one.

Cheers

@panticmilos

@panticmilos thank you!

@panticmilos

@panticmilos what do you think about updating these values too?

setup-python/src/cache-distributions/poetry-cache.ts

Line 36 in 592a7a7

    
           const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`;

setup-python/src/cache-distributions/pipenv-cache.ts

Line 35 in 592a7a7

    
           const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`;

@illia-v

Hi @illia-v, sorry for the delayed answer. At first I have implemented that part as well. But then as I and my team discussed, we realized we are saving virtualenv for poetry and pipenv. For pip we are saving global cache and that's why we had to do it.

Navigation Menu

Repository navigation

Dependencies cache key has to be specific to OS version #432

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`;
	const restoreKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}`;

Dependencies cache key has to be specific to OS version #432

Description

Activity

vsafonkin commented on Jun 13, 2022

dsame commented on Jun 27, 2022

panticmilos commented on Jul 19, 2022

illia-v commented on Jul 21, 2022

illia-v commented on Jul 21, 2022

panticmilos commented on Aug 10, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions