Repository navigation
Document how to properly setup Codecov Github Action for OpenSource repositories #301
Description
Describe the bug
PRs from external contributors fail CI.
The PR is happening in a forked repo. Using tokenless upload.
Tokenless has reached GitHub rate limit.
Environment (please complete the following information):
- CodeCov GitHub-Action v4.1.0
- https://github.com/faker-js/faker/blob/a672d277b883a445a852f9b6db18818104e8de7e/.github/workflows/ci.yml#L212-L216
To Reproduce
Steps to reproduce the behavior:
- Go to an OpenSource project with CodeCov GitHub Action
- Open a PR as external contributor
- The CI will very likely fail (~10 CI failures, sometimes multiple in a row)
External contributors have a near 100% chance at failing. Internal contributors have a near 0% chance at failing.
Expected behavior
The CI is expected to work.
Or alternatively the guide describing how to setup the action should provide hints on how to deal with the error.
Screenshots
info - 2024-03-12 15:43:46,354 -- ci service found: github-actions
info - 2024-03-12 15:43:46,690 -- The PR is happening in a forked repo. Using tokenless upload.
info - 2024-03-12 15:43:47,076 -- Process Commit creating complete
error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}
Error: Codecov: Failed to properly create commit: The process '/home/runner/work/_actions/codecov/codecov-action/54bcd8715eee62d40e33596ef5e8f0f48dbbccab/dist/codecov' failed with exit code 1Additional context
Created on request here:
Affected CI pipelines:
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
kdarkhan commentedon Mar 20, 2024
It might be hard to find the workaround I came up with from the linked issue, adding it here if that becomes useful.
I ended up using 2 workflow approach which seems to work pretty stable with token based auth for forked repos.
This workflow uploads coverage report to Github artifacts and this workflow uses that report and the token to upload it to codecov.
adamjstewart commentedon Apr 12, 2024
Any updates on this? We're seeing a 100% failure rate on most recent PRs. Is the open source community expected to transition to a different coverage provider, or are there future plans to once again support forks?
rohan-at-sentry commentedon Apr 12, 2024
@ST-DDT thanks for raising.
For context (which you probably already realized from the screenshot - but may benefit others), v4 allows tokenless uploads for forks ; enabling open-source contributors to upload coverage.
Because this happens without a token, Codecov uploads from forks upload using Codecov's own token - you can see how quickly that can get consumed leading to a failure. That is most likely what is happening when you see
In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?
Also - I'm curious if the workaround here worked for you
ST-DDT commentedon Apr 12, 2024
I think the rate remained the same.
It just happened a lot at the time of reporting (and for an extended period of time).
Previously it was an undefimed error, now it is a specific error.
No, we are currently in a feature freeze and thus have very few external contributions.
We generally have a low need for code coverage and thus consider removing it instead due to the additional work required.
adamjstewart commentedon Apr 12, 2024
I tried to use the above workaround. Didn't try the full thing with hundreds of lines of JavaScript code, but it was too complicated to get working with the usual upload/download artifact actions. TorchGeo gets 99% external contributions, so we're very keen on getting this working.
rohan-at-sentry commentedon Apr 15, 2024
Hey folks cc @ST-DDT @adamjstewart
We're exploring the best path forward for a fix here. Here's the current thinking - codecov/engineering-team#1574
We'd expect this to be ~ 4 weeks of work.. I'll report back here once this is complete. We're shooting for something that doesn't require more config.
thank you for your patience.
rohan-at-sentry commentedon Apr 29, 2024
Meanwhile, we've better documented the tokenless flow in our docs . In general, there shouldn't be a need to implement workarounds to pass the token to forks that can then use it to upload to Codecov.
LebedevRI commentedon Apr 29, 2024
It would be nice to expand
with an information about the actual first version that supports that behaviour.
adamjstewart commentedon Apr 29, 2024
Could also replace:
with:
as the former is much more common (at the moment) 😏
But seriously though, thanks for all of your hard work to improve the situation. I know how difficult it is when GitHub refuses to increase the rate limit for a token and security is involved. I'm sure you're drowning in bug reports at the moment. Looking forward to a working solution, and happy to help test on our repo!
rohan-at-sentry commentedon Apr 29, 2024
Agreed, updated
rohan-at-sentry commentedon Apr 29, 2024
Thanks for your patience as we work through this. <3
jotak commentedon May 22, 2024
Hi folks,
I was wondering if it would be possible to use codecov on PRs (via
pull_requestGH event) only for posting comment and without uploading the report? Report uploads being done via another workflow, onpushevents, which doesn't have the same impact on secrets visibility. If that's technically doable, I guess it would be beneficial not only to fix the failures we're currently seeing, but also in terms of lowering the traffic & attempts between codecov and github api ?adamjstewart commentedon May 22, 2024
That is the workaround discussed in #126. See #126 (comment) for why that's currently way too difficult to get working properly. Of course, if there was a documented solution or simplification of this that worked, that would be fine. But a solution that allowed PR coverage to be uploaded without a secret and upstream repo coverage to be only uploaded with a secret would also be fine with me.
jotak commentedon May 22, 2024
@adamjstewart I was thinking something slightly different - but maybe this is naive as I don't really know how codecov works under the hood :
pushevent). There's no report uploaded during the PR opened lifetimepull_requestevent), the report is generated, and diff is performed locally against the base, comment is posted, but nothing being uploadedI guess that's not possible to do with the current state of the codecov-action, but my question is, would it be a good solution to implement. Or perhaps it's not technically doable :-) What I'm not sure is if the action has the ability by itself to run the diff against the base locally.
adamjstewart commentedon May 22, 2024
A comment would be better than nothing, but the ability to view the diff and missing coverage on the codecov website would be better. I don't think it would be a security issue to upload PR coverage without a secret, as long as a secret is still required for protected branches like main.
kdarkhan commentedon May 22, 2024
Based on previous comments it seems Codecov is working on a solution to this. I don't know how they are planning to resolve this but I believe something like the workaround I used in #301 (comment) can be packaged as reusable actions. The workaround works pretty well but is verbose to implement.
They could define 2 GitHub actions:
If done properly then users could just use 2 actions without extra verbosity of JS scripting.