[sruggier]

Summary

This PR increases the podman driver's API timeout: Experience has shown that normal container creation can take longer than 30 seconds, for some users, so this change doubles the timeout, for now, until the source of the slowness is found and fixed.

Related Issue

Relates to #2913. (Edited to not auto-close)

This only partially addresses the issue, though, so it would probably make sense to keep it open, for now.

Release Notes

The hard-coded API timeout in the podman driver has been bumped from 30s to 60s, to reduce the likelihood of timeouts during sandbox creation (until the cause is investigated).

Testing

None yet.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[sruggier]

I have read the DCO document and I hereby sign the DCO.

[TaylorMutch]

/ok to test 9c091d9

[TaylorMutch]

@sruggier please see the failing lint jobs

Failing lint jobs prevent merge

[sruggier]

@sruggier please see the failing lint jobs

Thanks, sorry I didn't catch that locally. I've pushed a fix (and slightly updated the commit description).

[TaylorMutch]

/ok to test d31866d

[TaylorMutch]

@sruggier thanks! I edited the description slightly to not auto close. Please let me know further how we can help investigate the root cause issue further!

[sruggier]

@sruggier thanks! I edited the description slightly to not auto close. Please let me know further how we can help investigate the root cause issue further!

From this document, it looks like it's a known issue:

Choosing a storage driver

The following storage drivers are listed from fastest to slowest:

1. native overlayfs
2. fuse-overlayfs
3. vfs

There is one notable exception to this speed ranking. Creating a container takes significantly longer with native overlayfs than fuse-overlayfs when these conditions are all met:

* rootless Podman is used
* a modified UID/GID mapping is used
* native overlayfs is used
* no container has yet been created with the specified container image and UID/GID mapping

My setup definitely ticks all of these boxes. I'm not sure about the fourth one, though - I tried creating a second sandbox, and it was still slow enough that it might not be any faster than the first one. I can see in top that most of the slowness seems to involve a couple of buildah-copier processes.

From my perspective as a user, bumping the timeout is enough to solve the problem for me, the actual container creation time is acceptable as it is now.

I'm able to try switching to fuse-overlayfs at some point, but it requires me to run podman system reset (which throws out all images) or set up openshell on a secondary account, which would be inconvenient as well, because I probably don't have space in my VM to replicate the same setup.