Concurrent deployments refresh the same GitLab token, so one fails with "Failed to refresh token: Bad Request" #5653

To Reproduce

1. Install Dokploy v0.30.8 on a self-hosted server.
2. Add a GitLab provider.
3. On the Concurrent Builds page, set "Dokploy Server" to 2. Webhook deployments then run two at a time, see Webhook deployments do not set serverId, so all servers share one build queue #5586.
4. Create two applications that use the GitLab provider. Use the same repository and branch.
5. Set Autodeploy to on for the two applications, and add the webhook URL of each application in GitLab.
6. Wait until the GitLab access token expires, which takes 2 hours. To skip the wait, run update gitlab set expires_at = 0 where "gitlabId" = '<id>'; in the Dokploy database.
7. Push a commit to the branch.

Current vs. Expected behavior

One deployment works. The other fails in less than a second, before the clone starts:

Initializing deployment
Building on Build Server
Failed to refresh token: Bad Request
Error occurred ❌, check the logs for details.

I expected the two deployments to use one new token, with only one refresh request to GitLab.

This happened on two pushes to our branch. The two applications are on two servers and use the same GitLab provider. Times are UTC:

After the push on Oct 12, expires_at of the provider is 1791798415. That is 07:46:55 plus 7200 seconds, so the deployment on server A did the refresh.

Provide environment information

Operating System:
  OS: CentOS Linux 7 (Dokploy server), AlmaLinux 9.5 (remote servers)
  Arch: x86_64
Dokploy version: v0.30.8
GitLab: 18.11.1-ee, self-hosted
VPS Provider: Self-hosted virtual machines
What applications/services are you trying to deploy?
  Dockerfile applications from GitLab, with Autodeploy, on two remote servers

Which area(s) are affected? (Select all that apply)

Application, Docker Compose

Are you deploying the applications where Dokploy is installed or on a remote server?

Remote server

Additional context

Cause

refreshGitlabToken reads the provider, checks expiresAt, sends the stored refresh token to GitLab, and saves the new tokens (gitlab.ts#L14-L57). Nothing stops two calls from doing this at the same time.

GitLab rotates the refresh token. A refresh request "Invalidates the existing access_token and refresh_token" (GitLab OAuth 2.0 API). When two deployments start with an expired token:

1. The two calls read the same refreshToken from the database.
2. The first request succeeds. GitLab invalidates the old refresh token, and Dokploy saves the new tokens.
3. The second request sends the old refresh token. GitLab answers 400, and refreshGitlabToken throws (gitlab.ts#L41-L43).

The clone (L140), the repository list (L167) and Test Connection (L276) all call this function. So a deployment and the repository list in the UI can collide the same way.

Today this needs "Dokploy Server" set to 2 or more, because all webhook deployments use one partition, see #5586. With #5587, webhook deployments on different servers run in parallel with the default settings, so this will happen more often.

Possible fix

Let one call refresh at a time for each provider. There are two ways to do this.

1. A map of the refreshes in progress. A call for a provider that is already refreshing waits for the same promise:

const refreshes = new Map<string, Promise<unknown>>();

export const refreshGitlabToken = (gitlabProviderId: string) => {
	let refresh = refreshes.get(gitlabProviderId);
	if (!refresh) {
		refresh = refreshToken(gitlabProviderId).finally(() => refreshes.delete(gitlabProviderId));
		refreshes.set(gitlabProviderId, refresh);
	}
	return refresh;
};

refreshToken is the current body of refreshGitlabToken. A call that starts after the refresh reads the new expiresAt and returns.

2. A Postgres advisory lock. Calls for a provider wait for the lock, then read the provider again:

export const refreshGitlabToken = async (gitlabProviderId: string) => {
	await db.transaction(async (tx) => {
		await tx.execute(sql`select pg_advisory_xact_lock(hashtext(${gitlabProviderId}))`);

		const gitlabProvider = await findGitlabById(gitlabProviderId);
		// the current expiresAt check, request and updateGitlab
	});
};

The second call gets the lock after the first call saved the new tokens. It reads the new expiresAt and returns without a request to GitLab.

Note: as written, this sketch can deadlock. findGitlabById and updateGitlab use the global db, so the call that has the lock needs a second connection. Each call that waits for the lock keeps a connection, and the postgres-js pool has 10 connections by default. When all 10 are in use, the call that has the lock waits for a connection forever. The reads and the write in the lock must use tx.

The difference is where the calls run. The map works only in one process. On a self-hosted instance, deployments, the repository list and Test Connection all run in apps/dokploy, so the map is enough. Cloud runs deployments in apps/api, which is a different process. So with the map, a deployment in apps/api and the repository list in apps/dokploy can still collide.

The advisory lock works across processes. But it keeps a database transaction open during the request to GitLab, and Dokploy does not use database locks anywhere else today.

Workaround

Deploy the failed application again. The token is valid then, so the deployment works.

Will you send a PR to fix it?

Yes