Skip to content

Feat: Service Principal (client credentials) auth mode to the ADLS Gen2 sink connector #2270

Open

Description

Summary

The kafka-connect-azure-datalake sink currently supports three auth modes (config/AuthModeSettings.scala):
Credentials (account key), ConnectionString, and Default (DefaultAzureCredential, worker-level).
There is no way to authenticate to storage with a per-connector Service Principal (Entra ID App Registration:
client_id / client_secret / tenant_id) using client-credentials OAuth.

I'd like to add a ServicePrincipal auth mode backed by ClientSecretCredentialBuilder from com.azure.identity,
which is already a project dependency (azure-identity, see project/Dependencies.scala) and already used for
DefaultAzureCredentialBuilder in auth/DatalakeClientCreator.scala. No new dependency is required.

Rational

Usually security standards require secrets to be short-lived and scoped. A storage account key or connection string is a long-lived,
account-wide credential. DefaultAzureCredential is a worker-level setting, not something that can be configured per connector.

A Service Principal with a scoped RBAC role and short-lived OAuth tokens, configurable per connector instance, closes
that gap while keeping all the existing sink functionality.

Proposed change

I think the actual changes should be quite small and dependeny-free.

  1. Configuration — config/AuthModeSettings.scala: add an AuthMode.ServicePrincipal(clientId, tenantId, clientSecret) case, plus config keys (e.g. connect.datalake.azure.client.id, .tenant.id, .client.secret),
    wired into getAuthMode.
  2. Authentication — auth/DatalakeClientCreator.scala: add a
    createDataLakeClientWithServicePrincipal(...) path using
    new ClientSecretCredentialBuilder().clientId(..).tenantId(..).clientSecret(..).build().
  3. Tests & docs — unit tests for the new auth mode, and an update to the public connector docs (auth mode
    table currently lists Credentials / ConnectionString / Default).

I'm happy to implement this myself and submit a PR per the project's "Contributing" guidance.
Before starting, I wanted to check:
is this something the maintainers would be open to accepting,
and is there any existing work or design preference I should be aware of (e.g. naming of config keys, whether ClientSecretCredential should be exposed directly or wrapped in a broader "explicit credential" auth mode)?

Happy to align on approach before opening a PR.

Activity

  1. marcel-radusch-vf commented on Oct 6, 2026

    @marcel-radusch-vf
    Author

    Hi,
    since I didn't get any response yet. I try to tag some of the previous contributors for guidance.

    Casey W Crites (@caseycrites) Caio Oliveira (@caiooliveiraeti) Hubert Behaghel (@behaghel)
    Are you willing to accept or review this contribution or are you not open for it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions