Skip to content

S3 File Writer: custom header values stick at the first message #511

Open

Description

@pacmano1

A ${...} variable in a File Writer's S3 custom headers is filled in for every message, but every write after the first sends the first message's values, until the channel is redeployed.

To reproduce, with access to an S3 bucket:

  1. Make a channel whose source transformer has a JavaScript step $c('clinic', String(msg));, with a File Writer in S3 mode, file name ${message.messageId}.txt, and custom headers x-amz-meta-clinic: ${clinic} and x-test-id: ${message.messageId}.
  2. Send "clinic A", then "clinic B".
  3. Run aws s3api head-object on 2.txt. Its metadata says clinic: clinic A and x-test-id: 1; it should say clinic B and 2.

Measured on main and on #506 (AWS SDK 2.55.12), against real S3. Turning off Keep Connection Open doesn't help. Only a redeploy resets it, and then only for one message.

Cause: the File Writer fills in the variables per message (FileDispatcher.replaceConnectorProperties), but the values never reach the connection. The S3 connection pool's key is only the credentials and region (FileSystemConnectionFactory.getPoolKey), and every connection in a pool is built from the settings of the first message, so S3Connection.getCustomHeaders() always returns that message's headers.

Originally reported to NextGen as nextgenhealthcare/connect#5116 (2022). A PR with the fix is coming.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions