Skip to content

Pause/Resume of sink causes publisher to continuously publish #257

Open

Description

@jblythe

Upon initial deployment of sink connector, processing flows as expected. If we pause the connector, wait, and then resume the connector we see an exponential increase in messages sent to PubSub. This is a high(er) volume connector (2.5-4MM messages per day) and when we paused we had about 2.5MM messages ready to be queued in Kafka (it had been paused for roughly 24 hours). Once we resumed the Consumer, PubSub non-Ack'd messages climbed to 9MM from 0 in 4 minutes time.

We believe this has something to do with the messages not being ACK'd from the connector and them being reprocessed immediately. This may be a configuration as we are using the default sink connector config from the repo, as well as the v.5 Alpha uber-jar.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions