Repository navigation
Multi Topic consumer not using client provided logger - MultiTopicsConsumerImpl:526 | Received Message from one of the topic #216
Description
Instantiating a multi topic consumer, using a client with provided logger, result undesired debug logging, which logs the a dump of the message. This can result in extremely high logging storage cost.
Here is how to reproduce :
...
pulsar_logger = logging.getLogger("pulsar")
pulsar_logger.setLevel(logging.INFO)
client = Client("your_pulsar_url", logger=pulsar_logger)
topics = ["topic1", "topic2", "topic3"]
consumer = client.subscribe(
topics,
subscription_name="subscription_name",
consumer_type=ConsumerType.KeyShared,
)
# generate some message using a producer
...What I expected to see :
No logging with log level lower than INFO
What I see :
Dump of every single message received by the consumer:
2024-07-10 16:13:22.773 DEBUG [137957839206144] MultiTopicsConsumerImpl:526 | Received Message from one of the topic - DUMP_OF_THE_MESSAGE
Environment :
pulsar-client==3.5.0 (python)
apachepulsar/pulsar-all:3.0.2 (docker image)
python3.9
Activity
BewareMyPower commentedon Jul 11, 2024
It seems that you need to set
logging.basicConfigfirst.In my local env, the following code print no logs:
However, after adding
logging.basicConfig(level=logging.INFO)before the 1st line, it worked.And I can also change the log level to bug with
pulsar_logger.setLevel(logging.DEBUG):But after removing the
basicConfigcall, the debug logs disappeared.Vincouux commentedon Jul 11, 2024
Thank you for your answer ! Passing a logger to Pulsar Client has always seem to work fine, and the associated level was always respected.
But for some reason, using a Multi Topic Consumer is not working as it should, and is logging this very specific DEBUG logging for every single message (
MultiTopicsConsumerImpl:526 | Received Message from one of the topic).Also, this is the only DEBUG statement being printed. That is why I was suspecting a bug in Pulsar Client. Another strange observation is that replacing this Multi Topic Consumer by a Single Topic Consumer (replacing the list of topics with a simple string), immediately remove this particular DEBUG statement.
I'm still not sure if it's a bug, a dependency race condition or simply an implementation issue on my end. For the time being, I'm replacing the Multi Topic Consumer with a home-made multi-threaded (multiple) Single Topic Consumers..
hugoallaine commentedon Sep 30, 2024
I have the same issue as @Vincouux mentioned using Python 3.12 and pulsar-client 3.5.0.
In my case, I have both file and console log but the issue affected the console log only, not the file.
stap-odin commentedon Oct 11, 2024
We found a workaround @hugoallaine , try using:
hugoallaine commentedon Oct 11, 2024
@stap-odin Thanks, this works well (and can be improved by using LoggerLevel from pulsar lib as a ConsoleLogger parameter if needed).