JournalHandler: Sanitize extra fields to be valid journald fields #69

I noticed an incontinence with JournalHandler when using it in an existing code base. I already use the Python logging module and all my logging already includes structured data like this:

LOG.info('test', extra={'structured_data': 23})

The structured_data field does not make it into journald and is silently omitted. I checked http://github.com/mosquito/cysystemd and it has sanitation code which works for me:

http://github.com/mosquito/cysystemd/blob/e0acede93387d51e4d6b20fdc278b2675052958d/cysystemd/_journal.pyx#L26-L34

As the status of this project and relationship with http://github.com/mosquito/cysystemd is not clear I just wanted to document this here.

Edit:

I ended up using python-systemd instead of cysystemd mainly because of the less cluttered fields it outputs so I needed to work around the issue described here. I needed to workaround another limitation which is that journald cannot handle nested fields so I combined the two workarounds into this (also I want to log sets so it includes a workaround for this as well):

import logging
import json

from systemd.journal import JournalHandler


class SetJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, set):
            return list(obj)
        return json.JSONEncoder.default(self, obj)


class LogExtraAsJsonDataAdapter(logging.LoggerAdapter):
    def process(self, msg, kwargs):
        if 'extra' in kwargs:
            kwargs['extra'] = {'JSON_SD': json.dumps(kwargs['extra'], cls=SetJSONEncoder)}
        return msg, kwargs


_LOG = logging.getLogger(__name__)
_LOG.addHandler(JournalHandler())
LOG = LogExtraAsJsonDataAdapter(_LOG, {})

The idea here is that Journalbeat sends those logs to Logstash where JSON_SD is JSON decoded and included into the log event.