Repository navigation
python_template_launcher.go:37: Failed to initialize fluentd logger #13213
Description
In which file did you encounter the issue?
python-docs-samples/dataflow/flex-templates/pipeline_with_dependencies
Did you change the file? If so, how?
Yes, used a different base image in docker file
Describe the issue
Jon start but I got following error:
2025/03/06 17:38:45.885541 python_template_launcher.go:37: Failed to initialize fluentd logger: failed to create log file /var/log/dataflow/template_launcher/runner-json.log: open /var/log/dataflow/template_launcher/runner-json.log: permission denied
python_template_launcher fails if running with non root user or an user that have no write access to /var/log
Activity
iennae commentedon Mar 8, 2025
Thanks @pbonito. Can you share more details about your Dockerfile configuration? which base image are you using?
pbonito commentedon Mar 8, 2025
This is an example, my real image is hardened and running with nonroot user and no access to /var/log.
FROM nvcr.io/nvidia/cuda:12.6.1-cudnn-devel-ubuntu24.04
ARG WORKDIR=/template
WORKDIR ${WORKDIR}
COPY requirements.txt .
COPY .py ./
RUN apt-get update
# Install Python and other system dependencies.
&& apt-get install -y --no-install-recommends software-properties-common
&& add-apt-repository ppa:deadsnakes/ppa -y
&& apt-get install -y --no-install-recommends
curl g++ python3.11-dev python3.11-venv apt-transport-https ca-certificates gnupg
&& rm -rf /var/lib/apt/lists/
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 11
&& curl https://bootstrap.pypa.io/get-pip.py | python
# Install the pipeline requirements.
&& pip install --no-cache-dir -r requirements.txt
&& pip check
COPY --from=gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:2.61.0 /opt/apache/beam /opt/apache/beam
COPY --from=gcr.io/dataflow-templates-base/python311-template-launcher-base:latest /opt/google/dataflow/python_template_launcher /opt/google/dataflow/python_template_launcher
COPY main.py .
COPY pyproject.toml .
COPY setup.py .
COPY src src
RUN pip install .
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
RUN pip freeze
RUN pip list
ENTRYPOINT [ "/opt/apache/beam/boot" ]
iennae commentedon Mar 13, 2025
@pbonito thanks so much for the information. I can see the challenge with the current sample for custom Docker images. We're investigating what the right path is here. If you have a support contract with Google, please create an issue in the support console as well. Thank you!