[OCTRL-1102] run dpl in k8s from ECS

control-operator/README.md

@@ -1,12 +1,12 @@
 # operator
 
-Folder with operators regarding Task and Environment deployment.
+Folder with operators for Task and Environment deployment. For a better understanding of how the operators are used by ECS and the Control repository, see [docs/kubernetes_ecs.md](../docs/kubernetes_ecs.md).
 
 ## Description
 
-In order to deploy Task and Environment workflows to the k8s cluster you need controllers and operators
-controlling custom CRDs defining ALICE custom workload. This Folder defines and implements all moving parts together with Makefile
-to build, deploy, install CRDs and operators.
+In order to deploy Task and Environment workflows to the k8s cluster, you need controllers and operators
+controlling the CRDs that define ALICE custom workloads. This folder defines and implements all the moving parts, together with a Makefile
+to build, deploy, and install the CRDs and operators.
 
 ## Architecture
 
@@ -18,42 +18,54 @@ The operator is split into two separate binaries with different deployment strat
 
 ## Getting Started
 
-You’ll need a Kubernetes cluster to run against. You can use [KIND](https://sigs.k8s.io/kind) to get a local cluster for testing, or run against a remote cluster. Author had the most success with K3s [see](/docs/kubernetes_ecs.md).
-**Note:** Your controller will automatically use the current context in your kubeconfig (usually ~/.kube/config) file (i.e. whatever cluster `kubectl cluster-info` shows).
+You’ll need a Kubernetes cluster to run against. You can use [KIND](https://sigs.k8s.io/kind) to get a local cluster for testing, or run against a remote cluster. The author had the most success with K3s ([see](/docs/kubernetes_ecs.md)).
+**Note:** Your controller will automatically use the current context in your kubeconfig file (usually `~/.kube/config`) (i.e. whatever cluster `kubectl cluster-info` shows).
 
 ### Running on the cluster
 
-Following commands show basic use of Makefile. However this isn't exhaustive list.
+#### Namespaces
 
-1. Install Instances of Custom Resources:
+This project expects two namespaces: `alice-tasks` and `operator-system`. `alice-tasks` is where the actual `Task` resources run, and it must be created before the first Task is deployed. `operator-system` is where all operator pods run, and it is created during deployment to the cluster.
+
+#### Secrets
+
+When deploying operator containers, you are required to set up proper secrets to access the registry where the operator images reside. At the moment they are built by a GitHub pipeline and pushed to the AliceO2Group GitHub [package registry](https://github.com/orgs/AliceO2Group/packages).
+
+The second required secret must be set up to access images used by `Task` CRs. At the moment these images are located in a GitLab registry, and the secret name is hardcoded in this repository as `gitlab-registry-secret`.
+
+#### make commands
+
+The following commands show basic usage of the Makefile. However, this isn't an exhaustive list.
+
+- Install instances of Custom Resources:
 
 ```sh
 kubectl apply -f config/samples/
 ```
 
-1. Build and push your images. Default image tags are defined in the Makefile via `TASK_IMG` and `ENVIRONMENT_IMG`. Override them only if you want to use a different registry or tag:
+- Build and push your images. Default image tags are defined in the Makefile via `TASK_IMG` and `ENVIRONMENT_IMG`. Override them only if you want to use a different registry or tag:
 
 ```sh
 make docker-build docker-push TASK_IMG=<some-registry>/task-manager:tag ENVIRONMENT_IMG=<some-registry>/environment-manager:tag
 ```
 
-1. Deploy the controllers to the cluster. Uses the same `TASK_IMG` and `ENVIRONMENT_IMG` defaults, override them if needed. Images are pulled using the secret named by `PULL_SECRET` (default `ghcr-pull`), which must exist in the `operator-system` namespace:
+- Deploy the controllers to the cluster. Uses the same `TASK_IMG` and `ENVIRONMENT_IMG` defaults; override them if needed. Images are pulled using the secret named by `PULL_SECRET` (default `ghcr-pull`), which must exist in the `operator-system` namespace:
 
 ```sh
 make deploy TASK_IMG=<some-registry>/task-manager:tag ENVIRONMENT_IMG=<some-registry>/environment-manager:tag PULL_SECRET=<secret-name>
 ```
 
 ### Uninstall CRDs
 
-To delete the CRDs from the cluster:
+To delete the CRDs from the cluster (careful: this also deletes all `Task`, `Environment` and `TaskTemplate` resources in the whole cluster):
 
 ```sh
 make uninstall
 ```
 
 ### Undeploy controller
 
-UnDeploy the controller from the cluster:
+Undeploy the controllers from the cluster (careful: this deletes the whole `operator-system` namespace, the RBAC resources and the CRDs together with all `Task`, `Environment` and `TaskTemplate` resources; this also applies to `make undeploy-task` and `make undeploy-environment` on their own):
 
 ```sh
 make undeploy
@@ -90,7 +102,7 @@ The task-manager requires a `NODE_NAME` environment variable to know which node
 NODE_NAME=<your-node-name> make run-task
 ```
 
-**NOTE:** You can also install CRDs and run in one step: `make install run-environment` or `make install run-task`
+**NOTE:** You can also install the CRDs and run in one step: `make install run-environment` or `make install run-task`.
 
 ### Modifying the API definitions
 
@@ -100,9 +112,9 @@ If you are editing the API definitions, generate the manifests such as CRs or CR
 make manifests
 ```
 
-**NOTE:** Run `make --help` for more information on all potential `make` targets
+**NOTE:** Run `make --help` for more information on all available `make` targets.
 
-More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html)
+More information can be found in the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html).
 
 ## License
 

control-operator/api/v1alpha1/environment_types.go

@@ -81,6 +81,7 @@ type EnvironmentStatus struct {
 type TaskReference struct {
 	Name           string            `json:"name"`
 	TaskID         string            `json:"taskID,omitempty"`
+	NameSuffix     string            `json:"nameSuffix,omitempty"`
 	Env            []v1.EnvVar       `json:"env"`
 	ArgsCLI        []string          `json:"argsCLI"`
 	ArgsTransition map[string]string `json:"argsTransition,omitempty"`

control-operator/config/crd/bases/aliecs.alice.cern_environments.yaml

@@ -4100,6 +4100,8 @@ spec:
                         type: array
                       name:
                         type: string
+                      nameSuffix:
+                        type: string
                       taskID:
                         type: string
                     required:

control-operator/ecs-manifests/control-workflows/stfbuilder-kube-direct.yaml

@@ -0,0 +1,65 @@
+name: stfbuilder
+control:
+  mode: kubernetes_fairmq
+defaults:
+  detector: TST
+  rdh_version: "6"
+  user: flp
+  fmq_severity: info
+  dd_detector_subspec: feeid
+  dd_discovery_endpoint: "no-op://" # location of consul docker instance
+  log_task_stdout: none
+  log_task_stderr: none
+  stfb_dd_mode: "physics"
+  stfb_fee_mask: "0xffff"
+  stfb_enable_datasink: "false"
+  stfb_datasink_dir: "/tmp"
+  stfb_datasink_filesize: "2048"
+  monitoring_dd_interval: "5"
+  _module_cmdline: >-
+    source /etc/profile.d/modules.sh && MODULEPATH={{ modulepath }} module load DataDistribution Control-OCCPlugin &&
+    StfBuilder
+  _plain_cmdline: "{{ o2_install_path }}/bin/StfBuilder"
+wants:
+  cpu: 0.15
+  memory: 128
+bind:
+  - name: dpl-chan
+    type: push
+    rateLogging: "{{ fmq_rate_logging }}"
+    transport: shmem
+    addressing: ipc
+    sndBufSize: "4"
+    global: "readout-proxy-{{ it }}"
+command:
+  env:
+    - O2_INFOLOGGER_MODE={{infologger_mode}}
+    - O2_DETECTOR={{ detector }}
+    - DATADIST_FEE_MASK={{ stfb_fee_mask }}
+    - O2_PARTITION={{ environment_id }}
+  stdout: "{{ log_task_stdout }}"
+  stderr: "{{ log_task_stderr }}"
+  shell: true
+  user: "{{ user }}"
+  value: "{{ len(modulepath)>0 ? _module_cmdline : _plain_cmdline }}"
+  arguments:
+    - "--session=default"
+    - "--shm-segment-id=2"
+    - "--shm-segment-size=33554432"
+    - "--transport=shmem"
+    - "--monitoring-backend={{ monitoring_dd_url }}"
+    - "--monitoring-process-interval={{ monitoring_dd_interval }}"
+    - "--discovery-partition={{ environment_id }}"
+    - "--discovery-endpoint={{ dd_discovery_endpoint }}"
+    - "--detector-rdh={{ rdh_version }}"
+    - "--detector-subspec={{ dd_detector_subspec }}"
+    - "--severity={{ fmq_severity }}"
+    - "--severity-infologger={{ infologger_severity }}"
+    - "--dpl-channel-name=dpl-chan"
+    - "--shm-monitor=false"
+    - "--detector={{ detector }}"
+    - "--run-type={{ stfb_dd_mode }}"
+    - "{{ stfb_enable_datasink == 'true' ? '--data-sink-enable' : ' ' }}"
+    - "{{ stfb_enable_datasink == 'true' ? '--data-sink-dir=' +  stfb_datasink_dir : ' ' }}"
+    - "{{ stfb_enable_datasink == 'true' ? '--data-sink-max-stfs-per-file=0' : ' ' }}"
+    - "{{ stfb_enable_datasink == 'true' ? '--data-sink-max-file-size=' +  stfb_datasink_filesize : ' ' }}"

control-operator/ecs-manifests/kubernetes-manifests/dpl-qccheck-test.yaml

@@ -0,0 +1,110 @@
+apiVersion: aliecs.alice.cern/v1alpha1
+kind: Task
+metadata:
+  labels:
+    environment: 34xmvqdqcsu
+    node: mtichak-ost
+    taskID: 34XmVrU4GeE
+  name: mtichak-ost-dpl-qc-check-daq-qccheck
+  namespace: alice-tasks
+spec:
+  control:
+    mode: fairmq
+    port: 31000
+  nodeName: mtichak-ost.cern.ch
+  pod:
+    containers:
+    - args:
+      - source /etc/profile.d/o2.sh &&  o2-dpl-raw-proxy -b --session default --dataspec
+        'x:TST/RAWDATA;dd:FLP/DISTSUBTIMEFRAME/0' --inject-missing-data --readout-proxy
+        '--channel-config "name=readout-proxy,type=pull,method=connect,address=ipc:///tmp/stf-builder-dpl-pipe-0,transport=shmem,rateLogging=10"'
+        | o2-qc -b --config apricot://mtichak-ost.cern.ch:32188/o2/components/qc/ANY/any/stfb_to_daqtask-mtichak-ost
+        | o2-dpl-output-proxy --environment DPL_OUTPUT_PROXY_ORDERED=1 -b --session
+        default --dataspec 'x:TST/RAWDATA;dd:FLP/DISTSUBTIMEFRAME/0' --dpl-output-proxy
+        '--channel-config "name=downstream,type=push,method=bind,address=ipc:///tmp/stf-pipe-0,rateLogging=10,transport=shmem"'
+        | o2-qc -b --exit-transition-timeout '40' --data-processing-timeout '20' --monitoring-backend
+        'influxdb-unix:///tmp/telegraf-dpl.sock' --session 'default' --infologger-severity
+        'info' --infologger-mode 'infoLoggerD' --driver-client-backend 'stdout://'
+        --shm-segment-size '10000000000' --shm-throw-bad-alloc 'false' --resources-monitoring
+        '15' --id 'qc-check-DAQ-QcCheck' --shm-monitor 'false' --log-color 'false'
+        --batch --aod-file '' --aod-memory-rate-limit '0' --aod-writer-json '' --aod-writer-keep
+        '' --aod-writer-maxfilesize '0' --aod-writer-ntfmerge -1 --aod-writer-resdir
+        '' --aod-writer-resfile '' --aod-writer-resmode 'RECREATE' --bad-alloc-attempt-interval
+        '50' --bad-alloc-max-attempts '1' --channel-prefix '' --clone '' --config
+        'apricot://mtichak-ost.cern.ch:32188/o2/components/qc/ANY/any/stfb_to_daqtask-mtichak-ost'
+        --configKeyValues '' --early-forward-policy 'never' --fairmq-ipc-prefix '@'
+        --fairmq-rate-logging '0' --fairmq-recv-buffer-size '4' --fairmq-send-buffer-size
+        '4' --forwarding-destination 'drop' --forwarding-policy 'dangling' --host
+        '' --io-threads '1' --labels '' --local-batch '' --override-values '' --pipeline
+        '' --readers '1' --remote-batch '' --severity 'info' --shm-allocation 'rbtree_best_fit'
+        --shm-metadata-msg-size '0' --shm-mlock-segment 'false' --shm-mlock-segment-on-creation
+        'false' --shm-no-cleanup 'false' --shm-segment-id '0' --shm-zero-segment 'false'
+        --signposts '' --spawners '1' --stacktrace-on-signal 'simple' --timeframes-rate-limit
+        '0' --timeframes-rate-limit-ipcid -1 --workflow-suffix '' --qcConfiguration
+        '{}' --runNumber '' -S $CONTROL_OCCPLUGIN_ROOT/lib/ -P OCClite --color false
+        --control-port 31000
+      command:
+      - bash
+      - -c
+      env:
+      - name: O2_DETECTOR
+        value: TST
+      - name: O2_PARTITION
+        value: 34XmVqDQCsU
+      - name: HOME
+        value: /tmp
+      - name: O2_SYSTEM
+        value: FLP
+      - name: O2_ROLE
+        value: mtichak-ost
+      - name: OCC_CONTROL_PORT
+        value: "31000"
+      image: gitlab-registry.cern.ch/pkonopka/dockerfiles/flp-sw:20260729-1
+      imagePullPolicy: IfNotPresent
+      name: dpl
+      resources: {}
+      securityContext:
+        privileged: true
+        runAsGroup: 1100
+        runAsUser: 1100
+      volumeMounts:
+      - mountPath: /dev/shm
+        name: host-shm
+      - mountPath: /etc/group
+        name: group
+        readOnly: true
+      - mountPath: /etc/passwd
+        name: passwd
+        readOnly: true
+      - mountPath: /tmp
+        name: tmp
+      - mountPath: /lib/modules
+        name: modules
+    hostIPC: true
+    hostNetwork: true
+    imagePullSecrets:
+    - name: gitlab-registry-secret
+    nodeName: mtichak-ost.cern.ch
+    securityContext:
+      fsGroup: 1100
+      supplementalGroups:
+      - 10
+      - 1105
+    volumes:
+    - hostPath:
+        path: /dev/shm
+        type: Directory
+      name: host-shm
+    - hostPath:
+        path: /etc/group
+      name: group
+    - hostPath:
+        path: /etc/passwd
+      name: passwd
+    - hostPath:
+        path: /tmp
+      name: tmp
+    - hostPath:
+        path: /lib/modules
+      name: modules
+  state: standby

control-operator/ecs-manifests/task-templates/dpl-tasktemplate.yaml

@@ -0,0 +1,62 @@
+apiVersion: aliecs.alice.cern/v1alpha1
+kind: TaskTemplate
+metadata:
+  name: dpl
+  namespace: alice-tasks
+spec:
+  envVars:
+    - O2_DETECTOR
+    - O2_PARTITION
+    - OCC_CONTROL_PORT
+    - O2_SYSTEM
+    - O2_ROLE
+  pod:
+    hostNetwork: true
+    hostIPC: true
+    securityContext:
+      fsGroup: 1100
+      supplementalGroups: [10, 1105]
+    containers:
+      - name: dpl
+        image: gitlab-registry.cern.ch/aliceo2group/dockerfiles/alma9-flp-node/o2suite:20261001-1
+        command: ["bash", "-c"]
+        securityContext:
+          privileged: true
+          runAsUser: 1100
+          runAsGroup: 1100
+        volumeMounts:
+          - name: host-shm
+            mountPath: /dev/shm
+          - name: group
+            mountPath: /etc/group
+            readOnly: true
+          - name: passwd
+            mountPath: /etc/passwd
+            readOnly: true
+          - name: tmp
+            mountPath: /tmp
+          - name: modules
+            mountPath: /lib/modules
+        imagePullPolicy: IfNotPresent
+    volumes:
+      - name: host-shm
+        hostPath:
+          path: /dev/shm
+          type: Directory
+      - name: group
+        hostPath:
+          path: /etc/group
+      - name: passwd
+        hostPath:
+          path: /etc/passwd
+      - name: tmp
+        hostPath:
+          path: /tmp
+      - name: modules
+        hostPath:
+          path: /lib/modules
+    imagePullSecrets:
+      - name: gitlab-registry-secret
+  control:
+    mode: "fairmq"
+    # port: ${OCC_CONTROL_PORT} # to be filled in automatically by TaskController