Skip to content

feat(supervisor): NFLOG-based bypass detection to drop the CAP_SYSLOG requirement #2382

Closed as not planned

Description

@gaveezy

Problem Statement

Sandbox pods/containers currently request CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_PTRACE, and CAP_SYSLOG. With Kubernetes user namespaces (GA in 1.36, already supported via enable_user_namespaces) the first three become namespaced and lose their host-level power — but CAP_SYSLOG does not. The kernel checks syslog(2)//dev/kmsg access against the initial user namespace (capable(), not ns_capable()), so in a hostUsers: false pod the granted CAP_SYSLOG is a no-op and the bypass detection monitor silently degrades.

The only consumer of CAP_SYSLOG is the bypass detection monitor in openshell-supervisor-process, which tails dmesg --follow for nftables log entries (prefix openshell:bypass:<ns>:) emitted by the per-workload netns REJECT rules. Reading the host-global kernel ring buffer from inside a sandbox is also a mild information leak, and requires the nf_log_all_netns sysctl on the node.

Proposed Design

Switch bypass log delivery from kernel-printk to nfnetlink_log (NFLOG):

  • During netns setup (privileged startup, before supervisor seccomp hardening), bind an AF_NETLINK/NETLINK_NETFILTER socket to an NFLOG group inside the workload netns. Netns affinity is fixed at socket() time; binding needs only CAP_NET_ADMIN over the netns owner, which works under user namespaces.
  • Generate the nftables log rules as log prefix <p> group <N> when the socket bind succeeds; keep the existing log ... flags skuid printk rules as an automatic fallback when it fails (e.g. kernel without nfnetlink_log).
  • The bypass monitor consumes NFLOG packet messages (prefix attribute for filtering, NFULA_UID for workload UID, raw IP payload for dst/ports/proto) and reports through the existing OCSF dual-emit path. The dmesg path remains as fallback (still useful in the VM driver, which runs with full capabilities).
  • Drop CAP_SYSLOG from the Kubernetes, Docker, and Podman driver capability sets.

Enforcement is unaffected: the nftables REJECT rules do not depend on the log delivery mechanism. NFLOG delivery is netns-scoped, so the monitor no longer reads host-global kernel logs, and nf_log_all_netns is no longer needed on the NFLOG path.

Alternatives Considered

  • Keep dmesg, relax kernel.dmesg_restrict on nodes — exposes host kernel logs to every pod on the node; strictly worse.
  • eBPF-based detection — bpf() is also initial-userns-gated (CAP_BPF/CAP_SYS_ADMIN in init ns), so it has the same problem as CAP_SYSLOG, plus a new dependency.
  • Netlink crate dependency (netlink-packet-netfilter / libnetfilter_log bindings) — the required protocol subset (group bind, copy mode, packet attr parsing) is ~stable kernel ABI and small; a hand-rolled libc implementation avoids new dependencies in a security-sensitive crate and matches its existing raw-FFI style.

Agent Investigation

  • CAP_SYSLOG's only consumer is bypass_monitor::spawn (crates/openshell-supervisor-process/src/bypass_monitor/mod.rs); drivers request it in openshell-driver-kubernetes/src/driver.rs, openshell-driver-docker/src/lib.rs, openshell-driver-podman/src/container.rs.
  • No eBPF usage anywhere in the workspace; netns/nftables management shells out to iproute2/nft, so NFLOG covers the whole detection path.
  • The supervisor seccomp prelude does not restrict socket(); the workload runtime filter blocks non-NETLINK_ROUTE netlink sockets, so the sandboxed workload cannot bind the NFLOG group itself.
  • The sidecar topology has no bypass log listener today (no CAP_SYSLOG there either); it keeps printk rules unchanged.
  • Implementation is complete on a branch with unit tests (netlink message layout, datagram parsing, IPv4/IPv6 packet parsing, ruleset generation); Linux cargo check/clippy (pedantic+nursery)/tests pass. PR to follow.

Activity

  1. added
    area:sandboxSandbox runtime and isolation work
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Jul 22, 2026
  2. drew commented on Jul 22, 2026

    @drew
    Collaborator

    📋 triage-agent

    Triage Assessment

    Classification: feature-valid

    Summary

    The capability problem is real and the NFLOG proposal is technically sound. Current code tails dmesg --follow for bypass diagnostics while Kubernetes, Docker, and Podman request CAP_SYSLOG; under a pod user namespace the kernel permission check still uses capable(CAP_SYSLOG), so that grant cannot authorize access to the host-global kernel log.

    Investigation

    • crates/openshell-supervisor-process/src/bypass_monitor/mod.rs checks that dmesg exists, then runs dmesg --follow with stderr suppressed. An EPERM therefore ends monitoring without surfacing the permission failure.
    • crates/openshell-supervisor-process/src/netns/nft_ruleset.rs keeps LOG commands non-required and REJECT commands required, so changing log delivery does not weaken enforcement.
    • crates/openshell-supervisor-process/src/netns/mod.rs currently enables nf_log_all_netns, confirming the host kernel-log dependency.
    • The three combined/container drivers request SYSLOG; no other runtime consumer was found.
    • The architecture supports the proposal: network namespace setup occurs before supervisor startup hardening, and the workload seccomp policy blocks non-route netlink sockets. A supervisor-owned NFLOG socket can therefore be created during privileged setup and consumed afterward.
    • No duplicate or merged fix was found. Proposal: Split Supervisor and Agent into Separate Pods with gVisor Isolation #981 is broader architecture work. PR feat(supervisor): deliver bypass detection logs via NFLOG and drop CAP_SYSLOG #2383 implements this proposal but was self-closed, unreviewed, and unmerged.

    Recommendation

    Proceed as a valid supervisor/sandbox security improvement. Before merge, require live-kernel E2E coverage that proves nft log group packet delivery and behavior when NFLOG support is unavailable. Also describe the fallback precisely: after removing CAP_SYSLOG, ring-buffer fallback cannot restore diagnostics in restricted Kubernetes, Docker, or Podman environments; it remains useful only where kernel-log access is independently available, such as the VM path.

    Routing applied: area:supervisor, area:sandbox, and topic:security; removed state:triage-needed. state:agent-ready remains a human-only decision.

  3. gaveezy commented on Jul 22, 2026

    @gaveezy
    Author

    @drew thanks for the triage, I will submit the PR for the implementation soon

  4. drew commented on Jul 22, 2026

    @drew
    Collaborator

    Before implementing, can you talk a little more about how this would get configured and when we would use it? If understand this replaces an existing implementation and is used for all drivers.

  5. gaveezy commented on Jul 23, 2026

    @gaveezy
    Author

    @drew for the VM and Docker backend, no configuration changes, for Kubernetes if we want to opt-in user namespace then need to set pod.spec.hostUsers field to false

  6. github-actions commented on Aug 7, 2026

    @github-actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  7. github-actions commented on Oct 13, 2026

    @github-actions

    Closing this issue after 14 days with no activity since it was marked stale. Reopen it if the work is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:sandboxSandbox runtime and isolation workarea:supervisorProxy and routing-path workstate:staleInactive item at risk of automatic closure.topic:securitySecurity issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions