Disclaimer: We will use the term “k8s user” to refer to the Kubernetes control plane “user” (human or serviceaccount) who makes API calls. This user is different from the traditional Linux users and shall not be confused with the uids Falco logs today (e.g. user.uid, user.name or user.loginuid, see definitions on the Supported Fields for Conditions and Outputs page).

Motivation

When a user runs an interactive kubectl exec session like the following within a Kubernetes cluster:

$ kubectl exec -it nginx2-78848c9dcb-4ptf8 -- /bin/bash
root@nginx2-78848c9dcb-4ptf8:/# sleep 123123
^C
root@nginx2-78848c9dcb-4ptf8:/# exit

The built-in k8s audit controller will log an audit event with the command captured as follows:

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "RequestResponse",
    "auditID": "7b36f2ea-2e64-4b4a-a946-f387c75b7d72",
    "stage": "ResponseStarted",
    "requestURI": "/api/v1/namespaces/default/pods/nginx2-78848c9dcb-4ptf8/exec?command=%2Fbin%2Fbash&container=nginx&stdin=true&stdout=true&tty=true",
    "verb": "create",
    "user": {
        "username": "kubernetes-admin",
        "uid": "[user-id]",
        "groups": [
            "A",
            "B"
        ]
    },
    <...> # removed other parts not relevent
    "requestReceivedTimestamp": "2023-10-30T18:57:12.826789Z",
    "stageTimestamp": "2023-10-30T18:57:12.899734Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

If you see from the exec session above, the user ran sleep command on the k8s pod, but only /bin/bash is captured as part of the audit log entry, which creates a gap in k8s auditing solution where any other commands run in the interactive session won't be logged by the Kubernetes audit controller.

Note this auditing gap is called out and is out of scope by design for Kubernetes Auditing controller, k8s maintainers discussion here: kubernetes/kubernetes#29443 (comment)

This is where tools like Falco come in as they monitor syscalls. However, looking back to the new interactive shell that was spawned from the container entrypoint -- in Falco we won't see the Kubernetes control plane user (which can be a person or a serviceaccount), but the k8s audit log event does contain the user who made the API call (see the user.username or user.uid k8s fields from the Kubernetes Audit Logs). This is mainly because the Linux kernel does not know about anything that happens at layers above the kernel such as the Kubernetes control plane.

Knowing the user who made the k8s API calls is crucial for Incident Response and it would be nice to have the k8s user as part of each Falco log even though the Linux kernel does not know about it. This would enable us to do the following:

1. Disable specific user credentials as a response to an agent alert
   1. e.g. If there’s an agent alert about accessing a secrets file on pod, having correct k8s user will allow admins to quickly disable the credentials to stop an attack
2. Look up all commands executed by specific k8s user, this will allow for more accurate and complete forensics analysis in case of suspected compromised user credentials
   1. e.g. If a specific user or machine is suspected to have installed malicious software, having the ability to query all commands they have executed is very critical in understanding possible damage done.

Feature

One solution that can address the gap above is to provide the k8s user to the security agent running on the respective k8s cluster node. If the agent is able to correctly and securely fetch the k8s user of the user running the exec command at the time of when the new interactive shell is created from the container entrypoint, the agent can keep track of the k8s user and automatically attribute all sub-processes that the user runs in the interactive session with the same k8s user.

[1] The proposed solution involves modifying the k8s cluster, else there is nothing we can do.

We could take advantage of the mutation webhook functionality of k8s that would mutate the PodExecOptions request payload to modify the command parameter to include the k8s user extracted from the userInfo object that the API server includes in mutation AdmissionReview request.

Yes we would tamper with the container entrypoint command that will be run (drawbacks and alternatives to be discussed later).

For example:

Normally the PodExecOptions object looks like the following

        "object": {
            "kind": "PodExecOptions",
            "apiVersion": "v1",
            "stdin": true,
            "stdout": true,
            "tty": true,
            "container": "my-pod",
            "command": [
                "/bin/bash" # this is user controlled through `kubectl`
            ]
        }

After adding the mutation logic the new mutated object from API server will look like the following

        "object": {
            "kind": "PodExecOptions",
            "apiVersion": "v1",
            "stdin": true,
            "stdout": true,
            "tty": true,
            "container": "my-pod",
            "command": [
                "/bin/sh",
                "-c",
                "export SEC_USER_ID=12345;",
                "/bin/bash" # user provided command appended at end of list
            ]
        }

Note mutation above assumes that /bin/sh will always be available on target container, which is not always the case. There might be a way to remove this dependency by using the following pattern instead /usr/bin/env SEC_USER_ID=12345 /bin/bash which would rely on the env binary instead which is typically available on all POSIX systems.

[2] In Falco's event processing loop, we can parse the command line (proc.args) of execve* system calls to find special tokens that provide the Kubernetes user (e.g., SEC_USER_ID). Falco will then cache the k8s user using the regular threadtable caching mechanisms. This will allow all subsequent processes in the kubectl exec session to find the k8s user by traversing the parent process lineage, and as a result, include this information in Falco's output logs.