Recovering Proxmox VMs from Encrypted Raid Array

Initial Thoughts: Blog Reboot

It has been a few years since this Blog has been active, or frankly seen the light of day, beyond some web caches I enabled awhile back. The biggest reason for the blog phasing out, was the first server I ever built (back in 2010) having a failing mother board shortly after the birth of our first child. Now, shortly after the birth of our second child and with the abundance of free time this crisis has awarded us all, I’ve decided to give the blog and my good old Proxmox server a reboot.

Recovering Raid Array

Now since I didn’t have the original working hardware and I didn’t retain good VM backups, I had to recover the VMs from just the encrypted drives. That meant the first step to beginning this journey of recovering Proxmox VMs, was dealing with an encrypted raid 1 storage array.

To begin this recovery process I simply put the two ”high-end” 120 GB SSD drives, in the new server I built to run the latest version of Proxmox. Next we need to boot up the server and use mdadm to examine the drives to identify the raid format.

lsblk # to list all the current block devices

mdadm -E /dev/sd[c-d] #use mdadm to examine 

Then we can use mdadm to re-assemble the raid. In this case, since the the drives are mirrors it likely doesn’t matter, but for some individuals this will likely be a required step in order to complete a full recovery.

mdadm --assemble /dev/md1 /dev/sd[c-d]1 # use mdadm to re-assemble
# the raid as md1

Decrypting LVM Structure

Next we need to decrypt the raid drive so we can access the logical volumes and mount them to begin recovering Proxmox VMs from the encrypted file system.

cryptsetup luksOpen /dev/md1 encrypted_pve # Open the luks encrytped device

Now that the the encrypted device has been opened and mounted as the device encrypted_pve. We can now access the volume group within the encrypted partition with standard LVM commands.

vgdisplay --short # for instance just viewing the volume group

Now if your somewhat lazy like me and doing all of this work on the a new installation of Proxmox. Your likely going to start having a bad time and encounter issues because by default, Proxmox wants its primary volume group to be named ‘pve’ and now there are two different ones with the same name. In order to avoid confusion going forward and stop our new Proxmox server from crashing, we should rename the old ‘pve’ volume group to something else.

vgdisplay | egrep -i "uuid|name|VG size" # use vgdisplay to 
# find the UUID of the VG

vgrename QiOPy3-WhF8-RYns-44dY-FKKN-8orV-bEi3m3 pve-old # use vgrename
# to simply rename the VG using the appropriate UUID

vgscan  # lastly we can use vgscan to reload all the vgs as a 
# sanity check

Recovering Proxmox VMs Config files

Now that we have access to the original LVM structure we can go about recovering VMs from the file system. To start this process we should mount the two logical volumes Proxmox has by default. These two would be the root and data logical on our old ‘pve’ volume group.

mkdir /mnt/old-data /mnt/old-root #create directories to mount
# logical volumes

mount /dev/mapper/pve--old-root /mnt/old-root # mount old pve root

mount/dev/mapper/pve--old-data /mnt/old-data # mount old pve data

Here is the point where things diverge a bit depending on what version of your recovering from and how your VMs where originally setup. Regardless there are really only two components to a given VM in the Proxmox world, a config file and a disk image.

To recover the VM config files in newer versions of Proxmox you would just go into the /mnt/old-root/etc/pve/qemu-server/ directory. There you would likely see a bunch of VM configuration file named after the VM ID number. Generally speaking you should be able to copy these configs over to your new Proxmox server without too much of an issue.

cd /mnt/old-root/etc/pve/qemu-server/ # go to the mounted config 
# directory

cp 100.conf /etc/pve/qemu-server/

However, if your like me and you are looking around for these VMID.conf files and they aren’t on the old root filesystem. That’s because in older versions Proxmox the VM configs were stored in a sqlite database. Instead we can use the sqlite3 command on the pve-cluster config database in order to view all the VM config files and extract the ones we want directly to workable config files.

sqlite3 /mnt/old-root/var/lib/pve-cluster/config.db \
'SELECT * FROM tree;' # use sqlite3 to view all of the 
# config data in the cluster config database

sqlite3 /mnt/old-root/var/lib/pve-cluster/config.db \
'SELECT data FROM tree WHERE name = "100.conf";' \
> /etc/pve/qemu-server/100.conf 
# use a sql query to extract just the config file data 
# we need and write it to the appropriate file/code>

Recovering Proxmox VMs Raw Disks and Disk Images

When it comes to VM disk images they can be either raw, meaning they are logical volumes provisioned within a volume group using LVM or disk image files like qcow2's. Regardless of which type of VM disk image or if like me and had both. The path forward is basically the same, just copy it over to appropriate place on the never server.

In the case of raw disks, all we really need to do is copy the logical volume that was provisioned in the old volume group over to the new volume group. To do this we need to create a new logical volume on the the new volume group with the same size and name. Then use dd to copy over the raw data from the old logical volume to the new one.

lsblk # look at all our physical and logical devices to
# make sure we use the right devices

lvcreate -n vm-100-disk-1 -L 10G pve # Create the new logical 
# volume with the same name and size

# dd if=/dev/pve-old/vm-100-disk-1 bs=4096 of=/dev/pve/vm-100-disk-1 
# Use dd to complete a bit by bit copy of the LV data, ! Caution !

If you need to deal with the VM disk files, its pretty straight forward as well. Just go into the old-data directory we mounted and copy the disk image files over to the new storage location. These can be a various formats like qcow2 or vmdk, but the process is the same.

mkdir /var/lib/vz/images/100 # create the folder for your VM

cp /mnt/old-data/images/100/vm-100-disk-1.qcow2  \
/var/lib/vz/images/100/vm-100-disk-1.qcow2 
# copy over the disk image to appropriate local folder

Getting VMs to Boot

At this point Proxmox should have seen the VM configuration files added to the local nodes configuration directory and it should be visible in web UI and/or qm at the command line. The very last step in recovering Proxmox VMs is making sure your VM configuration is correct so it can boot up. I could probably do research and a whole blog post on this topic alone. So its kind of difficult to provide detailed examples of what could be wrong with a given configuration file. Since the configuration files are from working VMs, most likely issues are either device statements or the boot order.

The boot order can be forced with the bootdisk option, to a given device, such as ide0. Devices are registered in the config file as device statements like 'ide0: local:vm-100-disk-1'. Make sure your device statements are correct and your cdrom is empty such as 'ide2: none,media=cdrom' to avoid boot issues. For other errors make be sure review the documentation or ask in the comments bellow.

The Future of the CAPTCHA: Interactive Advertising

I’ve recently done quite a bit of research around using CAPTCHA’s to protect against automated form submissions on web pages. Unfortunately CAPTCHA’s are a thing of the past and it’s my belief that interactive advertising is next big alternative.

CAPTCHA’s have been common place on logon and registration forms for almost a decade now. However recent research conducted by industry leaders has shown that CAPTCHA’s are no longer very effective. In fact, there are several company’s that now offer CAPTCHA solving as a service, where humans will solve them for pennies. In fact just this year Google’s Recaptcha service moved from a user input driven model to a completely data analytic model for verification. This change was largely because of one of Google’s own research fellows released a paper describing how to programmatically beat the recaptcha over 70% of the time.

So what’s next? It’s my belief that one of the new interactive advertising firm on the internet will likely soon pull through as a leader in automated form submission protection. This may sound like an unusual combination, but it really does make since. With the payouts for conventional advertising online dropping rapidly, due to increased add relestate, site owners are looking for more ways to supplement hosting costs. Advertisers are also looking for new ways to engage their audience and new social science research shows that getting people to interact with your brand makes them more likely to buy.

I guess the the other question is will interactive advertising really protect against programmatic form submissions and increase the overall security of said website? I believe if site owns force users to participate in the add and only process form submissions once they have validated with the adds API. It should still stop rapid form submissions from being processed and make it significantly harder for attackers to find injection vulnerabilities.

The only question that remains to be answered is will these interactive advertising provide enough entropy and data analytics to ensure that they can’t easily be solved or bypassed.

So You Got a Shell: SSH for Linux Post Exploitation

So you worked hard to find that cool web application exploit and got a shell on the box. Many people not so familiar with the Linux operating system, quickly find themselves throwing there hand in the air and saying; Now what do I do? By popular demand, this is a list of some of my favorite SSH for Linux Post Exploitation techniques that I’ve used over and over in past Cyber Defense competitions and my career.

Note: I’m assuming that you will likely exploit a service level account, such as ftpd, www-data, ldapadmin, squid, apache or syslog, and not immediately have root level access on the system.

One of my overall favorites is utilizing SSH for linux post exploitation,  as a mean of developing access to systems on the internal network and to maintain access. Almost all Linux distro’s come with openssh-client baked in and many also have openssh-server set to run at startup by default. Fortunately, openssh’s config files ship with fairly basic security measures in place and expect system administrators to utilize the many additional security features, that are available, to defend systems. Most simply don’t bother.  Additionally the permissions on the /etc/ssh/ssh_conf and /etc/ssh/sshd_conf files are world readable by default. So start by just checking them out.

User$ cat /etc/ssh/ssh.conf

User$ cat /etc/ssh/sshd.conf

Within /etc/ssh/sshd_conf the default configuration allows for public file authentication; with the following line.

PubkeyAuthentication yes

This line allows for a public key to be passed to the system in order to authenticate as a specified user. This is done by adding your public key to the .ssh/authorized_keys file; under the users home directory structure. Once added you can use the simply ssh to the system, using your private key, while specifying the username.

Note: Most default system service accounts ship with /bin/false or /sbin/nologin as their default, which means ssh wont allow you to login directly via ssh.

So you are looking issued the command cat /etc/passwd and she that your user has no login shell and you say Michael this whole maintaining access with public keys in the authorized_keys file is useless to me. So how about another user of SSH for linux post exploitation; using your user to port forward into the local network over ssh?

So you get a web shell using some fancy new wordpress vulnerability and then you cat out the wp-config.php file and see that the database isn’t hosted on the local system. Now you have to pop another box to access that database right? Wrong. You can create a local port forward the the mysql port with ssh. by using the following command.

ssh -R 9000:<mysql server ip>:3306 www-data@localhost

So now maybe people are thinking of great I have to have a shell again…but www-data has /sbin/nologin! Thats fine we dont need a shell. Just follow the steps above. Use your web shell from the wordpress vulnerability to use ssh-keygen to generate the www-data users keys and .ssh directly. Then simply cat your public key into the .ssh/authorized_keys file, to give you a means of local authentication. You may not be able to get a local shell, but you can still ssh port forward by telling ssh not to use interactive login or allocate a tty with the following command.

ssh -v -nNT -R 9000:<mysql server ip>:3306 www-data@localhost

This will create an ssh tunnel that will port forward all traffic from 9000 to the mysql server on its default port. So simply issue the mysql command to connect on port 9000 and use the credentials in the wp-config.php file.

mysql -u wordpress -p -h <web server ip> -p 9000

Profit!

If you have any questions or want to know more just leave a comment or hit me up on social media.