Recovering Proxmox VMs from Encrypted Raid Array

Initial Thoughts: Blog Reboot

It has been a few years since this Blog has been active, or frankly seen the light of day, beyond some web caches I enabled awhile back. The biggest reason for the blog phasing out, was the first server I ever built (back in 2010) having a failing mother board shortly after the birth of our first child. Now, shortly after the birth of our second child and with the abundance of free time this crisis has awarded us all, I’ve decided to give the blog and my good old Proxmox server a reboot.

Recovering Raid Array

Now since I didn’t have the original working hardware and I didn’t retain good VM backups, I had to recover the VMs from just the encrypted drives. That meant the first step to beginning this journey of recovering Proxmox VMs, was dealing with an encrypted raid 1 storage array.

To begin this recovery process I simply put the two ”high-end” 120 GB SSD drives, in the new server I built to run the latest version of Proxmox. Next we need to boot up the server and use mdadm to examine the drives to identify the raid format.

lsblk # to list all the current block devices

mdadm -E /dev/sd[c-d] #use mdadm to examine 

Then we can use mdadm to re-assemble the raid. In this case, since the the drives are mirrors it likely doesn’t matter, but for some individuals this will likely be a required step in order to complete a full recovery.

mdadm --assemble /dev/md1 /dev/sd[c-d]1 # use mdadm to re-assemble
# the raid as md1

Decrypting LVM Structure

Next we need to decrypt the raid drive so we can access the logical volumes and mount them to begin recovering Proxmox VMs from the encrypted file system.

cryptsetup luksOpen /dev/md1 encrypted_pve # Open the luks encrytped device

Now that the the encrypted device has been opened and mounted as the device encrypted_pve. We can now access the volume group within the encrypted partition with standard LVM commands.

vgdisplay --short # for instance just viewing the volume group

Now if your somewhat lazy like me and doing all of this work on the a new installation of Proxmox. Your likely going to start having a bad time and encounter issues because by default, Proxmox wants its primary volume group to be named ‘pve’ and now there are two different ones with the same name. In order to avoid confusion going forward and stop our new Proxmox server from crashing, we should rename the old ‘pve’ volume group to something else.

vgdisplay | egrep -i "uuid|name|VG size" # use vgdisplay to 
# find the UUID of the VG

vgrename QiOPy3-WhF8-RYns-44dY-FKKN-8orV-bEi3m3 pve-old # use vgrename
# to simply rename the VG using the appropriate UUID

vgscan  # lastly we can use vgscan to reload all the vgs as a 
# sanity check

Recovering Proxmox VMs Config files

Now that we have access to the original LVM structure we can go about recovering VMs from the file system. To start this process we should mount the two logical volumes Proxmox has by default. These two would be the root and data logical on our old ‘pve’ volume group.

mkdir /mnt/old-data /mnt/old-root #create directories to mount
# logical volumes

mount /dev/mapper/pve--old-root /mnt/old-root # mount old pve root

mount/dev/mapper/pve--old-data /mnt/old-data # mount old pve data

Here is the point where things diverge a bit depending on what version of your recovering from and how your VMs where originally setup. Regardless there are really only two components to a given VM in the Proxmox world, a config file and a disk image.

To recover the VM config files in newer versions of Proxmox you would just go into the /mnt/old-root/etc/pve/qemu-server/ directory. There you would likely see a bunch of VM configuration file named after the VM ID number. Generally speaking you should be able to copy these configs over to your new Proxmox server without too much of an issue.

cd /mnt/old-root/etc/pve/qemu-server/ # go to the mounted config 
# directory

cp 100.conf /etc/pve/qemu-server/

However, if your like me and you are looking around for these VMID.conf files and they aren’t on the old root filesystem. That’s because in older versions Proxmox the VM configs were stored in a sqlite database. Instead we can use the sqlite3 command on the pve-cluster config database in order to view all the VM config files and extract the ones we want directly to workable config files.

sqlite3 /mnt/old-root/var/lib/pve-cluster/config.db \
'SELECT * FROM tree;' # use sqlite3 to view all of the 
# config data in the cluster config database

sqlite3 /mnt/old-root/var/lib/pve-cluster/config.db \
'SELECT data FROM tree WHERE name = "100.conf";' \
> /etc/pve/qemu-server/100.conf 
# use a sql query to extract just the config file data 
# we need and write it to the appropriate file/code>

Recovering Proxmox VMs Raw Disks and Disk Images

When it comes to VM disk images they can be either raw, meaning they are logical volumes provisioned within a volume group using LVM or disk image files like qcow2's. Regardless of which type of VM disk image or if like me and had both. The path forward is basically the same, just copy it over to appropriate place on the never server.

In the case of raw disks, all we really need to do is copy the logical volume that was provisioned in the old volume group over to the new volume group. To do this we need to create a new logical volume on the the new volume group with the same size and name. Then use dd to copy over the raw data from the old logical volume to the new one.

lsblk # look at all our physical and logical devices to
# make sure we use the right devices

lvcreate -n vm-100-disk-1 -L 10G pve # Create the new logical 
# volume with the same name and size

# dd if=/dev/pve-old/vm-100-disk-1 bs=4096 of=/dev/pve/vm-100-disk-1 
# Use dd to complete a bit by bit copy of the LV data, ! Caution !

If you need to deal with the VM disk files, its pretty straight forward as well. Just go into the old-data directory we mounted and copy the disk image files over to the new storage location. These can be a various formats like qcow2 or vmdk, but the process is the same.

mkdir /var/lib/vz/images/100 # create the folder for your VM

cp /mnt/old-data/images/100/vm-100-disk-1.qcow2  \
/var/lib/vz/images/100/vm-100-disk-1.qcow2 
# copy over the disk image to appropriate local folder

Getting VMs to Boot

At this point Proxmox should have seen the VM configuration files added to the local nodes configuration directory and it should be visible in web UI and/or qm at the command line. The very last step in recovering Proxmox VMs is making sure your VM configuration is correct so it can boot up. I could probably do research and a whole blog post on this topic alone. So its kind of difficult to provide detailed examples of what could be wrong with a given configuration file. Since the configuration files are from working VMs, most likely issues are either device statements or the boot order.

The boot order can be forced with the bootdisk option, to a given device, such as ide0. Devices are registered in the config file as device statements like 'ide0: local:vm-100-disk-1'. Make sure your device statements are correct and your cdrom is empty such as 'ide2: none,media=cdrom' to avoid boot issues. For other errors make be sure review the documentation or ask in the comments bellow.

Linux Administration Certifications: LPIC 1 and LFCS

LPIC 1 and LFCS

TLDR; The LPIC 1 and LFCS certifications can both be used to validate your skills, however the LFCS provides a robust and uniquely hands-one, testing approach.

I recently passed the LPIC 1 (Linux Professional Institute Certified System Administrator) and LFCS (Linux Foundation Certified System Administrator) certification exams. I’m now planning to pursue the LPIC 2 and LFCE certifications this coming year. Several individuals have approached me interested in hearing more about my experiences and some of big differences between the LPIC 1 and LFCS. I’ll attempt to address those questions here and also share my opinions on the perceived value in the market place today.

Big Differences

The biggest differences between the LPIC 1 and LFCS certifications, definitely come down to the testing methods they each use. The LPIC 1 is a standard multiple choice style examination, with a few fill the blank questions. The LPIC features two exams with 50 knowledge base and practical application question, over one and half hours. The LFCS on the other hand, is a interactive practical applications exam. Wherein the tester is given 40 practical multi-step tasks, within an actual Linux terminal, with two hours to complete as many as possible.

Another major difference between the LPIC 1 and LFCS is how the testing is conducted. The two LPIC 1 exams are proctored by Pearson Vue, so they take place in your standard testing center. Since it’s a standard multiple choice exam, in a standard testing center, you will receive your test results right after completing the exam. You are scored based on whether or not you select the correct answers to the exam questions and the respective weight in each of the tested categories. The LFCS is a online exam which utilizes a web cam, a screen share, a task portal, and a live connection to a Linux system to conduct the exam. Throughout the exam you have terminal access to your own Linux virtual machine, to complete your various tasks.  The entire system is graded upon completion and delays receiving your final score. Thus your score is based on whether each step of the tasks and the tasks themselves are completed correctly. Its also rumored, that points lost on one task can be recovered on others based on the methods used, cleanliness, and overall efficiency.

Difficulty Level

The difficulty level of the LPIC 1 and LFCS is heavily debated, however I think it comes down to how you study and your experience within the Linux terminal. That being said, the LPIC 1 is largely a test of base knowledge, so if one puts forth the time and effort to review some of the coursework out there, they shouldn’t have any problem completing the exam. I honestly don’t believe your experience in the Linux terminal is going to help you out anyone more then one of the official books. The exam is all about knowing the command names and what they do. On the other hand, the LFCS exam, and its largely based on weather or not you can complete a business operations related task, in a timely manner. There is no official book for the LFCS exam, although there is online coursework which introduces you to commands and then provides lab activities for completion all on your own. Having completed all of the online course work, I believe its likely sufficient to pass the exam. However I think the real world Linux experience would be quite a bit more useful during the LFSC exam, simple because your being indirectly scored on timeliness and efficiency. Addtionally, on top of having to understand what the names of commands are and what they do, one also needs to understand how to effectively use each command to successfully pass the LFSC exam. Overall I would say the LFCS is going to be far more difficult for those newer to Linux, if only because of more intimidating structure of the exam and the review of ones efficiency.

Market Value

When it comes to the market value of the LPIC 1 and LFCS certifications, I think the total value depends on your individual goals. For instances, if your goal is to get your foot in the door at a large institution, I would recommend the LPIC 1 since it has been around longer and thus has a greater chance of being recognized by a recruiter or HR. The LPIC 1 is also going to be better if your goal is to continue on and become more specialized within the Linux space. If your goal is instead to provide validation of your skills and experience to a future or current employer I would highly recommend the LFCS. In addition to the certification being run by the Linux Foundation themselves, they now have a partnership with Microsoft. This new partnership creates a great opportunity for those working within more diverse environments, by allowing for canadits to take both Linux foundation and Microsoft certifications to become specialized in mix environments and/or cloud. Overall I think if your really trying to project your worth to the market, the LPIC 1 is a better bet, simply because its been around longer and currently has more recognize then LFCS. However, I’d bet the LFCS will soon take its place at the top, due to the growing relationships being fostered by the Linux Foundation.

Honorable Mentions

Although I have not attempted these exams, because they are distribution specific, the OCA (Oracle Certified Administrator) and the RHCSA (Red Hat Certified System Administrator) both seem to have more visibility in the market place. This is likely due to the huge brand recognition associated with these respective certifications. If your already employed with an organization that mostly utilizes either of these distributions, they may provide more bang for your buck.

My Honest Advice About Information Security Education

Now that I am a security professional and mentor, there is one question that still haunts me to this day. It’s that simple moment when an aspiring young hacker (formal use, as in enjoys figure out how things work), looks to the future and asks for advice in furthering their education. The question is normally placed in the form of “Where should I go to school?” or “What college do you recommend”. I hesitate to answer these types of questions, because honestly my answer would be none. I find it hard to bring myself to recommend any of the schools I’ve gone to or have heard of to an InfoSec geek like myself. In fact I wish someone would have sat me down a few years ago and told me the truth about prospects for education. Instead, I got the same old sales pitches for degree’s that severe all your needs, from the big universities. So, for anyone out there who is looking for barely honest advice on the topic of information security education, I offer up my experiences in clear text for your consumption.

The first and last piece of advice I would give anyone is that you will need to do a substance amount of learning on your own time. I don’t care what college one goes to, or what program they belong to, they will not cover the material you need to know to work in the field. Most of the schools I’ve heard of do not offer any classes in information security and if they do, it’s a theory class that goes to about the level of a CompTIA Security+ certification. I think there are two main causes of these issues, based upon the colleges I’ve attended. First Is just fear, they are scared of teaching the “dark arts” to students, because of both the legality that exists and the fact that students may use their knowledge for unethical means. The second is the shear lack of student interest, accreditation, and standardization that has led to a truly sluggish development of InfoSec course work. That being said, there has been a noticeable push by several government bodies to form a basis for preparing security professionals for the future. This effort, known as Centers of Academic Excellence in Information Security, is in its infancy and still only has the backing of a few large, slow-moving institutions. So needless to say, it stands to be quite a while before we see fully developed information security degrees. In fact based on speaking with several universities, I don’t expect to see a fully accredited Information Security Bachelor’s degree from a top 10 school until around 2020.

That being said, there is a real need in the current education landscape to supplement your education, with additional work outside the class room. The good news is for us InfoSec geeks, there is a lot of free and cost effective training out there for us. The bad news is, you will have to prove you were engages in it and/or justify it to any future employer.  This learning commonly takes place online and can be video lectures like those found on Udemy.com and coursera.org. Others can be rather cost effective courses taught by security professionals online, like Georgia Weidman, Joseph McCray, and Marcus Carey. I also highly recommend going to as well InfoSec conferences as possible as well, as they are a wealth of information and networking opportunities for a future career. Many of these conferences are free of charge, the bsides events, and many others have limit student rate tickets as well. Also be sure to get involved in as many of the InfoSec competitions as possible to brush up your skills. Some may require you to be a full time student at an accredited university, but still do as many as possible. Some of the major competitions for college students are CCDC, CyberWars, and NCL.

The other huge source of knowledge is technical certifications. These certifications can be a great learning opportunity and great proof of knowledge to an employer. However, in the Information Security Industry not all companies give certifications the same weight. In fact some may require you have or acquire a certification, while others many not even give you a pat on the pack for them. In fact, many certification exams have been getting easier and more expensive over the years as demand has gone up. This devalues the certification itself, and makes the cost much higher. For this reason, I recommend not getting cert happy while in school. As a side bar, I will recommend taking all certifications for any courses that will gain you credit for passing the certification exam itself. This will save you a signification amount of money and give you something else to really shine on a resume. There is nothing wrong with doing the course work for a certification and explaining that to an employer.  They will understand and you will find that a lot of employers will pay for certifications when you are in the work force. As such I recommend doing just that, if you take a course and like it, find the certification and study the material. Simply state you completed the course work on your resume instead of the certification itself and explain the cost barrier to any prospective employer. This makes a nice conversation piece during the interview process and will help with the cost. Just do not feel the need to go out and get all the certifications, they are rather costly to maintain and may not produce the desired result.

Now I would not necessarily say there is a requirement in this industry, to go to a collegiate institution. However from personal experience I will say many large corporations will require at least a 4 year degree before they will consider you. Most of the time however, the hiring process comes down to what you have done in the past and how you present that information to a hiring manager. Never the less if you are looking for a college or need to go, here are some helpful questions to find the right place.

  1. 1.       Does the college offer a course over open source software and operating systems (linux)?

This question is where I think many colleges currently fall short. Most universities I talked to, during my search for a school, were all about teaching Windows, Cisco, and Java.  These technologies may be the most common, but they will severally limit your abilities when it comes to a career in the field. I put Linux in as an example, because if they do not even teach an introduction to Linux you are going to be really hurting. The truth of the matter is most security professionals use Linux every day and develop their tools for it as well.

  1. How many courses do you have that relate to the security, integrity, confidentiality and availability of computer networks?

This question is almost self-explanatory, it seeks to drill down and ask how many courses you will get that will be directly related to your field. This will hopefully give you some additional information about the types of course you will be taking and how they might relate to your prospects for a career.

  1. Is there a student organization on campus that focuses computing, security, InfoSec, or Cyber Defense? If so, do they engage in competitions?

It is very import to be involved with active student organizations that exist on a school’s campus. In my experience these organizations offer more of a learning experience then most classes. If a school does not have any such organization it most likely is not that school for you. If it does, try and set up a meeting with several of the members. It is just as important to make sure you will mesh with any groups that exist there. Also check and see what professional or national organizations meet on campus or nearby, these groups will offer great networking opportunities and can be a good source of knowledge as well. Most importantly, figure out if any of the organizations participate in collegiate competitions. If they don’t, check with the dean or a few professors about the policies on engaging in such events. You might find that some school will not support these academic competitions and in which case, the school is probably not a good choice.

  1. Is this school accredited or do they do research in the realm of Information security?

Once again this will really speak to the quality of any program that exists at the prospective institution. Just be sure to ensure that the school holds a current version of the accreditation that they claim they have. If, they are doing information security research ask about the projects they are working on and who is involved. If it’s a single grad student or a professor who has an interest in security and are doing some research it might not be a good reason to go. Also be sure to check that the research is current and producing useable results, there is no reason to go to an institution with bad research practices.

  1. Are there any courses that require a certification for completion or follow a certification track?

As I stated earlier in my tangent about certifications, they are a double edged sword. If the institution focuses too heavily on certifications it may not be the best choice. However, on the other hand if they encourage and support their students in acquiring certifications as opposed to requiring it, they may very well be a good choice. Be sure to ask if they school has a certification center on campus. If they do, be sure to ask which certification companies they are contacted with. Also ask if vouchers or reduced rates are available for students.

 

Once you have chosen a university or school the next step is making the most out of the experience.  Now there are several ways to maximize your learning, and I will continue to update this list with more over time, but here are my top few.

  1. Get involved with student organizations and always strive to make them that one step better.
  2. Try your best to pass your knowledge on to the other students and expect the same in return.
  3. Find the professors who encourage you to orient your assignments toward your prospective career and take as many classes with them as possible.
  4. If your professor does not allow you to orient your work or doesn’t let you do it your way (and you still fulfill the assignment criteria), do it anyway, and if issues arise go straight to the dean.
  5. Try to get local or regional security professionals to give a talk at your institution.
  6. Never forget to have fun or you will burn out.
  7. Find internship opportunities and do the work you love.