Linux Upskill Challenge
The Linux Upskill Challenge is a month-long course on Reddit (r/linuxupskillchallenge), about Linux Server Administration.
It is not a formal course, neither gives a certificate to share on social media. I think this makes the course more valuable, because the people taking it are doing to learn, and not to brag online.
It is a hands-on course with a lot of extra resources for the curious. Although I’ve been using GNU/Linux in my personal computer for a long time, I don’t consider myself an expert. I know my way around, but I’ve never dug deeper. This is a nice opportunity to learn more and also use a remote VPS for free. 😎
Daily logs
Here are some thoughts about each lecture. It is not meant to be a collection of study notes, as use a different system for that, but a brain dump of random ideas.
Days 0 and 1
Goal: get a remote server for free and learn how to manage it.
Decided to use Google Cloud Compute, just because. That dashboard is so chaotic… The most difficult task of the day was setting up the SSH public key to log to my new VM.
SSH keys are awesome. I’ve been using them for a long time for remote access
and they make life way easier. ssh-copy-id command is easier than
cat ~/.ssh/id_rsa.pub | ssh remote_username@server_ip_address "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys",
but for old servers is not an option. The machine I’m using at my university to
run the simulations for my Master’s studies did not accept ssh-copy-id…
Day 2
Goal: learn to navigate in the machine using only the terminal.
I’ve been using GNU/Linux for more than 10 years, I’m pretty confident with the
terminal. But there’s always something new to learn: ls -ltra is a nice
example to find the most recently altered files.
Day 3
Goal: get familiar with root things.
Mind blow of the day: I saw lots of attempts to connect to this
VM in /var/log/auth.log file. Awesome.
Bots everywhere.
I am running ArchLinux on my Raspberry Pi 3, it uses journalctl for the logs:
journalctl /usr/bin/sshd.
It was nice to learn about sudo -i. I’ve been using sudo su - to become
root and is good to know there are other ways to do it as well. The details
about the differences and other comments are explained
here.
Day 4
Goal: install stuff, explore the file structure and get familiar with “walking” around.
man hier blew my mind. Oh, this is awesome. I googled about the filesystem
hierarchy so many times and did not know I already had all the information in
front of me.
Day 5
Goal: get familiar with TAB, less, more, and hidden/dot files.
I was never a big fan of more. I normally use less as a pager, because it
allows me to scroll up and down easier, search for a string, jump to a
position, and quite fast for huge files.
The history command is nice. I know about it and never used directly. I
prefer to ctrl + r and a trick to use the Up/Down arrows to use the text that
has already been typed as the prefix for searching commands. Only works with
Bash, though. Check readline on
ArchWiki for more
details.
Day 6
Goal: learn Vim and how to exit it.
I’ve been using Vim for quite a long time. I use it for almost everything related to text: coding, writing my Master’s Thesis, writing TODO lists, writing these notes, and probably other things I can’t remember now. And despite those years of usage, every week I learn something new about it. Learning Vim is a hard task, but definitely worth it.
I started the vimtutor but never finished it. So many times. They say it
takes about half an hour to go through it, I’ll try to do it (again). Actually,
I’ll try neovim’s tutorial (:Tutor<Enter>).
The insane new thing about Vim that I learned today is that it has seven (!) basic modes:
- Normal mode
- Visual mode
- Select mode
- Insert mode
- Command-line mode (or Cmdline mode)
- Ex mode
- Terminal-Job mode
And then there are seven additional modes:
- Operator-pending mode
- Replace mode
- Virtual Replace mode
- Insert Normal mode
- Terminal-Normal mode
- Insert Visual mode
- Insert Select mode
You can read more about them with the internal help: :help vim-modes.
Curiously, neovim has one fewer additional mode (at least as of nvim 0.4.4).
Day 7
Goal: install a web server and use it to host a page.
The instructions are for Apache2, a.k.a. httpd, but I decided to use Caddy
instead. I’m curious about it’s automatic conversion from Markdown to HTML.
I got the binary from the Download page and
followed the official documentation to install it, creating a systemd service
file.
I am using this Caddyfile to serve a static-website:
:80 {
root * /var/lib/caddy/www/
encode zstd gzip
templates
file_server
}
And my /var/lib/caddy/www/index.html is a very complex one:
{ { include "index.md" | markdown }}
Afterthoughts
I should just follow the course. Lets play with Apache2. It is also simple, for some reason I expected it to be a quite complex to set up. On the other hand, I need to write html instead of Markdown :( but I can have a nice text only web page using only one tag:
<pre>
# Title
## Section
bla
Day 8
Goal: search for specific patterns in text files.
The Apache2 logs are in /var/log/apache2. To get all IP addresses that
connected to Apache:
$ sed -E 's/ (.*)//g' access.log | sort -n | uniq
27.123.241.20
45.148.121.28
45.148.121.85
54.93.196.88
62.33.138.6
66.249.75.147
66.249.75.149
74.120.14.52
75.76.251.71
91.193.5.58
92.118.160.17
99.10.249.153
103.49.188.166
116.28.214.153
125.41.10.86
125.43.93.10
125.47.64.145
139.162.119.197
142.118.76.99
173.174.64.235
174.51.18.48
177.76.230.216
179.99.69.186
186.71.77.9
192.241.239.84
193.118.53.210
194.87.138.60
209.126.3.185
209.17.97.34
Grepping /var/log/auth.log I can see 72 different IPs trying to log as
root and 10 other random users. Hahah, nice.
I liked to know that cut can also remove words from a string, I finally got
what the --fields option mean.
Day 9
Goal: First look at network security.
The first task is to find open ports. I’ve used nmap for that before and
let’s see what happens, when running it on the remote server:
$ nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2020-10-15 11:32 -03
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000086s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds
Nice, it can see ssh and something for http, that is apache2 on port 80
here. But it is interesting how different it is from my laptop:
$ nmap 35.228.166.73
Starting Nmap 7.80 ( https://nmap.org ) at 2020-10-15 11:32 -03
Nmap scan report for 73.166.228.35.bc.googleusercontent.com (35.228.166.73)
Host is up (0.25s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
19/tcp filtered chargen
22/tcp open ssh
25/tcp filtered smtp
80/tcp open http
5555/tcp filtered freeciv
Nmap done: 1 IP address (1 host up) scanned in 23.00 seconds
It finds three other filtered services. The manual says:
The state is either open, filtered, closed, or unfiltered. Open means that
an application on the target machine is listening for connections/packets
on that port. Filtered means that a firewall, filter, or other network
obstacle is blocking the port so that Nmap cannot tell whether it is open
or closed.
So, those ports might be or might not be open.
I tried to host something at those ports and could not get a connection from my laptop, so they are blocked by Google by default and there’s nothing (?) I can do. I can use all other ports though.
I’ve never used ufw before. It is time to learn
how to properly use a firewall and this one is simple and powerful enough,
although it uses iptables and not the “modern” nftables. I liked how simple
it is to allow or deny traffic in a port. But I can’t compare with other tools.
Changing my SSH port to a random number. I thought it was a bit pointless, but
apparently it
is not.
Using nmap, it shows the port open, but with unknown as the service. Sounds
good.
Day 10
Goal: run stuff periodically with cron jobs.
Lazy people automate stuff :)
systemd can also run jobs at specific times, using systemd/timers.
A fun cron job to add is to periodically change the SSH banner:
$ cat /etc/cron.hourly/banner
#!/bin/sh
/usr/games/fortune | /usr/games/cowsay > /etc/banner.txt
Day 11
Goal: searching for files and patterns.
I’ve been using blindly find for quite a while. It was time to finally read
it’s manual. There’s more to it than find . -iname "bla*".
zcat, zgrep, zless, and zmore are awesome! Did not know I could
cat/less/grep gzip compressed files.
Day 12
Goal: copy stuff around.
I like rsync to copy lots of small files (actually, also big and medium ones)
as it can create a “stream” of data instead of one new connection for each
file. It can also compress the files as it sends to the destination. This can
save a lot of time. I normally use rsync -azvP src dst meaning rsync --archive --compress --verbose --partial --progress src dst. Sometimes the
-P flag can be annoying, but YMMV.
Something funny I found out is that rsync shows the SSH banner of the remote
server:
$ rsync -avz linuxUpskill-gc:/var/www/html .
_________________________________________
/ Be free and open and breezy! Enjoy! \
| Things won't get any better so get used |
\ to it. /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
receiving incremental file list
html/
html/404.html
html/index.html
html/robots.txt
sent 104 bytes received 3,710 bytes 693.45 bytes/sec
total size is 11,324 speedup is 2.97
Day 13
Goal: familiarize with the Linux permission system.
It is way simpler to use u+w or o-rwx than playing around with octal
numbers for chmod. I didn’t know I could change the permission to more than
one letter of ugo, as in chmod go-wr [files], it never occurred to me to
try that either.
$ echo bla, blabla > file.txt
$ ls -la file.txt
-rw-rw-r-- 1 user user 12 Oct 21 12:11 file.txt
$ chmod ug-w file.txt
$ ls -la file.txt
-r--r--r-- 1 user user 12 Oct 21 12:11 file.txt
$ echo hehe >> file.txt
-bash: file.txt: Permission denied
Day 14
Goal: user management and permissions.
I always confuse useradd and adduser, after all they are anagrams.
This ServerFault
question has a nice answer about it:
adduser is a friendlier interactive frontend to useradd
I found out it is possible to enable offensive messages when an user type an
incorrect password when sudoing. Just add Defaults insults to
/etc/sudoers, obviously using visudo.
Nice to know that it is possible to enable specific commands for a user to run
with sudo without granting full sudo, like:
fred ALL = NOPASSWD:/sbin/reboot
fred ALL = NOPASSWD:/usr/bin/ls
This allows user fred to sudo ls and sudo reboot, without typing the
password, but does not allow any other command with sudo.
Day 15
Goal: package management, with apt.
I used a few different package managers before:
synaptic(when I used Kurumin, back then)apt-get(when I switched to Ubuntu)portage(when I used Gentoo)apt-get(again, briefly, when I used Debian in 2011)pacman(when I switched to ArchLinux)
I wish I remember the first time I tried Kurumin, but I don’t. I think I compiled Gentoo in 2008, I’m not sure. But I remember how fast my computer became. It was an AMD 32-bit single-core CPU, 1.2 GHz, 384 MiB RAM DDR, GeForce 4 MM 440, with hand soldered electrolytic capacitors in the motherboard (because they overheated and needed to be replaced). This computer was bought in ~2000, and when I finally managed to have a working Kernel, it needed ~12 seconds to fully boot! It had a very slow hard drive, slow RAM, slow CPU, and still was insanely fast! How I miss Gentoo…
ArchWiki has a Rosetta page, that “translates” commands from one package manager to other. It is a nice reference when dealing with different systems.
Day 16
Goal: managing tape archives, a.k.a tarballs.
Vim is always surprising me. Today I learned you can open file archives in Vim.
It is possible to open .zip, .tar.gz, .tar.bz, etc., and even modify its
content on the fly. No need to extract, edit, archive again.
I have a targz function in one of my dotfiles to create an archive of a bunch
of files or a directory. It’s time to get rid of it and learn how to properly
use tar and gzip/bzip.
Day 17
Goal: compiling stuff.
I liked to ./configure nmap:
$ CFLAGS="-O2 -march=native" CXXFLAGS="-O2 -march=native" ./configure
# skipping lots of lines...
. .
\`-"'"-'/
} 6 6 {
==. Y ,==
/^^^\ .
/ \ ) Ncat: A modern interpretation of classic Netcat
( )-( )/
-""---""--- /
/ Ncat \_/
( ____
\_.=|____E
Configuration complete.
( ) /\ _ (
\ | ( \ ( \.( ) _____
\ \ \ ` ` ) \ ( ___ / _ \
(_` \+ . x ( .\ \/ \____-----------/ (o) \_
- .- \+ ; ( O \____
(__ +- .( -'.- <. \_____________ ` \ /
(_____ ._._: <_ - <- _- _ VVVVVVV VV V\ \/
. /./.+- . .- / +-- - . (--_AAAAAAA__A_/ |
(__ ' /x / x _/ ( \______________//_ \_______
, x / ( ' . / . / \___' \ /
/ / _/ / + | \ /
' (__/ / \/
/ \
NMAP IS A POWERFUL TOOL -- USE CAREFULLY AND RESPONSIBLY
Cool ASCII art!
Day 18
Goal: cleaning-up old logs.
Logs can save your system. If there’s a weird behaviour somewhere, the first thing to check is the logs. But logs can also flood your system. In fact, this was what crashed some servers I was responsible for at my job and it was not nice. I wrote about that incident and how to tame Docker logs [here]({% post_url 2020-09-13-careful_with_docker_logs %}).
In /etc/cron.daily/logrotate there’s a line that checks if there are any
systemd/timers, if so, cron does nothing and let systemd do the job.
That’s clever. The timer is configured in
/etc/systemd/system/timers.target.wants/logrotate.timer.
logrotate’s log file is in a different path: /var/lib/logrotate/status.
That’s funny!
Day 19
Goal: overview of the Linux Virtual File System.
Inodes. Inodes everywhere. If I got it correctly, an inode is a bunch of information about the file/directory. It contains filename, file owner, file permissions, creation/modification times, where the file is on disk, and maybe more. So, it is metadata.
To find a file by it’s inode number, you have to scan to entire filesystem. The inode number is per filesystem, so there might be more than one file with the same number. To locate a file by its inode number, without traversing to a different filesystem:
$ sudo find / -mount -inum 1
/sys
/boot/efi
/proc
$ stat /sys /boot/efi /proc
File: /sys
Size: 0 Blocks: 0 IO Block: 4096 directory
Device: 18h/24d Inode: 1 Links: 13
Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-10-22 18:38:03.292000000 -0300
Modify: 2020-10-22 18:38:03.292000000 -0300
Change: 2020-10-22 18:38:03.292000000 -0300
Birth: -
File: /boot/efi
Size: 512 Blocks: 1 IO Block: 512 directory
Device: 80fh/2063d Inode: 1 Links: 4
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 1969-12-31 21:00:00.000000000 -0300
Modify: 1969-12-31 21:00:00.000000000 -0300
Change: 1969-12-31 21:00:00.000000000 -0300
Birth: -
File: /proc
Size: 0 Blocks: 0 IO Block: 1024 directory
Device: 5h/5d Inode: 1 Links: 120
Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-10-22 18:38:01.390534452 -0300
Modify: 2020-10-22 18:38:01.390534452 -0300
Change: 2020-10-22 18:38:01.390534452 -0300
Birth: -
I just found out, empirically, that there’s no file with inode 0. This number is reserved.
I think I finally got the difference between a hard and soft link. But I am definitely not sure about it. The following copy-pasta blob from a terminal might shred some light on it:
$ ls > /tmp/bla.txt
$ ln /tmp/bla.txt link_hard
$ ln -s /tmp/bla.txt link_soft
$ stat /tmp/bla.txt link_hard link_soft
File: /tmp/bla.txt
Size: 82 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 3122 Links: 2
Access: (0664/-rw-rw-r--) Uid: ( 1001/ h) Gid: ( 1002/ h)
Access: 2020-10-29 16:13:43.003122821 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:13:55.123855608 -0300
Birth: -
File: link_hard
Size: 82 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 3122 Links: 2
Access: (0664/-rw-rw-r--) Uid: ( 1001/ h) Gid: ( 1002/ h)
Access: 2020-10-29 16:13:43.003122821 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:13:55.123855608 -0300
Birth: -
File: link_soft -> /tmp/bla.txt
Size: 12 Blocks: 0 IO Block: 4096 symbolic link
Device: 801h/2049d Inode: 292697 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1001/ h) Gid: ( 1002/ h)
Access: 2020-10-29 16:14:14.681037964 -0300
Modify: 2020-10-29 16:14:03.624369542 -0300
Change: 2020-10-29 16:14:03.624369542 -0300
Birth: -
$ mv /tmp/bla.txt /tmp/bla2.txt
$ stat /tmp/bla2.txt link_hard link_soft
File: /tmp/bla2.txt
Size: 82 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 3122 Links: 2
Access: (0664/-rw-rw-r--) Uid: ( 1001/ h) Gid: ( 1002/ h)
Access: 2020-10-29 16:31:42.488385936 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:31:39.232189075 -0300
Birth: -
File: link_hard
Size: 82 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 3122 Links: 2
Access: (0664/-rw-rw-r--) Uid: ( 1001/ h) Gid: ( 1002/ h)
Access: 2020-10-29 16:13:43.003122821 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:14:38.554481313 -0300
Birth: -
File: link_soft -> /tmp/bla.txt
Size: 12 Blocks: 0 IO Block: 4096 symbolic link
Device: 801h/2049d Inode: 292697 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1001/ h) Gid: ( 1002/ h)
Access: 2020-10-29 16:14:14.681037964 -0300
Modify: 2020-10-29 16:14:03.624369542 -0300
Change: 2020-10-29 16:14:03.624369542 -0300
Birth: -
$ file link_*
link_hard: ASCII text
link_soft: broken symbolic link to /tmp/bla.txt
Day 20
Goal: ultimate laziness with scripts.
This is the last lesson of the Linux Upskill Challenge.
Scripting is an endless topic. As Bilbo would say, You start with a shebang, and if you don’t keep your feet, there is no knowing where you might be swept off to. This is a very powerful tool, but without paying attention to all details, a mistake can be the Destroyer of All Servers. Powerful, efficient, and simple.
Conclusions
I joined this course hoping I’d learn more about managing a server, in particular to improve my backup strategy. I ended up learning a lot more than I expected, not only about a VPS but also general Linux-related stuff.
It did not “consume” much time daily, it was very pleasant and time efficient. I liked how small, yet complete, each lecture was, and it was up to the learner to dig deeper as it pleased.
This course was totally worth it.
I have some final thoughts that don’t belong to any specific daily log, but are still relevant.
Never get a VPS on the other side
of the world, if possible. I decided to get one in Finland (because, why not?)
and I can feel the latency to type any commands in the terminal. I can type
normally, it is not that slow. But the typing latency is there. I can’t measure
it, but I can feel it. The average ping to this server is 297 ms, versus 19
ms for a server in Brazil (same country as I am now). 300 ms is definitely
something we can note, but it is not a problem for what I was doing.
It doesn’t matter how long ago you started using Vim (or Neovim), take some
time to go through vim-tutor. It takes about half an hour and it covers much
more than the basic usage. There are some great small new things to learn
there.
I recommend this course for anyone that wants to learn more about Linux, even if you are not planning to be a sysadmin. The lessons are very well written and there are many good tips to help a casual Linux user.