Heitor's log


Linux Upskill Challenge


The Linux Upskill Challenge is a month-long course on Reddit (r/linuxupskillchallenge), about Linux Server Administration.

It is not a formal course, neither gives a certificate to share on social media. I think this makes the course more valuable, because the people taking it are doing to learn, and not to brag online.

It is a hands-on course with a lot of extra resources for the curious. Although I’ve been using GNU/Linux in my personal computer for a long time, I don’t consider myself an expert. I know my way around, but I’ve never dug deeper. This is a nice opportunity to learn more and also use a remote VPS for free. 😎

Daily logs

Here are some thoughts about each lecture. It is not meant to be a collection of study notes, as use a different system for that, but a brain dump of random ideas.

Days 0 and 1

Goal: get a remote server for free and learn how to manage it.

Decided to use Google Cloud Compute, just because. That dashboard is so chaotic… The most difficult task of the day was setting up the SSH public key to log to my new VM.

SSH keys are awesome. I’ve been using them for a long time for remote access and they make life way easier. ssh-copy-id command is easier than cat ~/.ssh/id_rsa.pub | ssh remote_username@server_ip_address "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys", but for old servers is not an option. The machine I’m using at my university to run the simulations for my Master’s studies did not accept ssh-copy-id

Day 2

Goal: learn to navigate in the machine using only the terminal.

I’ve been using GNU/Linux for more than 10 years, I’m pretty confident with the terminal. But there’s always something new to learn: ls -ltra is a nice example to find the most recently altered files.

Day 3

Goal: get familiar with root things.

Mind blow of the day: I saw lots of attempts to connect to this VM in /var/log/auth.log file. Awesome. Bots everywhere.

I am running ArchLinux on my Raspberry Pi 3, it uses journalctl for the logs: journalctl /usr/bin/sshd.

It was nice to learn about sudo -i. I’ve been using sudo su - to become root and is good to know there are other ways to do it as well. The details about the differences and other comments are explained here.

Day 4

Goal: install stuff, explore the file structure and get familiar with “walking” around.

man hier blew my mind. Oh, this is awesome. I googled about the filesystem hierarchy so many times and did not know I already had all the information in front of me.

Day 5

Goal: get familiar with TAB, less, more, and hidden/dot files.

I was never a big fan of more. I normally use less as a pager, because it allows me to scroll up and down easier, search for a string, jump to a position, and quite fast for huge files.

The history command is nice. I know about it and never used directly. I prefer to ctrl + r and a trick to use the Up/Down arrows to use the text that has already been typed as the prefix for searching commands. Only works with Bash, though. Check readline on ArchWiki for more details.

Day 6

Goal: learn Vim and how to exit it.

I’ve been using Vim for quite a long time. I use it for almost everything related to text: coding, writing my Master’s Thesis, writing TODO lists, writing these notes, and probably other things I can’t remember now. And despite those years of usage, every week I learn something new about it. Learning Vim is a hard task, but definitely worth it.

I started the vimtutor but never finished it. So many times. They say it takes about half an hour to go through it, I’ll try to do it (again). Actually, I’ll try neovim’s tutorial (:Tutor<Enter>).

The insane new thing about Vim that I learned today is that it has seven (!) basic modes:

  • Normal mode
  • Visual mode
  • Select mode
  • Insert mode
  • Command-line mode (or Cmdline mode)
  • Ex mode
  • Terminal-Job mode

And then there are seven additional modes:

  • Operator-pending mode
  • Replace mode
  • Virtual Replace mode
  • Insert Normal mode
  • Terminal-Normal mode
  • Insert Visual mode
  • Insert Select mode

You can read more about them with the internal help: :help vim-modes. Curiously, neovim has one fewer additional mode (at least as of nvim 0.4.4).

Day 7

Goal: install a web server and use it to host a page.

The instructions are for Apache2, a.k.a. httpd, but I decided to use Caddy instead. I’m curious about it’s automatic conversion from Markdown to HTML.

I got the binary from the Download page and followed the official documentation to install it, creating a systemd service file.

I am using this Caddyfile to serve a static-website:

:80 {
	root * /var/lib/caddy/www/
	encode zstd gzip
	templates
	file_server
}

And my /var/lib/caddy/www/index.html is a very complex one:

{ { include "index.md" | markdown }}

Afterthoughts

I should just follow the course. Lets play with Apache2. It is also simple, for some reason I expected it to be a quite complex to set up. On the other hand, I need to write html instead of Markdown :( but I can have a nice text only web page using only one tag:

<pre>
# Title

## Section

bla

Day 8

Goal: search for specific patterns in text files.

The Apache2 logs are in /var/log/apache2. To get all IP addresses that connected to Apache:

$ sed -E 's/ (.*)//g' access.log | sort -n | uniq
27.123.241.20
45.148.121.28
45.148.121.85
54.93.196.88
62.33.138.6
66.249.75.147
66.249.75.149
74.120.14.52
75.76.251.71
91.193.5.58
92.118.160.17
99.10.249.153
103.49.188.166
116.28.214.153
125.41.10.86
125.43.93.10
125.47.64.145
139.162.119.197
142.118.76.99
173.174.64.235
174.51.18.48
177.76.230.216
179.99.69.186
186.71.77.9
192.241.239.84
193.118.53.210
194.87.138.60
209.126.3.185
209.17.97.34

Grepping /var/log/auth.log I can see 72 different IPs trying to log as root and 10 other random users. Hahah, nice.

I like to know that cut can also remove words from a string, I finally got what the --fields option mean.

Day 9

Goal: First look at network security.

The first task is to find open ports. I’ve used nmap for that before and let’s see what happens, when running it on the remote server:

$ nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2020-10-15 11:32 -03
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000086s latency).
Not shown: 998 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds

Nice, it can see ssh and something for http, that is apache2 on port 80 here. But it is interesting how different it is from my laptop:

$ nmap 35.228.166.73
Starting Nmap 7.80 ( https://nmap.org ) at 2020-10-15 11:32 -03
Nmap scan report for 73.166.228.35.bc.googleusercontent.com (35.228.166.73)
Host is up (0.25s latency).
Not shown: 995 closed ports
PORT     STATE    SERVICE
19/tcp   filtered chargen
22/tcp   open     ssh
25/tcp   filtered smtp
80/tcp   open     http
5555/tcp filtered freeciv

Nmap done: 1 IP address (1 host up) scanned in 23.00 seconds

It finds three other filtered services. The manual says:

The state is either open, filtered, closed, or unfiltered. Open means that
an application on the target machine is listening for connections/packets
on that port. Filtered means that a firewall, filter, or other network
obstacle is blocking the port so that Nmap cannot tell whether it is open
or closed.

So, those ports might be or might not be open.

I tried to host something at those ports and could not get a connection from my laptop, so they are blocked by Google by default and there’s nothing (?) I can do. I can use all other ports though.

I’ve never used ufw before. It is time to learn how to properly use a firewall and this one is simple and powerful enough, although it uses iptables and not the “modern” nftables. I liked how simple it is to allow or deny traffic in a port. But I can’t compare with other tools.

Changing my SSH port to a random number. I thought it was a bit pointless, but apparently it is not. Using nmap, it shows the port open, but with unknown as the service. Sounds good.

Day 10

Goal: run stuff periodically with cron jobs.

Lazy people automate stuff :)

systemd can also run jobs at specific times, using systemd/timers.

A fun cron job to add is to periodically change the SSH banner:

$ cat /etc/cron.hourly/banner
#!/bin/sh

/usr/games/fortune | /usr/games/cowsay > /etc/banner.txt

Day 11

Goal: searching for files and patterns.

I’ve been using blindly find for quite a while. It was time to finally read it’s manual. There’s more to it than find . -iname "bla*".

zcat, zgrep, zless, and zmore are awesome! Did not know I could cat/less/grep gzip compressed files.

Day 12

Goal: copy stuff around.

I like rsync to copy lots of small files (actually, also big and medium ones) as it can create a “stream” of data instead of one new connection for each file. It can also compress the files as it sends to the destination. This can save a lot of time. I normally use rsync -azvP src dst meaning rsync --archive --compress --verbose --partial --progress src dst. Sometimes the -P flag can be annoying, but YMMV.

Something funny I found out is that rsync shows the SSH banner of the remote server:

$ rsync -avz linuxUpskill-gc:/var/www/html .
 _________________________________________
 / Be free and open and breezy! Enjoy!     \
 | Things won't get any better so get used |
 \ to it.                                  /
  -----------------------------------------
          \   ^__^
           \  (oo)\_______
              (__)\       )\/\
                  ||----w |
                  ||     ||
receiving incremental file list
html/
html/404.html
html/index.html
html/robots.txt

sent 104 bytes  received 3,710 bytes  693.45 bytes/sec
total size is 11,324  speedup is 2.97

Day 13

Goal: familiarize with the Linux permission system.

It is way simpler to use u+w or o-rwx than playing around with octal numbers for chmod. I didn’t know I could change the permission to more than one letter of ugo, as in chmod go-wr [files], it never occurred to me to try that either.

$ echo bla, blabla > file.txt
$ ls -la file.txt
-rw-rw-r-- 1 user user 12 Oct 21 12:11 file.txt
$ chmod ug-w file.txt
$ ls -la file.txt
-r--r--r-- 1 user user 12 Oct 21 12:11 file.txt
$ echo hehe >> file.txt
-bash: file.txt: Permission denied

Day 14

Goal: user management and permissions.

I always confuse useradd and adduser, after all they are anagrams. This ServerFault question has a nice answer about it:

adduser is a friendlier interactive frontend to useradd

I found out it is possible to enable offensive messages when an user type an incorrect password when sudoing. Just add Defaults insults to /etc/sudoers, obviously using visudo.

Nice to know that it is possible to enable specific commands for a user to run with sudo without granting full sudo, like:

fred    ALL = NOPASSWD:/sbin/reboot
fred    ALL = NOPASSWD:/usr/bin/ls

This allows user fred to sudo ls and sudo reboot, without typing the password, but does not allow any other command with sudo.

Day 15

Goal: package management, with apt.

I used a few different package managers before:

  • synaptic (when I used Kurumin, back then)
  • apt-get (when I switched to Ubuntu)
  • portage (when I used Gentoo)
  • apt-get (again, briefly, when I used Debian in 2011)
  • pacman (when I switched to ArchLinux)

I wish I remember the first time I tried Kurumin, but I don’t. I think I compiled Gentoo in 2008, I’m not sure. But I remember how fast my computer became. It was an AMD 32-bit single-core CPU, 1.2 GHz, 384 MiB RAM DDR, GeForce 4 MM 440, with hand soldered electrolytic capacitors in the motherboard (because they overheated and needed to be replaced). This computer was bought in ~2000, and when I finally managed to have a working Kernel, it needed ~12 seconds to fully boot! It had a very slow hard drive, slow RAM, slow CPU, and still was insanely fast! How I miss Gentoo…

ArchWiki has a Rosetta page, that “translates” commands from one package manager to other. It is a nice reference when dealing with different systems.

Day 16

Goal: managing tape archives, a.k.a tarballs.

Vim is always surprising me. Today I learned you can open file archives in Vim. It is possible to open .zip, .tar.gz, .tar.bz, etc., and even modify its content on the fly. No need to extract, edit, archive again.

I have a targz function in one of my dotfiles to create an archive of a bunch of files or a directory. It’s time to get rid of it and learn how to properly use tar and gzip/bzip.

Day 17

Goal: compiling stuff.

I liked to ./configure nmap:

$ CFLAGS="-O2 -march=native" CXXFLAGS="-O2 -march=native" ./configure
# skipping lots of lines...
            .       .
            \`-"'"-'/
             } 6 6 {
            ==. Y ,==
              /^^^\  .
             /     \  )  Ncat: A modern interpretation of classic Netcat
            (  )-(  )/
            -""---""---   /
           /   Ncat    \_/
          (     ____
           \_.=|____E
Configuration complete.
   (  )   /\   _                 (
    \ |  (  \ ( \.(               )                      _____
  \  \ \  `  `   ) \             (  ___                 / _   \
 (_`    \+   . x  ( .\            \/   \____-----------/ (o)   \_
- .-               \+  ;          (  O                           \____
(__                +- .( -'.- <.   \_____________  `              \  /
(_____            ._._: <_ - <- _- _  VVVVVVV VV V\                \/
  .    /./.+-  . .- /  +--  - .    (--_AAAAAAA__A_/                |
  (__ ' /x  / x _/ (                \______________//_              \_______
 , x / ( '  . / .  /                                  \___'          \     /
    /  /  _/ /    +                                       |           \   /
   '  (__/                                               /              \/
                                                       /                  \
  NMAP IS A POWERFUL TOOL -- USE CAREFULLY AND RESPONSIBLY

Cool ASCII art!

Day 18

Goal: cleaning-up old logs.

Logs can save your system. If there’s a weird behaviour somewhere, the first thing to check is the logs. But logs can also flood your system. In fact, this was what crashed some servers I was responsible for at my job and it was not nice. I wrote about that incident and how to tame Docker logs here.

In /etc/cron.daily/logrotate there’s a line that checks if there are any systemd/timers, if so, cron does nothing and let systemd do the job. That’s clever. The timer is configured in /etc/systemd/system/timers.target.wants/logrotate.timer.

logrotate’s log file is in a different path: /var/lib/logrotate/status. That’s funny!

Day 19

Goal: overview of the Linux Virtual File System.

Inodes. Inodes everywhere. If I got it correctly, an inode is a bunch of information about the file/directory. It contains filename, file owner, file permissions, creation/modification times, where the file is on disk, and maybe more. So, it is metadata.

To find a file by it’s inode number, you have to scan to entire filesystem. The inode number is per filesystem, so there might be more than one file with the same number. To locate a file by its inode number, without traversing to a different filesystem:

$ sudo find / -mount -inum 1
/sys
/boot/efi
/proc

$ stat /sys /boot/efi /proc
  File: /sys
  Size: 0               Blocks: 0          IO Block: 4096   directory
Device: 18h/24d Inode: 1           Links: 13
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-10-22 18:38:03.292000000 -0300
Modify: 2020-10-22 18:38:03.292000000 -0300
Change: 2020-10-22 18:38:03.292000000 -0300
 Birth: -
  File: /boot/efi
  Size: 512             Blocks: 1          IO Block: 512    directory
Device: 80fh/2063d      Inode: 1           Links: 4
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1969-12-31 21:00:00.000000000 -0300
Modify: 1969-12-31 21:00:00.000000000 -0300
Change: 1969-12-31 21:00:00.000000000 -0300
 Birth: -
  File: /proc
  Size: 0               Blocks: 0          IO Block: 1024   directory
Device: 5h/5d   Inode: 1           Links: 120
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-10-22 18:38:01.390534452 -0300
Modify: 2020-10-22 18:38:01.390534452 -0300
Change: 2020-10-22 18:38:01.390534452 -0300
 Birth: -

I just found out, empirically, that there’s no file with inode 0. This number is reserved.

I think I finally got the difference between a hard and soft link. But I am definitely not sure about it. The following copy-pasta blob from a terminal might shred some light on it:

$ ls > /tmp/bla.txt
$ ln /tmp/bla.txt link_hard
$ ln -s /tmp/bla.txt link_soft
$ stat /tmp/bla.txt link_hard link_soft
  File: /tmp/bla.txt
  Size: 82              Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 3122        Links: 2
Access: (0664/-rw-rw-r--)  Uid: ( 1001/       h)   Gid: ( 1002/       h)
Access: 2020-10-29 16:13:43.003122821 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:13:55.123855608 -0300
 Birth: -
  File: link_hard
  Size: 82              Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 3122        Links: 2
Access: (0664/-rw-rw-r--)  Uid: ( 1001/       h)   Gid: ( 1002/       h)
Access: 2020-10-29 16:13:43.003122821 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:13:55.123855608 -0300
 Birth: -
  File: link_soft -> /tmp/bla.txt
  Size: 12              Blocks: 0          IO Block: 4096   symbolic link
Device: 801h/2049d      Inode: 292697      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1001/       h)   Gid: ( 1002/       h)
Access: 2020-10-29 16:14:14.681037964 -0300
Modify: 2020-10-29 16:14:03.624369542 -0300
Change: 2020-10-29 16:14:03.624369542 -0300
 Birth: -

$ mv /tmp/bla.txt /tmp/bla2.txt
$ stat /tmp/bla2.txt link_hard link_soft
  File: /tmp/bla2.txt
  Size: 82              Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 3122        Links: 2
Access: (0664/-rw-rw-r--)  Uid: ( 1001/       h)   Gid: ( 1002/       h)
Access: 2020-10-29 16:31:42.488385936 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:31:39.232189075 -0300
 Birth: -
  File: link_hard
  Size: 82              Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 3122        Links: 2
Access: (0664/-rw-rw-r--)  Uid: ( 1001/       h)   Gid: ( 1002/       h)
Access: 2020-10-29 16:13:43.003122821 -0300
Modify: 2020-10-29 16:13:43.003122821 -0300
Change: 2020-10-29 16:14:38.554481313 -0300
 Birth: -
  File: link_soft -> /tmp/bla.txt
  Size: 12              Blocks: 0          IO Block: 4096   symbolic link
Device: 801h/2049d      Inode: 292697      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1001/       h)   Gid: ( 1002/       h)
Access: 2020-10-29 16:14:14.681037964 -0300
Modify: 2020-10-29 16:14:03.624369542 -0300
Change: 2020-10-29 16:14:03.624369542 -0300
 Birth: -

$ file link_*
link_hard: ASCII text
link_soft: broken symbolic link to /tmp/bla.txt

Day 20

Goal: ultimate laziness with scripts.

This is the last lesson of the Linux Upskill Challenge.

Scripting is an endless topic. As Bilbo would say, You start with a shebang, and if you don’t keep your feet, there is no knowing where you might be swept off to. This is a very powerful tool, but without paying attention to all details, a mistake can be the Destroyer of All Servers. Powerful, efficient, and simple.

Conclusions

I joined this course hoping I’d learn more about managing a server, in particular to improve my backup strategy. I ended up learning a lot more than I expected, not only about a VPS but also general Linux-related stuff.

It did not “consume” much time daily, it was very pleasant and time efficient. I liked how small, yet complete, each lecture was, and it was up to the learner to dig deeper as it pleased.

This course was totally worth it.

I have some final thoughts that don’t belong to any specific daily log, but are still relevant.

Never get a VPS on the other side of the world, if possible. I decided to get one in Finland (because, why not?) and I can feel the latency to type any commands in the terminal. I can type normally, it is not that slow. But the typing latency is there. I can’t measure it, but I can feel it. The average ping to this server is 297 ms, versus 19 ms for a server in Brazil (same country as I am now). 300 ms is definitely something we can note, but it is not a problem for what I was doing.

It doesn’t matter how long ago you started using Vim (or Neovim), take some time to go through vim-tutor. It takes about half an hour and it covers much more than the basic usage. There are some great small new things to learn there.

I recommend this course for anyone that wants to learn more about Linux, even if you are not planning to be a sysadmin. The lessons are very well written and there are many good tips to help a casual Linux user.