UNIX/Linux System Administration 101

The history of this adage probably comes from multi-user, shared machines, where two or more users can access one UNIX machine. If user root run a long running application under its UID, any users on the same machine may try to exploit it to gain a root access.

This practice still applicable if you have a service that is open to public, even thought that you are the only user in the machine.

Beside security, one of the reason to not run as user root is to prevent your application or script or yourself wipe out data accidentally.

Operating system (OS) and your application should be stored in different disk. There are two major reason of this.

First, read-write operation will be increased and not affect each other. A write to log on OS level does not have effect on read operation on application level.

Second, in case the OS disk is corrupt, we will only need to recreate new OS, reinstall the packages, reapply the last good configuration, and reattach the application disk. In case the application disk corrupt, we will only need to create new disk and deploy new application. But if your OS and application is on the same disk, you need to create OS and redeploy your application.

By having a snapshot, we have a stable packages and configuration that ready to use, either as quick fix in case of disk corrupt (like in previous case) or if we want to horizontally scale the application.

Never use password.

Most cloud provider already enforce this practice and I think I have never seen someone share user:password to access remove server anymore.

Instead of opening two or more SSH sessions, use screen or tmux to create new virtual terminal.

I don’t remember why, but I prefer tmux over screen (probably window split features, which screen already have now; or probably the tmux shortcut make more sense to me).

Each time we type a key on SSH, client send the key, server render the screen, and send the whole text back to client.

Mosh use UDP, client did not wait for server to reply before the character that we type printed on screen. Also, the data that send by server only the diff, not the whole text.

In summaries, mosh provide faster and network resistance connection for SSH.

This is 2020, we are not running application with nohup anymore and big, major Linux distro already moved from SysV init to systemd.

Systemd provides sane configuration and states to manage services, jobs (like cron jobs), and historical logs.

Use TCP/HTTP proxy, do not directly serve your web server to public.

A good proxy will allow load balancing, rate limiting, temporary redirect, and other flexibility things without touching application code.

This is also provide separation of concern or layer.

Tips: HAProxy is my favourite proxy server.

Choose the one that,

Ubuntu (or any Debian derivate?) violate all of the above checklists.

If you want additional security and does not have a problem managing SELinux, use RedHat derivative. If you want user friendly distro, use SuSE. If you want latest packages with more flexibility, use Arch.

VPN provide a layer of access for your internal network in the cloud. Setting up VPN with WireGuard is now easier than ever, but its require latest (or supported) kernel, both on server and client.

Combine that with DNS server (like rescached), we will have a complete isolate environment with custom internal domain (for example, myapp.internal, doc.myapp.internal, and so on).

Rationale: Your server may be accessed from different time zones by different people, by different application. Storing and providing UTC to different peoples or application make it consistent that everyone know the time on the server (or database) is always in UTC.

It is the responsibilities of the application that convert the server time back to user locale when accessed from their local server.