kilabit.info
| AmA | Build | Email | GitHub | Mastodon | Projects | SourceHut

This article collect DevOps practices that I found bewildering and amusing.

On container

Use cases,

1) A developer just finish implementation of the new feature or fix an issue. He commit his works in the predefined branch names and push it to upstream repository. The Continuous Integration (CI) triggered, and one of the step is testing the code inside a container.

2) A developer just tagged a new release and push the "production" branch with new versioned tag to upstream. The Continuous Deployment (CD) triggered, and one of the step is creating a new image from the latest tag.

The CI or CD steps involve building a container image. The subsections below describe weird things that developers/operations do on the image script.

Installing packages on every build

The image script contains command that install packages.

...
RUN apt-get install -qq --no-install-recommends -y \
        apt-utils \
        locales \
        build-essential gcc sshpass vim
...

I don’t understand this. Why? ლ(゚ д゚ლ ) Why do you need to install packages every time some developer push to new branch.

What you should do is creating a base image that contains predefined packages installed, so the next time you need to run test or create release image, it use that base image as starter.

FROM baseimage

COPY src /
RUN testing

Building other tool (not the application) inside image

This is the worst than installing packages, because it will includes installing compilers and development packages.

...
RUN if test "$dev" = "yes"; then \
    apt-get update && apt-get install -y --no-install-recommends \
        libncursesw5-dev && \
    wget -qO- https://some-tools | \
    tar -xvzf - && \
    cd some-tools-2.0.0 && \
    ./configure && \
    make -j$(nproc) && \
    make install && \
    cd .. && rm -rf some-tools-2.0.0; fi
...

Why? ლ(゚ д゚ლ )

If you need it, you should build it on the base image and reuse that base image during CI/CD.

Updating system on every build

...
RUN apt-get update -qq && \
...

This is much worse.

Installing update without knowing what is being updated may hit you in many direction.

The application at version 1.50.0 may works because no updates. But, then after new patch release 1.50.1, the image contains an update that break the application. You spend night and day looking at the commits that cause it, without knowing that something has changes during system update.

Installing dependencies on every build

...
RUN cd ${APP_DIR}/ && npm install
...
RUN pip3 install -r /tmp/requirements.base && \
...

This case is the same with the first one.

Unnecessarily re-installing the same packages every time CI/CD triggered, where you should install it only once on the base image.

But, but, developer may changes their package.json/requirements/go.mod anytime…​

You can keep the "npm install/pip install" command in the Dockerfile. When the dependencies file updated, the npm/pip/other should pick up whats new and only download the new packages from external network (internet). The rest of packages that does not changes, which already installed during building of base image, should be fetched from local cache.

Anyway, the developer should communicate when new dependencies changes, so you, as DevOps should prepare new base image.

Using Alpine for the sake of smallest image

In 2015, I wrote a shell scripts to create Arch Linux images. The smallest possible size I can achieve at that time is around 118 MB.

In the README then I said,

> Arch Linux is become bloated, I recommend to use Alpine Linux for small size and probably faster container.

Later, I retract this statement.

Arch Linux is not bloated. Unlike other Linux distro, Arch Linux include documentation and development files in one package, while other distro split it into "-doc" and/or "-devel" packages.

DO NOT USE Alpine Linux just because you want smaller images. Alpine Linux use Musl libc, the core library where every single program depends on, which completely different with glibc that used by most Linux distro where you probably develop and test your program. And, no, Musl is not always faster than glibc. If you did not know what is libc and why it will affect your program, please do not use it for the sake of smaller images.

Using the "latest" tag

Instead of using the known version, they use "latest" tag on the script,

FROM image:latest
...

When the first time the image created, it pulls the image:3.12, everything works. But months later, it may pull "image:3.21" that breaks the build.

Using and mounting system host directory

Let say the application inside the container read the configuration from "/etc/app.conf" and write the logs into "/var/log/app.log". That does not means we should store the application configuration on the host system "/etc" and bind mount them into container like this

    ...
    volumes:
      - /etc/app.conf:/etc/app.conf
      - /var/log/:/var/log/
    ...

No, no, no, no.

Unless your application run directly on the host, you should never touch and mix the between host and container.

Instead, create a working directory for your application, let say "/data/app" with the same directory structure needed by container,

/data/app
  |
  + etc/app.conf
  |
  + var/log/

and use them in volumes like this,

    ...
    volumes:
      - /data/app/etc/app.conf:/etc/app.conf
      - /data/app/var/log/:/var/log/
    ...

On system administration

Running as "root" everywhere

(╯°□°)╯︵ ┻━┻

This is a basic 101 Linux system administration that they just ignore it completely. Not only managing the system as root but they also running the service (application) as root.