Running auto-updatable services in rootless containers with podman on Oracle Linux/RHEL/Fedora with SELinux enabled

These are my notes about configuring services with Podman on RHEL and related OSes with SELinux enabled, using compute instance in the Oracle Cloud with Oracle Linux 8. Information presented below is readily available elsewhere – see references – however, the intent of this opus is to condense all of that into palatable chunks to serve as a somewhat quick answer to the question “How do I get this container running on my instance?” without needing to spend hours reading pages and pages of documentation.

System configuration

To install podman, follow documentation for your OS. For Oracle Linux 8 this boils down to¹:

sudo dnf module install container-tools:ol8
sudo dnf install podman-docker

The second one is not necessary, but handy to have docker command for compatibility

On Oracle Linux 9 and 10, install the container-tools metapackage instead of the module:

sudo dnf install container-tools podman-docker

We will be running containers as a user, rootless, and therefore we need to allow processes launched by our user to persist²:

sudo loginctl enable-linger opc

While at it, also set timezone, so that logs make more sense:

timedatectl list-timezones
sudo timedatectl set-timezone America/Los_Angeles

SELinux

If SELinux is in enforcing mode, (as it should be) few things need to be done:

1. For systemd to be able to manage container add container_manage_cgroup permission³:

   sudo setsebool -P container_manage_cgroup on
   

2. Allow podman to relabel the content of the directories to be mapped into the container⁴. We’ll discuss this in the next section

Podman and systemd

podman generate systemd creates a systemd service file from an existing container. Quadlet generates services from .container configuration files. The podman generate systemd command is deprecated but supported.

Quadlet requires Podman 4.4 or newer and cgroup v2; check with podman --version and podman info --format '{{.Host.CgroupsVersion}}'. For rootless containers on Oracle Linux 8, use podman generate systemd: Oracle documents a rootless Quadlet limitation.

When switching an existing service between these approaches, stop it and move its old unit definition out of the configuration directory before reloading systemd.

As an example, we’ll set up two applications:

1. An awesome uptime monitor, “Uptime Kuma” https://hub.docker.com/r/louislam/uptime-kuma, that runs services as root inside the container
2. “UniFi Network Application” container, https://docs.linuxserver.io/images/docker-unifi-network-application/ that allows to specify the UID and GID for the user to be running the container as.

Uptime Kuma

Existing v1 installations need the project’s v1-to-v2 migration, including a backup of the data directory before starting v2.

This is the suggested docker run command from the project’s page:

docker run \
    -d \
    --restart=always \
    -p 3001:3001 \
    -v uptime-kuma:/app/data \
    --name uptime-kuma \
    docker.io/louislam/uptime-kuma:2

mkdir -p ~/.config/systemd/user
podman generate systemd \
    --new \
    --name uptime-kuma \
    --restart-policy=always \
    > ~/.config/systemd/user/container-uptime-kuma.service
podman rm uptime-kuma

systemctl --user enable container-uptime-kuma.service
systemctl --user start container-uptime-kuma.service
systemctl --user status container-uptime-kuma.service

mkdir -p ~/uptime-kuma ~/.config/containers/systemd
podman pull docker.io/louislam/uptime-kuma:2

[Container]
Image=docker.io/louislam/uptime-kuma:2
ContainerName=uptime-kuma
PublishPort=3001:3001
Volume=%h/uptime-kuma:/app/data:Z
AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user start container-uptime-kuma.service
systemctl --user status container-uptime-kuma.service

Once the container started, look at the content of the ~/uptime-kuma folder:

$ ls -lZ uptime-kuma
total 11884
-rwxr-xr-x. 1 opc opc system_u:object_r:container_file_t:s0:c30,c974 4681728 Dec 25 20:58 kuma.db
-rwxr-xr-x. 1 opc opc system_u:object_r:container_file_t:s0:c30,c974   61440 Dec 22 00:22 kuma.db.bak0
-rwxr-xr-x. 1 opc opc system_u:object_r:container_file_t:s0:c30,c974 3043328 Dec 24 12:07 kuma.db.bak20221224200743

1. The stuff inside Uptime Kuma runs as root. However, the owner of these files is the current user on the host – opc. That is not surprising, of course. If you look at what container sees, we will see those files as owned by root:

   $ podman unshare ls -l uptime-kuma
   total 11900
   -rwxr-xr-x. 1 root root 4698112 Dec 25 21:23 kuma.db
   -rwxr-xr-x. 1 root root   61440 Dec 22 00:22 kuma.db.bak0
   -rwxr-xr-x. 1 root root 3043328 Dec 24 12:07 kuma.db.bak20221224200743
   

   The podman-unshare is a little handy utility to run commands in the modified namespace. We can use it to look up the mapping in the /proc/self/uid_map file:

   $ id opc
   uid=1000(opc) gid=1000(opc) groups=1000(opc),4(adm),190(systemd-journal)
   $ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
   

   The user 0 is mapped to us (user 1000), and users between 1 and 65536 are mapped to 100000 + userID - 1. We’ll see a better illustration in the next chapter, Unifi Controller.

   So, is it ok to run the stuff inside the container as root? See an excellent discussion here⁶.

2. Note the container_file_t label that was added by podman as instructed by the Z flag.

Unifi Controller

Now let’s do the same thing with unifi-network-application. LinuxServer retired unifi-controller; its replacement requires an external MongoDB database. Existing installations migrate by exporting a UniFi backup and restoring it into a fresh instance.

Create separate application and database directories, and a network for the two containers:

mkdir -p ~/unifi ~/unifi-db ~/.config/unifi
podman network create unifi

Save ~/.config/unifi/unifi-db.env, replacing the two passwords with URL-safe values:

MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=change-this-root-password
MONGO_USER=unifi
MONGO_PASS=change-this-unifi-password
MONGO_DBNAME=unifi
MONGO_AUTHSOURCE=admin

Save ~/.config/unifi/unifi.env, using the same application password:

PUID=1000
PGID=1000
TZ=America/Los_Angeles
MONGO_USER=unifi
MONGO_PASS=change-this-unifi-password
MONGO_HOST=unifi-db
MONGO_PORT=27017
MONGO_DBNAME=unifi
MONGO_AUTHSOURCE=admin
MEM_LIMIT=2048
MEM_STARTUP=1024

Save LinuxServer’s database initialization script as ~/.config/unifi/init-mongo.sh:

#!/bin/bash
if which mongosh > /dev/null 2>&1; then
    mongo_init_bin='mongosh'
else
    mongo_init_bin='mongo'
fi
"${mongo_init_bin}" <<EOF
use ${MONGO_AUTHSOURCE}
db.auth("${MONGO_INITDB_ROOT_USERNAME}", "${MONGO_INITDB_ROOT_PASSWORD}")
db.createUser({
    user: "${MONGO_USER}",
    pwd: "${MONGO_PASS}",
    roles: [
        "clusterMonitor",
        { db: "${MONGO_DBNAME}", role: "dbOwner" },
        { db: "${MONGO_DBNAME}_stat", role: "dbOwner" },
        { db: "${MONGO_DBNAME}_audit", role: "dbOwner" },
        { db: "${MONGO_DBNAME}_restore", role: "dbOwner" }
    ]
})
EOF

The script runs only when MongoDB initializes an empty data directory. Both images are marked for auto-updates; MongoDB stays on 8.0. Major database upgrades require a separate migration.

Starting with LinuxServer’s docker run example, the application command with the environment file and network above looks like this:

docker run -d \
  --name=unifi-controller \
  --network=unifi \
  --env-file ~/.config/unifi/unifi.env \
  -p 8443:8443 \
  -p 3478:3478/udp \
  -p 10001:10001/udp \
  -p 8080:8080 \
  -p 1900:1900/udp \
  -p 8843:8843 \
  -p 8880:8880 \
  -p 6789:6789 \
  -p 5514:5514/udp \
  -v /home/opc/unifi:/config \
  --restart unless-stopped \
  lscr.io/linuxserver/unifi-network-application:latest

The UniFi container developers expect the container to start as root, and then launch the actual application using the passed PUID and GUID of 1000. Which is the same as our default opc user. Or is it?

mkdir -p ~/.config/systemd/user
podman create \
  --label "io.containers.autoupdate=registry" \
  --name=unifi-db \
  --network=unifi \
  --env-file ~/.config/unifi/unifi-db.env \
  -v /home/opc/unifi-db:/data/db:Z \
  -v /home/opc/.config/unifi/init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh:ro,Z \
  docker.io/mongo:8.0

podman create \
  --label "io.containers.autoupdate=registry" \
  --name=unifi-controller \
  --network=unifi \
  --env-file ~/.config/unifi/unifi.env \
  -p 8443:8443 \
  -p 3478:3478/udp \
  -p 10001:10001/udp \
  -p 8080:8080 \
  -p 1900:1900/udp \
  -p 8843:8843 \
  -p 8880:8880 \
  -p 6789:6789 \
  -p 5514:5514/udp \
  -v /home/opc/unifi:/config:Z \
  lscr.io/linuxserver/unifi-network-application:latest

podman generate systemd --new --name unifi-db --restart-policy=always > ~/.config/systemd/user/container-unifi-db.service
podman generate systemd --new --name unifi-controller --restart-policy=always \
  --requires container-unifi-db.service \
  --after container-unifi-db.service \
  > ~/.config/systemd/user/container-unifi-controller.service
podman rm unifi-db unifi-controller

systemctl --user enable container-unifi-db.service container-unifi-controller.service
systemctl --user start container-unifi-controller.service
systemctl --user status container-unifi-controller.service

mkdir -p ~/.config/containers/systemd
podman pull docker.io/mongo:8.0
podman pull lscr.io/linuxserver/unifi-network-application:latest

[Container]
Image=docker.io/mongo:8.0
ContainerName=unifi-db
AutoUpdate=registry
Network=unifi
EnvironmentFile=%h/.config/unifi/unifi-db.env
Volume=%h/unifi-db:/data/db:Z
Volume=%h/.config/unifi/init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh:ro,Z

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
Requires=container-unifi-db.service
After=container-unifi-db.service

[Container]
Image=lscr.io/linuxserver/unifi-network-application:latest
ContainerName=unifi-controller
Network=unifi
EnvironmentFile=%h/.config/unifi/unifi.env
PublishPort=8443:8443
PublishPort=3478:3478/udp
PublishPort=10001:10001/udp
PublishPort=8080:8080
PublishPort=1900:1900/udp
PublishPort=8843:8843
PublishPort=8880:8880
PublishPort=6789:6789
PublishPort=5514:5514/udp
Volume=%h/unifi:/config:Z
AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user start container-unifi-controller.service
systemctl --user status container-unifi-controller.service

We expect the data files created by the container to be owned by user 1000. As it is understood by the container. Indeed, files are owned by user 1000, just as we requested via container environment variables:

$ podman unshare ls -ln unifi
total 0
drwxr-xr-x. 6 1000 1000 176 Dec 25 21:07 data
drwxr-xr-x. 3 1000 1000  77 Dec 21 23:34 logs
drwxr-xr-x. 3 1000 1000  62 Dec 25 21:07 run

Which user does actually own then on our host?

$ ls -ln unifi
total 0
drwxr-xr-x. 6 100999 100999 176 Dec 25 21:07 data
drwxr-xr-x. 3 100999 100999  77 Dec 21 23:34 logs
drwxr-xr-x. 3 100999 100999  62 Dec 25 21:07 run

Reasonable, according to the mapping:

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

Auto-update

With all the prerequisites in place, all is left to do is enable and start the auto-update timer:

systemctl --user enable podman-auto-update.timer
systemctl --user start podman-auto-update.timer
systemctl --user status podman-auto-update.timer

By default, the version check will be performed daily, and if the new container image is available in the registry, it will be downloaded, and the container service restarted. Use systemctl --user edit podman-auto-update.timer for customizations.

Misc

Firewall configuration

In the context of the examples chosen, the next logical step would be to allow the mapped ports through the firewall, some of them permanently, some temporarily until the further application configuration is made (such as configuring cloudflare tunnel for Uptime Kuma, and registering Unifi Controller with the UniFi cloud account).

Oracle linux has firewalld enabled by default, so this task is trivial:

sudo firewall-cmd \
    --permanent \
    --add-port=8080/tcp \
    --add-port=3478/udp

sudo firewall-cmd \
    --add-port=8443/tcp \
    --add-port=8080/tcp \
    --add-port=3478/udp \
    --add-port=3001/tcp

Instance hostname:

Edit /etc/oci-hostname.conf and set PRESERVE_HOSTINFO=2. Then set the hostname:

$ sudo hostnamectl set-hostname MYHOSTNAME

TL;DR

# Install podman (Oracle Linux 8; see above for 9 and 10)
sudo dnf module install container-tools:ol8
# Enable linger
sudo loginctl enable-linger opc
# Set timezone
sudo timedatectl set-timezone America/Los_Angeles
# Allow systemd mess with containers
sudo setsebool -P container_manage_cgroup on

# Create a container (label, Z)
podman create \
       --label "io.containers.autoupdate=registry" \
       --name mycontainer \
       ... \
       -v xxx:yyy:Z \
       registry/thatcontainer:latest

# Generate systemd config
mkdir -p ~/.config/systemd/user
podman generate systemd \
  --new \
  --name mycontainer \
  --restart-policy=always \
  > ~/.config/systemd/user/container-mycontainer.service

# Remove the container 
podman rm mycontainer

# Enable, start, and check the container service:
systemctl --user enable container-mycontainer.service
systemctl --user start container-mycontainer.service
systemctl --user status container-mycontainer.service
      
      
# Enable and start auto-updater.
systemctl --user enable podman-auto-update.timer
systemctl --user start podman-auto-update.timer

# Install podman (Oracle Linux 9 or 10)
sudo dnf install container-tools
# Enable linger
sudo loginctl enable-linger opc
# Set timezone
sudo timedatectl set-timezone America/Los_Angeles
# Allow systemd mess with containers
sudo setsebool -P container_manage_cgroup on

# Create a Quadlet (label, Z)
mkdir -p ~/.config/containers/systemd
podman pull registry/thatcontainer:latest
cat > ~/.config/containers/systemd/container-mycontainer.container <<'EOF'
[Container]
Image=registry/thatcontainer:latest
ContainerName=mycontainer
Volume=xxx:yyy:Z
AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=default.target
EOF

# Enable, start, and check the container service:
systemctl --user daemon-reload
systemctl --user start container-mycontainer.service
systemctl --user status container-mycontainer.service


# Enable and start auto-updater.
systemctl --user enable podman-auto-update.timer
systemctl --user start podman-auto-update.timer
systemctl --user status podman-auto-update.timer

References

History