Patching software via OCI Hooks

Sometimes you want to tweak some settings in a Pod, like test etcd with longer timeouts to improve stability in a virtualized environment. (High-Availability means it's better to wait a second or five in the unlikely case of required failover than having unexpected restarts every other minute.)

In this example, the configuration items are tantalizingly listed as environment variables. But as the Pod gets managed by an operator, changing the settings is not as easy as it should be - any manual modification gets immediately rolled back.

So, what can we do?

As etcd's configuration is controlled by environment variables, it's enough to modify these before the etcd process is run. But how? Environment settings inherited from the outside do not work, as the shell script started in the container overrides all these options; we need to use a lower-level to patch the Pod, ie. use an OCI hook.

Choose some writable path on your (master) nodes, like /run/brz/patch-etcd.sh, and deploy this script to the relevant nodes:

#!/usr/bin/bash
 
set -e
root="$(jq -r '.annotations["io.kubernetes.cri-o.MountPoint"]')"
cd "$root"
 
mv usr/bin/etcd usr/bin/etcd.bin
echo '#!/usr/bin/bash
 
export ETCD_ELECTION_TIMEOUT=5000
export ETCD_HEARTBEAT_INTERVAL=500

exec /usr/bin/etcd.bin "$@"
' > usr/bin/etcd
chmod +x usr/bin/etcd

This moves the etcd binary aside and creates a shell script in its place, which overrides two environment variables before calling the original executable.

[Note for eager readers of this blog: if the container image would be minimized, there'd be no shell to call. But as long as the configured container "command" starts with "#!/bin/sh" (and so requires a shell to work), there's little to worry.]

To activate this patch, we'll configure an OCI hook by deploying a JSON file as /run/containers/oc/hooks.d/patch-etcd.json:

{
  "version": "1.0.0",
  "when": {
    "annotations": {
       "kubectl.kubernetes.io/default-container": "etcd"
    }
  },
  "stages": [
    "createRuntime"
  ],
  "hook": {
    "path": "/run/brz/patch-etcd.sh"
  }
}

It's as simple as it sounds - when a container with the given annotation gets to the "createRuntime" stage, this shell script is run, and the Pod gets modified - without any OpenShift process knowing about that.

Let's see the difference. Here's a command with its output for the old (original, unmodified) state:

egrep -o "ETCD_(HEARTBEAT_INTERVAL|ELECTION_TIMEOUT)=[0-9]+" \
  /proc/$(pgrep --full 'etcd --logg')/environ -a
ETCD_HEARTBEAT_INTERVAL=100
ETCD_ELECTION_TIMEOUT=1000

After activating the OCI hook and restarting etcd (eg. by killing the process), we see the new, modified state (note the executable name in pgrep!):

egrep -o "ETCD_(HEARTBEAT_INTERVAL|ELECTION_TIMEOUT)=[0-9]+" \
  /proc/$(pgrep --full 'etcd.bin --logg')/environ -a
ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=5000

While in this specific case the patch might not have been necessary (there's another mechanism to use a set of longer timeout values), knowing about the option might be helpful for other cases in the future.