File Encryption in OpenShift

NIS2 (eur-lex) requires all countries to have a cyber security strategy. To enhance the confidentiality of application data, we provide ecryptfs support in our OpenShift installation.

Data needs protection, but where?

Without going into details about data protection basics and C, I, A stuff: "data in transit" is typically protected against unauthorized reading and modification (i.e. the C and I attributes are enhanced) via TLS (the modern version of SSL). The oldest "good enough" version is TLS1.2, but TLS1.3 has been available since 2017 as well and should also be accepted everywhere (though I know of some deviations) - there are already some websites who only provide TLS 1.3.

"Data at rest" is a different beast, with different considerations depending on your threat model.

If you want to protect against a thrown-away harddisk being found by your scrap dealer (yeah, that's already quite a process failure if it didn't get wiped or destroyed before that!), encryption on any layer is good enough - even only on the storage box.

By moving a bit higher up in the stack we next get to the worker nodes with their Linux kernel. If these encrypt your data, even the transit (independent of the transport layer, be it Fibre Channel, NFS, SMB, iSCSI, IB, etc.) will be protected. Though I need to note that with file encryption like ecryptfs only the data content itself is safe — metadata like "new file created, file opened, file read at some offset" might still be visible. Also, the key material can be protected very well, once it got loaded by some userspace process into the kernel. Using a FUSE encryption filesystem is very similar, but has slower performance and the key isn't as secure in the userspace process.

The last option for data encryption would be the application. While this is useful in some special use cases (eg. see PostgreSQL options for column encryption), in general I advise against it — first of all, the key will be visible in the application itself and so might be compromised if it gets hacked, and secondly, there is no real good standard for doing such encryption across the various ecosystems and applications (Java, Node.js, Go, Apache, PostgreSQL, ...).

So, using the second option isolates the application and the storage from any encryption concerns — this sounds like a local optimum to us.

But why ecryptfs? That's so old!

We chose ecryptfs because:

• It encrypts individual files. This allows for easy incremental backups, rsync, and so on.
  Block device encryption (LUKS, or Thales) hides the filesystem structure in a big blob of many GB to TB - which makes backup/restore/etc. much harder.
• The encryption metadata is stored in the first 8 kiB of the actual (non-virtual) file. Here the big advantage is that the encrypted file (on the lower-level storage) can be transferred via rsync, tar, ftp, dragging in a file manager, HTTP, zip, etc.
  In contrast, encFS and FSCrypt use POSIX xattr for the per-file encryption key - which is all too easily lost when moving stuff around.
• Because ecryptfs is so old, it is available in very "stable" kernel sources (like RHEL7, RHEL8) as well - if it doesn't readily exists as a kernel module, it just needs to be compiled (but see below).

Technicalities

No, sorry, this time I won't bother with them here - go see the project page (again on OpenCode.de), it includes much more details.

RedHat Compatibility

Unfortunately Red Hat does not support this approach anymore. While most other Linux distributions (Suse, Debian, Ubuntu, Arch, even CentOS (Streams)!) provide the eCryptFS kernel module and userspace utilities by default, RedHat doesn't support it in any way (yet, see below) - although the full project name is "The enterprise cryptographic filesystem for Linux" and it originated at IBM, which owns RedHat, at least right now.

In OpenShift 4.12 with RHEL8 underneath the combination of RedHat Linux Kernel sources, KMM, and the DTK image was sufficient to "just build it"; starting with RHEL9.2 (which is used by OpenShift 4.13) kernel module compilation is broken. There's a "Request for Enhancement" for ecryptfs open at RedHat (link) - you won't be able to see this RFE, but you can surely vote for it via your support channel! Please help us get support for this neat piece of software.