Keycloak readOnlyRootFilesystem Implementation

Secure Keycloak deployment with read-only root filesystem using optimized GHCR image

Keycloak readOnlyRootFilesystem Implementation

Overview

This document describes the implementation of readOnlyRootFilesystem: true for Keycloak pods using a pre-optimized GHCR image. This security hardening prevents filesystem modifications at runtime, reducing attack surface.

Current Status

Date: 2025-12-08 Status: ✅ IMPLEMENTED - readOnlyRootFilesystem: true enabled Image: ghcr.io/vishnu2kmohan/keycloak-optimized:26.4.2 File: deployments/base/keycloak-deployment.yaml

Solution Summary

Approach: Pre-Built Optimized Keycloak Image

The solution uses a custom Keycloak image built via GitHub Actions that pre-compiles Quarkus at build time, eliminating the need for JIT compilation at runtime.

Key Benefits:

• ✅ readOnlyRootFilesystem: true - Full security hardening
• ✅ Faster container startup (no runtime augmentation)
• ✅ Matches docker-compose.test.yml for dev/prod parity
• ✅ Reduced attack surface

Architecture

┌─────────────────────────────────────────────────────────────┐
│  GitHub Actions (.github/workflows/build-keycloak-image.yaml)│
├─────────────────────────────────────────────────────────────┤
│  1. Build Stage: kc.sh build --db=postgres                  │
│  2. Push to GHCR: ghcr.io/vishnu2kmohan/keycloak-optimized │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│  Kubernetes Deployment                                       │
├─────────────────────────────────────────────────────────────┤
│  image: ghcr.io/vishnu2kmohan/keycloak-optimized:26.4.2    │
│  args: ["start", "--optimized"]                             │
│  readOnlyRootFilesystem: true                               │
└─────────────────────────────────────────────────────────────┘

Implementation Details

Dockerfile (docker/Dockerfile.keycloak)

# Stage 1: Build optimized Keycloak
FROM quay.io/keycloak/keycloak:26.4.2 as builder

WORKDIR /opt/keycloak

# Build optimized configuration
RUN /opt/keycloak/bin/kc.sh build \
    --db=postgres \
    --http-enabled=true \
    --health-enabled=true \
    --metrics-enabled=true \
    --http-relative-path=/authn

# Stage 2: Runtime image
FROM quay.io/keycloak/keycloak:26.4.2

# Copy pre-built artifacts
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
COPY --from=builder /opt/keycloak/lib/lib/ /opt/keycloak/lib/lib/

# Set non-root user
USER 10000

ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

Kubernetes Deployment (deployments/base/keycloak-deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
spec:
  template:
    spec:
      containers:
      - name: keycloak
        image: ghcr.io/vishnu2kmohan/keycloak-optimized:26.4.2
        args:
        - start
        - --optimized
        - --http-enabled=true
        - --http-port=8080
        - --hostname-strict=false
        - --health-enabled=true
        - --metrics-enabled=true

        env:
        - name: KC_HTTP_RELATIVE_PATH
          value: /authn  # For Traefik gateway routing

        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true  # ✅ Enabled!
          runAsNonRoot: true
          runAsUser: 10000
          runAsGroup: 10000
          capabilities:
            drop:
            - ALL

        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: var-tmp
          mountPath: /var/tmp
        - name: cache
          mountPath: /opt/keycloak/data/tmp
        - name: work-dir
          mountPath: /opt/keycloak/data
        - name: providers
          mountPath: /opt/keycloak/providers
        - name: themes
          mountPath: /opt/keycloak/themes

      volumes:
      - name: tmp
        emptyDir: {}
      - name: var-tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}
      - name: work-dir
        emptyDir: {}
      - name: providers
        emptyDir: {}
      - name: themes
        emptyDir: {}

Health Check Paths

With KC_HTTP_RELATIVE_PATH=/authn, health checks use:

Probe	Path	Purpose
startupProbe	/authn/health/started	Container initialization
livenessProbe	/authn/health/live	Process alive check
readinessProbe	/authn/health/ready	Ready to serve traffic

Security Controls

Pod Security Context

securityContext:
  runAsNonRoot: true
  runAsUser: 10000
  runAsGroup: 10000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

Container Security Context

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 10000
  runAsGroup: 10000
  capabilities:
    drop:
    - ALL

EmptyDir Volume Isolation

Writable paths are isolated to ephemeral emptyDir volumes:

• /tmp - Temporary files
• /var/tmp - Additional temporary storage
• /opt/keycloak/data/tmp - Keycloak cache directory
• /opt/keycloak/data - Keycloak work directory
• /opt/keycloak/providers - Custom provider JARs
• /opt/keycloak/themes - Custom themes

IMPORTANT: Do NOT mount emptyDir at /opt/keycloak/lib - this overwrites Quarkus runtime JARs and causes ClassNotFoundException.

GitHub Actions Workflow

Image is built automatically via .github/workflows/build-keycloak-image.yaml:

name: Build Keycloak Optimized Image

on:
  push:
    paths:
      - 'docker/Dockerfile.keycloak'
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: docker/Dockerfile.keycloak
          push: true
          tags: |
            ghcr.io/${{ github.repository_owner }}/keycloak-optimized:26.4.2
            ghcr.io/${{ github.repository_owner }}/keycloak-optimized:latest

Verification

Test Read-Only Filesystem

# Should fail - filesystem is read-only
kubectl exec -it keycloak-xxx -c keycloak -- touch /test
# Expected: touch: cannot touch '/test': Read-only file system

# Should succeed - emptyDir is writable
kubectl exec -it keycloak-xxx -c keycloak -- touch /tmp/test
# Expected: Success

Test Health Endpoints

kubectl exec -it keycloak-xxx -c keycloak -- curl http://localhost:8080/authn/health/ready
# Expected: {"status": "UP", "checks": [...]}

Test Admin Console

Access via Traefik gateway:

https://<ingress-host>/authn/admin/master/console/

Troubleshooting

Issue: Container CrashLoopBackOff

Symptoms: Pod fails to start with Quarkus errors

Cause: Using stock Keycloak image without --optimized flag

Solution: Ensure using GHCR optimized image:

image: ghcr.io/vishnu2kmohan/keycloak-optimized:26.4.2
args: ["start", "--optimized"]

Issue: ClassNotFoundException

Symptoms: Java class loading errors at startup

Cause: emptyDir mounted at /opt/keycloak/lib overwrites JAR files

Solution: Remove /opt/keycloak/lib volume mount. Only mount:

• /opt/keycloak/data
• /opt/keycloak/providers
• /opt/keycloak/themes

Issue: Health Check Failures

Symptoms: Pods never become ready

Cause: Wrong health check paths (missing /authn prefix)

Solution: Update probe paths:

readinessProbe:
  httpGet:
    path: /authn/health/ready  # Include KC_HTTP_RELATIVE_PATH
    port: http

References

• Keycloak Server Installation
• Keycloak Containers
• Quarkus Native Images
• Kubernetes Pod Security Standards

Related Files

• docker/Dockerfile.keycloak - Optimized image build
• .github/workflows/build-keycloak-image.yaml - CI/CD pipeline
• deployments/base/keycloak-deployment.yaml - Base deployment
• deployments/base/.trivyignore - Security scan suppressions
• docker-compose.test.yml - Test environment parity

---

Last Updated: 2025-12-08 Status: ✅ Implemented Owner: DevOps/Security Team Priority: Complete (Security Hardening)