Deploying on K8S

Running MetaDefender Cluster on Kubernetes gives you service discovery, storage orchestration, automated rollouts, secret management, self-healing and horizontal scaling without building any of it yourself. This page explains how the charts use those mechanisms, so that the behaviour you observe during install, upgrade and scaling is predictable.

Health probes

Every MetaDefender Cluster service container — MetaDefender Cluster Control Center, MetaDefender Cluster Identity Service, MetaDefender Cluster File Storage and the MetaDefender Cluster Workers — defines all three Kubernetes probes. They share a single executable, healthcheck.sh, invoked with the probe phase as its argument. Each invocation performs one check and exits — Kubernetes owns the retry interval and the failure threshold. (The bundled postgres, redis and rabbitmq use their own readiness and liveness checks with different timing.)

The probe command is fixed by the chart. Only the timing is tunable.

What each phase checks

For MetaDefender Cluster Control Center, MetaDefender Cluster Identity Service and MetaDefender Cluster File Storage, the checks are built around the service's /readyz endpoint:

The distinction between "answers HTTP" and "returns 200" is deliberate. A service that is up but not yet ready still answers, so liveness passes and Kubernetes does not restart it; readiness fails, so it receives no traffic until it is genuinely ready.

For MetaDefender Cluster workers, the three phases check different things, because a MetaDefender Cluster Worker pod is really two processes: the worker agent, and the instance it deploys and supervises.

The readiness distinction is what makes rollouts safe. A MetaDefender Cluster Worker pod is not "ready" when its agent starts — it is ready only once the instance it deployed (the MetaDefender Core engine, for example) is itself answering /readyz. Traffic is withheld until then.

Liveness is a deny-list of exactly two statuses, not a positive health check. Every other status passes — including unlicensed, incompatible, busy and isolated — and so does a MetaDefender Cluster Worker the MetaDefender Cluster Control Center has no record of. This is deliberate: an unlicensed or busy worker is something for an operator to resolve, not a reason to restart the container. It also means an unlicensed ometascan pod will sit there indefinitely without Kubernetes flagging anything.

The liveness check additionally fails open: if the MetaDefender Cluster Control Center cannot be reached, or returns an unexpected status code, the MetaDefender Cluster Worker is treated as healthy. A MetaDefender Cluster Control Center outage therefore does not cascade into every worker pod being killed and restarted.

Default probe timing

All probes poll every 5 seconds. The failure thresholds differ per component, reflecting how long each one legitimately takes to become available.

The MetaDefender Cluster Control Center's generous startup window covers everything that happens before it answers HTTP at all: waiting for its database port, creating and migrating the four cluster databases, then waiting for the MetaDefender Cluster Identity Service and every MetaDefender Cluster File Storage endpoint. Uploading the installers and registering the license happen after the startup probe has already passed — they gate readiness, not startup, so a slow upload delays Ready without ever restarting the pod.

ometascan raises its liveness threshold to 90 failures (roughly 7.5 minutes) because deploying and initialising the MetaDefender Core engines takes far longer than any other instance, and engine updates can briefly make the instance unresponsive.

The startup phase has no timeout of its own, so Kubernetes applies its 1-second default — shorter than the 5 seconds the check script allows itself. On slow or heavily loaded nodes, set it explicitly:

Tuning probes

Override any phase per component. Unspecified fields keep their defaults.

periodSeconds, timeoutSeconds and failureThreshold are tunable. The probe command itself is not.

Rollout behaviour

The MetaDefender Cluster Control Center rolls out with maxUnavailable: 1 and maxSurge: 0. This is a recreate-style rollout: the old pod is terminated before the new one starts, guaranteeing that two MetaDefender Cluster Control Centers never run at once. Expect a short orchestration outage during any MetaDefender Cluster Control Center upgrade. MetaDefender Cluster Workers already running continue to serve traffic through it.

MetaDefender Cluster File Storage and MetaDefender Cluster
Worker

Both are StatefulSets with podManagementPolicy: Parallel, so replicas start and stop concurrently rather than one ordinal at a time. Initial install and scale-out are correspondingly faster.

MetaDefender Cluster File Storage additionally sets minReadySeconds: 15 — a replica must stay ready for 15 seconds before Kubernetes counts it as available. This prevents a flapping replica from being treated as a successful rollout step.

Graceful shutdown

A MetaDefender Cluster Worker pod carries state that lives outside the pod: it is registered with the MetaDefender Cluster Control Center, and it has an instance deployed. Simply killing the pod would leave a stale registration and a consumed license activation behind.

On SIGTERM, a MetaDefender Cluster Worker therefore runs an ordered shutdown:

1. Isolate the instance — tell the MetaDefender Cluster Control Center to stop dispatching new work to it, then wait for in-flight work to drain.

2. Undeploy the instance — remove the deployed instance from this MetaDefender Cluster Worker. For ometascan, the MetaDefender Cluster Control Center also deactivates this deployment on the OPSWAT Activation Server at this point, which is what returns the license activation.

3. Unregister the worker — delete the MetaDefender Cluster Worker from the MetaDefender Cluster Control Center's inventory, which re-runs the deactivation as a fallback for anything still attached.

This is why MetaDefender Cluster Workers default to terminationGracePeriodSeconds: 1800 (30 minutes). The isolation step alone waits up to isolate.timeout seconds (1600 by default) for work to drain.

Isolation can be disabled per MetaDefender Cluster Worker if you prefer faster, less graceful shutdowns:

To skip isolation, set isolate.enabled: false. Do not set isolate.timeout: 0 expecting that — 0 means no timeout, so isolation would wait indefinitely until SIGKILL arrives, which is exactly the case that strands a license activation.

Budget the grace period as: isolation (up to isolate.timeout) + undeploy (a single attempt, bounded by GLOBAL_WAIT_TIMEOUT, 150 s by default) + unregister. Note that even with isolation disabled the undeploy step alone can consume a full GLOBAL_WAIT_TIMEOUT, which is why the example above uses 300 rather than something tighter. Revisit the grace period whenever you raise either bound. An undeploy failure does not stop the unregister step.

The chart sets isolation on with a 1600-second timeout. The MetaDefender Cluster Worker image's own built-in defaults are the opposite — isolation off, timeout 0 meaning wait forever — so a worker container run outside these charts behaves differently. Set both explicitly if you ever run the image directly.