[K8s] How to enable Horizontal Pod Autoscaling based on RabbitMQ queue metrics
This guide explains how to scale the MDSS processing services (Scanning, Discovery and Remediations) with the number of messages waiting in RabbitMQ, so the number of pods grows when a backlog builds up and shrinks when the work is done.
CPU is a poor signal for these services: they spend most of their time waiting on the storages and on MetaDefender Core, so a large backlog can build up while CPU usage stays low. The services exchange work through RabbitMQ queues, which makes the queue length the direct measure of pending work.
Two ways to do it are described here:
Option | Best for | Effort |
|---|---|---|
A. KEDA (recommended) | Any cluster where KEDA can be installed. Built into the MDSS Helm chart. | Install KEDA, then set |
B. Prometheus + Prometheus Adapter | Clusters that already run Prometheus and cannot run KEDA. | Manual setup of scraping, adapter rules and an HPA per service. |
This documentation provides a general implementation approach that will need to be adapted to your specific environment. The exact commands, configurations, and values presented here should be reviewed and modified according to your organization's specific Kubernetes infrastructure, network architecture, and operational requirements.
Use one autoscaling mechanism per service. Two autoscalers on the same Deployment fight each other. When KEDA is enabled the chart skips the CPU based HPA (hpa.enabled) for the services KEDA manages, and a manually created HPA (Option B) must not target a service that KEDA manages.
Prerequisites
Kubernetes cluster with HPA support (Kubernetes 1.23 or higher for
autoscaling/v2)MDSS deployed with the Helm chart (the KEDA integration is available from chart version 4.6.0)
The RabbitMQ instance used by MDSS (in-cluster or external) must be reachable from the cluster
Option A (KEDA):
KEDA installed in the cluster (on OpenShift: the Custom Metrics Autoscaler operator from OperatorHub)
KEDA must be able to reach RabbitMQ over AMQP (port 5672). The RabbitMQ management plugin and the Prometheus plugin are not required
Option B (Prometheus):
If using an external RabbitMQ instance: version 3.8.0 or higher, the RabbitMQ Prometheus plugin enabled, port 15692 accessible for metrics scraping and the management interface available
Basic understanding of Kubernetes and Prometheus
Option A: Autoscaling with KEDA
Step 1. Install KEDA (if not already deployed)
Check that the operator is running and the CRDs are registered:
On OpenShift, install the Custom Metrics Autoscaler operator from OperatorHub and create its KedaController instance instead.
Step 2. Enable KEDA in the MDSS Helm values
Add the following to your values file (or pass the same keys with --set):
Then install or upgrade the release:
The chart creates, in the release namespace:
one
ScaledObjectfor each service that has queues configured (scanningservice,discoveryservice,remediationsservice),one
TriggerAuthentication(mdss-rabbitmq),one Secret (
mdss-keda-rabbitmq) with the AMQP connection string, built fromRABBITMQ_HOST,RABBITMQ_PORT,RABBITMQ_DEFAULT_USERandRABBITMQ_DEFAULT_PASS.
To use your own Secret instead, create one containing the full amqp://user:password@host:5672/ URI under the key host and set keda.existingSecret to its name. An external RabbitMQ needs no other change, as long as RABBITMQ_HOST points to it.
A service managed by KEDA is created without a fixed replicas value, so a later helm upgrade does not reset the replica count chosen by the autoscaler.
Step 3. Which queues are monitored
Every work queue exists in three priorities (_High, _Medium, _Low). Each one is a separate trigger, and the service is scaled to whatever the busiest queue requires.
Service | Queues (each with |
|---|---|
scanningservice |
|
discoveryservice |
|
remediationsservice |
|
Step 4. Verify
Each ScaledObject should report READY=True. KEDA creates and manages an HPA for it, named keda-hpa-<service> (for example keda-hpa-scanningservice).
To see the queue depth that drives the scaling:
Step 5. Tune the scaling
Every setting can be overridden per service, which is also how you change the thresholds for a single service:
queueLengthis the number of ready messages one replica is expected to handle, evaluated for each queue separately. A lower value scales up earlier.minReplicaCount/maxReplicaCountbound the number of pods.cooldownPeriodis how long the queues must stay empty before the service scales back tominReplicaCount. Scale-down between those limits follows the standard HPA stabilization window; usebehaviorto change it.
Scanning throughput is ultimately bound by the MetaDefender Core instances behind it. Scaling the Scanning service beyond what Core can absorb only moves the backlog, so scale MetaDefender Core as well (see the MetaDefender Core Helm chart).
Scaling behavior with KEDA
When the number of ready messages in any monitored queue exceeds
queueLengthper replica, KEDA scales the service up, up tomaxReplicaCount.When the queues drain, the HPA scales the service down gradually; after
cooldownPeriodseconds with empty queues it returns tominReplicaCount.
Troubleshooting KEDA
kubectl describe scaledobject <service>shows the conditions and events. AREADY=Falsestatus usually means KEDA cannot reach RabbitMQ.Check the KEDA operator logs:
kubectl logs -n keda -l app=keda-operatorThe AMQP connection string is stored in the
mdss-keda-rabbitmqSecret (keyhost). KEDA runs in its own namespace, so a short in-cluster host such asrabbitmqis qualified automatically with the release namespace; use a fully qualified host name for an external broker.If the service does not scale, compare
queueLengthwith the real queue depth (rabbitmqctl list_queues).
Option B: Autoscaling with Prometheus and the Prometheus Adapter
Use this option when KEDA is not available. It scales the Scanning service on the total number of messages in its scan queues, using a manually created HPA. Set hpa.enabled: false for the service in the Helm values (it is disabled by default) so the chart does not create a CPU based HPA for the same Deployment, and do not enable keda for it.
Step 1. Install Prometheus (if not already deployed)
If Prometheus is not already deployed in your cluster, install it using Helm:
Step 2. Configure Prometheus to Scrape RabbitMQ Metrics
Add the following scrape configuration to your Prometheus ConfigMap to collect metrics from RabbitMQ:
Apply the updated ConfigMap and restart Prometheus to apply the changes.
Note for External RabbitMQ: If using an external RabbitMQ instance, you'll need to modify the scrape configuration to target your external instance instead of using Kubernetes service discovery.
Step 3. Install Prometheus Adapter (if not already deployed)
The Prometheus Adapter is required to expose Prometheus metrics to the Kubernetes metrics API:
Please update prometheus.url and prometheus.port according to your environment
Step 4. Configure Prometheus Adapter
Update the Prometheus Adapter ConfigMap to expose the RabbitMQ queue metric:
This configuration:
Queries metrics with name "rabbitmq_queue_messages_ready"
Associates metrics with Kubernetes namespaces
Renames the metric to "rabbitmq_scan_queue_messages"
Filters for the scan queues (both the
scanning.object_ready_for_scan.v2_queueandobject_ready_for_scan_queuequeues) with Low, Medium, and High prioritiesSums the total number of messages across these queues
After modifying the ConfigMap, restart the adapter:
Step 5. Verify the Custom Metric
Check that the metric is properly exposed to the Kubernetes API:
You should see "rabbitmq_scan_queue_messages" listed in the output.
To get the metric value:
You should see the actual value of the queue size:
Step 6. Create a Horizontal Pod Autoscaler
Create an HPA resource that uses the custom metric to scale the Scanning service:
Apply the HPA:
Step 7. Monitor the HPA
Check the status of your HPA:
Scaling Behavior (Option B)
When the total number of messages in the scan queues exceeds the configured target (5000 in the example), the HPA scales up the number of scanning pods, up to the maximum of 10.
When the number of messages decreases, the HPA will gradually scale down the number of pods.
Troubleshooting (Option B)
If the HPA isn't working as expected:
Verify Prometheus is collecting RabbitMQ metrics:
Then visit http://localhost:9090 and query:
rabbitmq_queue_messages_ready{queue=~"(scanning\\.object_ready_for_scan\\.v2_queue|object_ready_for_scan_queue)_(Low|Medium|High)"}
Check that the Prometheus Adapter can access the metrics:
Verify the custom metric is available:
Ensure the HPA is targeting the correct deployment and namespace.