Performance Tunning Guide
About This Guide
This guide is designed to assist you in fine-tuning various configuration settings to achieve desired throughput and performance levels. By default, MetaDefender Storage Security operates in a balanced mode, optimizing both speed and functionality for maximum value. However, in scenarios where certain features may be unnecessary and system throughput is paramount, this guide will provide you with the necessary steps to adjust your settings for optimal performance outcomes.
Scanning Performance
The throughput of the system largely depends on the speed of the scan instance. If you are using MetaDefender Cloud, our SaaS offering, you are automatically benefiting from optimal performance. For users of MetaDefender Core, we recommend visiting the MetaDefender Core Performance Tuning Guide to explore ways to reduce the response time of your scan instance.
Increasing the number of concurrent file scans
A critical configuration parameter in MetaDefender Storage Security that significantly affects throughput is the number of concurrent scan requests sent to the scan instance. By default, the system processes 20 files simultaneously. As each file completes scanning, another is promptly queued. If your scan instance has additional capacity, we recommend gradually increasing this count to optimize throughput until you reach the desired performance level.
When adjusting the number of parallel scan requests, it's important to monitor the scan instance queue closely. An excessive number of messages in the queue indicates that the setting is too high, which can lead to performance degradation due to overloading the scan instance. Please adjust accordingly to maintain optimal performance.
To adjust this setting please modify the RABBITMQ_SCANNING_PREFETCH_COUNTenvironment variable as described in: Custom Configuration | Increase the number of concurrent scans.
Discovery Performance
Discovery Throttling
In scenarios where system load is exceptionally high, you may encounter a notification that discovery has been temporarily throttled. This measure halts the discovery of new files when the queue already contains a large volume of pending files. It is designed to optimize system resources and prevent excessive RAM usage. By setting this limit, we ensure efficient processing and reduce the risk of an endlessly growing queue, which can occur if the rate of file discovery exceeds the scanning capacity.
The threshold for maximum pending files is controlled by the environment variable DISCOVERY_THROTTLER_MAX_PENDING_FILES which defaults to 5000. This value can be adjusted upward or downward depending on your system's specifications to optimize memory usage effectively.
Performance Mode
Several configuration options are available that can accelerate system performance while slightly reducing functionality. When certain features are not required, enabling Performance Mode can be an effective way to enhance speed.
To activate Performance Mode, modify the environment variable PERFORMANCE_MODEas outlined in Custom configuration and set the value to PERFORMANCE_MODE=true .
Functionality Impacted with Performance Mode Activated
Before enabling Performance Mode, please review the table below, which details all areas that will be impacted. This information will help you make an informed decision about whether the trade-offs in functionality are acceptable for your needs.
| Functionality | Details |
|---|---|
| Amazon S3 | Tags are not being retrieved automatically for all files which means they are not available on the response of /api/file/{fileId} and /api/file/{scanId}/{startIndex}/{count} APIs. |