MetaDefender Distributed Cluster Overview
Introduction
More features will be supported along the road. Learn more details at Roadmap.
The MetaDefender Distributed Cluster (MDDC) is an approach to serve very large deployments while offering improved auto-scaling, high availability and fault tolerant capabilities for MetaDefender Core.
Distributed Cluster consists of several components:
| Component | Functionalities |
|---|---|
| MDDC Control Center | Assist administrators with user management, system health monitoring, and deploying or upgrading MetaDefender Core or API Gateway without any downtime. |
| MDDC Identity Service | Assist Control Center and API Gateway in client authentication, managing user activity sessions and authorization. |
| MDDC File Storage | Securely store and share files asynchronously across components in the cluster. The component manages the duration and duplication of files. |
| MDDC Worker | Deploy and monitor activities of MetaDefender Core and API Gateway. |
| MDDC API Gateway | Accept file scans, fetch scan statuses, and process download requests from clients. |
| MetaDefender Core | Scan the accepted files. |
| RabbitMQ - Message broker | Receives tasks from the API Gateway and forwards them to MetaDefender Core instances for processing. |
| Redis - Caching server | Store in-progress results in memory for rapid retrieval. |
| PostgreSQL - Database sever | Permanently store scan results, configuration and executive reports. |
The Distributed Cluster offers users two distinct interfaces. The first is a RESTful interface provided by MDDC API Gateway for applications to upload files for scanning, retrieve scan status, download processed files, or abort file scanning. The other is a Web UI provided by MDDC Control Center for the system administrator to manage licenses and users, modify workflow configurations, monitor the overall system, and remotely deploy or upgrade MDDC API Gateway or MetaDefender Core.
When a file is submitted to the MDDC API Gateway for scanning, its body content is securely transmitted to MDDC File Storage for subsequent use. API Gateway submits a scan task in RabbitMQ queue, and responds to the application with data_id. The task is delivered to healthy MetaDefender Core instances and one of them will accept the task. The file corresponding to the task is transmitted from MDDC File Storage to the instance's local storage, and the processing of the file takes place. Scan results produced by the processing are continuously recorded in Redis for fast retrieval and are finally stored in the PostgreSQL database for long-term storage. If created, the sanitized or watermarked file is securely transmitted to MDDC File Storage for future download by API Gateway.
In certain rate situations, if one of the MetaDefender Core instances unexpectedly ceases operation, its 'broken' files are delivered to other MetaDefender Core instances for continued processing without the need for applications to resubmit the files. By leveraging MDDC File Storage and RabbitMQ, MetaDefender Core instances within MetaDefender Distributed Cluster can collaborate in distributing the workload of archive extraction, greatly decreasing the overall time required to process archive files while utilizing the resources much more efficiently.
Using the Web Console from by MDDC Control Center, the system administrator is able to adjust workflow settings centrally and, after which the updates are automatically synced across all MetaDefender Core instances. The administrator can scale out the number of MDDC API Gateway or MetaDefender Core instances if additional power is required. He or she can also upgrade the instances seamlessly while the file processing is occurring. All statistical data and health information for components, along with executive reports, can be accessed easily through the Web UI of MDDC Control Center.