Shared DB - Linux

Environment

Using AWS environment with the specification below:

MetaDefender Core

Database Server

Deployment Model

Using a AWS Load Balancer to distribute files sent from the client tool to two (2) different MetaDefender Core servers applying Round Robin algorithm. With this algorithm, each MetaDefender Core server is supposed to receive same number of requests.

Client tool

A simple tool written in Python to collect files in a designated folder and submit requests to Load Balancer mentioned above.

files_to_scan = list of files to scan

scan_futures =[]

for file_path in files_to_scan:

    scan_futures.append(asyncio.create_task(self.scan_file(file_path, self.load_balancer_url))

​

async def scan_file(self, file_path, core_url):

    api_url = f'{core_url}/file'

​

    with open(file_path, 'rb') as file:

        file_content = file.read()

​

    headers = {'Content-Type': 'application/octet-stream', 'filename': file_path}

​

    max_retries = 20  # Maximum number of retry attempts

    retry_delay = 120  # Delay in seconds before retrying

​

    for attempt in range(max_retries + 1):

        try:

            starttime = time.time()

            response = requests.post(api_url, data=file_content, headers=headers)

            endtime = time.time()

            status_code = response.status_code

            if status_code == 200:

                self.post_wait = self.post_wait + endtime - starttime

                self.num_post_req += 1

                response_json = response.json()

                data_id = response_json.get('data_id')

                return status_code, response_json, data_id

​

            if status_code == 503:

                print(f"Received status code {status_code}. Retrying in {retry_delay} seconds...")

                await asyncio.sleep(retry_delay)

            else:

                return None, None, None

​

        except requests.RequestException as e:

            return None, None, None

​

    return None, None, None

Dataset

Detailed information of dataset below will be used for testing:

Product Information

Product versions:

• MetaDefender Core version 5.6.0
• Engines:
  • 10 AV engines: Ahnlab, Avira, ClamAV, ESET, Bitdefender, K7, Quick Heal, IKarus, Cyren, Tachyon (nProtect)
  • Deep CDR: 6.6.2
  • Proactive DLP: 2.16.0
  • Archive: 6.3.2
  • File type analysis: 6.3.2
  • File-based vulnerability assessment: 4.2

MetaDefender Core settings

General settings

• Turn off data retention
• Turn off engine update

Archive extraction settings

• Max recursion level: 99999999
• Max number of extracted files: 99999999
• Max total size of extracted files: 99999999
• Timeout: 10 minutes
• Handle archive extraction task as Failed: true
  • Extracted partially: true

Metascan AV settings

• Max file size: 99999999
• Scan timeout: 20 minutes
• Per engine scan timeout: 5 minutes

Performance test results

Summary

MetaDefender Core with single engine (technology):

MetaDefender Core with common engine packages:

Recommendations

Controlling total processing time of each MD Core server:

In this deployment model, we should organize and send files in the way that it best utilizes the load of each MD Core server. It is not a good practice if one Core server is free while the other one is busy. By optimizing the distribution of files, we can ensure that each Core server is utilized efficiently, thereby improving overall system performance. Furthermore, this approach can help prevent bottlenecks and minimize the chances of system overload.

Adding proper number of MD Core servers to the cluster:

Adding more Core servers to this model will increase more load on the shared database. When adding a new MD Core server, users should monitor performance of database server such as memory/CPU consumption, disk usage, network bandwidth, request response time and so on… to see if it still can handle the load. This is important in order to maintain optimal performance and ensure that the database server can continue to efficiently serve the needs of the system.

Optimizing database server for better performance:

Continuing to add more Core servers to this model may result in increased strain on the shared database. As such, it is crucial to ensure that the database is optimized to handle the additional load effectively. Users can consider adjusting default database settings of PostgresSQL to optimize for more data load if needed. Here is where we can adjust PostgresSQL database settings: <PostgreSQL install location\version>\data\postgresql.conf.

Besides that, MD Core also supports a parameter (db_connection) for users to specify max connections that MD Core can handle, take a look at this guideline:

https://docs.opswat.com/mdcore/configuration/metadefender-configuration#internal

System resource utilization charts

MetaDefender Core with Metascan (8 AV engines):

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2

MetaDefender Core with Metascan (8 AV engines) + Deep CDR:

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2

MetaDefender Core with Metascan (8 AV engines) + Deep CDR + Proactive DLP:

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2

MetaDefender Core with Metascan (8 AV engines) + Deep CDR + Proactive DLP + Vulnerability:

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2

MetaDefender Core with Deep CDR:

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2

MetaDefender Core with Proactive DLP:

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2

MetaDefender Core with Vulnerability:

Memory usage

MD Core #1

MD Core #2

CPU usage

MD Core #1

MD Core #2

Network usage

MD Core #1

MD Core #2