Shared DB - Linux

Environment

Using AWS environment with the specification below:

MetaDefender Core

RDS

Deployment Model

Using a AWS Load Balancer to distribute files sent from the client tool to two (2) different MetaDefender Core servers applying Round Robin algorithm. With this algorithm, each MetaDefender Core server is supposed to receive same number of requests.

Client tool

A simple tool written in Python to collect files in a designated folder and submit requests to Load Balancer mentioned above.

xxxxxxxxxx

files_to_scan = list of files to scan

scan_futures =[]

for file_path in files_to_scan:

    scan_futures.append(asyncio.create_task(self.scan_file(file_path, self.load_balancer_url))

async def scan_file(self, file_path, core_url):

    api_url = f'{core_url}/file'

    with open(file_path, 'rb') as file:

        file_content = file.read()

    headers = {'Content-Type': 'application/octet-stream', 'filename': file_path}

    max_retries = 20  # Maximum number of retry attempts

    retry_delay = 120  # Delay in seconds before retrying

    for attempt in range(max_retries + 1):

        try:

            starttime = time.time()

            response = requests.post(api_url, data=file_content, headers=headers)

            endtime = time.time()

            status_code = response.status_code

            if status_code == 200:

                self.post_wait = self.post_wait + endtime - starttime

                self.num_post_req += 1

                response_json = response.json()

                data_id = response_json.get('data_id')

                return status_code, response_json, data_id

            if status_code == 503:

                print(f"Received status code {status_code}. Retrying in {retry_delay} seconds...")

                await asyncio.sleep(retry_delay)

            else:

                return None, None, None

        except requests.RequestException as e:

            return None, None, None

    return None, None, None

Dataset

Detailed information of dataset below will be used for testing:

Product Information

Product versions:

• MetaDefender Core version 5.18.0
• Engines:
  • Metascan 5: Ahnlab, Bitdefender, ClamAV, ESET, K7
  • Metascan 10: Metascan 5 + Avira, Varist, IKARUS, Quick Heal, TACHYON
  • Metascan MAX: Metascan 10 + Lionic, CMC, CrowdStrike Falcon ML, Aurora, Trellix, NANOV, RocketCyber, Sophos, Webroot SMD, Xvirus Anti-Malware
  • Deep CDR: 7.8.0
  • Proactive DLP: 3.2.0
  • Archive: 7.8.0
  • File type analysis: 7.8.0
  • File-based vulnerability assessment: 4.57-236
  • Adaptive Sandbox: 3.0.0

MetaDefender Core settings

General settings

• Turn off data retention
• Turn off engine update
• Max file size: 99999999

Archive extraction settings

• Max recursion level: 99999999
• Max number of extracted files: 99999999
• Max total size of extracted files: 99999999
• Timeout: 10 minutes
• Handle archive extraction task as Failed: true
  • Extracted partially: true

Metascan AV settings

• Scan timeout: 10 minutes
• Per engine scan timeout: 1 minutes

Adaptive Sandbox

• Max file size: 18446744073709 MB (max)
• Timeout: 86400 (max)

Performance test results

MetaDefender Core with single engine (technology)

Summary metrics:

System resource utilization:

MetaDefender Core with common engine packages

Summary metrics:

System resource utilization:

Recommendations

Controlling total processing time of each MD Core server:

In this deployment model, we should organize and send files in the way that it best utilizes the load of each MD Core server. It is not a good practice if one Core server is free while the other one is busy. By optimizing the distribution of files, we can ensure that each Core server is utilized efficiently, thereby improving overall system performance. Furthermore, this approach can help prevent bottlenecks and minimize the chances of system overload.

Adding proper number of MD Core servers to the cluster:

Adding more Core servers to this model will increase more load on the shared database. When adding a new MD Core server, users should monitor performance of database server such as memory/CPU consumption, disk usage, network bandwidth, request response time and so on… to see if it still can handle the load. This is important in order to maintain optimal performance and ensure that the database server can continue to efficiently serve the needs of the system.

Optimizing database server for better performance:

Continuing to add more Core servers to this model may result in increased strain on the shared database. As such, it is crucial to ensure that the database is optimized to handle the additional load effectively. Users can consider adjusting default database settings of PostgresSQL to optimize for more data load if needed. Here is where we can adjust PostgresSQL database settings: <PostgreSQL install location\version>\data\postgresql.conf.

Besides that, MD Core also supports a parameter (db_connection) for users to specify max connections that MD Core can handle, take a look at this guideline: MetaDefender Configuration.