Rectify malformed FileType data in PostgreSQL database

FileType engine v6.0.2 sometimes creates malformed data after analyzing files. When being written into PostgreSQL database, those malformed data cause MetaDefender Core warehouse stuck and result in negative impacts:

  • Executive Report in Dashboard gets frozen and changed back to zero.
  • CPU usage will go too high.
  • A dramatical decrease in file processing performance.

Even though a recently released version of FileType engine v6.1.1 fixes the malformed data issue, there might be a few remaining data stored in PostgreSQL database.

This tool will handle those data by rectifing and converting all the malformed data of filetype_details in PostgreSQL database to a true format.

Note: If you do fresh installation of MetaDefender Core v5.4.1, you can skip the following content since we already enhance and harden the service to prevent malformed data issue.

Parameters

ParameterMandatory
--help/-hNodisplay all the options and flags associated with the guides.

Usage

Besides an enhancement, the upcoming version 5.4.1 of MetaDefender Core will also bundle the tool ometascan-data-normalizer into its installation folder. For older versions, we provide following separate tools:

MetaDefender Core versionCentOSDebianWindows
5.2.0 or newerdownload linkdownload linkdownload link
5.0.0 - 5.1.2download linkdownload linkdownload link
4.21.xdownload linkdownload linkdownload link

Step-by-step

  1. Copy a relevant ometascan-data-normalizer.exe(Windows) or ometascan-data-normalizer (Linux) into the installation folder of MetaDefender Core

    1. Default Windows path: C:\Program Files\OPSWAT\MetaDefender Core\
    2. Default Linux path: /usr/sbin/

  3. Stop posting files to MetaDefender Core.

  4. Run the cmd or Terminal as Administrator or root.

  5. Execute the command:

    1. Windows: ometascan-data-normalizer.exe
    2. Linux: ./ometascan-data-normalizer

  6. Start MetaDefender Core service. A few components might take time to start and refresh, so let's wait for a couple minutes.

Notes:

  • After running the ometascan-data-normalizer, it might take considerable extra time to update data if database size is large.
  • Ensure that MetaDefender Core server is running after the tool finishes. If it is stopped, please start it and wait for a while
  • If you encounter error or warning during the execution, please wait for a couple of minutes before executing it again.

Log file

Filename format: data_normalizer-yyyyMMdd.log

  • On Windows, log file will be placed at C:\Program Files\OPSWAT\MetaDefender Core\data\logs\
  • On Linux, log file will be placed at /var/log/ometascan/
Log levelMessageTroubleshooting
INFO[INFO ]: Data normalization is done, data: filetype, instance id: 1The process is completed successfully.
WARNING[WARNING]: Data normalization is failed, data: filetype, instance id: 1,...The data cannot be updated. Collect all the log files and send the support package to us for troubleshooting.
ERROR[ERROR ]: Select all last_request___id unsuccessfully....The data cannot be updated. Collect all the log files and send the support package to us for troubleshooting.
ERROR

[ERROR ]: Cannot connect to Postgres server: metadefender_core...

[ERROR ]: Failed to init Postgres service

  1. Check the privilege when executing the tool and make sure that it has the true privilege to read the .dat file in the installation path.
  2. Collect all the log files and send the support package to us for troubleshooting.
ERROR on consoleERROR: Cannot open file log: <logpath>/data normalizer-20230128.log
  1. Check the log path and make sure that its existence is valid.
  2. Check the privilege when executing the tool and make sure that it has the true privilege to create and edit the log file in the log path.
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
Next to read:
Migrate MetaDefender Core from one machine to another machine
On This Page
Rectify malformed FileType data in PostgreSQL databaseParametersUsageLog file