Troubleshoot log storage

There are several possible causes and solutions when alerts indicate that the disk of an engine or a Log Server is filling up with log files.

The “Log spool filling” alert indicates that logs are not being transferred from the engine at all, or the engine is generating logs more quickly than they can be transferred to the Log Server.

Steps

  1. If an engine is filling up with logs, do one of the following:
    1. Check that the Log Server is running. If it is not running, try to start it. If the Log Server is running, check for network problems between the engine and the Log Server. The log entries are spooled on the engines if they cannot be sent to the Log Server. Stopping and restarting the Log Server process can help in resetting the connection.
    2. If the volume of logs is high, they might not be transferred quick enough, and logs must be spooled even though they are being transferred. If you suspect this is the case, turn off all diagnostics logs for all engines that you are not actively troubleshooting. Also turn off logging for all rules that have connection tracking set to off (because these rules log each packet individually). Finally, check if logs that are currently pruned could be prevented from being generated in the first place.
  2. If the Log Server is filling up with logs, do one of the following:
    1. Set up log management tasks that archive and remove the oldest Log entries from the Log Server hard disk. To avoid problems in the future, set up tasks to run automatically at regular intervals.
    2. In an emergency, you can also move or delete old log entries manually. The logs are stored on the Log Server machine under a folder structure based on dates and times (default location is <installation directory>/data/storage). You should always avoid manual handling of the newest entries.