Log and event storage best practices

Best practices for event data storage depend on the data compliance regulations you must meet, such as PCI and HIPAA. Workload Security retains security events for 32-39 days and system events for 13-17 weeks (depending on when database maintenance is scheduled).

Steps to consider:

  1. Forward security events to external storage. See Forward events to an external Syslog or SIEM server.

  2. Set thresholds in the log inspection module for event storage or event forwarding. Severity clipping allows you to send events to a Syslog server (if enabled) or to store events based on the severity level of the log inspection rule. See Thresholds for Event Storage or Event Forwarding.

Limit log file sizes

You can set the maximum size of each individual log file and how many of the most recent files are kept. Event log files will be written to until they reach the maximum allowed size, at which point a new file will be created and written to until it reaches the maximum size and so on. Once the maximum number of files is reached, the oldest will be deleted before a new file is created. Event log entries usually average around 200 bytes in size and so a 4 MB log file will hold about 20,000 log entries. How quickly your log files fill up depends on the number of rules in place.

  1. Open the Computer or Policy editor for the policy that you want to configure.
  2. Go to Settings > Advanced > Events.
  3. Configure these properties:

    • Maximum size of the event log files (on Agent/Appliance): Maximum size that the log file can reach before a new log file is created.
    • Number of event log files to retain (on Agent/Appliance): Maximum number of log files that will be kept. Once the maximum number of log files is reached, the oldest file will be deleted before a new one is created.
    • Do Not Record Events with Source IP of: This option is useful if you don't want Workload Security to make record events for traffic from certain trusted computers.
      The following three settings let you fine tune event aggregation. To save disk space, Deep Security Agents will take multiple occurrences of identical events and aggregate them into a single entry and append a "repeat count", a "first occurrence" timestamp, and a "last occurrence" timestamp. To aggregate event entries, Deep Security Agents need to cache the entries in memory and then write them to disk.
    • Cache Size: Determines how many types of events to track at any given time. Setting a value of 10 means that 10 types of events will be tracked (with a repeat count, first occurrence timestamp, and last occurrence timestamp). When a new type of event occurs, the oldest of the 10 aggregated events will be flushed from the cache and written to disk.
    • Cache Lifetime: Determines how long to keep a record in the cache before flushing it to disk. If this value is 10 minutes and nothing else causes the record to be flushed, any record that reaches an age of 10 minutes gets flushed to disk.
    • Cache Stale time: Determines how long to keep a record whose repeat count has not been recently incremented. If Cache Lifetime is 10 minutes and Cache Staletime is 2 minutes, an event record which has gone 2 minutes without being incremented will be flushed and written to disk.
      Regardless of the above settings, the cache is flushed whenever events are sent to Workload Security.
  4. Click Save.

Event logging tips

  • On computers that are less important, modify the amount of logs collected. This can be done in the Events and Advanced Network Engine Options areas on the Computer or Policy editor > Settings > Advanced tab.
  • Consider reducing the event logging of firewall rule activity by disabling the event logging options in the firewall stateful configuration. (For example, if you disable UDP logging, it will eliminate unsolicited UDP log entries.)
  • For intrusion prevention rules, the best practice is to log only dropped packets. If you log packet modifications, it may cause too many log entries.
  • For intrusion prevention rules, only include packet data (an option in the intrusion prevention rule's Properties window) when you are interested in examining the behavior of a specific attack. Packet data increases log sizes, so it shouldn't be used for everything.