Forwarding log data to an Elasticsearch cluster

Elasticsearch is an open-source search engine that runs on an external Elasticsearch server cluster. You can forward log data from Log Servers and Management Servers to an Elasticsearch cluster to improve the performance of browsing and searching for log entries, report generation, and other log-related features.

Important: Forwarding log data to an Elasticsearch cluster is an advanced feature that requires knowledge of how to configure Elasticsearch. You must already have an Elasticsearch cluster deployed and configured in your environment.

For more information about requirements for using Elasticsearch with the SMC, see Knowledge Base article 17583.

You can browse log entries that have been forwarded to an Elasticsearch cluster using the Management Client in the same way as for other log entries. The Log Server automatically maps log fields to the corresponding Elasticsearch fields.

Elasticsearch indexes the following kinds of log fields:

  • Log fields that can be used for filtering and browsing log entries.
  • Log fields that can be used for reporting.

Elasticsearch indexes log data only when a log data file on the Log Server is complete. Typically, log data files are completed about once every hour. If a large number of log entries are received, the Log Server might create multiple log files each hour.

The configuration consists of these general steps:

  1. Configure Elasticsearch in your environment.
    1. Deploy and configure an external Elasticsearch cluster.
    2. (Recommended) Configure TLS and client authentication in Elasticsearch.
  2. In the Management Client, create an Elasticsearch Cluster element.
  3. (Optional) Override the settings for client authentication that are defined in the Elasticsearch Cluster element in the properties of the Log Server or the Management Server.