Stream logs using Elasticsearch data input - Advanced guide - Support and Troubleshooting

Stream logs using Elasticsearch data input - Advanced guide Summary This knowledge base article will help you stream your logs data from ElasticSearch indices into ServiceNow's Health Log Analytics and avoid common pitfalls.

Note: A single ElasticSearch Data Input is used to fetch logs from a single index (including its rolling indices).

Instructions Validate ElasticSearch Access : Currently, the Elastic Data Input only supports the basic authentication method (token authentication). Client certificate authentication is not yet supported.

The user provided to access the Elastic index must have the correct permissions:

1. If you have a licensed ElasticSearch, grant the following permissions:

Cluster permissions:

monitor Note : When the Use minimal privileges check box is selected in the Elasticsearch data input configuration, you don't need to grant this permission

Indices permissions:

view_index_metadata monitor read

Note : When the Use minimal privileges check box is selected in the Elasticsearch data input configuration, grant only the read permission.

In case you receive the following error g o to Query Settings --> check the " Use search-after API " checkbox. " *** WARNING *** [Elastic Octobus Playground] - clearScroll not succeeded, errorCode: {"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/indices/scroll/clear_all] is unauthorized for user ### "

2. If you are using an Open Distro ElasticSearch, grant the following permissions:

Cluster permissions:

cluster:monitor/* indices:data/read/scroll indices:data/read/scroll/clear Note : When the Use minimal privileges check box is selected in the Elasticsearch data input configuration, you don't need to grant the cluster:monitor/* permissions.

Index permissions:

indices:data/read* indices:admin/exists indices:admin/aliases* indices:admin/get indices:monitor/stats indices:admin/mappings/fields/get* indices:admin/mappings/get indices:monitor/settings/get Note : When the Use minimal privileges check box is selected in the Elasticsearch data input configuration, grant only the indices:data/read* and indices:admin/get permissions.

Fill in the correct Timestamp field & format:

To start streaming logs from Elastic, you must configure the relevant timestamp details which fit your index.

The common default timestamp field is: @timestamp The common default timestamp format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

To support multiple timestamp formats, list the different formats separated by: $$

Validate Connectivity :

The network path between the MID Server and Elastic must be open for communication on the 9200 port (or any other port). This can be tested by running "curl http:// :9200 " from the MID.

Scroll Slice API (default behavior):

The Scroll slice API ( https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/scrolling-documents.html ) is used to fetch a large amount of documents, to cope with real-time streaming.

When using this API, it's recommended to set the 'Max scroll slices' to the number of shards of the relevant index. 'Max scroll slices' has to be less than or equal to the 'Max connections per route' .

Search after API

The Search after API ( https://www.elastic.co/guide/en/elasticsearch/reference/8.0/paginate-search-results.html - search-after ) is used by specifying the 'Search-after tie breaker' to handle data that contains thousands of events (docs) which have the same timestamp. The tie breaker is usually either _id (es >6.x) or _uid (es

Related Links For basic configuration steps of the Elastic Data Input click here