Requirements to send data to HLA from Amazon Data Firehose - Support and Troubleshooting

Requirements to send data to HLA from Amazon Data Firehose Introduction ServiceNow Health Log Analytics (HLA) predicts IT issues from log data before they affect your users. The application helps you solve problems faster by collecting, analyzing, and correlating machine-generated log data in real time.

The Amazon Data Firehose connector is a way for AWS customers to securely stream CloudWatch logs directly to HLA from AWS their accounts without a MID server. This connector requires a new Data Firehose stream to be created in each AWS account that contains log data you would like to be analyzed by HLA.

Planning for an HLA integration Identify AWS accounts with application and infrastructure AWS logs that you want to send to HLA. Some types of log data sources that are of particular interest: Logs from applications that support critical business functions. Logs from databases, serverless functions. Logs from cloud virtual machines. Review documentation and the example CloudFormation template to understand the IAM polices, roles, and Data Firehose configuration. Make appropriate changes or modifications to comply with your internal cloud security policies. Understand the impact on your AWS bill: use the Data Firehose pricing calculator to calculate the price of streaming your expected volume of CloudWatch logs to HLA. Complete prerequisites for configuring your instance. Solution architecture

Note that a Data Firehose stream is required for each account that sends data to a specific ServiceNow HTTPS destination with an integration identifier and access token. We recommend one HLA Data Input per Data Firehose stream.

What’s needed in AWS A new AWS Data Firehose stream needs to be created with a destination that points to an HTTPS endpoint in the ServiceNow Datacenter. The destination must include headers that specify a unique ServiceNow destination and integration. IAM policies to allow Firehose to read from specific AWS CloudWatch log groups. Configuration via AWS Console or the example CloudFormation template to complete the above steps. The CloudFormation template is available for download during setup of the integration in ServiceNow. Subscription filters on your log groups to forward logs to Data Firehose streams. What’s needed in ServiceNow Version 36.0.19 of the HLA (or greater) HLA scale is enabled on your instance Security model The Data Firehose connector to HLA is push-based, meaning Amazon Data Firehose is responsible for streaming data to a regional ServiceNow datacenter. Only data configured to be sent to Firehose is forwarded to ServiceNow. Data is authorized by a token set on the header, and the token can be revoked by a ServiceNow administrator at any time. By default, it expires after 6 months. ServiceNow customers are responsible for managing and rotating tokens used in Amazon Data Firehose per their internal security policies.

ServiceNow does not need any direct API access to your AWS accounts for this integration to function : the only AWS configuration needed is a new Data Firehose Destination that is configured to send data to a remote HTTPS endpoint managed by ServiceNow.

Setup and configuration in the AWS Console See KB1957134 for setup instructions in the AWS console.

Troubleshooting See KB1957226 for troubleshooting steps and recommendations. For additional documentation, including AWS documentation suggestions, see KB2117152.

Appendix: IAM Permissions needed to configure this integration At minimum, a new Data Firehose must be given permission to read from appropriate Amazon CloudWatch log groups. S3 is also used to log delivery failures. Additional permissions are needed to run CloudFormation if using the example provided by ServiceNow.

Always use the latest CloudFormation template attached to the data input for the latest permissions.

AWS Identity and Access Management (IAM) - needed to create roles for Firehose

iam:CreateRole iam:DeleteRole iam:AttachRolePolicy iam:DetachRolePolicy iam:PutRolePolicy iam:DeleteRolePolicy iam:PassRole iam:GetRole iam:GetPolicy iam:GetPolicyVersion iam:ListAttachedRolePolicies iam:ListRolePolicies Amazon Kinesis Firehose - needed to create and manage a new Firehose object

firehose:CreateDeliveryStream firehose:DeleteDeliveryStream firehose:DescribeDeliveryStream firehose:ListDeliveryStreams firehose:PutRecord firehose:PutRecordBatch firehose:UpdateDestination Amazon CloudWatch Subscription Filter - n eeded to forward AWS logs to your Firehose

logs:PutSubscriptionFilter Amazon CloudWatch Logs - needed if enabling logging for Firehose

logs:CreateLogGroup logs:DeleteLogGroup logs:DescribeLogGroups logs:CreateLogStream logs:DeleteLogStream logs:DescribeLogStreams logs:PutLogEvents logs:PutRetentionPolicy logs:GetLogEvents logs:ListTagsLogGroup Amazon S3 - needed if enabling error logs and diagnostics for Firehose

s3:CreateBucket s3:DeleteBucket s3:GetBucketLocation s3:GetBucketPolicy s3:ListBucket s3:ListBucketMultipartUploads s3:GetObject s3:PutObject s3:DeleteObject s3:AbortMultipartUpload s3:PutBucketPolicy s3:PutLifecycleConfiguration s3:GetLifecycleConfiguration s3:DeleteLifecycleConfiguration AWS Security Token Service (STS) - needed for IAM roles

sts:AssumeRole AWS CloudFormation (if using) - needed to run the provided Cloudformation template

cloudformation:CreateStack cloudformation:UpdateStack cloudformation:DeleteStack cloudformation:DescribeStacks cloudformation:DescribeStackEvents cloudformation:DescribeStackResources cloudformation:ListStacks cloudformation:GetTemplate cloudformation:GetTemplateSummary cloudformation:ValidateTemplate