Troubleshoot sending data from Amazon Data Firehose to HLARecommended troubleshooting workflow There are many components involved in MID-less ingest for HLA. It's best to try and narrow down where the issue is as early as possible, which could include: Your corporate and/or cloud network is preventing data egress to the ServiceNow data center.There is a misconfiguration of CloudWatch and logs are not being forwarded to a Data Firehose.There is a misconfiguration of Firehose and data is not being forwarded correctly to ServiceNow.There is a misconfiguration of your ServiceNow instance and/or the HLA backend and data is not able to be received correctly. HLA-specific and instance-specific troubleshooting suggestions Review the requirements and ensure HLA is property deployed to your instance and app versions are up-to-date in KB2117152. Amazon Data Firehose-specific troubleshooting suggestions Additional links to AWS tutorials and product documentation are available in KB2117152. As of the May 2025 release, only Amazon CloudWatch logs are supported as an input to HLA via Amazon Data Firehose. Check Firehose Configuration Delivery Stream Status: Ensure the Firehose delivery stream is active and not in a failed state.Delivery Stream Type: Verify if the stream type is CloudWatch Logs.Buffering Configuration: If logs are sent in batches, check buffer size (MB) and buffer interval (seconds) settings.Try reducing the buffer interval to test if data gets sent faster. Validate IAM Permissions IAM Role for Firehose (DeliveryStreamRole) Ensure it has permissions for firehose:PutRecord, firehose:PutRecordBatch, and firehose:DescribeDeliveryStream.If writing to CloudWatch Logs or S3 as a backup, check logs:PutLogEvents and s3:PutObject. IAM Role for CloudWatch Logs (CloudWatchLogsRole) If logs are coming from CloudWatch, ensure firehose:PutRecordBatch permission is granted. IAM PassRole: Ensure the execution role has iam:PassRole for Firehose to assume the necessary IAM roles. Verify CloudWatch Input into Firehose Check Incoming Data Volume If Firehose isn't receiving data, confirm if the data source CloudWatch is properly pushing data.Use aws firehose describe-delivery-stream to check IncomingBytes and IncomingRecords. Inspect Firehose Logs and Metrics Check CloudWatch Metrics for Firehose DeliveryToS3.Bytes → If data is going to S3 as a backup, the remote endpoint may be failing.DeliveryToHttpEndpoint.Success → Indicates successful delivery to the remote HTTP endpoint.DeliveryToHttpEndpoint.Failure → Indicates failed requests to the endpoint. Validate Remote Endpoint Configuration Check HTTP Endpoint URL Ensure the endpoint URL is correct and accessible.If using authentication (e.g., API key, OAuth token), verify credentials. Check HTTP Response Codes A high number of 4xx (client errors) → Issue with request format, missing headers, or authentication.A high number of 5xx (server errors) → Issue with the remote service, check API availability. Enable Firehose Retry Mechanism Check RetryOptions.DurationInSeconds (default: 900s).If retries are exhausted, data may be lost or stored in S3 backup. Check Backup & S3 Destination Verify S3 Backup for Failed Data If S3BackupMode: FailedDataOnly is enabled, inspect the backup bucket for errors.Check the S3Configuration section in the Firehose setup. Inspect Data Format & Compression If using GZIP or JSON encoding, ensure the remote endpoint can process the format. Test with a Different Destination Temporarily configure Firehose to send data to an S3 bucket instead of the remote endpoint.If data reaches S3 but not the endpoint, the issue is with the remote service. Review AWS Service Limits & Quotas Firehose has throughput limits (e.g., PutRecord is limited to 5 MiB per second per shard).If using multiple delivery streams, verify account limits using: aws service-quotas get-service-quota --service-code firehose --quota-code L-XXXXXXX