How to validate Kafka topic connectivity using native Apache Kafka - Product Knowledge

How to validate Kafka topic connectivity using native Apache Kafka For Hermes topics Step 1:

Install native Apache Kafka, if testing for connectivity from a specific host, install this on the relevant host.

https://kafka.apache.org/

Step 2:

Use the IPKI Cert Generator tool to create a truststore and keystore:

.service-now.com/now/ipki-cert-generator/main

Download this on the host that will be running kafka.

Step 3:

Navigate to and open consumer.properties file in the config folder of the kafka directory where you installed native Apache Kafka.

Populate it with the truststore and keystore file path downloaded from step 2 and the password details, i.e.:

security.protocol=SSL

ssl.truststore.password=

ssl.truststore.location= /truststore.p12

ssl.truststore.type=PKCS12

ssl.keystore.password=

ssl.keystore.location= /keystore.p12

ssl.keystore.type=PKCS12

ssl.key.password=

Note: If group and bootstrap are defined in the consumer.properties file, comment these out as they are set in the kafka-console-consumer.sh scirpt invocation examples.

Step 4:

In your terminal navigate to the Kafka directory from Step 1 and run the following to utilize the kafka-console-consumer.sh script to consume messages for the topic and bootstrap server defined:

Unix:

./bin/kafka-console-consumer.sh --consumer.config ./config/consumer.properties --topic snc. . .sn_ . --group snc. . --from-beginning --bootstrap-server .service-now.com:4100, .service-now.com:4101, .service-now.com:4102, .service-now.com:4103 Windows:

bin\windows\kafka-console-consumer.bat --consumer.config config\consumer.properties --topic snc. . .sn_ . --group snc. . --from-beginning --bootstrap-server .service-now.com:4100, .service-now.com:4101, .service-now.com:4102, .service-now.com:4103

For Kafka SSL Credential record for external Kafka broker topics Step 1:

Install native Apache Kafka, if testing for connectivity from a specific host, install this on the relevant host.

https://kafka.apache.org/

Step 2:

Gather the relevant SSL credential details from the Kafka Topic owners/admins and cross reference with the Kafka SSL credentials record created in ServiceNow.

https://www.servicenow.com/docs/r/washingtondc/integrate-applications/integration-hub/create-kafka-ssl-credentials.html?contentId=Ehv7Z4xGc6bRUSiVTTCWyQ

https://www.servicenow.com/docs/r/washingtondc/it-operations-management/health-log-analytics/hla-data-input-kafka-credentials.html

These fields will directly correlate to the properties and their values in the consumer.properties file to facilitate the connection to the relevant Kafka broker.

Step 3:

Navigate to and open consumer.properties file in the config folder of the kafka directory where you installed native Apache Kafka.

Populate it with the SSL credentials details provided by the Kafka Topic owners/admins and what is configured on the ServiceNow Kafka SSL credentials record.

The fields directly correspond to the consumer.properties file properties.

Example consumer.properties For Security protocol = SSL:

security.protocol=SSL

ssl.truststore.password=

ssl.truststore.location= /truststore.p12

ssl.truststore.type=PKCS12

ssl.keystore.password=

ssl.keystore.location= /keystore.p12

ssl.keystore.type=PKCS12

ssl.key.password=

Example consumer.properties For Security protocol = SASL_SSL:

security.protocol=SASL_SSL

ssl.truststore.password=

ssl.truststore.location= /truststore.p12

ssl.truststore.type=PKCS12

# The sasl.mechanism value depends on your broker configuration

sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \

username=" " \

password=" ";

# Optional: Keystore if using client certificate authentication

# ssl.keystore.password=

# ssl.keystore.location= /keystore.p12

# ssl.keystore.type=PKCS12

# ssl.key.password=

Example consumer.properties For Security protocol = SASL_PLAINTEXT:

security.protocol=SASL_PLAINTEXT

# The sasl.mechanism value depends on your broker configuration

sasl.mechanism=PLAIN

sasl.username=

sasl.password=

Note: If group and bootstrap are defined in the consumer.properties file, comment these out as they are set in the kafka-console-consumer.sh scirpt invocation examples.

Step 4:

In your terminal navigate to the Kafka directory from Step 1 and run the following to utilize the kafka-console-consumer.sh script to consume messages for the topic and bootstrap server defined:

Unix:

./bin/kafka-console-consumer.sh --consumer.config ./config/consumer.properties --topic --group --from-beginning --bootstrap-server : Windows:

bin\windows\kafka-console-consumer.bat --consumer.config config\consumer.properties --topic --group --from-beginning --bootstrap-server :

Note:

For external Kafka topics, the keystore and truststore will need to be provided by the relevant Kafka broker owners/admins.

If the host is reachable by a client with openssl you can use openssl to retrieve the certificates the broker presents i.e.:

openssl s_client -connect : -showcerts