Troubleshooting Virtual Server Provisioning - Support and Troubleshooting

Troubleshooting Virtual Server Provisioning Issue  There are a myriad of issues that can come up before, during, and after the provisioning process. This article will attempt to detail common issues/errors encountered and how to resolve them.

This will primarily focus on the cloud aspects and will lightly touch on, if at all, anything regarding platform usability and/or issues. This is also written from the perspective of AWS & Linux but the concepts broadly cover what you would experience when provisioning Windows machines or on Azure.

Quick Points: - Correct username must be used for basic auth credentials attached to OS profile record - MID servers must have 443 access to REST API endpoints for AWS/Azure - MID servers must be in the allow-liste (public IP) for SSH and ICMP for whichever Security Group the provisioned EC2 is in (default SG if not defined) - MID servers must be in the allow-list on their internal firewalls to allow SSH/ICMP inbound/outbound to the EC2 - Make sure the user attempting to SSH to the EC2 is allowed in the correct Security Group list

Resolution

The first step of provisioning is executing a REST API call over HTTPS (443) to the Cloud Provider. If networking is not properly configured to allow the MID server to reach the Cloud Provider endpoint, then provisioning will fail. Troubleshooting this is covered in more detail in KB0713124

If provisioning is successful, issues are most likely to arise as part of the client attempting to login to the server (using the SSH key they selected during provisioning) and being unable to do so (it's also likely as Day 2 Orchestration becomes more prominent that it'll be a result of issues with those operations). In this example of attempting to SSH into a Linux EC2 and failing, the follow error is viewed when attempting to SSH with verbose debugging enabled: debug1: Trying private key: client.pem debug3: sign_and_send_pubkey: RSA SHA256:+wciJTNlkvd2pCirj6BkG5T39qHMa5VwkbxsUpH7ZBc debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey).

The cause here is that the selected SSH key did not have it's public key properly "infused" onto the created EC2. This is because, during provisioning, AWS does not allow more than 1 SSH key to be assigned to a server. What we do is provision the server with the management key (found in sn_cmp_preference). Then, after provisioning is complete, the MID server attempts to SSH to the EC2 using the management key and then add the publickey for the selected SSH key in provisioning to the server's .ssh/authorized_keys file.

We can verify this by doing the following: - Have the client change sn_cmp_preference to an SSH key credential they have a privatekey for - Provision a new Linux EC2 - After provisioning is successful have the client SSH into the machine using the key set for sn_cmp_preference (e.g: ssh -i ".ssh/[keyname].pem" [default username]@[public IP address] - Run the command "cat .ssh/authorized_keys" and note the output. If you're having issues, it should look similar to this: ubuntu@[hostname of target]:~/.ssh$ cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0MPyPBUO7LuXvOBvSbxCr385i1xhO8Oa2d4UQKpF+xQxchyMyOfuvw2gO8ZN1S/Z7W46jl+4elKMK1VlUSUb7Gg4nUjmmwrFbLI5XFYp1ldJAcgG9nmtW/A9n5rq/J42W7ab4z8Zf3+BjPp5KOHaEdyVbCTuJe6fkD+u4/UD//FGZ/EqDg2aavNpdJdeZsS9rvReUo0xzCav8CplXt45DFO0ptxnsPHKRggylHsTkNwIBCdzF5DizA+R1XRplPdQILS5MqEB5d+YB2RUuwV8CvmmmgXaxIr8Br7jvTOluveZbfbAt9yKdGzmVoUcPCD9sP2KHsLsIxwmw2MSnL3AZ replacement-mgt ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0MPyPBUO7LuXvOBvSbxCr385i1xhO8Oa2d4UQKpF+xQxchyMyOfuvw2gO8ZN1S/Z7W46jl+4elKMK1VlUSUb7Gg4nUjmmwrFbLI5XFYp1ldJAcgG9nmtW/A9n5rq/J42W7ab4z8Zf3+BjPp5KOHaEdyVbCTuJe6fkD+u4/UD//FGZ/EqDg2aavNpdJdeZsS9rvReUo0xzCav8CplXt45DFO0ptxnsPHKRggylHsTkNwIBCdzF5DizA+R1XRplPdQILS5MqEB5d+YB2RUuwV8CvmmmgXaxIr8Br7jvTOluveZbfbAt9yKdGzmVoUcPCD9sP2KHsLsIxwmw2MSnL3AZ replacement-mgt

- Note that this only includes the management key, and not the public key of the selected user key

The reason we use the management key during provisioning is that the Instance needs to be able to access the EC2 for management/lifecycle purposes. If we didn't add the management key, we would have no way to access/administrate the server.

This means we now have a whole other set of factors we need to account for that clients likely haven't considered (particularly if they have strict security/firewall rules)

Verify networking configurations. Does the client connect to AWS over the Internet or do they have a private VPN gateway (AWS VPN, AWS Direct Connect)? This will determine if the client needs to allow-list the MID server(s) public IPs or their private IPs. Also note that they may need to check their internal firewall rules to ensure they are not dropping SSH/ICMP between the MID and EC2.

If the MID server is not allowed to connect to the EC2 with SSH, then you may encounter this NPE from the AWS Java JDK: java.lang.NullPointerException at com.amazonaws.services.ec2.model.transform.EC2RequestHandler.beforeRequest(EC2RequestHandler.java:46)

If the MID server is not allowed to connect to the EC2 with SSH and ICMP, then may see the following errors in the MID agent logs: - 01/04/19 21:35:59 (298) Camel (camel-1) thread #3813 - ProducerTemplate WARNING *** WARNING *** Could not reach the VM on the following IP addresses ([public ip]); ([private ip]). Check the Security Group settings to allow inbound ICMP and SSH traffic.&#13; - 01/04/19 23:46:34 (917) Camel (camel-1) thread #3814 - ProducerTemplate WARNING *** WARNING *** Node is not ready to accept SSH connections.. waiting.. Error com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out: connect&#13;

These originate from AWSComputeProvider.java, which is attempting to make the SSH connection to add the selected SSH publickey to the EC2

Once these issues are resolved, you can change the sn_cmp_preference credential back to the default management key, provision, and attempt to connect with the SSH key selected during provisioning. Do note that the client needs to make sure that they themselves have SSH access to the Security Group, otherwise they're going to see errors when attempting to connect. Related Links - Make sure to use the correct default username when attempting to SSH. Be sure to verify that the username is correct for the Basic Auth credentials attached to the OS Profile record. This varies by Linux distribution (RHEL: ec2-user, Ubuntu: ubuntu): https://alestic.com/2014/01/ec2-ssh-username/

- When trying to provision with a custom Security Group (ComputeFirewall variable) a subnet must be defined or the provisioning will not properly apply the Security Group and will use the default SG. Take note of this if attempting to provision with a specific SG.

- Since provisioning activities can use any MID server with defined Cloud capabilities, ensure that networking configurations are valid for all Cloud Management MID servers

- Private images can have a pre-defined SSH key. This prevents the system from infusing the default management key from sn_cmp_preference and prevents the Instance from being able to access the EC2 at all. This is a limitation on AWS' end. The workaround here would be to change sn_cmp_preference to the default SSH key in the private image so that the Instance can access the server post provisioning.