Cloud Cost Management 6.0 Features - Container Spend Management and Shared Cost Allocation<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Container spend management Description Cloud-based Kubernetes services from AWS, GCP, and Azure have become increasingly utilized by companies deploying workloads on cloud infrastructure. This mirrors the broader trend of rising adoption of Kubernetes container orchestration technology. With this rise in adoption, there is also a rise in spend for these services. However reporting and allocation of this spend has been challenging because of the nature of this service : Containers are shared resources and its hard to find out which application consumed how much of the underlying resources to be able to allocate it fairly.Providers show the spend at the abstract level (kubernetes cluster) whereas for accurate allocation, customers need breakdown at more granular levels like namespaces or cost of each underlying asset that is part of the Kubernetes cluster. Hence, this feature leverages the data and relationships discovered by ServiceNow discovery to provider deeper breakdown of the Kubernetes spend. Log Location and Debug Properties Validate in the Cloud Cost Management Workspace Job execution(sn_clin_core_insights_execution) to verify if the Spend job executions is completed successfully.Verify the Execution Log (sn_cld_intg_core_execution_log) ,Flow engine context (sys_flow_context) and Flow engine log entry(sys_flow_log) tables for any errors related to spend flow execution.Verify the errors in system logs (syslog.list) table with the Source prefix labelled as sn_cld_intg, sn_clin and sn_cld_spend to see all the unknown/technical errors around the job execution time. Symptoms and Facts The spend for a Kubernetes cluster will be generated only if cost allocation tags are enabled for a Kubernetes cluster in their respective consoles for AWS and GCP providers. For AWS provider, cost allocation tags need to be enabled at the account level and for GCP, it is at the Kubernetes Engine. For the Azure provider, the Kubernetes spend will be generated only if the default node resource group is provisioned during AKS deployment and has this naming convention MC_{name of the cluster resource group}_{name of the cluster}_{location of the cluster}. Troubleshooting Path/Diagram Billing data download tiggers.Billing data download fetches Kubernetes cluster line items and maps each asset or resource of the cluster to Kubernetes cluster based on the tags for AWS and GCP providers. CI placement triggers as part of billing download for CI type Kubernetes cluster cmdb_ci_Kubernetes_cluster.After the billing download completes successfully, Spend execution will trigger. Tag Category 40 column should be populated with the name of the cluster for each asset or resource in the cluster and to the cluster resources in the 'sn_cld_spend_core_monthly_aggregated_cost' table for all 3 providers. Note : Here assets/resources refer to Network/VM/Load Balancer.. Etc Technical details AWS Provider: Enable below mentioned cost allocation tags at account level for a Kubernetes cluster in AWS Management Console before you run a AWS Billing download job to view the Kubernetes spend. Static tag keys containing cluster name : aws:eks:cluster-name, user:eks:cluster-name, eks:cluster-name Dynamic tag keys of the following format : kubernetes.io/cluster/<Cluster-Name> : shared/owned alpha.eksctl.io/cluster-name : <Cluster-Name> Tag "sn_ccm_k8_cluster_name" will be added to the resources already having tags specified above during aws billing download. GCP Provider: Enable cost allocation tag "goog-k8s-cluster-name" for each Kubernetes cluster before you run a Google Cloud Billing download job to view the Kubernetes spend.Tag "sn_ccm_k8_cluster_name" will be added to the resources already having tags "goog-k8s-cluster-name" during GCP billing download. Azure Provider: Tag "sn_ccm_k8_cluster_name" will be added to the Kubernetes cluster during billing download based on meter category "Azure Kubernetes Service". Tag value will be the name of the cluster. Tag "sn_ccm_k8_cluster_name" will be added to the assets/resources of cluster present in the infrastructure/managed resource group after ‘Spend Report monthly Aggregated Cost’ table population as part of post spend execution. The default name for this infrastructure resource group is MC_<resourcegroupname>_<clustername>_<location>. Script includes: Mid Server Script includes AWSCSVRecordProcessor: This is the script containing the logic to add cost allocation tags and the 'sn_ccm_k8_cluster_name' tag to the assets/resources of the Kubernetes cluster and 'sn_ccm_k8_cluster_name' to the Kubernetes cluster for AWS provider. AzureAsynCostUsageNodeDataConstructor: This is the script containing the logic to add 'sn_ccm_k8_cluster_name' to the Kubernetes cluster for Azure provider. GCPBillingNodeDataConstructor: This is the script containing the logic to add cost allocation tag 'goog-k8s-cluster-name' and the 'sn_ccm_k8_cluster_name' tag to the assets/resources of the Kubernetes cluster and 'sn_ccm_k8_cluster_name' to the Kubernetes cluster for GCP provider Script Includes AWSCIHandlerForKubernetesClusterIREObject: This is the script containing the logic for AWS Kubernetes cluster CI Placement AzureCIHandlerForKubernetesClusterIREObject : This is the script containing the logic for Azure Kubernetes cluster CI placement. GCPCIHandlerForKubernetesClusterIREObject: This is the script containing the logic for GCP Kubernetes cluster CI placement. AggregatedMonthlySpendUtil: This is the script containing the logic to add 'sn_ccm_k8_cluster_name' tag to the assets/resources of cluster present in the infrastructure/managed resource group after ‘Spend Report monthly Aggregated Cost’ table population as part of post spend execution. Data Broker Server Script Table:sys_ux_data_broker_transform Cluster Cost Transformer: Transformer for Cloud Insights Workspace to get the service category costs for a cluster. Spend dashboard transformer: Transformer for Cloud Insights Workspace to get Kubernetes cluster cost in Kubernetes Spend Analytics dashboard Tables sn_cld_spend_core_monthly_cost sn_cld_spend_core_monthly_aggregated_cost sn_cld_intg_core_tag_category sn_cld_intg_core_tag_name_value Flow AWS: Generate AWS Spend Report Azure and GCP: Spend Orchestrator Troubleshooting CI Placement: CI placement happens as part of the billing download. All related logs are available as part of the insight pipeline execution stage logs.To look for the CI placement information or error search with the class name for Kubernetes cluster cmdb_ci_Kubernetes_cluster. Once the billing download is complete, verify that tag 'sn_ccm_k8_cluster' is added to each asset or resource of the cluster and to the cluster resources in the AWS (sn_cld_intg_aws_cost_usage) and GCP (sn_cld_intg_gcp_cost_usage) node tables. Once the spend execution is complete, verify that the Tag Category 40 column should be populated with the name of the cluster for each asset or resource in the cluster and to the cluster resources in the 'sn_cld_spend_core_monthly_aggregated_cost' table for all 3 providers. Verify that out of the box tag category is created for Kubernetes cluster with the name ‘Kubernetes cluster name’ and tag name‘sn_ccm_k8_cluster’ mapped with Tag category 40.Verify the errors in system logs (syslog.list) table related to identification engine while doing CI placement for cmdb_ci_kubernetes_cluster. Shared cost allocation feature overview Description: In any organization, cloud resources are shared across multiple business units, departments, cost centers or projects. It is therefore necessary to do the following: Identify shared cloud resources using tagsDownload billing details for these shared cloud resourcesDefine policies that specify how the costs associated with these cloud resources are shared across Business units, Divisions, Departments or Cost Centers.Apply these policies on the billing details downloadedGenerate reports that provide the breakup of direct and shared costs across all cloud resourcesFor Kubernetes containers, the costs can be shared across multiple namespaces. Shared Cost Policies for Kubernetes Namespaces: Prerequisite: To get all the namespaces of a Kubernetes cluster, we need to run Auto K8s Discovery using patterns. We query the namespace column from the cmdb_ci_kubernetes_cluster table while displaying namespace values during the creation of Shared Cost Policies for Kubernetes clusters. Please find the reference link below for Kubernetes Discovery using Patterns: https://www.servicenow.com/docs/bundle/zurich-it-operations-management/page/product/service-mapping/concept/kubernetes-discovery.html Below are the steps to be followed for creating the Kubernetes Shared Cost policies: Navigate to Cloud Cost Management Workspace → Operations → Shared Cost Allocation PoliciesClick New/Edit buttonSelect Service Category: Kubernetes ServiceSet Resource Type: Kubernetes ClusterEnter the Kubernetes cluster name in the Kubernetes Cluster fieldFill in all mandatory fieldsReapply policies or run billing download to populate the spend grouped by namespaces for the newly created Shared Cost policy Shared Cost policies should be created to view spend grouped by Kubernetes namespaces in the spend analytics page. Out of the box (OOB), the system automatically creates Shared Cost policies with the Even allocation type for each Kubernetes cluster that has namespaces. These are created after the billing execution completes or when Shared Cost policies are reapplied. Customers can create their own custom shared cost allocation policies for Kubernetes clusters with the highest priority order. Log Location and Debug Properties Verify the errors in system logs (syslog.list) table with the Source prefix labelled as sn_cld_intg, sn_clin and sn_cld_spend to see all the unknown/technical errors around the job execution time. Troubleshooting Path/Diagram Validate active shared cost policy is present Validate policy matching records is present in monthly aggregated spend table with active state. Technical details SharedCostController This is the main script which will be called once user will click on re-apply policy. This will clear all existing record present in shared cost table and re-run policy execution. SharedCostPolicyExecution This script contains the policy execution part it will figure out all the unique policy depending upon run order and other cost distribution parameters. This will also find out cost for source and target costs. SharedCostSplitProcessor This script include will split the cost depending upon allocation type and spilt the cost and create records in shared cost policy. SharedCostk8sPolicyHandler This script include will check all the records present in spend table having tag 40 populated and service category is Kubernetes Service. It will generate Auto generated policy for namespace only. SharedCostk8sPolicyExecution This script will handle all the kubernetes policies execution it will list all policy depending upon run order. SharedCostK8SplitProcessor This script include will split the cost depending upon allocation type and spilt the cost and create records in shared cost policy for kubernetes cluster. Tables: 1. sn_cld_spend_core_sc_policy2. sn_cld_spend_core_sc_policy_target3. sn_cld_spend_core_monthly_shared_cost Known Errors and Workarounds: Exception while executing request: Transaction cancelled: maximum execution time exceededTo resolve this error follow the below steps: 1. Go to Transaction Quota Rules 2. Search for 'REST Batch API request timeout' 3. Increase the timeout to 300 seconds. Kubernetes Spend for “Group by Kubernetes namespace”: We reserved TagCategory40 to store the Kubernetes cluster name for reporting spend by Kubernetes clustername and Kubernetes namespace. OOB, all Kubernetes cluster resources are tagged with sn_ccm_k8_cluster_name.However, the current issue is that this tag is added only when the Kubernetes resource already has at least one tag key-value pair. If a resource has no existing tags, the sn_ccm_k8_cluster_name tag is not populated, causing spend to not appear in the Spend Analytics dashboard for Group By Kubernetes clustername or Kubernetes namespace. A fix will be available in the upcoming release. Customer currently affected can contact the support to get a fix via update sets.