ECC Queue database footprint and disk space usage explainedHuge database tables for the ECC Queue can have a performance impact on a ServiceNow instance. ServiceNow charges for additional database storage for instances as well, and we sometimes have questions related to that. This article aims to explain that footprint and answer the most common questions. Table of Contents What is the ECC QueueWhat is the ECC Queue table used byDatabase Rotation, and retention periodAn example What is the ECC Queue The External Communication Channel (ECC) Queue is a table, named ecc_queue, and is the connection point between an instance and the MID Server, or directly to an integration endpoint. It is 2 queues in 1 table, with the "queue" field defining if it is Input or Output queue. Any application or platform feature that is doing an integration with a 3rd party system, will be using the ECC Queue for defining the job to run, and usually also as a way of transferring the sent and/or received data. The Topic field defines the "probe" to run, which means what function/code. Other fields such as Source, Name and Payload define other parameters for that probe. If there is output data, such as data to be sent to a customer's internal or 3rd party systems, the Payload field will be used for that data. Input data will usually use the payload field, if it is below 500k of data, otherwise will put the data in an attachment payload.txt. The exception is when other instance REST or SOAP APIs are used to ingest the data directly to the target tables instead. More on the fields, and process by which the MID Server processes jobs in this queue is explained in detail in:KB0855595 How the ECC Queue table records get processed: from output ready to input processed What is the ECC Queue table used by This will depend on the Applications and Plugins you have installed, and how you have configured them. The most common are: LDAP Integration, to keep your user lists updated from your LDAP/Active Directory server.IntegrationHub, and other REST/SOAP integrations, where workflows or business logic trigger integrations.System Import Sets, to bringing data into the instance from databases of 3rd party systems, including Service Graph Connectors for populating the CMDB.ITOM Visibility features, including Discovery, Service Mapping and Agent Client Collector. If this is used it usually accounts for most of the data.ITOM Health features such as Event Monitoring/Operational Intelligence and Health Log Analytics, where Events and Logs are brought into the instance. If you are looking at the records in the ECC Queue table, you can work out what feature they are from by using this article:KB0727132 How to link an ECC Queue record back to a specific Feature or Job Database Rotation, and retention period Once the job has run to completion, and the input data has been processed in the instance, these records are just a log of what has been run, and no longer needed. To keep the table size under control, and to help optimise queries for current/today's records, the ECC Queue table uses Table Rotation. The Table Rotation documentation explains this in more detail, together with the advantages/disadvantages. The default setup is 7 rotations of 1 day each, and we call these 7 sql tables "shards". That means there is 5-6 days of data present, because 1 shard is empty ready for tomorrow, and one shard is being filled now. Of those 5-6 days of data, 4-5 days are visible from the application, because 1 shard is offline waiting to be truncated. An admin role user can see how ecc_queue table rotation is set up by navigating to System Definition -> Table Rotations, and opening the ecc_queue record. You can see from there which the current shard is from the dates in the related list. For 7 rotations, the shards are named ecc_queue0000 to ecc_queue0006, and you can also see what data is in each of these from the instance by entering e.g. "ecc_queue0000.FILTER" in the navigation. Note: Some customers have customised this rotation, usually to increase the retention period. A few other customers are not using rotation, and have a table cleaner instead. An example The following is an example of a seemingly small change to how Discovery is configured, causing a considerable increase in the table size. Note: This customer had already chosen to increase the retention period of records by setting the rotations to 14, which is why shards numbered up to ecc_queue0013 are in this graph. That would mean more than double the amount of storage usage would be expected, compared to the default 7 rotations of most customers. Up to Jan 11th, a number of quite big Discovery Schedules were running daily. A few of the schedules were set to run Daily, with a lot of others set to "Run After Discovery" in a long sequence. In total, all schedules were launching approx. 400,000 probes per day, which is something ServiceNow can handle fine, and that resulted in approx. 700GB storage for the ecc_queue shards. On Jan 11, one daily schedule was changed to run every 3 hours, so all the the run after schedules also run every 3 hours. That sequence was 150,000 probes, and so now instead of 150k per day there were 1,200,000 probes per day. Then on Jan 26th another daily sequence of discovery schedules launching 100,000 probes was was changed to every 3 hours, and the full effect of that won't be realised until we rotate through all the shards over the subsequent 14 days. That has added >2TB to the database footprint and will probably grow to over 3TB before it settles down. So be wary.