Exporting Bulk Data from Servicenow via REST Web Service Pagination


Description

Servicenow always recommends exporting data in chunks in case of large data set with millions of records to avoid any performance implications and impact on other services running on the system. The Procedure below will explain how to Export Bulk Data from Servicenow via REST Web Service Pagination.

Procedure

By default, ServiceNow has the max limit of 10000 records which will be returned in any rest call and is being caused by the omission of the parameter sysparm_limit which default value is 10000. If you specify a higher value in the URL then you can get the desired amount of records.

Example: /api/now/table/incident?sysparm_limit=40000 

The best practice to retrieve large amounts of records is to by using pagination. That is, get sub-sets of records in different calls, for example, first, you get records from 1 to 10000 in a call, then you get from 10001 to 20000, then from 20001 to 30000, etc. 

 Note: Increasing the maximum returned record (10000 by default) may have performance implications.

To do so, you can make use of the parameters 

In this case, the first call would be something like 

/api/now/table/incident?sysparm_limit=10000&sysparm_offset=0&sysparm_query=ORDERBYsys_created_on 

the second one 

/api/now/table/incident?sysparm_limit=10000&sysparm_offset=10000&sysparm_query=ORDERBYsys_created_on 

the third one 

/api/now/table/incident?sysparm_limit=10000&sysparm_offset=20000&sysparm_query=ORDERBYsys_created_on 

. and so on...

You can put that in a loop to get all the records using pagination until all the records are read.

Resolution

It is always a best practice not to keep the value of sysparm_limit in check as the default Quota Rules for REST API transactions is set to 60 secs per transaction and increasing sysparm_limit drastically can cause the transaction to time out and get canceled. Default quota rules