Automated Test Framework - Frequently asked Questions (FAQ)DescriptionThe goal of this article is to answer generic frequent requests/questions ServiceNow Technical Support receives in relation to Automated Test Framework (ATF). What is Automated Test Framework (ATF)? When and how should I use ATF?How do I get started with ATF? What are the available ATF Features for the ServiceNow Releases?What are Test Suites?What are Quick Start Tests?Should my Quick Start Tests pass after applying Customizations ?What best practices should I consider when creating ATF Tests?ATF related system propertiesWhat are the recommendations when running ATF Tests?What is Headless Browser for Automated Test Framework?What is ATF Test Generator and Cloud Runner?Can I run my ATF Tests in Production?Can I disable Rollback for ATF Tests?Can I design one test to rely on the outcome of another test?How do we migrate ATF Tests to other environments?How can custom UI such as UI Pages be tested via ATF?How do we verify that custom UI components are testable?What are some of the common issues reported in ATF ? What is Automated Test Framework (ATF) ? The Automated Test Framework (ATF) enables you to create and run automated tests to confirm that your instance works after making a change. For example, after an upgrade, during application development, or when deploying instance configurations with update sets. Review failed test results to identify the changes that caused the failure and the changes that you should review. When and how should I use ATF? You should run ATF tests to confirm that the instance still works as designed whenever you upgrade, modify an instance (e.g., customize or make field changes), or have internal development releases. This includes running tests on any scoped apps, custom apps, or out-of-the-box products. With ATF, besides running test suites on demand, you can also schedule them to run during off hours. You can run an entire suite of tests and, in moments, analyze the results to determine if remediation is required. You can use the same tests again for your next release or upgrade if you keep the tests in sync with the configurations and customizations they test. ATF is intended to test the changes you make to your instance, but not to test base ServiceNow functionality. How do I get started with ATF? Review the available documentation: Getting started with the Automated Test FrameworkComplete the training : Automated Test Framework TrainingReview the following videos: Getting Started with the Automated Test Framework (ATF) Setting Up Your First ATF Test What are the available ATF Features for the ServiceNow Releases? You may refer to the Release Notes for ATF features introduced in the specific release : TokyoUtahVancouverWashington What are Test Suites? A test suite can contain both individual tests and other test suites. It allows you to group tests in a specific order to test an application or a group of related features. This way, you're able to run tests and see results as one job. Grouping tests into test suites offers these benefits: Allows testers to run every test in a test suite with one action.Allows testers to run all child test suites in a parent test suite.Allows testers to see test results for every test in a test suite.Allows test designers and testers to schedule when to run test suites.Allows test designers and testers to schedule starting client test runners to support test runs. For more information refer to : Building and running automated test suites What are Quick Start Tests? Quick Start Tests are base system examples of how a feature developer recommends test designers to write tests against their feature. Each application or feature has its own plugin activation requirements for enabling quick start tests. Refer to links below for Available quick start tests by application or feature for the available releases: Tokyo - Available quick start tests by application or feature Utah - Available quick start tests by application or feature Vancouver - Available quick start tests by application or feature Washington - Available quick start tests by application or feature For more information , refer to the Getting started with quick start tests video . This demonstrates how to install and customize quick start test templates that are available for the Automated Test Framework. Should Quick Start Tests pass after applying Customizations ? Customers are expected to alter their test to meet their customization changes What best practices should I consider when creating ATF tests? Refer to this link : ATF best practices ATF related system properties Refer to this link : KB0720524-List of ATF [ Automated Test Framework ] related system properties What are the recommendations when running ATF Tests? When running ATF tests, make sure your browser has plenty of power. ServiceNow has no control over browser throttling behavior, but we do have some workarounds. Here are a few: Run unattended tests on a machine that does not automatically lock the screen.Browser test through the ATF either by leaving the Client Test Runner as the foreground window or by making sure that the browser tab for the ATF is in its own window and has sufficient screen real estate (i.e., shows through to the user in a large enough percentage of the screen) that the browser grants it the full allotment of CPU time. Other workarounds are listed in KB0621889 on HI. For more information, see: PRB822056: ATF: Invisible browser throttling can cause poor test performance (KB0621889)Automated Test FrameworkAutomated Test Framework Client Test Runner moduleChrome will aggressively throttle background tabs — as a feature, not a bug Also , refer to How to avoid ATF testing failures. What is Headless Browser for Automated Test Framework? In the Rome release, ServiceNow® improved UI testing by automating the creation of browsers to process Automated Test Framework (ATF) User Interface (UI) tests. This feature is known as a headless browser, and helps you test UI functionality without having to manually open a browser to the Client Test Runner, which is what processes the UI tests in your local browser. Refer to : Headless Browser for Automated Test Framework What is ATF Test Generator and Cloud Runner? ATF Test Generator and Cloud Runner is a tool that enables you to leverage ServiceNow’s ability to intelligently generate regression tests for a specific instance. The test generator intelligently creates tests by analyzing instance behavior and workflows. The ATF Test Generator integrates with the existing ATF framework providing you with a seamless user experience. You also have the option to execute these tests using ServiceNow’s Cloud Runner which runs ATF tests on a headless browser hosted by ServiceNow. Refer to the video below when using ATF Test Generator & Cloud Runner: Best Practices guide for ATF Test Generator & Cloud Runner: ATF Walkthrough Additional Information: ATF Test Generator and Cloud Runner KB1188991-Data handling with ServiceNow ATF Test Generator and Cloud Runner (FAQ) Can I run my ATF Tests in Production? No, it is not recommended to do so. By default, the system property that is used to run automated tests is disabled to prevent you from accidentally running these tests on a production system. To avoid data corruption or an outage, run tests only on development, test, and other non-production instances. Can I disable Rollback for ATF Tests? No. AFT Tests are designed to rollback test data and changes made after each test run. It is not supported to disable rollback. Can I design one test to rely on the outcome of another test? No. All changes are rolled back after each test, so each test is independent of others. How do I migrate ATF Tests to other environments? It is recommended to transfer test using update set, using preservers during the clone process is untested and therefore not recommended.Promoting ATF to production instances is an option as well since clients would be cloning over sub production, but remember run tests only on development, test, and other non-production instances.You may need to check the sys_variable_value.value field to make sure that the reference it is expecting exist on the target instance, otherwise a valid reference would need to be provide before the test can pass. NOTE: For existing ATF Tests , you may refer to KB0825608-Capture an existing ATF test in a new update set How can custom UI such as UI Pages be tested via ATF? Test customized user interfaces such as UI pages and UI macros by retrieving their HTML and JavaScript page components and identifying the test actions they support. For more information refer to: Custom UI test steps How do we verify that custom UI components are testable? Use the Manual Page Inspector to inspect UI pages and confirm that page components are testable. For more information refer to: Enable and use the page inspector What are some of the common issues reported in ATF ? ATF Tests results in client errors 1. Refer to: Identify and resolve client errors 2. How to use the Client Error Inclusion list in the Automated Test Framework Refer to KB0725054 If you are getting any of the below errors , this could be related to the use of custom widgets. Catalog items that have deprecated or custom widgets are not supported in ATF FAILURE: Failed to open Catalog Item. Either you don't have access to the Catalog Item or g_form is not definedFAILURE: Validations failed. Failed to Order ItemFAILURE: Failed to submit the record producerFAILURE: Failed to Add the catalog item to cart Refer to KB below for more information: KB0716457-When opening and ordering pages, forms, record producers in Service Portal through ATF steps, the error observed is "FAILURE: Failed to open Catalog Item. Either you don't have access to the Catalog Item or g_form is not defined" Unable to retrieve Custom UI Components Refer to KB below when you are unable to retrieve the UI Components when testing a UI Page. KB0755252-Components are not getting retrieved correctly using Custom UI in ATF ATF tests to not rollback the records which have been created/deleted/updated Refer to KB below : KB0714767-Automated Test Framework (ATF) tests do not rollback the records which have been created / deleted / updated