Automated Test Framework - Rescheduled ATF test suites and parameterized tests consume too much memory - Known Error

Automated Test Framework - Rescheduled ATF test suites and parameterized tests consume too much memory Description When an ATF test suite that contains many child suites and/or parameterized tests is rescheduled due to a mutually exclusive test running, the suite takes a long time to be rescheduled and will use a lot of the instance memory. In extreme cases, the instance may run out of memory and terminate the ATF test execution, leaving the test suite run incomplete.

This issue becomes worse with bigger test suite hierarchies and/or mixing child test suites with parameterized tests; for example, a test suite containing many child suites, each of which has child suites of their own with multiple parameterized tests in them will use a significant amount of memory when it is rescheduled.

Steps to Reproduce

1) Create a large ATF test suite containing multiple child suites (which may have child suites of their own) and parameterized tests. This can be done in the "Automated Test Framework (ATF) > Tests" and "Automated Test Framework (ATF) > Suites" modules

For context, this issue was observed with the following setup: One root suite containing 10 child suites, each of which has a single child suite of their own Every suite mentioned above (including the root suite) contained 10 parameterized tests, each of which had 3 parameter sets The grand total number of tests is 21 suites * 10 tests per suite = 210 tests

2) Run the test suite 3) At some point during the test suite run, force it to reschedule

NOTE: One way to do this is to create a long running test (e.g. a test with a "Run Server Side Script" step that uses gs.sleep() to sleep for 10 minutes), then navigate to the Mutually Exclusive Tests related list on the test form, click "Add Mutual Exclusion" and choose one of the tests in the suite. Then run the test and the suite in parallel; the mutual exclusion rule on the test will make the suite reschedule when it tries to run the mutually exclusive test (at that point the long-running test can be cancelled to allow the suite to run)

4) Wait ~10 seconds for the test suite to resume execution (at this point there's no need for it to reschedule again, so the test started in Step 3 can be completed)

Expected: The suite run should resume normal execution after being rescheduled Observed: The suite run takes a very long time (several minutes) to resume execution. While waiting for the test to resume execution, more and more memory is consumed; in extreme cases, the instance may run out of memory and kill the test run

Workaround As mentioned in this description, this issue occurs when a test suite execution is rescheduled, and gets worse with large test suite hierarchies. This means there are two potential workarounds for this issue:

Update the tests in the suite so they are less likely to be rescheduled. In most cases this means updating the tests so that they don't rely on updating pre-existing record data. Check the "Mutually Exclusive Tests" related list on the tests in the suite to see which tests have record conflicts with other tests. See below for some documentation on Parallel Testing and Mutually Exclusive Tests Break down large test suites hierarchies; in particular, removing child test suites from their parents ("flattening" the test suite) and running them separately will significantly reduce the impact of this issue Parallel testing documentation:

https://docs.servicenow.com/bundle/rome-application-development/page/administer/auto-test-framework/concept/parallel-testing.html

https://docs.servicenow.com/bundle/rome-application-development/page/administer/auto-test-framework/concept/mutual-exclusion-rule.html