·
8 min
Test data isolation: a checklist for stable parallel and repeated test runs
Roman Kirchmeier - Autemos
Test data isolation means every automated test works with data that belongs to it alone, and no other test or process changes that data. Only then do tests give the same result in parallel, in any order, and on the tenth run as on the first. Google's DORA research program (dora.dev) calls isolated test data “a prerequisite for tests to run in parallel” (dora.dev, Test data management, 2026). A Python study shows how often this goes wrong: order dependency caused 59% of 7,571 flaky tests (Gruber et al., 2021). The checklist below adds 12 checks to the guide on test data management.
TL;DR: Isolated test data belongs to exactly one test or one worker. Create data per run with unique IDs through the application's APIs, hand out vouchers and tokens only once, and clean up only your own records. To see whether isolation holds, compare the suite's results with one worker and with several.
Figure 1: Shared data makes tests depend on each other; own data per worker removes that dependency.
What is test data isolation?
Test data isolation is a property of automated tests in which each test owns data that no other test reads or changes. dora.dev asks that the data a test consumes be “explicitly associated with that test” and that it “isn't modified by other tests or processes” (dora.dev, 2026).
Test isolation is the wider term. Playwright wants each test to run with “its own local storage, session storage, data, cookies etc.” and names the benefit: isolation “improves reproducibility, makes debugging easier and prevents cascading test failures” (Playwright docs, 2026). The framework separates browser state itself. Databases, accounts and queues are your job.
dora.dev gives the reason for databases: “any changes to the data will persist across tests”. Worker counts and test distribution are covered in the article on parallel test execution.
Why does shared test data break parallel and repeated test runs?
Shared, mutable test data makes a test's result depend on which other test changed the same data before it or at the same time. In a fixed order with a single process, the bug often stays hidden. It shows up in parallel runs, in random order, or on the second run against the same environment.
Figure 2: 19 of 161 studied fixes concerned test order dependency (Luo et al. 2014).
Polluters and victims
Gruber et al. describe the pattern with two roles. A polluter p is a test that “changes a shared state that t tries to read from in a way that t fails”; test t is the victim (Gruber et al., 2021). Example: test A cancels order 4711, and test B wants to pay that order. If B runs first, both pass. If A runs first, B fails.
The second variant is the brittle test: it needs another test to set up the shared state, and it fails when run alone. Typical shared state in functional test suites:
database rows with fixed IDs, such as customer 1001 or order 4711
one shared test account with a cart, settings or a lock status
static and global variables in test code
queues, caches, mailboxes and files
one-time data such as voucher codes, registrations and tokens
How common are dependencies between tests?
Luo et al. studied fixes for flaky tests in 51 open-source projects. Of 161 studied commits, 19 (12%) concerned test order dependency, behind async wait with 74 (45%) and concurrency with 32 (20%) (Luo et al., 2014). The authors write: “In principle, all tests in a test suite should be properly isolated and independent of one another”.
Python looks different: 59% of the 7,571 flaky tests in the Gruber et al. dataset came down to test order. Zhang et al. analyzed 96 real-world dependent tests from 5 issue trackers; 59 of them (61%) came from shared static variables and 10 (10%) from database access (Zhang et al., 2014). The studies measure differently, so their numbers aren't directly comparable.
Checklist: 12 checks for test data isolation
The 12 checks cover who owns test data, how it's created, how it's consumed and who removes it. You can answer each one with yes or no in a code review.
Figure 3: The 12 checks for test data isolation at a glance.
Each test owns its data. A test creates the customers, accounts and orders it needs and reads no one else's records. Shared reference data stays unchanged: changing seed data later for a new test “may cause existing tests to fail” (Microsoft EF Core, 2026).
Unique IDs include a run ID and the worker index. A name like `customer-<run-id>-w3-0042` collides neither with the neighboring worker nor with yesterday's run. Playwright gives each worker “a unique worker index that starts with 0” (Playwright Parallelism, 2026), and pytest-xdist provides a `worker_id` such as “gw2” (pytest-xdist, 2026).
Setup creates state through the application's APIs. dora.dev recommends: “your tests should create the necessary state themselves as part of setup, using the application's APIs.” The API applies the business rules; a direct SQL insert skips them.
One-time data is never handed out twice. Voucher codes, registrations, one-time tokens and confirmation links are spent after the first test. The GTB Test Data Specialist syllabus calls irreversible cases “Verbrauch von Testdaten” (consumption of test data) (GTB syllabus Test Data Specialist (V1.1), 2021). Flag each record as used before a second test can get it.
Each worker has its own test account. Playwright uses the worker index “to isolate user data in the database between tests running on different workers”. pytest-xdist shows the same pattern: “use a different account in each xdist worker”. If a test changes account settings, it resets them in teardown.
Use transaction rollback only where it works. EF Core describes wrapping the test in a transaction and rolling it back at the end. If the code under test uses its own transactions, that fails: without nested transactions, you must clean up and disable parallelization (Microsoft EF Core, 2026).7. Writing tests get a schema or database per worker. Microsoft advises: “Use multiple databases to parallelize tests which modify the database”. The pytest-xdist example for `testrun_uid` creates “a unique database for this particular test run.” The cost is setup and migration time.
External resources are separated. Queues, cache keys, mailbox addresses and file paths carry the run ID or sit in their own instances. What you can't separate gets a lock, such as JUnit's `@ResourceLock` for “a specific shared resource that requires synchronized access” (JUnit, 2026).
Cleanup deletes only your own data, and a job removes orphans. Teardown deletes by run ID, never with `TRUNCATE` on a shared environment. Aborted runs leave records behind; a scheduled job deletes run IDs older than one day. The GTB syllabus lists the deletion of data sets no longer needed as a TDM task in test automation.
No test assumes an order. Playwright gives “no guarantee about the order of test execution across the files”. Martin Fowler states the goal: “Properly isolated tests can be run in any sequence.” (Martin Fowler, 2011). Run the suite in random order on a regular schedule.
Test data is separated per environment. An ID from TEST often doesn't exist on STAGING. The GTB syllabus calls test data inseparably linked to the test environments. Bind records to the environment and keep fixed IDs out of test code.
Isolation is checked with 1 and N workers. Run the suite once with one worker (`-n0` in pytest-xdist) and once with several. If the results differ, tests share state. Add the comparison run as a nightly job to your CI/CD pipeline.
Which isolation strategy fits which test?
Unique data per test works at every test level; rollback, a database per worker and locks each have a clear limit.
Figure 4: Six isolation strategies, what they fit and where they stop working.
Strategy | Fits | Limit | Source |
|---|---|---|---|
Own data with a unique ID per test | API and UI tests | data volume grows without cleanup | |
Test account per worker | UI tests with login | tests in one worker share the account | |
Transaction with rollback | tests with a direct database connection | no nested transactions; misses writes the application makes | |
Schema or database per worker | integration tests that write data | setup and migration time | |
Lock or serial execution | resources without concurrent access | costs parallelism; Playwright: “Using serial is not recommended” | |
Cleanup after each test, filtered by run ID | shared test environments | aborted runs leave orphans |
For UI and E2E tests, a blank starting state “can be much harder with functional tests” than with unit tests (Martin Fowler, 2011). Unique data, accounts per worker and cleanup by run ID carry most of the load there.
Why do reruns hide missing isolation?
Automatic reruns often turn a data-dependent test green on the second try, and the shared state stays in place. Gruber et al. did the math: a 95% confidence that a passing test isn't flaky would take 170 reruns on average (Gruber et al., 2021). At Google, about 1.5% of all test runs reported a flaky result (Google Testing Blog, 2016).
Use reruns as a measuring point: every test that passes only on the second try goes on a list and gets examined with check 12. Red tests after changed selectors have a different cause; self-healing locators handle those and do nothing for data conflicts.
How does Autemos support test data isolation, and where doesn't it?
Autemos supports part of the checklist: it hands out one-time data only once (check 4), assigns data to a run, project or tenant, and binds it per environment (check 11). Autemos is AI-assisted software for functional test automation across web, mobile, API and desktop. Autemos is not a data masking or test data generation tool.
One-Time Use: used vouchers, registrations, tokens and test users are flagged automatically and never handed out again, built for parallel runs.
Scopes RUN, PROJECT and TENANT: they decide whether a record lives per run, per project or per tenant.
Bindings per environment: records attach to DEV, TEST, STAGING or PROD.
Runtime bindings: customers, orders, users or tokens load at runtime from REST APIs, JSON files, Excel or databases and arrive as variables.
Datasets can be reused across workflows (Test Data Handling); the article on data-driven testing explains the principle. Autemos doesn't roll back databases and doesn't create schemas or databases per worker. Checks 6 and 7 stay with your database and pipeline setup.
Frequently asked questions
What is the difference between test isolation and test data isolation?
Test isolation covers a test's whole state, from cookies to global variables; test data isolation covers data in databases, accounts and external systems. Database changes, per dora.dev, persist across tests until someone resets them.
Is transaction rollback enough for E2E tests?
Usually not: in E2E tests, the application writes through its own database connections, which the test's rollback doesn't reach. For E2E tests, unique data with cleanup by run ID fits better.
How many test accounts do parallel tests need?
Parallel tests need at least one test account per worker. The Playwright example binds a user to the worker index, and every test in that worker uses it. If a test changes the account state for good, such as locking it, that test needs its own account.
Can I use a production copy for isolated test data?
You don't need a full production copy for isolation, and it carries risk. dora.dev lists “Using a full copy of the production database” as a common pitfall and warns that copied production data might contain sensitive information. If you do need production data, the article on data masking covers the techniques.
How do I find the test that breaks another one?
Run the failing test together with each test that ran before it, one pair at a time. Zhang et al. found that at least 82% of dependent tests can be revealed with no more than two tests (Zhang et al., 2014). Once you've found the polluter, give the victim its own data.
Conclusion
Isolated test data makes parallel and repeated test runs reproducible. The studies show the scale: 19 of 161 studied commits in Luo et al., 59% of the 7,571 Python flaky tests in the Gruber et al. dataset The fixes are simple. Each test creates its own data with a run ID and worker index, one-time data is handed out once, and cleanup deletes only your own records. Rollback and a database per worker help where the test controls the database.
Start with check 12: one run with one worker, one run with several. Every difference points to shared state you can now fix. If you want to try one-time data, scopes and environment bindings on your own test workflows, talk to the Autemos team.
