·
8 min
Data-driven testing: how to separate test logic from test data in automation
Roman Kirchmeier - Autemos
Data-driven testing separates the test logic of an automated test from its test data: one script describes the flow once, a table supplies the records, and each row runs as its own test case. You check the same bank transfer with 20 amounts without maintaining 20 scripts. Getting there takes eight steps, from a stable flow to a readable report. The basis is the official ISTQB glossary, the ISTQB Test Automation Engineering syllabus v2.0, the German GTB Test Data Specialist syllabus and the docs of pytest, JUnit, TestNG and Playwright. Planning, roles and provisioning are covered in the guide to test data management.
TL;DR: Data-driven testing (DDT) is, per ISTQB, a test automation approach that separates test data from the test script. You automate a stable flow once and feed it a table of inputs and expected results derived from equivalence partitions, boundary values and negative cases. No reliable study measures the savings; the benefit shows when a flow has many variants.
Figure 1: One script describes the flow once, every table row runs as its own test case
What is data-driven testing?
Data-driven testing (DDT) is a test automation approach that separates test data from the test script, so the same script can run repeatedly with different data. The official ISTQB glossary defines it as “A test automation approach in which test data is separated from a test script to allow its repeated execution with various test data.” (ISTQB Glossary, 2026) The ISTQB Test Automation Engineering syllabus v2.0 adds: “Data-driven testing (DDT) builds upon the structured scripting approach.” (ISTQB CTAL-TAE v2.0, 2024)
How does an abstract test case become a series of test case variants?
An abstract test case becomes a series of variants when you fill its parameters with different values from the permitted range; the German GTB Test Data Specialist syllabus describes this logic. An abstract test case holds parameters as placeholders, and concrete values make it a concrete test case. Then, in our translation: “By inserting different values within the permitted value range of a parameter, a single concrete test case becomes a series of test case variants.” (GTB Test Data Specialist syllabus (V1.1), 2021)
A data table writes that series down: each column is a parameter, each row a variant. A well-written test case with preconditions and an expected result shows at once which values may vary.
When is data-driven testing worth it?
DDT pays off once a stable flow has to be checked with many value combinations. The ISTQB TAE v2.0 syllabus lists the advantage “Allows quick and easy test case expansion through data feeds” and states: “The cost of adding new automated tests can be significantly reduced” (ISTQB CTAL-TAE v2.0, 2024).
Good candidates share these traits:
Same steps, changing inputs: login per role, fee calculation, mandatory fields.
Business rules with limits: transfer limits, age limits, interest tiers.
Variants per country, language or tenant.
If each row needs a different click path, logic ends up in control columns, and separate tests are clearer.
How much effort DDT saves has not been measured reliably. An MSR 2026 registered report on parameterized tests (PTs) states: “there is still no large-scale empirical evidence on how PTs are used within modern JUnit-based ecosystems.” (MSR 2026, 2026) Its planned study of more than 260 Apache Java projects has no results yet.
How do you implement data-driven testing step by step?
You implement DDT in eight steps, and the order matters: every row inherits the script's weaknesses, so a shaky flow with 20 rows fails 20 times.
Figure 2: Data-driven testing in eight steps
Pick a stable flow. Choose one that already runs reliably, such as login or payment entry.
Identify the parameters. Mark every value that may vary: amount, currency, role, account status. Everything else stays fixed.
Derive the rows. Write one row per equivalence partition and per boundary value. Add negative cases with invalid input, such as empty mandatory fields.
Enter the expected results. Each row carries its outcome, such as “booked” or the exact error message.
Choose the source. CSV for small versioned tables, XLSX for the business team, JSON for nested structures, a database or API for current states.
Bind the data. Use your framework's mechanism (table below) or variables in your test tool.
Name the rows. Every row gets an ID shown in the report, such as “TRF-05 amount above daily limit”.
Run and review. Check every red row: is the defect in the application, the data or the expected value?
Example: data table for a transfer with a daily limit
Steps 3 and 4 for a domestic transfer with an assumed daily limit of CHF 10,000 (example values, not a real bank rule):
ID | Amount (CHF) | Account | Origin of the row | Expected result |
|---|---|---|---|---|
TRF-01 | 500.00 | active | valid partition | booked |
TRF-02 | 0.01 | active | lower boundary, valid | booked |
TRF-03 | 0.00 | active | lower boundary, invalid | amount error |
TRF-04 | 10000.00 | active | upper boundary, valid | booked |
TRF-05 | 10000.01 | active | upper boundary, invalid | limit notice |
TRF-06 | empty | active | negative, mandatory field | field error |
TRF-07 | 500.00 | blocked | negative, account status | no booking |
Only the review needs the “Origin of the row” column: it stops someone from deleting a boundary row as a duplicate.
Which mechanisms do pytest, JUnit, TestNG and Playwright offer?
pytest, JUnit Jupiter (JUnit 5/6), TestNG and Playwright Test bind data rows through a decorator, an annotation, a data provider or a loop (quotes from the official docs, as of 2026).
Framework | Mechanism | What the docs say | Data sources |
|---|---|---|---|
pytest | `@pytest.mark.parametrize`, `pytest.param`, `pytest_generate_tests` | “@parametrize allows one to define multiple sets of arguments and fixtures at the test function or class.” (pytest) | Lists in code, files via the hook |
JUnit Jupiter (JUnit 5/6) | `@ParameterizedTest` with `@CsvSource`, `@CsvFileSource` or `@MethodSource` | “Parameterized tests make it possible to run a test method multiple times with different arguments.” (JUnit User Guide) | CSV literals, CSV files, factory methods (`junit-jupiter-params`) |
TestNG | `@DataProvider` plus the `dataProvider` attribute | “A Data Provider is a method on your class that returns an array of array of objects.” (TestNG) | Arrays, Iterator for lazy loading |
Playwright Test | Loop or `test.describe()` per record, project parameters | “You can either parameterize tests on a test level or on a project level.” (Playwright) | Arrays, CSV via Node.js, environment variables |
The opposite of all four is values fixed in the script (hardcoding). The TAE v2.0 syllabus says: “It can be avoided by using data-driven testing, so the test data comes from a common source” (ISTQB CTAL-TAE v2.0, §4.3.1, 2024).
How does data-driven differ from keyword-driven testing and BDD?
DDT varies data with fixed steps, keyword-driven testing (KDT) describes the steps as keywords, and BDD writes acceptance criteria in given-when-then form. Per the ISTQB glossary, KDT test scripts “contain keywords and supporting files that contain scripts that implement those keywords” (ISTQB Glossary, 2026).
Figure 3: DDT, KDT and BDD by core idea and typical artifact (ISTQB CTAL-TAE v2.0)
Approach | Core idea | Typical artifact | Position in TAE v2.0 |
|---|---|---|---|
Data-driven (DDT) | Data apart from the script | Script plus data table | Test analysts fill data files; test data management may be needed |
Keyword-driven (KDT) | Steps as keywords | Keyword table plus library | Business analysts contribute; “It creates a huge effort for smaller systems” |
BDD | Acceptance criteria with given, when, then | Scenarios, optionally with example table | Separate approach next to DDT and KDT |
Hybrid | Industry term, no ISTQB definition | Keywords plus data tables | Indirect only: KDT is often built on DDT |
Which mistakes make data-driven tests unreliable?
Four mistakes make data-driven tests unreliable: too many rows, unreadable test names, shared mutable data and data that ages. The TAE v2.0 syllabus warns: “Proper test data management may be necessary” (ISTQB CTAL-TAE v2.0, 2024).
Figure 4: Four mistakes that make data-driven tests unreliable
Too many rows
Full combinatorics multiply: 4 parameters with 5 values each give 5 × 5 × 5 × 5 = 625 rows. One representative per equivalence partition plus the boundary values is usually enough. Combine parameters only where they depend on each other.
Unreadable test names
Without explicit IDs, frameworks build names from the values. The pytest docs show it: “The test test_eval[1+7-8] passed, but the name is autogenerated and confusing.” (pytest docs, 2026) The fix is `pytest.param` with an ID or, in JUnit Jupiter, the `name` attribute with `{index}` and `{arguments}`.
Shared mutable data
Two rows that change the same customer affect each other once tests run in parallel or in another order. The Selenium docs put it in five words: “Do not share test data.” (Selenium docs, 2026) The test data isolation checklist shows how to separate data per test.
Data that ages
Fixed dates or expired tokens turn green tests red after a few weeks. Google's DORA research program (dora.dev) lists the pitfall “Relying on data that is out of date or no longer relevant.” (dora.dev, 2026) The GTB syllabus mentions relative values, such as “today” minus one day. For fresh states, see the methods compared in generating test data.
How does Autemos support data-driven testing?
In Autemos you keep data rows as datasets and bind them at runtime as variables to a visual test workflow. Autemos is AI-assisted software for functional test automation across web, mobile, API and desktop. Autemos is not a data masking or test data generation tool. Which rows belong in a table stays a test design decision for your team.
You create datasets in Autemos or import them as CSV, XLSX or JSON and reuse them across workflows; the test data handling page lists simple data-driven tests among the uses. External sources such as REST APIs, JSON files, Excel or databases are bound per environment (DEV, TEST, STAGING, PROD).
You build flows by drag and drop in visual test workflows, and test analysts change rows without code. The TAE syllabus describes the same split of work: “Test analysts can specify automated tests by populating one or more test data files that describe the tests” (ISTQB CTAL-TAE v2.0, 2024).
Test engineers add code blocks or embed existing Playwright tests. Autemos exports web tests as Playwright code in Java or TypeScript, synced through Git, and they run without Autemos. One-Time Use flags consumed vouchers or tokens so parallel runs don't get them twice.
Frequently asked questions
Is parameterized testing the same as data-driven testing?
Parameterized tests are the framework mechanism, and DDT is the approach behind it. `@pytest.mark.parametrize` or `@ParameterizedTest` apply the ISTQB idea to single test functions. In UI and API tests, the data usually lives outside the code, for example in CSV files.
Which data source works best for data-driven tests?
CSV works for small, versioned tables, XLSX for tables the business team maintains. JSON fits nested structures, a database or API fits states that must be current. The ISTQB TAE v2.0 syllabus names .csv and .xlsx files and database dumps as example sources.
How many data rows does a data-driven test need?
No fixed row count exists for a data-driven test; one representative per equivalence partition is usually enough, plus the boundary values and a few negative cases. In the transfer example above, 7 rows cover 2 parameters. Full cross products, 625 rows for 4 parameters with 5 values, rarely pay off.
Does the expected result belong in the data table?
Yes, every row should carry its expected result. The GTB Test Data Specialist syllabus lists expected data (Soll-Daten) as a test data type of its own, next to input, state and output data. A script that calculates expected values itself can repeat the application's error and still report green.
Does data-driven testing save measurable effort?
No reliable numbers on DDT savings exist yet. The TAE v2.0 syllabus only says the cost of adding new automated tests can be significantly reduced. An MSR 2026 registered report states that large-scale empirical evidence on parameterized tests is missing. Measure it yourself: time how long one new test variant takes before and after the switch.
Conclusion
Data-driven testing pays off where a stable flow has many business variants. Most of the work is test design: identify parameters, derive rows from test design, and write an expected result into every row. The built-in tools of pytest, JUnit Jupiter, TestNG or Playwright handle the binding. Keep tables small, give every row a readable name and don't share mutable records between tests. No reliable study backs savings percentages yet, so check the effect on your own test portfolio. To see how datasets and visual workflows in Autemos look with your flows, talk to our team.
