·
8 min
Test data generation: synthetic test data, masked production copies or hand-made fixtures?
Roman Kirchmeier - Autemos
For software tests, test data generation comes down to four approaches: hand-made fixtures, rule-based generation, statistical or ML-based synthesis and the masked production copy. For banking applications, rule-based generation from the data model is often the safer default. The masked copy fits migration tests and data constellations built up over years. ML synthesis gives you realistic distributions and a residual risk: NIST states that the disclosure risk of fully synthetic datasets is not zero (NIST SP 800-188, 2023). Which option fits depends on what the test has to prove.
TL;DR: Synthetic test data for software tests is safest when generated by rules from the data model, with a fixed seed and a pinned generator version. Masked production copies pay off for migrations and historical data constellations. ML synthesis needs its own privacy review: NIST says fully synthetic datasets do not have zero disclosure risk (NIST SP 800-188, 2023).
Figure 1: Four ways to generate test data, mapped to the test goal (own illustration)
What is synthetic test data?
Synthetic test data is artificially created test data for software tests that, according to the German Testing Board (GTB) syllabus, has “keinerlei inhaltlichen Bezug zu Produktivdaten” (no content relation to production data) (GTB Test Data Specialist syllabus V1.1, 2021). It is built from metadata: the data model, value ranges, formats and business rules. The syllabus notes it is often used in early test levels.
By origin, the GTB sets three other kinds against it: production data, anonymized data and pseudonymized data, plus mixed forms. In practice that leaves two directions. You either create data from scratch, or you alter an extract from production.
The GTB sees no regulatory obstacle to generated test data, provided it is used correctly. Quality depends on the metadata available. A data model with gaps produces test data with gaps. Our guide to test data management places the four origins in the full process.
What are the four ways to generate test data?
Four approaches are in common use: hand-made fixtures, rule-based generators, statistical or ML-based synthesis and the masked production copy. None wins on every criterion. They differ on realism, privacy risk and whether a test run sees the same data tomorrow as today.
Figure 2: Four approaches compared, assessment based on documentation, not measurements
Hand-made fixtures: A tester writes a few records by hand, usually as CSV or JSON. NIST describes test data “that are syntactically valid but do not convey accurate information when analyzed” as usable for software development (NIST SP 800-188, 2023).
Rule-based generation: A library such as Faker produces values from rules. With the `de_CH` locale it returns Swiss addresses with cantons, IBANs and AHV numbers (Faker docs, de_CH, 2026).
Statistical or ML synthesis: A model learns distributions from real data and produces new rows. According to the vendor documentation, the HMA synthesizer in SDV (Synthetic Data Vault) models each table and the connections between them (SDV docs, 2026).
Masked production copy: An extract from production with personal fields altered. The techniques are covered in our article on data masking for test data.
The ratings in the table are our assessment based on the linked documentation, not measurements.
Criterion | Hand-made fixtures | Rule-based (e.g. Faker) | ML synthesis (e.g. SDV) | Masked production copy |
|---|---|---|---|---|
Realism | low | medium, plausible per field | high for distributions | very high |
Privacy risk | very low | very low, no link to production | residual risk, model learns from real data | high, re-identification possible |
Edge-case coverage | targeted but few | high, boundaries written as rules | low, values stay within the source range | only what occurs in production |
Referential integrity | maintained by hand | via generation order and key rules | modeled, built for around 5 tables | present if extracted consistently |
Reproducibility | high, versioned in the repository | high with fixed seed and pinned version | repeatable with `reset_sampling` | low, every extract differs |
Effort | rises with every row | medium to build, low after that | high: training, checks, privacy review | high: extract, masking, approval |
Typical test level | component test | component to system test | system test of reports | migration, acceptance and E2E regression test |
When is a masked production copy the better choice?
A masked production copy is the better choice when the test needs real legacy data: data migrations and historical constellations grown over decades. The GTB syllabus calls a production extract problematic for data protection reasons “aber nicht zwangsweise unzulässig” (but not necessarily unlawful) and lists it for migration tests (§2.4.1, GTB syllabus V1.1, 2021).
Our reasoning: a migration has to cope with the data actually in the legacy database. That includes formats from older system versions, empty mandatory fields from before a check rule existed and codes that no current data model knows.
Historical constellations follow the same logic. Example: a customer with an account since 1987, three address changes, a repaid mortgage and a signing mandate. A generator only produces such combinations if someone writes them down as rules first. Often nobody on the team remembers them.
The copy has a cost. For partial extracts, the GTB names keeping referential integrity as a challenge, and system chains need a single timestamp across the extract. Countermeasures against stale data, per the syllabus: data aging, time travel and a set environment clock.
For EU financial entities, Art. 16(5) of Delegated Regulation (EU) 2024/1774 under the EU DORA regulation requires that “non-production environments only store anonymised, pseudonymised, or randomised production data” (EUR-Lex, 2024). Our article on GDPR-compliant test data covers what this means under GDPR and Swiss law.
How safe is synthetic test data from ML models?
Test data from ML models carries a residual risk: NIST writes that fully synthetic datasets “do not have a zero-disclosure risk” (NIST SP 800-188, 2023). The reason NIST gives: they still contain information derived from non-public data. A model that has seen real customer records can reproduce parts of them.
Figure 3: SDV HMA is built for around 5 tables; a banking model is larger (SDV docs 2026, NIST SP 800-188)
Stadler, Oprisanu and Troncoso (EPFL and UCL) found that synthetic data either fails to prevent inference attacks or loses its utility. Their conclusion: synthetic data is “far from the holy grail of privacy-preserving data publishing” (Stadler et al., USENIX Security, 2022). They found no better privacy-utility tradeoff than with traditional anonymization.
Practical limits come on top. The vendor documentation says SDV's HMA synthesizer is “optimized for smaller datasets with around 5 tables and 1 level of depth” (SDV docs, 2026). A banking model with customer, account, contract, booking, limit and signing mandate is already larger. SDV's diagnostic report checks that continuous values stay within the min/max range of the real data (SDV diagnostic, 2026). Add boundary values just outside that range as your own rules.
Our conclusion for bank tests: rule-based generation from the data model is often the safer default. NIST states that realistic test data not developed from confidential data will not compromise the content of that data (NIST SP 800-188, §4.4.3, 2023). A Faker generator built on data-model rules never sees a real customer.
How do you generate test data in eight steps?
Generating reproducible test data takes eight steps: from the data model through boundary values, referential integrity, seed and checks to provisioning per environment. The order applies to rule-based generators.
Figure 4: Reproducible test data in eight steps (own illustration)
Capture the data model and rules. Record tables, fields, value ranges, mandatory fields and business rules. Example: an account belongs to exactly one customer.
Derive boundary and negative values. Equivalence partitions and boundary value analysis give you targeted rows: age 17 and 18, amount 0.00 and -0.01, an IBAN with a wrong check digit. Random values rarely hit those boundaries.
Secure referential integrity across tables and systems. Create parent records first, then dependent rows. Shared keys, such as the customer number in CRM and the core banking system, come from one source.
Set a fixed seed and pin the generator version. The Faker docs state: “A Seed produces the same result when the same methods with the same version of faker are called.” Pin the version “down to the patch number” (Faker docs, 2026).
Use obviously fake names. NIST recommends a name such as “FIRSTNAME1 LASTNAME2” over “JOHN SMITH”, so nobody confuses test data with confidential data (NIST SP 800-188, 2023).
Check against business rules. An automated check confirms that every account has a customer and totals add up. Verify that Faker IBANs and AHV numbers pass your check-digit logic wherever the test expects valid values.
Version the generator configuration. Rules, seed and version number live in Git next to the tests. Any old run can then be repeated with the same data.
Provide data per environment. DEV needs different volumes and states than STAGING. Our article on the test environment covers how to keep environments apart.
The generated rows then feed test cases that read their inputs from a data table, where one test case runs against many records in turn.
How do you measure data minimization with the test data coefficient?
The test data coefficient (Testdatenkoeffizient) is a metric that shows the share of test data used out of the test data held: with 100 GB held and 20 GB used, it is 20 % (GTB syllabus V1.1, 2021). The syllabus lists it as a measure of data minimization.
A low value means you hold data no test needs. With a full masked copy that is the normal case, and every unused table with personal data adds risk. In our assessment, generated data usually scores higher: you only create what a test case asks for. Measure the coefficient per environment and per release.
What role does Autemos play in generating test data?
Autemos is AI-assisted software for functional test automation across web, mobile, API and desktop. Autemos is not a data masking or test data generation tool. It does not create synthetic data itself, does not mask production data and does not subset production databases.
Autemos comes in where finished test data enters test workflows. You connect external data sources: REST APIs, JSON files, Excel, databases and the source iSynth. Runtime bindings load the data at runtime as variables, such as customers, orders or tokens.
You set bindings per environment (DEV, TEST, STAGING, PROD). A Faker dataset for TEST and a masked extract for STAGING can then feed the same test workflow. One-Time Use flags consumed vouchers or test users so parallel runs don't receive them twice. Details are on the Autemos test data handling page.
Frequently asked questions
What is the difference between synthetic test data and anonymized data?
According to the GTB syllabus, synthetic records have no content relation to production data; anonymized data comes from production and has been altered. So anonymized data carries a re-identification risk if the anonymization has gaps. The GTB lists both as separate origins of test data (GTB syllabus V1.1, 2021).
Can Faker generate Swiss test data?
Yes, Faker ships a `de_CH` locale with providers for addresses with cantons and postcodes, persons, phone numbers, companies, IBANs and AHV numbers (Faker docs, de_CH, 2026). Confirm check-digit validity in your own check step; for negative tests, an invalid check digit is what you want.
Why does the same seed suddenly produce different test data?
Usually the generator version has changed. The Faker docs warn that results are “not guaranteed to be consistent across patch versions” (Faker docs, 2026). Pin the version to the patch number and keep seed and configuration in the test repository.
When is ML synthesis with tools such as SDV worth it?
ML synthesis is worth it when realistic distributions matter more than edge cases, for example when testing reports on small schemas. SDV describes its HMA synthesizer as built for around 5 tables and one level of depth (SDV docs, 2026). Plan a privacy review: the model learns from real data.
What is a mix of anonymized and synthetic test records?
A mix combines an anonymized production extract with targeted generated records. The GTB syllabus states that a mix of the two has often proven itself in practice (GTB syllabus V1.1, 2021). Typically the extract covers historical constellations and the generator adds boundary values and negative cases.
Conclusion
Generating test data starts with the test goal. For boundary values, negative cases and repeatable runs in banks, rule-based generators built on the data model are often the safer choice: no link to production, a fixed seed, a pinned version and versioned rules. The masked production copy keeps its place for migrations and historical constellations, with effort for referential integrity and approval. ML synthesis gives realistic distributions and a residual risk that NIST and peer-reviewed research name clearly. The test data coefficient shows whether your data holdings are getting leaner. If you want to see how Autemos binds generated or masked datasets per environment to functional test workflows, talk to our team.
