·
8 min
System testing explained: scope, test levels and automation
Roman Kirchmeier - Autemos
System testing checks a complete system against its requirements specification, covering functional and non-functional behavior. The ISTQB Foundation Level syllabus v4.0.1 lists it as one of five test levels, between component integration testing and system integration testing (ISTQB CTFL v4.0.1, 2024). Many German-language articles still show four levels, the state before version 4.0.
Within functional testing, this is the level where business flows run against the finished product. For banks and insurers, two questions follow. Which test data may sit in the test environment? And what can you automate? For the data, Delegated Regulation (EU) 2024/1774 under DORA applies to EU financial entities, and FINMA Circular 2023/1 to Swiss banks.
TL;DR: System testing is one of five test levels in ISTQB CTFL v4.0.1. It checks the overall behavior of a complete system against its specification, functional and non-functional. Banks separate the environment from production; under DORA it may hold only anonymised, pseudonymised or randomised production data. Functional flows and regression tests automate well; load and security tests need other tools.
Figure 1: The five test levels according to ISTQB CTFL v4.0.1
What is system testing?
System testing is a test level that checks the overall behavior of a complete system or product against its specification. Per the syllabus, it “focuses on the overall behavior and capabilities of an entire system or product” (ISTQB CTFL v4.0.1, 2024). It often includes “functional testing of end-to-end tasks and the non-functional testing of quality characteristics”.
Example from online banking: a customer opens an account in the browser, completes identification and then sees the account in the mobile app. The test runs this flow on the complete system, with UI, backend and database. The test basis is the system specification, and in the V-model the system test sits opposite it.
Which five test levels does ISTQB describe?
CTFL v4.0.1 describes five test levels:
Component testing (unit testing): tests components in isolation.
Component integration testing: tests interfaces and interactions between components.
System testing: tests the overall behavior of the complete system.
System integration testing: tests interfaces to other systems and external services.
Acceptance testing: focuses on “demonstrating readiness for deployment” against the user's business needs.
Since CTFL v4.0, system integration testing is a separate level. Teams that work with four levels put two different test objectives under the single label integration testing.
How does system testing differ from integration and E2E testing?
A system test checks the complete system against its specification, a component integration test checks how components interact, and a system integration test checks the interfaces to other systems. An E2E test describes the scope of a scenario and is not a test level.
Figure 2: System testing compared with component integration testing and E2E scenarios
Term | What it checks (per ISTQB) | Account-opening example |
|---|---|---|
Component integration testing | Interfaces and interactions between components | The account service calls the address check correctly |
System testing | Overall behavior of the complete system against its specification | Account opening in the browser through to the app |
System integration testing | Interfaces to other systems and external services | Data exchange with the identity provider and core banking |
E2E scenario | Not a test level: a complete business flow that can run at several levels | Registration through to the first payment |
Acceptance testing | Readiness for deployment, ideally shown by the intended users | The business unit checks account opening against its needs |
IBM Think describes a system test as “performance-based, end-to-end software testing” and treats it as black-box testing (IBM Think, accessed 2026). English Wikipedia equates it with E2E testing (Wikipedia, accessed 2026). Per ISTQB, it covers functional and non-functional tests, and performance efficiency is one of eight quality characteristics.
DORA Art. 25(1) lists “end-to-end testing” as a test type of its own, next to “performance testing” (Regulation (EU) 2022/2554, 2022). More on the neighboring topics: integration testing and its variants, E2E tests across complete business processes and the forms of acceptance testing.
What do tests at system level check?
System testing checks what the system should do (functional testing) and how well it behaves doing it (non-functional testing). Per the syllabus, “the main objective of functional testing is checking the functional completeness, functional correctness and functional appropriateness” (ISTQB CTFL v4.0.1, 2024).
For non-functional testing, the syllabus lists eight ISO/IEC 25010 characteristics: performance efficiency, compatibility, usability (interaction capability), reliability, security, maintainability, portability (flexibility) and safety. Safety is new to the list. For some characteristics, such as usability, ISTQB says “it is preferable to test them on a complete system in a representative test environment”.
Black-box test techniques fit well, since the test basis is a specification. Per CTFL, all four test types (functional, non-functional, black-box, white-box) “can be applied to all test levels”. The black-box testing guide with equivalence partitions and boundary values covers the techniques.
What test environment and test data does this level need?
System testing needs a representative environment separated from production, plus test data free of unprotected production data. For EU financial entities, Delegated Regulation (EU) 2024/1774 Art. 16(5)(a) requires that “non-production environments only store anonymised, pseudonymised, or randomised production data” (Commission Delegated Regulation (EU) 2024/1774, 2024). The syllabus allows simulations of sub-systems.
DORA has applied since 17 January 2025. Art. 16(6) allows an exception: “production data are stored only for specific testing occasions, for limited periods of time, and following the approval by the relevant function”, with each occasion reported to the ICT risk management function.
For Swiss banks, FINMA Circular 2023/1, in force since 1 January 2024, requires a separation between development or test environments and the ICT production environment (FINMA Circular 2023/1, 2022). It extends critical-data protection explicitly: “Dies gilt auch für kritische Daten in Testumgebungen” (the same applies to critical data in test environments). DORA reaches Swiss banks only through EU subsidiaries or EU business.
Autemos separates test data configuration per environment but doesn't anonymize data; that has to happen before the data reaches the test environment.
Who performs the tests at system level?
“System testing may be performed by an independent test team”, says CTFL v4.0.1; the syllabus doesn't make it mandatory. It describes several degrees of independence (ISTQB CTFL v4.0.1, 2024):
The author tests their own work.
Peers from the same team test.
Testers from outside the team test.
Testers from outside the organization test.
Independent testers tend to spot different failures but risk isolation and being seen as a bottleneck.
DORA Art. 24(4) requires that “tests are undertaken by independent parties, whether internal or external”, as part of a financial entity's resilience testing programme. Delegated Regulation (EU) 2024/1774 Art. 17(1)(b) asks for independence between the functions that approve changes and those that request and implement them. Neither passage uses that ISTQB term. If you use the results as evidence, record who tested and who approved.
What entry and exit criteria apply?
Entry criteria define when a test level may start; exit criteria define when it is done. ISTQB section 5.1.3 recommends defining both for every test level (ISTQB CTFL v4.0.1, 2024).
Figure 3: Typical entry and exit criteria per ISTQB section 5.1.3
Typical entry criteria:
Resources are available: people, tools, environments, test data, budget and time.
Testware is available: test basis, testable requirements, user stories and test cases.
Initial quality is in place, for example all smoke tests have passed.
Typical exit criteria:
Measures of thoroughness: coverage level, unresolved defects, defect density, failed test cases.
Binary yes/no criteria, for example all regression tests are automated.
Time or budget used up, if stakeholders accept the remaining risk.
For a bank, that means: environment separated, test data approved, smoke tests passed on the new build. Write these criteria into the test plan for the release so development, test and the business unit check the same conditions.
How can you automate tests at system level?
Functional tests with repeatable flows and clear expected results automate well; usability and exploratory tests stay mostly manual. The CTFL syllabus names less repetitive work, consistency and faster feedback as benefits, unrealistic expectations and underestimated maintenance as risks (ISTQB CTFL v4.0.1, 2024).
Figure 4: Which system-level tests to automate, keep manual or cover with other tools
Autemos, the AI-assisted test automation software by selementrix GmbH in Baar, automates functional tests across web, mobile, API and desktop.
Item | Approach | Note |
|---|---|---|
Business flows across web, mobile, API and desktop | Automate, for example with Autemos | One test can span several browsers, devices, API calls and user sessions |
Regression tests | Automate | ISTQB calls regression testing “a strong candidate for automation” |
Data variants (amounts, customer types, limits) | Automate, data-driven | One-time-use data is flagged so parallel runs don't reuse it |
Usability | Mostly manual | IBM describes usability testing as a largely manual process |
Exploratory testing | Manual | Depends on the tester's judgment |
Load and performance tests | Other tools | Autemos has no load or performance testing |
Security and penetration tests | Other tools | Autemos does not run security tests |
Code coverage (white-box) | Other tools | Autemos does not measure code coverage |
How do you get started?
Pick the flows from the system specification that must pass on every release.
Clear the test data and the configuration per environment.
Build each test as a workflow: business users by drag and drop, engineers with code blocks or embedded Playwright tests.
Let the AI Recorder draft a workflow from a Jira ticket if useful. Your team approves every AI-generated step before it runs.
Run the tests from GitLab, GitHub Actions or Jenkins and check the exit criteria against the results in Jira/Xray and Allure.
Autemos exports web tests as Playwright code (Java or TypeScript) and mobile tests as Appium code (Java), which runs without Autemos. The visual test workflows in Autemos show how a test is built.
Frequently asked questions
How does ISTQB define testing at system level?
According to ISTQB CTFL v4.0.1, system testing is one of five test levels, focused on the overall behavior and capabilities of a complete system or product. It often includes functional tests of complete tasks and non-functional tests of quality characteristics. Its test basis is the specification for the system.
How does integration testing differ from testing at system level?
Component integration testing checks interfaces and interactions between components; a system test checks the complete system against its specification. Interfaces to other systems belong to system integration testing, a separate level since CTFL v4.0.
Is an E2E test the same as a test of the whole system?
No, E2E describes the scope of a scenario, meaning a complete business flow, and is not an ISTQB test level. Such flows can run at system, system integration or acceptance level. A system test covers non-functional tests of quality characteristics as part of its scope.
Can production data be used in a test environment?
Under DORA, non-production environments may only store anonymised, pseudonymised or randomised production data (Delegated Regulation (EU) 2024/1774, Art. 16(5)(a)). Real production data is a time-limited, approved exception for specific testing occasions. FINMA Circular 2023/1 extends critical-data protection to test environments.
Can system tests be automated?
Yes, functional tests with repeatable flows automate well, regression tests first. Usability and exploratory tests stay mostly manual. Load, performance and security tests need specialized tools; Autemos doesn't cover them and automates functional tests only.
Conclusion
System testing checks the complete system against its specification and is one of five test levels in ISTQB CTFL v4.0.1. System integration testing is a separate level, and E2E describes the scope of a scenario. Plan functional and non-functional parts separately, since load, performance and security tests need their own tools.
At banks and insurers, environment and data matter too: Swiss banks separate test and production under FINMA Circular 2023/1, and EU financial entities keep only anonymised, pseudonymised or randomised production data in test environments under Delegated Regulation (EU) 2024/1774. Define entry and exit criteria in the test plan and record who tests and who approves.
Repeatable flows and regression tests belong in automation. If you want to find out which of your tests Autemos can automate, talk to our team.
