- Test executes and despite everything working as expected, the test tells us that there is a "bug"
- Tester / Test incorrectly concludes that the program failed or the intent is not being met.
Commonly observed in:
- Errors in automation test scripts
- Unstability in test environment
- Failure due to third party libraries or cooperating processes
- Automation Test Script Failure due to changing locators
- Test Pipeline Failure due to Build Issue(s)
Flaky Test Scripts
- Poor Design
- Non Modular Structure
- Non Clean Code, etc.
- Poor Testability
- JSON Paths
- XML Paths, etc.
Unstable Test Environment
- State of System
- Dependent Processes
- Version Changes
- Breaking Contracts, etc.
Impractical Test Sequence
- Write Software --> Read Software
- Delete Software --> Read Software
- Misconfigured Dependencies
- Does it consider the specific version of each one?
Does it consider environment states?
Script Development & Execution on one specific machine
- Scripts that only run on local machine are mostly FLAKY
- Programs don't do what they haven't been told to pay attention to!
Loss of time
- In going through logs
- In investigating the problem
- In reproducing the test steps
- In fixing the test scripts, environment, etc.
- Noise in Testing Cycles
- Loss of credibility & Integrity
- Defocus from actual information gathering process (testing)
- The more you go in one direction, the far you go from the opposite direction. - Jerry Weinberg
Avoiding False Positives
- Robust Locators (Automation Hooks)
- Automatic Retry / Rerun(s)
- Scheduling test runs periodically to ascertain stability.
Develop & Execute on Separate Machines
- Prevention > Investigation > Detection
- Headless Executions
- Docker Containers / Cloud Systems
Plan Testing across Layers
- Flakiness Index: UI > API > Unit
Refreshing Test Environment
- Resetting State
- Using Docker Images, etc.
- Verbose Logging & Observability
Using Contract Verification Scripts
- Monitor Contracts
- Monitor Dependencies
- Access to GOLD
- GOLD = Going one layer down
- Automation IDs, etc.
- By: Rahul Parwal