1. Testing is context dependent - Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site.
2. Exhaustive testing is impossible - Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases.
Instead of exhaustive testing, we use risks and priorities to focus testing efforts.
3. Early testing - Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives.
4. Defect clustering - A small number of modules contain most of the defects discovered during pre-release testing or show the most operational failures.
5. Pesticide paradox - If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs.
To overcome this 'pesticide paradox', the test cases need to be regularly reviewed and revised, and new and different tests need to be written
to exercise different parts of the software or system to potentially find more defects.
6. Testing shows presence of defects - Testing can show that defects are present, but cannot prove that there are no defects.
Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness.
7. Absence of errors fallacy - Finding and fixing defects does not help if the system built is unusable and does not fulfill the users' needs and expectations
Problems
Mistakes (see error)
Error: A human action that produces an incorrect result.
Defect: A flaw in a component or system that can cause the component or system to fail to
perform its required function, e.g. an incorrect statement or data definition. A defect, if
encountered during execution, may cause a failure of the component or system.
Failure: Deviation of the component or system from its expected delivery, service or result.
Causes:
errors in the specification, design and implementation of the software and system
errors in use of the system
environmental conditions
intentional damage
potential consequences of earlier errors, intentional damage, defects and failures
Test Process
planning and control
Project and test plans should include time to be spent on planning the tests, designing test cases, preparing for execution and evaluating status
test policies
gives rules for testing, e.g. 'we always review the design documents'
test strategy
test strategy is the overall high-level approach, e.g. 'system testing is carried out by an independent team reporting to the program quality manager. It will be risk-based and proceeds from a product (quality) risk analysis'
Test planning tasks
Determine the scope and risks and identify the objectives of testing
Determine the test approach (techniques, test items, coverage, identifying and interfacing with the teams involved in testing, testware)
Implement the test policy and/or the test strategy
Determine the required test resources (e.g. people, test environment, PCs)
Schedule test analysis and design tasks, test implementation, execution and evaluation
Determine the exit criteria: we need to set criteria such as coverage criteria (for example, the percentage of statements in the software that must be executed during testing)
Test control tasks (ongoing)
Measure and analyze the results of reviews and testing
Monitor and document progress, test coverage and exit criteria
Provide information on testing
Initiate corrective actions
Make decisions
to continue testing
to stop testing
to release the software
to retain it for further work
etc.
analysis and design
Review the test basis (such as the product risk analysis, requirements, archi tecture, design specifications, and interfaces), examining the specifications
Identify test conditions based on analysis of test items, their specifications, and what we know about their behavior and structure
Design the tests using techniques to help select representative tests based on the test conditions
Evaluate testability of the requirements and system
Design the test environment set-up and identify any required infrastructure and tools.
implementation and execution
Implementation tasks
Develop and prioritize our test cases, using the techniques, and create test data for those tests
Create test suites from the test cases for efficient test execution
Implement and verify the environment
Execution tasks
Execute the test suites and individual test cases, following test procedures
Log the outcome of test execution and record the identities and versions of the software under test, test tools and testware: report defects, test logs, etc.
Compare actual results with expected results
report discrepancies as incidents
Repeat test activities as a result of action taken for each discrepancy
confirmation testing or re-testing
regression testing
evaluating exit criteria and reporting (should be set and evaluated for each test level)
Check test logs against the exit criteria specified in test planning
Assess if more tests are needed or if the exit criteria specified should be changed
Write a test summary report for stakeholders
test closure activities
Check which planned deliverables we actually delivered and ensure all incident reports have been resolved through defect repair or deferral (open);
document the-acceptance or rejection of the software system
Finalize and archive testware, such as scripts, the test environment, and any other test infrastructure, for later reuse
Hand over testware to the maintenance organization
Evaluate how the testing went and analyze lessons learned for future releases and projects
Test Process glossary
Confirmation testing
re-testing: Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions.
Exit criteria
The set of generic and specific conditions, agreed upon with the stakeholders,
for permitting a process to be officially completed. The purpose of exit criteria is to
prevent a task from being considered completed when there are still outstanding parts of
the task which have not been finished. Exit criteria are used to report against and to plan
when to stop testing
Incident
Any event occurring that requires investigation
Regression testing
Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a
result of the changes made. It is performed when the software or its environment is changed
Test basis
All documents from which the requirements of a component or system can be
inferred. The documentation on which the test cases are based. If a document can be
amended only by way of formal amendment procedure, then the test basis is called a frozen test basis
Test condition
An item or event of a component or system that could be verified by one or
more test cases, e.g. a function, transaction, feature, quality attribute, or structural element
Test coverage
The degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite
Test data
Data that exists (for example, in a database) before a test is executed, and that
affects or is affected by the component or system under test
Test execution
The process of running a test on the component or system under test,
producing actual result(s)
Test log
A chronological record of relevant details about the execution of tests
Test plan
A document describing the scope, approach, resources and schedule of intended
test activities. It identifies amongst others test items, the features to be tested, the testing
tasks, who will do each task, degree of tester independence, the test environment, the test
design techniques and entry and exit criteria to be used, and the rationale for their choice,
and any risks requiring contingency planning. It is a record of the test planning process
Test strategy
A high-level description of the test levels to be performed and the testing within
those levels for an organization or programme (one or more projects)
Test summary report
A document summarizing testing activities and results. It also contains
an evaluation of the corresponding test items against exit criteria
Testware
Artifacts produced during the test process required to plan, design, and execute
tests, such as documentation, scripts, inputs, expected results, set-up and clear-up
procedures, files, databases, environment, and any additional software or utilities used in testing
The psychology of testing
Glossary
Independence
Separation of responsibilities, which encourages the
accomplishment of objective testing
Levels of
tests by the person who wrote the item under test
tests by another person within the same team, such as another programmer
tests by a person from a different organizational group, such as an independ ent test team
tests designed by a person from a different-organization or company, such as outsourced testing or certification by an external body
2. Testing throughout the software life cycle
Glossary
Section 2.1
(Commercial) off-the-shelf software (COTS)
A software product that is developed for the general market, i.e. for a large number of
customers, and that is delivered to many customers in identical format
incremental development model
A development lifecycle where a project is broken into a
series of increments, each of which delivers a portion of the functionality in the overall
project requirements. The requirements are prioritized and delivered in priority order in the
appropriate increment. In some (but not all) versions of this lifecycle model, each
subproject follows a ‘mini V-model’ with its own design, coding and testing phases
test level
A group of test activities that are organized and managed together. A test level is
linked to the responsibilities in a project. Examples of test levels are component test,
integration test, system test and acceptance test
validation
Confirmation by examination and through provision of objective evidence that
the requirements for a specific intended use or application have been fulfilled
verification
Confirmation by examination and through provision of objective evidence that
specified requirements have been fulfilled
V-model
A framework to describe the software development lifecycle activities from
requirements specification to maintenance. The V-model illustrates how testing activities
can be integrated into each phase of the software development lifecycle
Section 2.2
alpha testing
Simulated or actual operational testing by potential users/customers or an
independent test team at the developers’ site, but outside the development organization.
Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing
beta testing
Operational testing by potential and/or existing users/customers at an external
site not otherwise involved with the developers, to determine whether or not a component
or system satisfies the user/customer needs and fits within the business processes. Beta
testing is often employed as a form of external acceptance testing for off-the-shelf software
in order to acquire feedback from the market
component testing
The testing of individual software components
driver
A software component or test tool that replaces a component that takes care of the
control and/or the calling of a component or system
functional requirements
A condition or capability needed by a user to solve a problem or achieve an
objective that must be met or possessed by a system or system component to satisfy a
contract, standard, specification, or other formally imposed document
integration
The process of combining components or systems into larger assemblies
integration testing
Testing performed to expose defects in the interfaces and in the
interactions between integrated components or systems
non-functional testing
Testing the attributes of a component or system that do not relate to
functionality, e.g. reliability, efficiency, usability, maintainability and portability
operational testing
Testing conducted to evaluate a component or system in its operational environment
The process of testing to determine the compliance of the component or system
robustness testing
Testing to determine the robustness of the software product
stub
A skeletal or special-purpose implementation of a software component, used to develop
or test a component that calls or is otherwise dependent on it. It replaces a called component
system testing
The process of testing an integrated system to verify that it meets specified requirements
test-driven development
A way of developing software where the test cases are developed,
and often automated, before the software is developed to run those test cases
test environment
An environment containing hardware, instrumentation, simulators,
software tools, and other support elements needed to conduct a test
user acceptance testing
Formal testing with respect to user needs, requirements, and business
processes conducted to determine whether or not a system satisfies the acceptance criteria
and to enable the user, customers or other authorized entity to determine whether or not to
accept the system
Section 2.3
black-box testing
Testing, either functional or non-functional, without reference to the
internal structure of the component or system
code coverage
An analysis method that determines which parts of the software have been
executed (covered) by the test suite and which parts have not been executed, e.g. statement
coverage, decision coverage or condition coverage
confirmation testing (re-testing)
functional testing
Testing based on an analysis of the specification of the functionality of a component or system
interoperability testing
The process of testing to determine the interoperability of a software product
load testing
A type of performance testing conducted to evaluate the behavior of a
component or system with increasing load, e.g. numbers of parallel users and/or numbers
of transactions, to determine what load can be handled by the component or system
maintainability testing
The process of testing to determine the maintainability of a software product
performance testing
The process of testing to determine the performance of a software product
portability testing
The process of testing to determine the portability of a software product
regression testing
Testing of a previously tested program following modification to ensure
that defects have not been introduced or uncovered in unchanged areas of the software, as a
result of the changes made. It is performed when the software or its environment is changed
reliability testing
The process of testing to determine the reliability of a software product
security testing
Testing to determine the security of the software product
specification-based testing
black box testing: Testing, either functional or non-functional, without reference to the
internal structure of the component or system
stress testing
A type of performance testing conducted to evaluate a system or component at
or beyond the limits of its anticipated or specified work loads, or with reduced availability
of resources such as access to memory or servers
structural testing
white-box testing: Testing based on an analysis of the internal structure of the component or system
test suite
A set of several test cases for a component or system under test, where the post
condition of one test is often used as the precondition for the next one.
usability testing
Testing to determine the extent to which the software product is
understood, easy to learn, easy to operate and attractive to the users under specified conditions
white-box testing
Section 2.4
impact analysis
maintenance testing
Testing the changes to an operational system or the impact of a
changed environment to an operational system
SOFTWARE DEVELOPMENT MODELS
Glossary
Verification is concerned with evaluating a work product, component or system to determine whether it meets the requirements set.
In fact, verification focuses on the question 'Is the deliverable built according to the specification?'
Validation is concerned with evaluating a work product, component or system to determine whether it meets the user needs and requirements.
Validation focuses on the question 'Is the deliverable fit for purpose, e.g. does it provide a solution to the problem?'
V-model
Test levels
component testing
integration testing
system testing
acceptance testing
Iterative life cycles
incremental development models
prototyping
Rapid Application Development (RAD)
formally a parallel development of
functions and subsequent integration
Validation with the RAD development process is thus an early and major activity
Dynamic System Development Methodology [DSDM]
is a refined RAD process that allows controls to be put in place
in order to stop the process from getting out of control
Rational Unified Process (RUP)
agile development
Extreme Programming (XP)
generation of business stories to define the functionality
demands an on-site customer for continual feedback and to define and carry out functional acceptance testing
promotes pair programming and shared code ownership amongst the developers
states that component test scripts shall be written before the code is written and that those tests should be automated
states that integration and testing of the code shall happen several times a day
states that we always implement the simplest solution to meet today's problems
Example iteration
Define
Develop
Build
Test (increase each phase)
Implement
Testing within a life cycle model
for every development activity there is a corresponding testing activity
each test level has test objectives specific to that level
the analysis and design of tests for a given test level should begin during the corresponding development activity
testers should be involved in reviewing documents as soon as drafts are avail able in the development cycle
Test levels
Component testing (unit) - module and program testing (e.g. modules, programs, objects, classes, etc.) that are separately testable
Stub - is called from the software component to be tested
driver - calls a component to be tested
resource-behavior (e.g. memory leaks)
performance
robustness
structural testing (e.g. decision coverage)
In XP: test-first approach or test-driven development: prepare and automate test cases before coding
Integration testing - tests interfaces between components, interactions to dif-ferent parts of a system or interfaces between systems
component integration testing
system integration testing
Big-bang testing
advantage: everything is finished before integration testing starts
disadvantage: in general it is time-consuming and difficult to trace the cause of failures
Top-down: testing takes place from top to bottom, following the control flow or architectural structure (e.g. starting from the GUI or main menu).
Components or systems are substituted by stubs
Bottom-up: testing takes place from the bottom of the control flow upwards.
Components or systems are substituted by drivers.
Functional incremental: integration and testing takes place on the basis of the functions or functionality,
as documented in the functional specification
System testing - concerned with the behavior of the whole system/product as defined by the scope of a development project or product.
Requires a controlled test environment
functional
non-functional
performance
reliability
Specification-based (black-box) techniques
Decision table may be created for combinations of effects described in business rules
performed against the regulations which must be adhered to,
such as governmental, legal or safety regulations
2 stages of acceptance test of COTS
Alpha testing
at the developer's site
A cross-section of potential users and members of the developer's organization are invited to use the system
Beta testing
or field testing
sends the system to a cross-section of users who install it and use it under real-world working conditions
Test types
Functional testing (often - 'black-box')
'what it does'
Based upon ISO 9126, can be done focusing on:
suitability
interoperability
security
accuracy
compliance
Perspectives:
requirements-based
business-process-based
experienced-based
Non-functional testing
'how well' the system works
Characteristics (ISO/IEC 9126, 2001)
functionality
suitability
accuracy
security
interoperability
compliance
reliability
maturity (robustness)
fault-tolerance
recoverability
compliance
usability
understandability
learnability
operability
attractiveness
compliance
efficiency
time behavior (performance)
resource uti lization
compliance
maintainability
analyzability
changeability
stability
testability
compliance
portability
adaptability
installability
co-existence
replaceability
compliance
Types
portability testing
reliability testing
maintainability testing
usability testing
stress testing
load testing
performance testing
Structural testing ('white-box')
Testing related to changes
Confirmation testing (re-testing)
Regression testing
Maintenance testing (testing an OS)
different from maintainability testing,
which defines how easy it is to maintain the system
Levels:
component test
integration test
system test
acceptance test
Parts
testing the changes
regression tests
Based on:
Impact analysis
Risk analysis
Triggers
modifications
Planned modifications - 90% of work
(enhancement changes (e.g. release-based))
perfective modifications (adapting software to the user's wishes,
for instance by supplying new functions or enhancing performance)
adaptive modifications (adapting software to environmental changes
such as new hardware, new systems software or new legislation)
corrective planned modifications (deferrable correction of defects)
Ad-hoc corrective and emergency changes
<- patching up
planned modification
changes of environment
planned OS or DB upgrades
patches to newly exposed or discovered vulnerabilities of the OS
migration
(from one platform to another
operational testing of the new environment
testing of the changed software
retirement of the system
the testing of data migration or archiving
3. Static techniques
3.1 REVIEWS AND THE TEST PROCESS
Glossary
static testing
Testing of a component or system at specification or implementation level
without execution of that software, e.g. reviews or static analysis
dynamic testing
Testing that involves the execution of the software of a component or system
reviews
An evaluation of a product or project status to ascertain discrepancies from planned
results and to recommend improvements. Examples include management review, informal
review, technical review, inspection, and walkthrough
Types of defects
deviations from standards
missing requirements
design defects
non-maintainable code
inconsistent interface specifications
Advantages:
early feedback on quality issues
Cheap fix and improvement
development productivity increases
exchange of information between the participants
increased awareness of quality issues
3,2 REVIEW PROCESS
Glossary
entry criteria
The set of generic and specific conditions for permitting a process to go
forward with a defined task, e.g. test phase. The purpose of entry criteria is to prevent a
task from starting which would entail more (wasted) effort compared to the effort needed
to remove the failed entry criteria
exit criteria
The set of generic and specific conditions, agreed upon with the stakeholders,
for permitting a process to be officially completed. The purpose of exit criteria is to
prevent a task from being considered completed when there are still outstanding parts of
the task which have not been finished. Exit criteria are used to report against and to plan
when to stop testing
formal review
A review characterized by documented procedures and requirements, e.g. inspection
informal review
A review not based on a formal (documented) procedure. initiating
(IDEAL): The phase within the IDEAL model where the groundwork is laid for a
successful improvement effort. The initiating phase consists of the activities: set context,
build sponsorship and charter infrastructure
inspection
A type of peer review that relies on visual examination of documents to detect
defects, e.g. violations of development standards and non-conformance to higher level
documentation. The most formal review technique and therefore always based on a
documented procedure
moderator
The leader and main person responsible for an inspection or other review process
reviewer
The person involved in the review that identifies and describes anomalies in the
product or project under review. Reviewers can be chosen to represent different viewpoints
and roles in the review process
scribe
The person who records each defect mentioned and any suggestions for process
improvement during a review meeting, on a logging form. The scribe should ensure that
the logging form is readable and understandable
technical review
A peer group discussion activity that focuses on achieving consensus on
the technical approach to be taken
walkthrough
A step-by-step presentation by the author of a document in order to gather
information and to establish a common understanding of its content
3.2.1 Phases of a formal review
informal
not documented
formal
Phases
Planning
entry criterias:
A short check of a product sample by the moderator (or expert) does not reveal a large number of major defects. For example, after 30 minutes of checking, no more than 3 major defects are found on a single page or fewer than 10 major defects in total in a set of 5 pages.
The document to be reviewed is available with line numbers
The document has been cleaned up by running any automated checks that apply
References needed for the inspection are stable and available
The document author is prepared to join the review team and feels confident with the quality of the document
Focuses:
focus on higher-level documents
focus on standards
focus on related documents at the same level
focus on usage
Kick-off
Preparation
checking rate (pages per hour)
Review meeting
Phases
logging phase
discussion phase
decision phase
Severity classes
Critical: defects will cause downstream damage; the scope and impact of the defect is beyond the document under inspection
Major, defects could cause a downstream effect (e.g. a fault in a design can result in an error in the implementation)
Minor, defects are not likely to cause downstream damage (e.g. non-compli ance with the standards and templates)
exit criteria
the average number of critical and/or major defects found per page (e.g. no more than three critical/major defects per page)
Rework
Follow-up
3.2.2 Roles and responsibilities
The moderator
(or review leader) leads the review process
deter-mines, in co-operation with the author, the type of review
approach
the composition of the review team
performs the entry check
follow-up on the rework
schedules the meeting
disseminates documents before the meeting
coaches other team members
paces the meeting
leads possible discussions
stores the data that is collected
The author
learn as much as possible with regard to improving the quality of the document
to improve his or her ability to write future documents
to illuminate unclear areas
to understand the defects found
The scribe
(or recorder)
often the author
to record each defect mentioned and any suggestions for process improvement
The reviewers
(also called checkers or inspectors)
to check any material for defects
The manager
decides on the execution of reviews
allocates time in project schedules
determines whether review process objectives have been met
take care of any review training requested by the participants
can also be involved in the review itself depending on his or her background, playing the role of a reviewer
3.2.3 Types of review
Walkthrough
guiding the participants through the document by the author
useful for higher-level documents, such as requirement specifications and architectural documents
goals
to present the document to stakeholders both within and outside the soft ware discipline
to explain (knowledge transfer) and evaluate the contents of the document
to establish a common understanding of the document
to examine and discuss the validity of proposed solutions and the viability of alternatives, establishing consensus
Key characteristics
The meeting is led by the authors; often a separate scribe is present
Scenarios and dry runs may be used to validate the content
Separate pre-meeting preparation for reviewers is optional
Technical review
discussion meeting that focuses on achieving con-sensus about the technical content of a document
Compared to inspec-tions, technical reviews are less formal
little or no focus on defect identification on the basis of referenced documents
goals
assess the value of technical concepts and alternatives in the product and project environment
establish consistency in the use and representation of technical concepts
ensure, at an early stage, that technical concepts are used correctly
inform participants of the technical content of the document
Key characteristics
It is a documented defect-detection process that involves peers and technical experts
It is often performed as a peer review without management partici pation
Ideally it is led by a trained moderator, but possibly also by a technical expert
A separate preparation is carried out during which the product is examined and the defects are found
More formal characteristics such as the use of checklists and a logging list or issue log are optional
Inspection
the most formal review type
Weinberg's concept of egoless engineering
number of goals
if the time to market is extremely important, the emphasis in inspections will be on efficiency
In a safety-critical market, the focus will be on effectiveness
goals
help the author to improve the quality of the document under inspection
remove defects efficiently, as early as possible
improve product quality, by producing documents with a higher level of quality
create a common understanding by exchanging information among the inspection participants
train new employees in the organization's development process
learn from defects found and improve processes in order to prevent recur rence of similar defects
sample a few pages or sections from a larger document in order to measure the typical quality of the document,
leading to improved work by individuals in the future, and to process improvements
Key characteristics
It is usually led by a trained moderator (certainly not by the author)
It uses defined roles during the process
It involves peers to examine the product
Rules and checklists are used during the preparation phase
A separate preparation is carried out during which the product is examined and the defects are found
The defects found are documented in a logging list or issue log
A formal follow-up is carried out by the moderator applying exit criteria
Optionally, a causal analysis step is introduced to address process improve ment issues and learn from the defects found
Metrics are gathered and analyzed to optimize the process
3.2.4 Success factors for reviews
Find a 'champion'
Pick things that really count
Explicitly plan and track review activities
Train participants
Manage people issues
Follow the rules but keep it simple
Continuously improve process and tools
Report results
quantify the benefits as well as the costs
Just do it!
3.3 STATIC ANALYSIS BY TOOLS
Glossary
compiler
A software tool that translates programs expressed in a high order language into
their machine language equivalents
cyclo-matic complexity
The number of independent paths through a program. Cyclomatic
complexity is defined as: L – N + 2P, where
- L = the number of edges/links in a graph
- N = the number of nodes in a graph
- P = the number of disconnected parts of the graph (e.g. a called graph or subroutine)
control flow
A sequence of events (paths) in the execution through a component or system
data flow
An abstract representation of the sequence and possible changes of the state of
data objects, where the state of an object is any of: creation, usage, or destruction
static analysis
Analysis of software artifacts, e.g. requirements or code, carried out without
execution of these software development artifacts. Static analysis is usually carried out by
means of a supporting tool
Differents from dynamic testing
Static analysis is performed on requirements, design or code without actually executing the software artifact being examined
Static analysis is ideally performed before the types of formal review
Static analysis is unrelated to dynamic properties of the requirements, design and code, such as test coverage
The goal of static analysis is to find defects, whether or not they may cause failures
As with reviews, static analysis finds defects rather than failures
3.3.1 Coding standards
3.3.2 Code metrics
Complexity metrics identify high risk, complex areas
The cyclomatic complexity metric is based on the number of decisions in a program
It is important to testers because it provides an indication of the amount of testing
(including reviews) necessary to practically avoid defects
While there are many ways to calculate cyclomatic complexity, the easiest way is
to sum the number of binary decision statements (e.g. if, while, for, etc.) and add 1 to it
The control flow
3.3.3 Code structure
aspects
control flow structure
the sequence in which the instructions are executed
reflects the iterations and loops in a program's design
can also be used to identify unreachable (dead) code
number of code metrics, e.g.:
number of nested levels
cyclomatic complexity
data flow structure
follows the trail of a data item as it is accessed and mod-ified by the code
how the data act as they are transformed by the program
Defects can be found
referencing a variable with an undefined value
variables that are never used
data structure
organization of the data itself, independent of the program
provides a lot of information about the difficulty in writing programs to handle the data
and in designing test cases to show program correctness
the value of static analysis
early detection of defects prior to test execution
early warning about suspicious aspects of the code, design or requirements
identification of defects not easily found in dynamic testing
improved maintainability of code and design since engineers
work according to documented standards and rules
prevention of defects, provided that engineers are willing
to learn from their errors and continuous improvement is practised
4. Test design techniques
4.1 IDENTIFYING TEST CONDITIONS AND DESIGNING TEST CASES
Test Documentation Standard [IEEE829]
Glossary
test case
A set of input values, execution preconditions, expected results and execution
postconditions, developed for a particular objective or test condition, such as to exercise a
particular program path or to verify compliance with a specific requirement
test case specification
A document specifying a set of test cases (objective, inputs, test
actions, expected results, and execution preconditions) for a test item
test condition
An item or event of a component or system that could be verified by one or
more test cases, e.g. a function, transaction, feature, quality attribute, or structural element
test data
Data that exists (for example, in a database) before a test is executed, and that
affects or is affected by the component or system under test
test procedure specification
A document specifying a sequence of actions for the execution
of a test. Also known as test script or manual test script
test script
Commonly used to refer to a test procedure specification, especially an automated one
traceability
The ability to identify related items in documentation and software, such as
requirements with associated tests
4.1.1 Introduction
test conditions
documented in a Test Design Specification
test cases
documented in a Test Case Specification
test procedures (or scripts)
documented in a Test Procedure Specification
(also known as a test script or a manual test script)
4.1.2 Formality of test documentation
4.1.3 Test analysis: identifying test conditions
A test condition is simply something that we could test
the basic ideas
[Marick, 1994]: 'test requirements' as things that should be tested
[Hutcheson, 2003]: 'test inventory' as a list of things that could be tested
[Craig, 2002]: 'test objectives' as broad categories of things to test
and 'test inventories' as the actual list of things that need to be tested
ISTQB: test condition
traceability.
Test conditions should be able to be linked back to their sources in the test basis
horizontal
through all the test documentation for a given test level
(e.g. system testing, from test conditions through test cases to test scripts)
vertical
through the layers of development documentation
(e.g. from requirements to components)
Test conditions can be identified for test data as well as for test inputs and test outcomes,
IEEE 829 STANDARD: TEST DESIGN SPECIFICATION
4.1.4 Test design: specifying test cases
IEEE 829 Standard for Test Documentation
Oracle - source of informa-tion about the correct behavior of the system
IEEE 829 STANDARD: TEST CASE SPECIFICATION
4.1.5 Test implementation: specifying test procedures or scripts
test proce-dure in IEEE 829
also referred to as a test script
The document that describes the steps to be taken in running a set of tests (and specifies the executable order of the tests)
4.2 CATEGORIES OF TEST DESIGN TECHNIQUES
Glossary
white-box test design techniques
Procedure to derive and/or select test cases based on an
analysis of the internal structure of a component or system
experience-based test design techniques
Procedure to derive and/or select test cases based
on the tester’s experience, knowledge and intuition
specification-based test design techniques
Black box test design technique: Procedure to derive and/or select test cases based on an
analysis of the specification, either functional or non-functional, of a component or system
without reference to its internal structure.
structure-based test design techniques
See white box test design technique
white-box test design techniques
Procedure to derive and/or select test cases based on an
analysis of the internal structure of a component or system
3 types or categories of test design technique, distin-guished by their primary source:
a specification
the structure of the system or component
a person's experience
Techniques
Static (Chapter 3)
Dynamic techniques
4.2.2 Static testing techniques
specification-based (black-box,
also known as behavioral techniques
or input/output-driven testing techniques)
what the software does, not how it does it
Specification-based techniques are appropriate at all levels of testing
(component testing through to acceptance testing) where a specification exists.
Structure-based techniques can also be used at all levels of testing
4.2.5 Experience-based testing techniques
used to complement other techniques
used when there is no specifica-tion
or if the specification is inadequate or out of date
the only type of technique used for low-risk systems
4.3 SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES
Glossary
boundary value analysis
A black box test design technique in which test cases are designed
based on boundary values
decision table testing
A black box test design technique in which test cases are designed to
execute the combinations of inputs and/or stimuli (causes) shown in a decision table
equivalence partitioning
A black box test design technique in which test cases are designed
to execute representatives from equivalence partitions. In principle test cases are designed
to cover each partition at least once
state transition testing
A black box test design technique in which test cases are designed to
execute valid and invalid state transitions
use case testing
A black box test design technique in which test cases are designed to
execute scenarios of use cases
4.3.1 Equivalence partitioning and boundary value analysis
Equivalence partitions are also known as equivalence classes
Boundary value analysis (BVA) is based
on testing at the boundaries between partitions
Designing test cases
both equivalence partitioning and boundary value analysis
combinations of things (e.g. inputs, conditions, etc.)
other techniques with combination:
pairwise testing
orthogonal arrays
Creating a table listing all the combinations of True and False for each of the aspects
4.3.3 State transition testing
('finite state machine')
four basic parts:
the states that the software may occupy
(open/closed or funded/insufficient funds)
the transitions from one state to another
(not all transitions are allowed)
the events that cause a transition
(closing a file or withdrawing money)
the actions that result from a transition
(an error message or being given your cash)
model can be as detailed or as abstract as you need it to be
4.3.4 Use case testing
is a technique that helps us identify test cases that exercise the whole system
on a transaction by transaction basis from start to finish
4.4 STRUCTURE-BASED OR WHITE-BOX TECHNIQUES
Glossary
code coverage
An analysis method that determines which parts of the software have been
executed (covered) by the test suite and which parts have not been executed, e.g. statement
coverage, decision coverage or condition coverage
decision coverage
The percentage of decision outcomes that have been exercised by a test
suite. 100% decision coverage implies both 100% branch coverage and 100% statement
coverage
statement coverage
The percentage of executable statements that have been exercised by a test suite
structural testing
See white box testing
structure-based testing
See white-box testing
white-box testing
Testing based on an analysis of the internal structure of the component or system
4.4.1 Using structure-based techniques
to measure coverage and design tests
2 purposes:
test coverage measurement
100% coverage does not mean 100% tested!
structural test case design
Types of coverage
For testing levels
At integration level
coverage of interfaces
specific interactions
at system or acceptance level the coverage items may be:
require-ments
menu options
screens
typical business transactions
database structural elements
(records, fields and sub-fields) and files
For specification-based techniques
EP: percentage of equivalence partitions exercised
BVA: percentage of boundaries exercised
Decision tables: percentage of business rules or decision table columns tested
State transition testing:
Percentage of states visited
Percentage of (valid) transitions exercised
(this is known as Chow's 0- switch coverage)
Percentage of pairs of valid transitions exercised
('transition pairs' or Chow's 1-switch coverage)
transition triples
transition quadruples
etc.
Percentage of invalid transitions exercised
(from the state table)
Instrumentation to measure coverage
1 Decide on the structural element to be used, i.e. the coverage items to be counted
2 Count the structural elements or items.
3 Instrument the code.
4 Run the tests for which coverage measurement is required.
5 Using the output from the instrumentation, determine the percentage of elements or items exercised.
4.4.2 Statement coverage and statement testing
Example
READ A
2 READ B
3 C =A + 2*B
4 IF C> 50 THEN
5 PRINT large C
6 ENDIF
Then 100% coverage test will be:
Test 1_4: A = 20, B = 25
4.4.3 Decision coverage and decision testing
'sub-sumes' statement coverage - this means that 100% decision coverage always guarantees 100% statement coverage
but not the other way around!
Bit to achieve 100% decision coverage, at least 2 test cases are necessary to cover both True and False
Example
1 READ A
2 READ B
3 C=A-2*B
4 IFC <0THEN
5 PRINT "C negative"
6 ENDIF
Then 100% coverage test will be:
Test 2_1: A = 20, B = 15
Test 2_2: A = 10, B = 2
4.4.4 Other structure-based techniques
branch coverage
Branch coverage measures the coverage of both conditional and unconditional branches
Whilst decision coverage measures the coverage of conditional branches
path coverage
or 'independent path segment coverage'
4.5 EXPERIENCE-BASED TECHNIQUES
Glossary
error guessing
A test design technique where the experience of the tester is used to
anticipate what defects might be present in the component or system under test as a result
of errors made, and to design tests specifically to expose them
exploratory testing
An informal test design technique where the tester actively controls the
design of the tests as those tests are performed and uses information gained while testing to
design new and better tests
4.5.1 Error guessing
A structured approach to the error-guessing technique is to list possible defects or failures
and to design tests that attempt to produce them
can be built based on
the tester's own experience
experience of other people
available defect and failure data
from common knowledge about why software fails
4.5.2 Exploratory testing
is a hands-on approach in which testers are involved in
minimum planning and maximum test execution
A key aspect of exploratory testing is learning
Books
Kaner, 2002
Copeland, 2003
Whittaker, 2002 ('attacks')
4.6 CHOOSING A TEST TECHNIQUE
The best testing technique is no single testing technique
Internal factors
Models used
The models available (i.e. developed and used during the specification)
will govern which testing techniques can be used
Tester knowledge and experience
How much testers know about the system and about testing techniques
Likely defects
Knowledge of the likely defects
(since each technique is good
at finding a particular type of defect)
Test objective
simply to gain confidence? - Use cases
thorough testing? - more rigorous and detailed techniques
Documentation
Whether or not documentation exists and it is up to date
The content and style of the documentation will also influence the choice of techniques
Life cycle model
A sequential life cycle model? - more formal techniques
An iterative life cycle model? - an exploratory testing approach
External factors
Risk
The greater the risk (e.g. safety-critical systems)? - more thorough and more formal testing
Commercial risk
by quality issues? - more thorough testing
time-to-market issues? - exploratory testing
Customer and contractual requirements
Contracts may specify particular testing techniques
(most commonly statement or branch coverage)
Type of system
Financial application? - boundary value analysis
Regulatory requirements
regulatory standards or guidelines
the aircraft industry
equivalence partitioning
boundary value analysis
state transition testing
statement, decision or modified condition decision coverage
Time and budget
5. Test management
5.1 TEST ORGANIZATION
Glossary
tester
A skilled professional who is involved in the testing of a component or system
test leader
See test manager
test manager
The person responsible for project management of testing activities and
resources, and evaluation of a test object. The individual who directs, controls, administers,
plans and regulates the evaluation of a test object
5.1.1 Independent and integrated testing
Independent Test Team
benefits
see more defects
brings a different set of assumptions to testing and to reviews
brings a skeptical attitude of professional pessimism
reporting to a senior or execu-tive manager
a separate budget
risks
interpersonal isolation
stakeholders might see as a bottleneck and a source of delay
programmers can abdicate their responsibility for quality
5.1.2 Working as a test leader
planning, monitoring, and control of the testing activities
devise the test objectives, organizational test policies
(if not already in place), test strategies and test plans
estimate the testing to be done and negotiate with
management to acquire the necessary resources
recognize when test automation is appropriate plan the
effort, select the tools, and ensure training of the team
consult with other groups to help them with their testing
lead, guide and monitor the analysis, design, implementation and
execution of the test cases, test procedures and test suites
ensure proper configuration management of the testware
produced and traceability of the tests to the test basis
make sure the test environment is put into place before
test execution and managed during test execution
schedule the tests for execution and monitor, measure, control and
report on the test progress, the product quality status and the test results,
adapting the test plan and compensating as needed to adjust to evolving conditions
During test execution and as the project winds down,
they write summary reports on test status
5.1.3 Working as a tester
In the planning and preparation phases
review and contribute to test plans
review-ing and assessing requirements and design specifications
identifying test conditions and creating
test designs
test cases
test procedure specifications
test data
automate or help to automate the tests
set up the test envi-ronments
assist system administration network management staff
test execution
execute and log the tests
evaluate the results
document problems found
monitor the testing and the test environment
gather performance metrics
review each other's work, incl. test specifications,
defect reports and test results
5.1.4 Defining the skills test staff need
Application or business domain:
the intended behavior, the problem the system will solve, the process it will automate
Technology:
issues, limitations and capabilities of the chosen implementation technology
Testing:
know the testing topics
5.2 TEST PLANS, ESTIMATES AND STRATEGIES
Glossary
entry criteria
The set of generic and specific conditions for permitting a process to go
forward with a defined task, e.g. test phase. The purpose of entry criteria is to prevent a
task from starting which would entail more (wasted) effort compared to the effort needed
to remove the failed entry criteria
exit criteria
The set of generic and specific conditions, agreed upon with the stakeholders,
for permitting a process to be officially completed. The purpose of exit criteria is to
prevent a task from being considered completed when there are still outstanding parts of
the task which have not been finished. Exit criteria are used to report against and to plan
when to stop testing
exploratory testing
An informal test design technique where the tester actively controls the
design of the tests as those tests are performed and uses information gained while testing to
design new and better tests
test approach
The implementation of the test strategy for a specific project. It typically
includes the decisions made that follow based on the (test) project’s goal and the risk
assessment carried out, starting points regarding the test process, the test design techniques
to be applied, exit criteria and test types to be performed
test level
A group of test activities that are organized and managed together. A test level is
linked to the responsibilities in a project. Examples of test levels are component test,
integration test, system test and acceptance test
test plan
A document describing the scope, approach, resources and schedule of intended
test activities. It identifies amongst others test items, the features to be tested, the testing
tasks, who will do each task, degree of tester independence, the test environment, the test
design techniques and entry and exit criteria to be used, and the rationale for their choice,
and any risks requiring contingency planning. It is a record of the test planning process
test procedure
test procedure specification: A document specifying a sequence of actions for the execution
of a test. Also known as test script or manual test script
test strategy
A high-level description of the test levels to be performed and the testing within
those levels for an organization or programme (one or more projects)
5.2.1 The purpose and substance of test plans
Reasons
guides our thinking
forces us to confront the challenges that await us
and focus our thinking on important topics
cating with other members of the project team, testers,
peers, managers and other stakeholders
manage change
Master test plan
Test levels
integration test plan
system test plan
hardware test plan
software test plan
planning tasks
purposes
What is in scope and what is out of scope for this testing effort?
What are the test objectives?
What are the important project and product risks?
What constraints affect testing (e.g., budget limitations, hard deadlines, etc.)?
What is most critical for this product and project?
Which aspects of the product are more (or less) testable?
What should be the overall test execution schedule and how should we decide the order in which to run specific tests?
select strategies
split the testing work into various levels
fit your testing work in the level
inter-level coordination
integrate and coordinate all the testing work with the rest of the project
entry criteria factors
Acquisition and supply:
the availability of staff, tools, systems and other materials required
Test items:
the state that the items to be tested must be in to start and to finish testing
Defects:
the number known to be present, the arrival rate, the number predicted to remain, and the number resolved
Tests:
the number run, passed, failed, blocked, skipped, and so forth
Coverage:
the portions of the test basis, the software code or both that have been tested and which have not
Quality:
the status of the important quality characteristics for the system
Money:
the cost of finding the next defect in the current level of testing compared
to the cost of finding it in the next level of testing (or in production)
Risk:
the undesirable outcomes that could result from shipping too early
(such as latent defects or untested areas) - or too late (such as loss of market share)
5.2.3 Estimating what testing will involve and what it will cost
phases
planning and control
analysis and design
implementation and execution
evaluating exit criteria and reporting
test closure
risk analysis
identify risks and activities required to reduce them
performance-testing planning
5.2.4 Estimation techniques
consulting the people
'bottom up' estimation
who will do the work
other people with expertise on the tasks to be done
analyzing metrics
top-down estimation
from past projects
from industry data
Approaches
simplest approach
'How many testers do we typically have per developer on a project?'
classifying the project
in terms of size (small, medium or large)
complexity (simple, mod-erate or complex)
how long such projects have taken in the past
simple and reliable approach
the average effort per test case in similar past projects
use the estimated number of test cases to estimate the total effort
Sophisticated approaches
building mathematical models that look at historical or
industry averages for certain key parameters
number of tests run by tester per day
number of defects found by tester per day
etc.
tester-to-developer
estimation must be negotiated with management
5.2.5 Factors affecting test effort
project documentation
Complexity
The difficulty of comprehending and correctly handling the problem the system is being built to solve
The use of innovative technologies, especially those long on hyperbole and short on proven track records
The need for intricate and perhaps multiple test configurations, especially when these rely on the timely arrival of scarce software, hardware and other supplies
The prevalence of stringent security rules, strictly regimented processes or other regulations
The geographical distribution of the team, especially if the team crosses time-zones (as many outsourcing efforts do)
increasing the size of the product leads to increases in the size of the project and the project team
availability of test tools
life cycle of development model
Process maturity, including test process maturity
Time pressure
people factors
skills of the individ-uals and the team as a whole
the alignment of those skills with the project's needs
solid relationships
reliable execution of agreed-upon commitments and responsibilities
a determination to work together towards a common goal
the stability of the project team
The test results
The delivery of good-quality software at the start of test execution
quick, solid defect fixes during test execution
prevents delays in the test execution process
5.2.6 Test approaches or strategies
The major types
Analytical:
the risk-based strategy
project documents and stakeholder input
planning
esti mating
designing
prioritizing based on risk
the requirements-based strategy
planning
estimating
designing tests
have in common the use of some formal or infor mal analytical technique
usually during the requirements and design stages of the project
Model-based:
mathematical models
have in common the creation or selection of some formal or informal model for critical system behaviors
usually during the requirements and design stages of the project
Methodical:
checklist suggests the major areas of testing to run
industry-standard for software quality, e
.g. ISO 9126, to outline of major test areas
have in common the adherence to a pre-planned, systematized approach
developed in-house
assembled from various concepts developed
in-house and gathered from outside
or adapted significantly from outside ideas
may have an early or late point of involvement for testing
design
implement
execute
Process- or standard-compliant:
IEEE 829
one of the agile methodologies
e.g. Extreme Programming (XP)
have in common reliance upon an externally developed approach to testing
may have an early or late point of involvement for testing
Dynamic:
lightweight set of testing guide lines
exploratory testing
have in common concentrat ing on finding as many defects as possible
during test execution and adapt ing to the realities of the system under test
typically emphasize the later stages of testing
the attack-based approach
[Whittaker, 2002] and [Whittaker, 2003]
exploratory approach
[Kaner et al., 2002]
Consultative or directed:
ask the users or develop ers of the system to tell you what to test
or even rely on them to do the testing
have in common the reliance on a group of non-testers
to guide or perform the testing effort
typically emphasize the later stages of testing simply
due to the lack of recognition of the value of early testing
Regression-averse:
automate all the tests of system functionality
have in common a set of procedures (usually automated)
may involve automating functional tests prior to release of the function
sometimes the testing is almost entirely focused on
testing functions that already have been released
There is no one best way
adopt whatever test approaches
feel free to borrow and blend
Factors to consider
Risks:
For a well-established application that is evolving slowly
regression-averse strategies make sense
For a new application, a risk analysis may reveal different
risks if you pick a risk-based analytical strategy
Skills:
which skills your testers possess and lack for strategy execution
A standard- compliant strategy is a smart choice when you lack the time
and skills in your team to create your own approach
Objectives:
Testing must satisfy the needs of stakeholders
find as many defects as possible with a minimal amount of up-front
time and effort invested - dynamic strategy makes sense
Regulations:
satisfy regulators
methodical test strategy
Product:
weapons systems and contract-development software -
synergy with a requirements-based analytical strategy
Business:
can use a legacy system as a model for a new system -
can use a model-based strategy
5.3 TEST PROGRESS MONITORING AND CONTROL
Glossary
defect density
The number of defects identified in a component or system divided by the
size of the component or system (expressed in standard measurement terms, e.g. lines-ofcode,
number of classes or function points)
failure rate
The ratio of the number of failures of a given category to a given unit of
measure, e.g. failures per unit of time, failures per number of transactions, failures per
number of computer runs
test control
A test management task that deals with developing and applying a set of
corrective actions to get a test project on track when monitoring shows a deviation from
what was planned
test coverage
The degree, expressed as a percentage, to which a specified coverage item has been
exercised by a test suite
test monitoring
A test management task that deals with the activities related to periodically
checking the status of a test project. Reports are prepared that compare the actuals to that
which was planned
test report
test summary report: A document summarizing testing activities and results. It also contains
an evaluation of the corresponding test items against exit criteria
test progress report: A document summarizing testing activities and results, produced at
regular intervals, to report progress of testing activities against a baseline (such as the
original test plan) and to communicate risks and alternatives requiring a decision to management
5.3.1 Monitoring the progress of test activities
Test monitoring's purposes
Give the test team and the test manager feedback
to guide and improve the testing and the project
Provide the project team with visibility about the test results
Measure the status of the testing, test coverage and test items against
the exit criteria to determine whether the test work is done
Gather data for use in estimating future test efforts
small projects
gather test progress monitoring information manually using
documents
spreadsheets
simple databases
large teams
distributed projects
long-term test efforts
data collection is aided by the use of automated tools
Metrics
ultra-reliable software
thousands of source lines of code (KSLOC)
func-tion points (FP)
other metric of code size
common metrics
The extent of completion of test environment preparation
The extent of test coverage achieved, measured against requirements,
risks, code, configurations or other areas of interest
The status of the testing (including analysis, design and implementation)
compared to various test milestones
The economics of testing, such as the costs and benefits of continuing test
execution in terms of finding the next defect or running the next test
use the IEEE 829 test log template
5.3.2 Reporting test status
variations driven by
the pref-erences of the testers and stakeholders
the needs and goals of the project
reg-ulatory requirements
time and money constraints
limitations of the tools available
Enables conclusions, recommendations, and decisions
about how to guide the project forward
data gathering for test report
(should be identified at test
planning and preparation periods)
How will you assess the adequacy of the test objectives for a
given test level and whether those objectives were achieved?
How will you assess the adequacy of the test approaches taken and
whether they support the achievement of the project's testing goals?
How will you assess the effectiveness of the testing
with respect to these objectives and approaches?
test summary report
(at a key milestone or
at the end of a test level)
The IEEE 829 Standard Test Summary Report Template
5.3.3 Test control
guiding and corrective actions to try to achieve the best possible outcome for the project
5.4 CONFIGURATION MANAGEMENT
Glossary
configuration management
A discipline applying technical and administrative direction and
surveillance to: identify and document the functional and physical characteristics of a
configuration item, control changes to those characteristics, record and report change
processing and implementation status, and verify compliance with specified requirements
version control
configuration control: An element of configuration management, consisting of the
evaluation, co-ordination, approval or disapproval, and implementation of changes to
configuration items after formal establishment of their configuration identification
Goals
Determe clearly what the items are that
make up the software or system
source code
test scripts
third-party software
hardware
data
both development and test documentation
making sure that these items are managed carefully,
thoroughly and attentively throughout the
entire project and product life cycle
support the build process
to map what is being tested to the underlying
files and components that make it up
report defects against something which is version controlled
transmittal report or release notes
Should be planned during the project planning stage
As the project proceeds
the configuration process and mechanisms must be implemented
the key interfaces to the rest of the development process should be documented
IEEE 829 STANDARD: TEST ITEM TRANSMITTAL REPORT TEMPLATE
5.5 RISK AND TESTING
Glossary
product risk
A risk directly related to the test object
project risk
A risk related to management and control of the (test) project, e.g. lack of
staffing, strict deadlines, changing requirements, etc
risk
A factor that could result in future negative consequences; usually expressed as impact and likelihood
risk-based testing
An approach to testing to reduce the level of product risks and inform
stakeholders of their status, starting in the initial stages of a project. It involves the
identification of product risks and the use of risk levels to guide the test process
5.5.1 Risks and levels of risk
Risk is the possibility of a negative or undesirable outcome
5.5.2 Product risks
'quality risks'
Possibility that the system or software might fail to satisfy some
reasonable customer, user, or stakeholder expectation
Risk-based testing
starts early in the project, identifying risks to system quality
guide testing planning, specification, preparation and execution
involves both mitigation
testing to provide opportunities to reduce the likelihood
of defects, especially high-impact defects
testing to identify work-arounds to make the
defects that do get past us less painful
involves measuring how well we are doing at finding and removing defects in critical areas
involve using risk analysis to identify proactive opportunities to remove or prevent defects
through non-testing activities and to help us select which test activities to perform
product risk analysis techniques
a close reading of the requirements specification, design specifica-tions, user documentation and other items
brainstorming with many of the project stakeholders
a sequence of one-on-one or small-group sessions with the business and technology experts in the company
team-based approach that involves the key stakeholders and experts is preferable to a purely document-based approach
risks in the areas
functionality
localization
usability
reliability
performance
supportability
use the quality characteristics and sub-characteristics from ISO 9126
a checklist of typical or past risks that should be considered
review the tests that failed and the bugs that you found in a previous release or a similar product
A five-point scale to rate likelihood and impact vary tends to work well
Tips
to consider both likelihood and impact
calculate a risk priority number
a high likelihood and a medium impact = 6 (2 times 3)
risk analyses are educated guesses
5.5.3 Project risks
Examples of possible risks
the late delivery of the test items to the test team
availability issues with the test environment
excessive delays in repairing defects found in testing
problems with getting professional system administration support for the test environment.
4 typical options of risks:
Mitigate: Take steps in advance to reduce the likelihood (and possibly the impact) of the risk
Contingency: Have a plan in place to reduce the impact should the risk become an outcome
Transfer: Convince some other member of the team or project stakeholder to reduce the likelihood or accept the impact of the risk
Ignore: Do nothing about the risk, which is usually a smart option only when there's little that can be done or when the likelihood and impact are low
typical risks
Logistics or product quality problems that block tests:
These can be miti gated through careful planning, good defect triage and management, and robust test design
Test items that won't install in the test environment:
These can be mitigated through smoke (or acceptance) testing prior to starting test phases
or as part of a nightly build or continuous integration.
Having a defined uninstall process is a good contingency plan
Excessive change to the product that invalidates test results or requires
updates to test cases, expected results and environments:
These can be mit igated through good change-control processes,
robust test design and light weight test documentation.
When severe incidents occur, transference of the risk
by escalation to management is often in order
Insufficient or unrealistic test environments that yield misleading results:
One option is to transfer the risks to management by explaining
the limits on test results obtained in limited environments.
Mitigation - sometimes com plete alleviation - can be achieved by outsourcing
tests such as performance tests that are particularly sensitive to proper test environments
additional risks
Organizational issues such as shortages of people, skills or training,
problems with communicating and responding to test results,
bad expectations of what testing can achieve and
complexity of the project team or organization
Supplier issues such as problems with underlying platforms or hardware, failure to consider
testing issues in the contract or failure to properly respond to the issues when they arise
Technical problems related to ambiguous, conflicting or unprioritized requirements, an excessively large number of requirements
given other project constraints, high system complexity and quality problems with the design, the code or the tests
test items can also have risks, e.g.:
the test plan will omit tests for a functional area
that the test cases do not exercise the critical areas of the system
5.5.4 Tying it all together for risk management
assess or analyze risks early in the project
educated guesses
Do not confuse impact with likelihood or vice versa
5.6 INCIDENT MANAGEMENT
Glossary
incident logging
Recording the details of any incident that occurred, e.g. during testing
5.6.1 What are incident reports for and how do I write good ones?
causes
the system exhibits questionable behavior
a defect only when the root cause is some problem in tested item
misconfiguration or failure of the test environment
corrupted test data
bad tests
invalid expected results
tester
mistakes
can also log, report, track, and manage incidents
found during development and reviews
defect detection percentage (DDP) metric
5.6.2 What goes in an incident report?
5.6.3 What happens to incident reports after you file them?
6. Tool support for testing
Glossary
debugging tool
A tool used by programmers to reproduce failures, investigate the state of
programs and find the corresponding defect. Debuggers enable programmers to execute
programs step by step, to halt a program at any program statement and to set and examine
program variables
driver
A software component or test tool that replaces a component that takes care of the
control and/or the calling of a component or system
stub
A skeletal or special-purpose implementation of a software component, used to develop
or test a component that calls or is otherwise dependent on it. It replaces a called component
probe effect
The effect on the component or system by the measurement instrument when
the component or system is being measured, e.g. by a performance testing tool or monitor.
For example performance may be slightly worse when performance testing tools are being used
data-driven testing
A scripting technique that stores test input and expected results in a table
or spreadsheet, so that a single control script can execute all of the tests in the table. Data
driven testing is often used to support the application of test execution tools such as
capture/playback tools
keyword-driven testing
A scripting technique that uses data files to contain not only test
data and expected results, but also keywords related to the application being tested. The
keywords are interpreted by special supporting scripts that are called by the control script
for the test
scripting language
A programming language in which executable test scripts are written,
used by a test execution tool (e.g. a capture/playback tool)
Types of tools
test execution tools
performance testing tools
static analysis tools
test management tools
6.1 TYPES OF TEST TOOL
'probe effect'
'instrumenting the code'
different coverage tools get a slightly different coverage measure on the same program
'Heizenbugs'
If the code is run with the debugger, then the bug disappears
6.1.2 Tool support for management of testing and tests
Also known as:
'the management of tests'
'managing the testing process'
Test management tools
Features or characteristics
management of tests
keeping track of the associated data for a given set of tests
knowing which tests need to run in a common environment
number of tests planned, written, run, passed or failed
scheduling of tests to be executed
(manually or by a test execution tool)
management of testing activities
time spent in test design
test execution
whether we are on schedule or on budget
interfaces to other tools, such as:
test execution tools (test running tools)
incident management tools
requirement management tools
configuration management tools
traceability of tests, test results and defects to requirements or other sources
logging test results
summarize results from test execution tools that the test manage-ment tool interfaces with
preparing progress reports based on metrics (quantitative analysis), such as:
tests run and tests passed
incidents raised, defects fixed and outstanding
Requirements management tools
Features or characteristics
storing requirement statements
storing information about requirement attributes
checking consistency of requirements
identifying undefined, missing or 'to be defined later' requirements
prioritizing requirements for testing purposes
traceability of requirements to tests and tests to requirements, functions or features
traceability through levels of requirements
interfacing to test management tools
coverage of requirements by a set of tests (sometimes)
Incident management tools
also known as
a defect-tracking tool
a defect-management tool
a bug-tracking tool
a bug-management tool
Features or characteristics
storing information about the attributes of incidents (e.g. severity)
storing attachments (e.g. a screen shot)
prioritizing incidents
assigning actions to people (fix, confirmation test, etc.)
status, e.g.:
open
rejected
duplicate
deferred
ready for confirmation test
closed
reporting of statistics/metrics about incidents, e.g.:
average time open
number of incidents with each status
total number raised
open or closed
Configuration management tools
Features or characteristics
storing information about versions and builds of the software and testware
traceability between software and testware and different versions or variants
keeping track of which versions belong with which configurations, e.g.:
operating systems
libraries
browsers
build and release management
baselining (e.g. all the configuration items that make up a specific release)
access control (checking in and out)
6.1.3 Tool support for static testing
Review process support tools
a common reference for the review process or processes to use in different situations
storing and sorting review comments
communicating comments to relevant people
coordinating online reviews
keeping track of comments, including defects found, and providing statistical information about them
providing traceability between comments, documents reviewed and related documents
a repository for rules, procedures and checklists to be used in reviews, as well as entry and exit criteria
monitoring the review status (passed, passed with corrections, requires re- review)
collecting metrics and reporting on key factors
Static analysis tools (D)
*D - likely to be used by developers
calculate metrics such as cyclomatic complexity or nesting levels
(which can help to identify where more testing may be needed
due to increased risk)
enforce coding standards
analyze structures and dependencies
aid in code understanding
identify anomalies or defects in the code
Modeling tools (D)
identifying inconsistencies and defects within the model
helping to identify and prioritize areas of the model for testing
predicting system response and behavior under various situations,
such as level of load
helping to understand system functions and identify test conditions
using a modeling language such as UML
can be used before dynamic tests can be run:
- earlier defect detection and fix
- fewer defects left to propa-gate into later stages
6.1.4 Tool support for test specification
Test design tools
Types of tools
construct test cases
Computer Aided Software Engineering (CASE)
possible to identify the input fields, including the range of valid values
select combinations of possible factors to be used in testing, to ensure that
all pairs of combinations of operating system and browser are tested
'screen scraper'
coverage tool
which branches have been covered by a set of existing tests
identify the path that needs to be taken in order to cover the untested branches
Features or characteristics
generating test input values from:
requirements
design models (state, data or object)
code
graphical user interfaces
test conditions
generating expected results,
if an oracle is available to the tool
- This helps the testing to be more thorough
(if that is an objective of the test!)
- Unmanageable number of tests can be done by risk analysis
Test data preparation tools
enable data to be selected from an existing data-base or created,
generated, manipulated and edited for use in tests
The most sophisticated tools can deal with a range of files and database formats
Features or characteristics
extract selected data records from files or databases
'massage' data records to make them anonymous or not able
to be identified with real people (for data protection)
enable records to be sorted or arranged in a different order
generate new records populated with pseudo-random data, or data set up
according to some guidelines, e.g. an operational profile
construct a large number of similar records from a template,
to give a large set of records for volume tests, for example
6.1.5 Tool support for test execution and logging
Test execution tools
'test running tool' or
'regression testing tools'
'capture/playback' tools or
'capture/replay' tools or
'record/playback' tools
difficult to maintain because:
It is closely tied to the flow and interface presented by the GUI
It may rely on the circumstances, state and context of the system at the time the script was recorded
The test input information is 'hard-coded'
using programming skills
- tests can repeat actions (in loops) for different data values
- take different routes depending on the outcome of a test
- can be called from other scripts giving some structure to the set of tests
Features or characteristics
capturing (recording) test inputs while tests are executed manually
storing an expected result in the form of a screen or object
to compare to, the next time the test is run
executing tests from stored scripts and optionally data files accessed by the script
(if data-driven or keyword-driven scripting is used)
dynamic comparison (while the test is running) of screens,
elements, links, controls, objects and values
ability to initiate post-execution comparison
logging results of tests run (pass/fail, differences between expected and actual results)
masking or filtering of subsets of actual and expected results
measuring timings for tests
synchronizing inputs with the application under test
e.g. wait until the appli cation is ready to accept
the next input, or insert a fixed delay to
represent human interaction speed
sending summary results to a test management tool
Test harness/unit test framework tools (D)
test harness
stubs
drivers
unit test framework tools
Features or characteristics
supplying inputs to the software being tested
receiving outputs generated by the software being tested
executing a set of tests within the framework or using the test harness
framework tools
recording the pass/fail results of each test
storing tests
support for debugging
coverage measurement at code level
Test comparators
Dynamic comparison
Integrated with test execution tool
Post-execution comparison
'stand-alone' tool
comparing a large volume of data
comparing a large set of records from a database
with the expected content of those records
Features or characteristics
dynamic comparison of transient events that occur during test execution
post-execution comparison of stored data, e.g. in files or databases
masking or filtering of subsets of actual and expected results
identifying coverage items (instrumenting the code)
calculating the percentage of coverage items that were exercised by a suite of tests
reporting coverage items that have not been exercised as yet
identifying test inputs to exercise as yet uncovered items (test design tool functionality)
generating stubs and drivers (if part of a unit test framework)
Security tools
may focus on:
the network
the support software
the application code
the underlying database
Features or characteristics
identifying viruses
detecting intrusions such as denial of service attacks
simulating various types of external attacks
probing for open ports or other externally visible points of attack
identifying weaknesses in password files and passwords
security checks during operation
6.1.6 Tool support for performance and monitoring
Dynamic analysis tools (D)
detecting memory leaks
identifying pointer arithmetic errors such as null pointers
identifying time dependencies
Broken links research
'web spider'
Performance-testing, load-testing and stress-testing tools
Performance-testing tools
testing at system level to see whether or not the system will stand up to a high volume of usage
'load' test
checks that the system can cope with its expected number of transactions
'volume' test
checks that the system can cope with a large amount of data
'stress' test
beyond the normal expected usage of the system
Features or characteristics
generating a load on the system to be tested
measuring the timing of specific transactions as the load on the system varies
measuring average response times
producing graphs or charts of responses over time
Monitoring tools
For:
servers
networks
databases
security
performance
website and internet usage
applications
Features or characteristics
identifying problems and sending an alert message to the administrator
logging real-time and historical information
finding optimal settings
monitoring the number of users on a network
monitoring network traffic
6.1.7 Tool support for specific application areas (Kl)
web-based performance-testing tools
performance-testing tools for back-office systems
static analysis tools for specific development platforms and programming languages
dynamic analysis tools that focus on security issues
dynamic analysis tools for embedded systems
6.1.8 Tool support using other tools
word processor
spreadsheet
SQL
debugging tools
6.2 EFFECTIVE USE OF TOOLS: POTENTIAL BENEFITS AND RISKS
6.2.1 Potential benefits of using tools
Benefits
reduction of repetitive work
greater consistency and repeatability
objective assessment
ease of access to information about tests or testing
Examples of repetitive work
running regression tests
entering the same test data
checking against coding standards
creating a specific test database
Examples of beneficial usage of tools
to confirm the correctness of a fix to a defect
(a debugging tool or test execution tool)
enter-ing test inputs
(a test execution tool)
generating tests from requirements
(a test design tool or possibly a
requirements management tool)
assessing the cyclomatic complexity or nesting levels of a component
(a static analysis tool)
coverage
(coverage measurement tool)
system behavior
(monitoring tools)
incident statistics
(test management tool)
statistics and graphs about test progress
(test execution or test management tool)
incident rates
(incident management or test management tool)
performance
(performance testing tool)
6.2.2 Risks of using tools
Risks include:
unrealistic expectations for the tool
underestimating the time, cost and effort for the initial introduction of a tool
underestimating the time and effort needed to achieve significant and con tinuing benefits from the tool
underestimating the effort required to maintain the test assets generated by the tool
over-reliance on the tool
Two other important factors are:
the skill needed to create good tests
the skill needed to use the tools well, depending on the type of tool
6.2.3 Special considerations for some types of tools
Test execution tools
levels of scripting
linear scripts (which could be created manually or captured by recording a manual test)
structured scripts (using selection and iteration programming structures)
shared scripts (where a script can be called by other scripts so can be re-used
(also require a formal script library under configuration man agement)
data-driven scripts (where test data is in a file or spreadsheet to be read by a control script)
keyword-driven scripts (where all of the information about the test is stored in a file or spreadsheet,
with a number of control scripts that implement the tests described in the file)
Reasons why a captured test (a linear script) is not a good solution:
The script doesn't know what the expected result is until you program it in -
it only stores inputs that have been recorded, not test cases
A small change to the software may invalidate dozens or hundreds of scripts
The recorded script can only cope with exactly the same conditions as when it was recorded.
Unexpected events (e.g. a file that already exists) will not be interpreted correctly by the tool
when capturing test inputs is useful
exploratory testing
running unscripted tests with experienced business users
short term, where the context remains valid
to log everything that is done,
as an audit trail
Performance testing tools
Examples
the transaction throughput
the degree of accuracy of a given computation
the computer resources being used for a given level of transactions
the time taken for certain transactions
the number of users that can use the system at once
Issues
the design of the load to be generated by the tool
(e.g. random input or according to user profiles)
timing aspects (e.g. inserting delays to make simulated user input more realistic)
the length of the test and what to do if a test stops prematurely
narrowing down the location of a bottleneck
exactly what aspects to measure (e.g. user interaction level or server level)
how to present the information gathered
Static analysis tools
Test management tools
6.3 INTRODUCING A TOOL INTO AN ORGANIZATION
6.3.1 Main principles
Factors in selecting a tool:
assessment of the organization's maturity (e.g. readiness for change)
identification of the areas within the organization where
tool support will help to improve testing processes
evaluation of tools against clear requirements and objective criteria
proof-of-concept to see whether the product works as desired
and meets the requirements and objectives defined for it
evaluation of the vendor (training, support and other commercial aspects)
or open-source network of support
identifying and planning internal implementation (including coaching and
mentoring for those new to the use of the tool)
6.3.2 Pilot project
should experiment with different ways of using the tool
different settings for a static analysis tool
different reports from a test management tool
differ-ent scripting and comparison techniques for a test execution tool
different load profiles for a performance-testing tool
The objectives for a pilot project for a new tool are:
to learn more about the tool (more detail, more depth)
to see how the tool would fit with existing processes or documentation,
how those would need to change to work well with the tool and how
to use the tool to streamline existing processes
to decide on standard ways of using the tool that will work for all potential users, e.g.:
naming conventions
creation of libraries
defining modularity, where different elements will be stored
how modularity and the tool itself will be maintained
to evaluate the pilot project against its objectives (have the benefits been achieved at reasonable cost?)
6.3.3 Success factors
incremental roll-out (after the pilot) to the rest of the organization
adapting and improving processes, testware and tool artefacts to get
the best fit and balance between them and the use of the tool
providing adequate training, coaching and mentoring of new users
defining and communicating guidelines for the use of the tool,
based on what was learned in the pilot
implementing a continuous improvement mechanism as
tool use spreads through more of the organization
monitoring the use of the tool and the benefits achieved and adapting
the use of the tool to take account of what is learned
CHAPTER REVIEW
Section 6.1
Tools that support the management of testing and tests:
test management tool
A tool that provides support to the test management and control part
of a test process. It often has several capabilities, such as testware management, scheduling
of tests, the logging of results, progress tracking, incident management and test reporting
requirements management tool
A tool that supports the recording of requirements,
requirements attributes (e.g. priority, knowledge responsible) and annotation, and
facilitates traceability through layers of requirements and requirements change
management. Some requirements management tools also provide facilities for static
analysis, such as consistency checking and violations to pre-defined requirements rules
incident management tool
A tool that facilitates the recording and status tracking of
incidents. They often have workflow-oriented facilities to track and control the allocation,
correction and re-testing of incidents and provide reporting facilities
configuration management tool
A tool that provides support for the identification and
control of configuration items, their status over changes and versions, and the release of
baselines consisting of configuration items
Tools that support static testing:
review process support tool
static analysis tool (D)
static analyzer: A tool that carries out static analysis
modeling tool (D)
A tool that supports the creation, amendment and verification of models of the
software or system
Tools that support test specification:
test design tool
A tool that supports the test design activity by generating test inputs from a
specification that may be held in a CASE tool repository, e.g. requirements management
tool, from specified test conditions held in the tool itself, or from code
test data preparation tool
A type of test tool that enables data to be selected from existing
databases or created, generated, manipulated and edited for use in testing
Tools that support test execution and logging:
test execution tool
A type of test tool that is able to execute other software using an
automated test script, e.g. capture/playback
test harness and unit test framework tool (D)
test harness: A test environment comprised of stubs and drivers needed to execute a test.
unit test framework: A tool that provides an environment for unit or component testing in
which a component can be tested in isolation or with suitable stubs and drivers. It also
provides other support for the developer, such as debugging capabilities
test comparator
A test tool to perform automated test comparison of actual results with
expected results
coverage measurement tool (D)
coverage tool: A tool that provides objective measures of what structural elements, e.g.
statements, branches have been exercised by a test suite.
security tool
A tool that supports operational security
Tools that support performance and monitoring:
dynamic analysis tool
A tool that provides run-time information on the state of the software
code. These tools are most commonly used to identify unassigned pointers, check pointer
arithmetic and to monitor the allocation, use and de-allocation of memory and to flag
memory leaks
performance-testing, load-testing and stress-testing tool
performance testing tool: A tool to support performance testing that usually has two main
facilities: load generation and test transaction measurement. Load generation can simulate
either multiple users or high volumes of input data. During execution, response time 32
measurements are taken from selected transactions and these are logged. Performance
testing tools normally provide reports based on test logs and graphs of load against response times
See performance testing tool.
stress testing tool: A tool that supports stress testing.
monitoring tool
monitor: A software tool or hardware device that runs concurrently with the component or
system under test and supervises, records and/or analyses the behavior of the component or system
Section 6.3
main principles of introducing a tool into an organization, e.g.:
assessing organizational maturity
clear requirements and objective criteria
proof-of-concept
vendor evaluation
coaching and mentoring
goals of a proof-of-concept or
piloting phase for tool evaluation. e.g.: