1. Perceived Quality
2. Test Techniques
   1. Functional Testing
      1. Identify things that the product can do (functions and subfunctions).
      2. Determine how you’d know if a function was capable of working.
      3. Test each function, one at a time.
      4. See that each function does what it’s supposed to do and not what it isn’t supposed to do.
   2. Domain Testing
      1. Look for any data processed by the product. Look at outputs as well as inputs.
      2. Decide which particular data to test with. Consider things like boundary values, typical values, convenient values, invalid values, or best representatives.
      3. Consider combinations of data worth testing together.
   3. Stress Testing
      1. Look for sub-systems and functions that are vulnerable to being overloaded or “broken” in the presence of challenging data or constrained resources.
      2. Identify data and resources related to those sub-systems and functions.
      3. Select or generate challenging data, or resource constraint conditions to test with: e.g., large or complex data structures, high loads, long test runs, many test cases, low memory conditions.
   4. Flow Testing
      1. Perform multiple activities connected end-to-end; for instance, conduct tours through a state model.
      2. Don’t reset the system between actions.
      3. Vary timing and sequencing, and try parallel threads.
   5. Scenario Testing
      1. Begin by thinking about everything going on around the product.
      2. Design tests that involve meaningful and complex interactions with the product.
      3. A good scenario test is a compelling story of how someone who matters might do something that matters with the product.
   6. Claims Testing
      1. Identify reference materials that include claims about the product (implicit or explicit). Consider SLAs, EULAs, advertisements, specifications, help text, manuals, etc.
      2. Analyze individual claims, and clarify vague claims.
      3. Test the veracity of each claim about the product.
      4. If you’re testing from an explicit specification, expect it and the product to be brought into alignment.
   7. User Testing
      1. Identify categories and roles of users.
      2. Determine what each category of user will do (use cases), how they will do it, and what they value.
      3. Get real user data, or bring real users in to test.
      4. Otherwise, systematically simulate a user (be careful—it’s easy to think you’re like a user even when you’re not).
      5. Powerful user testing is that which involves a variety of users and user roles, not just one.
   8. Risk Testing
      1. What kinds of problems could the product have?
      2. Which kinds matter most? Focus on those.
      3. How would you detect them if they were there?
      4. Make a list of interesting problems and design tests specifically to reveal them.
      5. It may help to consult experts, design documentation, past bug reports, or apply risk heuristics.
   9. Automatic Checking
      1. Look for or develop tools that can perform a lot of actions and check a lot things.
      2. Consider tools that partially automate test coverage.
      3. Consider tools that partially automate oracles.
      4. Consider automatic change detectors.
      5. Consider automatic test data generators.
      6. Consider tools that make human testing more powerful.
3. Quality Criteria Categories
   1. Capability
   2. Reliability
      1. Robustness: the product continues to function over time without degradation, under reasonable conditions.
      2. Error handling: the product resists failure in the case of errors, is graceful when it fails, and recovers readily.
      3. Data Integrity: the data in the system is protected from loss or corruption.
      4. Safety: the product will not fail in such a way as to harm life or property.
   3. Usability
      1. Learnability: the operation of the product can be rapidly mastered by the intended user.
      2. Operability: the product can be operated with minimum effort and fuss.
      3. Accessibility: the product meets relevant accessibility standards and works with O/S accessibility features.
   4. Charisma
      1. Aesthetics: the product appeals to the senses.
      2. Uniqueness: the product is new or special in some way.
      3. Necessity: the product possesses the capabilities that users expect from it.
      4. Usefulness: the product solves a problem that matters, and solves it well.
      5. Entrancement: users get hooked, have fun, are fully engaged when using the product.
      6. Image: the product projects the desired impression of quality.
   5. Security
      1. Authentication: the ways in which the system verifies that a user is who he says he is.
      2. Authorization: the rights that are granted to authenticated users at varying privilege levels.
      3. Privacy: the ways in which customer or employee data is protected from unauthorized people.
      4. Security holes: the ways in which the system cannot enforce security (e.g. social engineering vulnerabilities)
   6. Scalability
   7. Compatibility
      1. Application Compatibility: the product works in conjunction with other software products.
      2. Operating System Compatibility: the product works with a particular operating system.
      3. Hardware Compatibility: the product works with particular hardware components and configurations.
      4. Backward Compatibility: the products works with earlier versions of itself.
      5. Resource Usage: the product doesn’t unnecessarily hog memory, storage, or other system resources
   8. Performance
   9. Installability
      1. System requirements: Does the product recognize if some necessary component is missing or insufficient?
      2. Configuration: What parts of the system are affected by installation? Where are files and resources stored?
      3. Uninstallation: When the product is uninstalled, is it removed cleanly?
      4. Upgrades/patches: Can new modules or versions be added easily? Do they respect the existing configuration?
      5. Administration: Is installation a process that is handled by special personnel, or on a special schedule?
   10. Development
       1. Supportability: How economical will it be to provide support to users of the product?
       2. Testability: How effectively can the product be tested?
       3. Maintainability: How economical is it to build, fix or enhance the product?
       4. Portability: How economical will it be to port or reuse the technology elsewhere?
       5. Localizability: How economical will it be to adapt the product for other places?
4. Project Environment
   1. Mission
      1. Do you know who your customers are? Whose opinions matter? Who benefits or suffers from the work you do?
      2. Do you know what your customers expect of you on this project? Do you agree?
      3. Maybe your customers have strong ideas about what tests you should create and run.
      4. Maybe they have conflicting expectations. You may have to help identify and resolve those.
   2. Information
      1. Whom can we consult with to learn about this project?
      2. Are there any engineering documents available? User manuals? Web-based materials? Specs? User stories?
      3. Does this product have a history? Old problems that were fixed or deferred? Pattern of customer complaints?
      4. Is your information current? How are you apprised of new or changing information?
      5. Are there any comparable products or projects from which we can glean important information?
   3. Developer Relations
      1. Hubris: Does the development team seem overconfident about any aspect of the product?
      2. Defensiveness: Is there any part of the product the developers seem strangely opposed to having tested?
      3. Rapport: Have you developed a friendly working relationship with the programmers?
      4. Feedback loop: Can you communicate quickly, on demand, with the programmers?
      5. Feedback: What do the developers think of your test strategy?
   4. Test Team
      1. Do you know who will be testing? Do you have enough people?
      2. Are there people not on the “test team” that might be able to help? People who’ve tested similar products before and might
      3. have advice? Writers? Users? Programmers?
      4. Are there particular test techniques that the team has special skill or motivation to perform?
      5. Is any training needed? Is any available?
      6. Who is co-located and who is elsewhere? Will time zones be a problem?
   5. Equipment & Tools
      1. Hardware: Do we have all the equipment you need to execute the tests? Is it set up and ready to go?
      2. Automation: Are any test tools needed? Are they available?
      3. Probes: Are any tools needed to aid in the observation of the product under test?
      4. Matrices & Checklists: Are any documents needed to track or record the progress of testing?
   6. Schedule
      1. Test Design: How much time do you have? Are there tests better to create later than sooner?
      2. Test Execution: When will tests be executed? Are some tests executed repeatedly, say, for regression purposes?
      3. Development: When will builds be available for testing, features added, code frozen, etc.?
      4. Documentation: When will the user documentation be available for review?
   7. Test Items
      1. Scope: What parts of the product are and are not within the scope of your testing responsibility?
      2. Availability: Do you have the product to test? Do you have test platforms available? When do you get new builds?
      3. Volatility: Is the product constantly changing? What will be the need for retesting?
      4. New Stuff: What has recently been changed or added in the product?
      5. Testability: Is the product functional and reliable enough that you can effectively test it?
      6. Future Releases: What part of your tests, if any, must be designed to apply to future releases of the product?
   8. Deliverables
      1. Content: What sort of reports will you have to make? Will you share your working notes, or just the end results?
      2. Purpose: Are your deliverables provided as part of the product? Does anyone else have to run your tests?
      3. Standards: Is there a particular test documentation standard you’re supposed to follow?
      4. Media: How will you record and communicate your reports?
5. Product Elements
   1. Structure
      1. Code: the code structures that comprise the product, from executables to individual routines.
      2. Hardware: any hardware component that is integral to the product.
      3. Non-executable files: any files other than multimedia or programs, like text files, sample data, or help files.
      4. Collateral: anything beyond software and hardware that is also part of the product, such as paper documents, web links and content, packaging, license agreements, etc.
   2. Function
      1. Application: any function that defines or distinguishes the product or fulfills core requirements.
      2. Calculation: any arithmetic function or arithmetic operations embedded in other functions.
      3. Time-related: time-out settings; daily or month-end reports; nightly batch jobs; time zones; business holidays; interest calculations; terms and warranty periods; chronograph functions.
      4. Transformations: functions that modify or transform something (e.g. setting fonts, inserting clip art, withdrawing money from account).
      5. Startup/Shutdown: each method and interface for invocation and initialization as well as exiting the product.
      6. Multimedia: sounds, bitmaps, videos, or any graphical display embedded in the product.
      7. Error Handling: any functions that detect and recover from errors, including all error messages.
      8. Interactions: any interactions between functions within the product.
      9. Testability: any functions provided to help test the product, such as diagnostics, log files, asserts, test menus, etc.
   3. Data
      1. Input: any data that is processed by the product.
      2. Output: any data that results from processing by the product.
      3. Preset: any data that is supplied as part of the product, or otherwise built into it, such as prefabricated databases, default values, etc.
      4. Persistent: any data that is stored internally and expected to persist over multiple operations. This includes modes or states of the product, such as options settings, view modes, contents of documents, etc.
      5. Sequences/Combinations: any ordering or permutation of data, e.g. word order, sorted vs. unsorted data, order of tests.
      6. Cardinality: Numbers of objects or fields may vary (e.g. zero, one, many, max, open limit). Some may have to be unique (e.g. database keys).
      7. Big/Little: variations in the size and aggregation of data.
      8. Noise: any data or state that is invalid, corrupted, or produced in an uncontrolled or incorrect fashion.
      9. Lifecycle: transformations over the lifetime of a data entity as it is created, accessed, modified, and deleted
   4. Interfaces
      1. User Interfaces: any element that mediates the exchange of data with the user (e.g. displays, buttons, fields, whether physical or virtual).
      2. System Interfaces: any interface with something other than a user, such as other programs, hard disk, network, etc.
      3. API/SDK: Any programmatic interfaces or tools intended to allow the development of new applications using this product.
      4. Import/export: any functions that package data for use by a different product, or interpret data from a different product.
   5. Platform
      1. External Hardware: hardware components and configurations that are not part of the shipping product, but are required (or optional) in order for the product to work: systems, servers, memory, keyboards, the Cloud.
      2. External Software: software components and configurations that are not a part of the shipping product, but are required (or optional) in order for the product to work: operating systems, concurrently executing applications, drivers, fonts, etc.
      3. Internal Components: libraries and other components that are embedded in your product but are produced outside your project.
   6. Operations
      1. Users: the attributes of the various kinds of users.
      2. Environment: the physical environment in which the product operates, including such elements as noise, light, and distractions.
      3. Common Use: patterns and sequences of input that the product will typically encounter. This varies by user.
      4. Disfavored Use: patterns of input produced by ignorant, mistaken, careless or malicious use.
      5. Extreme Use: challenging patterns and sequences of input that are consistent with the intended use of the product.
   7. Time
      1. Input/Output: when input is provided, when output created, and any timing relationships (delays, intervals, etc.) among them.
      2. Fast/Slow: testing with “fast” or “slow” input; fastest and slowest; combinations of fast and slow.
      3. Changing Rates: speeding up and slowing down (spikes, bursts, hangs, bottlenecks, interruptions).
      4. Concurrency: more than one thing happening at once (multi-user, time-sharing, threads, and semaphores, shared data).