Modern Testing (MT)

principles
1. https://www.angryweasel.com/ABTesting/modern-testing-principles/
2. Why Do We have MT Principles?
  1. WHAT GOT US HERE WON"T GET AS THERE
    1. MT is looking how to get "THERE"
      1. Moving test from cost center to cost benefit
      2. Being Force multiplier
  2. Software is much easier to produce
  3. Thanks to advancements in Continuous Delivery, the customer has much more options to choose from
    1. Companies are shipping faster
  4. It's much easier to compete with any company
    1. For customers, the switching cost has gone way down
  5. I.E World is changing and Tradiotnal Testing is not catching up
    1. Any time there is a change there is opportunity
3. The principles are not just theory
  1. They are based on Experience and observation
4. Why modern testing
  1. To make contrast with Traditional
    1. Traditional testing is not only a cost but also delay
5. Principle 1
  1. Our priority is improving the business.
    1. What it means to improve the business?
      1. MTer needs to be seen as positive influence
      2. Boiling Frogs
      3. Adoption has to be gradual
      4. Delta from Idea to deployments is rapid
      5. Get that mvp out with just enough quality to get feedback from customers
      6. “You don’t get value from engineering effort until it is in customer hands”
      7. Think not only how to get it out but what’s next
      8. No one tests your product better than your customer
      9. what in case of software houses? They don't have product per se
      10. There are bugs and bugs that customer cares about
      11. The GOAL
      12. Short Term
      13. Provide Value to customer Frequently
      14. Long Term
      15. Making sure Principle are Followed
6. Principle 2
  1. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
    1. WHAT
      1. Our goal is to accelerate team
      2. that doesn't mean "we have to go fast"
      3. Instead we are force multiplier
      4. its about combining
      5. adaptiblity
      6. Reaction
      7. Quickness
      8. Lean Principles
      9. Deliver fast by managing the flow
      10. source: https://www.lean.org/WhatsLean/Principles.cfm
      11. Books to read
      12. The Goal: A Business Graphic Novel
      13. Phonix Project
      14. Lean Thinking
    2. WHY
      1. Don’t forget MT context
      2. By focusing the MT Practitioner on identifying their role as acceleration”
      3. this model were chosen because they are proven
      4. It’s important cause far to much of The TEST is reverse of this
      5. they create the bottleneck instead of resolving them
      6. Example: stabilization period just before release
      7. It’s easier to "build quality In" instead of trying to test "quality in"
      8. Test Complete date
      9. No i don't mean the tool
      10. “If you test after test complete you will find bugs and that will cause us to miss the deadline"
    3. HOW
      1. This mantra is About completion and predictability not about value in order to accelerate you need mechanism to identify what value is and what the next bottleneck
      2. Mt don’t want testing to be viewed as bottle neck
      3. Quality is problem solved to humans being
      4. Fast feedback loop is critical in context of understanding what is quality
      5. By Balancing fallacy of value with failure to launch
7. Principle 3
  1. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.
    1. WHAT
      1. “We are going to favour continuously adapting and improving above safety net presentation model.”
      2. Thou shall kill your safety net
      3. Traditional testing is treated as safty net
      4. It’s wrong that testers feel validated by the fact the developers don’t want to test.
      5. If we are going to move quality upstream, we have to retrain developers without the safety net
      6. “Too many people think the quality is limited to testing “
      7. Code correctnes
      8. The tester can coach code correctness, but devs own it.
      9. Developers have to learn from their mistakes.
      10. Continuous improvement is about not being afraid of making mistakes but embracing them as a learning opportunity.
      11. Isn’t unit testing a safety net too?
      12. Yes it is. And it should be there. TESTERS aren't though.
      13. Safty net is a problem
      14. Example
      15. Brent test harness lead to developers stopped using design patterns and ended doing code reviews. Cause automation would catch mistakes. - it’s terrible cause sloppy architecture can’t scale and is hard to maintain.
      16. To remove safety net, you need also ensure minimal risk of getting harm and consequences of damage cannot be severe
      17. Continuous improvement
      18. getting better all the time
      19. Every time, *not* every half year, *not* every two weeks
      20. ALL THE TIME
      21. Actionable vs vanity
      22. Kaizen
      23. looking for improvement every time everywhere
      24. reducing the waste.
      25. Efficiency vs productivity
    2. WHY
      1. It’s a critical process change directly related to Quality
      2. Testers are well suited to drive continuous improvement
      3. Tester has to see the system as a whole because of this they can easily spot place for improvement
      4. Testers are specialists in asking questions
    3. HOW
      1. By measuring
      2. Visibility
      3. Unified engineering
      4. Specialists is a bottleneck!
      5. Lots of small changes
      6. Frequent retrospective (only 1-3 action)
      7. Testers are well suited for driving retros
      8. Also starfish model for long term retros
      9. A small vertical slices of architecture
      10. MVP!
      11. Master branch owned by the business they can release it whenever they want
8. Principle 4
  1. We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture.
    1. WHAT
      1. Quality culture and leadership
      2. Testers need to stop being passive
      3. Quality culture
      4. Quality
      5. is not Bugs
      6. is not code correctnes
      7. is user reaction to what you built
      8. What is a higher quality app with no bugs but with few users, or app with lots of bugs but 100k users?
      9. is about eliminating friction (enchanting positive, reducing negativities)
      10. Alan Defintion
      11. The shared mindset that delivering high-quality software to the customer is our top priority and all our practices support this effort
      12. Bren Definiton
      13. IT is how do we build healthy, actives, not intuition based, useful view of users empathy within the organisation, caring about problems that users have
      14. Getting people closer to understanding the problems that users are facing, how software developers are geared to solve it and how to stay current.
      15. Getting people closer to understanding the problems that users are facing, how software developers are geared to solve it and how to stay current.
      16. “Hey, customer are you happier ?”
      17. What it mature Quality culture
      18. It is maturity model use with caution
      19. its alan model - it doesn't has to work for you
      20. Testing breadth
      21. infancy example
      22. functional testing
      23. adulthood example
      24. contextual testing
      25. Quality test ownership ship
      26. infancy example
      27. Tester are doing all the testing
      28. adulthood example
      29. Testers are coaches
      30. No testeres needed, cause team gets it
      31. Bug life cycle
      32. infancy example
      33. bug backlog
      34. adulthood example
      35. zero bug policy
      36. how do you use code corretnes tools
      37. Data analysis
      38. infancy example
      39. oblivous
      40. adulthood example
      41. data centric
      42. Development approach
      43. infancy example
      44. ad-hoc
      45. adulthood example
      46. super small pathes
      47. Learn and improvement
      48. infancy example
      49. no learning happens
      50. retrospectives are for whining
      51. adulthood example
      52. every failure is oportunity to learn
      53. and oportunity is taken
      54. Leadership support
      55. infancy example
      56. adulthood example
      57. good quality pratices are praised suported and celebrated
      58. read book
      59. Book leadership on the line
      60. ADKAR Model
      61. source
      62. https://www.prosci.com/adkar
      63. Awareness
      64. Desire
      65. Knowledge
      66. Abilty
      67. Reinforcment
    2. WHY
      1. The reason is we want feedback from our customers more often
      2. we want to understand and react to what they are doing,
      3. we want to reduce risk in the ability to do that
      4. To work in the way we want to function, to keep, retain and engage more users we need the quality culture
      5. to move testing
      6. From Risk minimalisation
      7. to reacting to knowledge gained.
    3. HOW
      1. Unless the whole team has a common understanding of the problem trying to solve it is pointless
      2. Small Batches
      3. Knowledge sharing (software is knowledge works)
      4. Try stuff -> fail -> learn
      5. Experiment
      6. Don’t make sweeping changes at once
      7. Small changes, and don’t give up
      8. Don’t try to change all at once
      9. Reaping the band-aid works only when people are ready.
      10. Brent example on slack TL:DR verion
      11. my history: most time as QA, then as dev, now DS. I went to dev as a manager. Half of my dev were old school we dont test and half were ex Testers. 3 things I did to start: 0 bug backlog allowed , trained everyone on TDD , and lastly, bugs will not be fixed by the code author.
      12. After learning/rumbling period was over, the whole team mentioned they would not ever go back to old ways....
      13. bugs will NOT be fixed by the code author?
      14. is the social pressure... teammates do not appreciate having to fix a bug in code *you* should have tested.... motivates correct behavior in long term.
      15. Even better is to assign bug fixing to the peeps who approved the checkin
      16. What about toxicty and hostily
      17. combine with retrospective
      18. and a clear non-negotiable expectation that the team works as a team
      19. mitigateds the problem
      20. great way for knowledge sharing about code.
9. Principle 5
  1. We believe that the customer is the only one capable to judge and evaluate the quality of our product
    1. WHAT
      1. Definition customer - it is a human who takes a dependency on what you are releasing, and that human can make decisions what quality is
      2. It is not about preventing harm
      3. How do we help business to go faster
    2. WHY
      1. Customers don’t want software; they want the problem solved.
      2. It is seductive to believe “hey I am doing it forever I know what I am doing
      3. There is a risk in trust that QA belife in requirements doc and PMs , are solving a customer problem
      4. which is not often true
      5. Features are developed cause we think they are cool, not because It solves problems.
      6. Quite often asking teams
      7. Q: “What problem it solves? “
      8. lead to answer
      9. “They will use it”
      10. Q2:“What value does it bring? “
      11. “They will use it”
      12. Need vs want
      13. Usually, you need to decompose the want to Find the need
      14. B2B
      15. Even in B2B case delivering what customer want not what they need, may end with customer leaving and never returning
      16. Example of building wrong thing
      17. Microsoft Kin
      18. https://en.wikipedia.org/wiki/Microsoft_Kin
    3. HOW
      1. Ask your self
      2. Who is my customer ?
      3. What is the case when I am delivering software to business?
      4. Who is customer then? Buissness or its customer?
      5. Both
      6. You need to be able to help your customers help their customers.
      7. End users who benefit from software you are producing.
      8. Testers are not customers.
      9. Unless you are selling software for testers.
      10. Requirements from the business should be treated as hypothesis
      11. For each feature request you can ask question:
      12. “Imagine you had that what would you do with it?”
      13. Leaders ship is disappointing people at level they can absorb
      14. ergo
      15. Small changes!
10. Principle 6
  1. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
    1. WHAT
      1. Validating hypotheses doesn’t mean pushing buggy software.
      2. Your reaction time is important.
      3. we need to be able to know
      4. What metrics meter
      5. What part of your product
    2. WHY
      1. Data is critical to scale
      2. As long as you have good telemetry, you can find and fix bugs
      3. Relaying on telemetry and customer data doesn’t mean it is using the customer as testers
      4. Customer don’t care about test cases run, nor about a code coverage
      5. They care if theirs problem was solved
    3. HOW
      1. Instrument the hell out of your software
      2. Use automation as a load generator and let the telemetry be a test oracle.
      3. Data tells you where to focus.
      4. Check episode 82
      5. Once the product ship - look at the data and try to figure out which testing effort was wasted.
      6. Which feature is barely used?
      7. It will help you count What is the ROI of current testing
      8. Who values your product?
      9. tl:Dr data is important
11. Principle 7
  1. We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.
    1. WHAT
      1. Modern Tester doesn’t test
      2. It is also transitory role
      3. If you follow all principles you may not need testing specialists
      4. No testing isn’t dead
      5. but testers are on their way out
    2. WHY
      1. Testing is not a unique skill for testers others can do it too
      2. Testing is an activity, not a job.
    3. HOW
      1. There is a risk that Dev manager may not be able to deal with the mixed team - he will need a boiling frog
      2. You need people that genuinely understand testing to help with the transition.
      3. At the beginning you need experts to teach others how to test.
      4. Aka Bootstrapping effort
      5. There is social pressure to conform
      6. when you achieve quality culture
      7. it will be kept with minimal help, unless some one will actively try to change it
      8. If you are a manager start working with your people. and easing them into transition
      9. Give them a new place for their career to go
      10. And that new place is a positive one.
Historical information
1. Episode 77
  1. General
    1. Based on agile principles
      1. 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
      2. 2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
      3. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
      4. 4. Business people and developers must work together daily throughout the project.
      5. 5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
      6. 6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
      7. 7. Working software is the primary measure of progress.
      8. 8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
      9. 9. Continuous attention to technical excellence and good design enhances agility.
      10. 10. Simplicity--the art of maximizing the amount of work not done--is essential.
      11. 11. The best architectures, requirements, and designs emerge from self-organizing teams.
      12. 12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
    2. Who is customer for MT Principles?
    3. Modern Tester
      1. Existed in this version
      2. Tester as coach
      3. He is not Champion of quality
      4. Responsible for quality culture
    4. Principle importance
      1. Alan think all are equaly important
      2. Brent thinks principle 1 is most important
    5. Legacy current version doesn't have modern tester role per se
  2. Principlaces first version
    1. 1. The customer is the only one suitable to judge and evaluate the quality of our software
      1. it is combination of Testing Discipline , agile and Budisness as data
    2. 2. We apply the theory of constraints to identify, prioritize and mitigate bottlenecks from the system (and accelerate the team).
      1. Constant re-evaluate of bottle necks
      2. Testing often is a bottle neck but rarly is a root cause of one.
    3. 3. We are not the safety net for software correctness – we focus on improving the business, and not the code.
      1. Testing is not responsible for code correctness
      2. Software quality!=code correctness
    4. 4. We are responsible for the quality culture of our team, and are accountable for helping and coaching the team in this regard.
    5. 5. We use data to deeply understand customer usage and customer pain.
    6. 6. Embrace continuous improvement, and help the team adapt and optimize in order to solve customer pain
      1. neither It doesn’t need to be now, nor it doesn’t have to be perfect- balance
      2. its about ballance
    7. 7. We strive to reduce or eliminate the need for a dedicated testing specialist on our teams by increasing testing abilities and knowhow across the team.
  3. Legacy Principles
Discusion on Revison of MTP
1. Note those are drafts not actual changes
2. MTP
  1. 1
    1. Our priority is improving the business.
      1. no change
      2. Our priority is improving the business.
  2. 2
    1. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
      1. orginalny "we" was about testers - there are concerns if "we" includes developers. - NO CHANGE
      2. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
  3. 3
    1. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.
      1. Again issue with point being about testing
      2. the team -> our team
      3. rather than providing a safety net to catch failures. -> activly removing safety net to catch failures.
      4. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.
  4. 4
    1. We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture.
      1. the team -> each other (again issue we vs them)
      2. We care deeply about the quality culture of our team, and we coach, lead, and nurture each other towards a more mature quality culture.
      3. alternative proposition
      4. brent would swap principle 4 and 5
      5. We care deeply about the quality of our product, and we coach, lead, and nurture each other towards a more mature quality culture.
  5. 5
    1. We believe that the customer is the only one capable to judge and evaluate the quality of our product
      1. no change
      2. We believe that the customer is the only one capable to judge and evaluate the quality of our product
  6. 6
    1. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
      1. no change
      2. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
  7. 7
    1. We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.
      1. again issue we vs them - removed testing from pricniples
      2. We expand abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated specialist.
3. reason for changes
  1. Principles were writen with testers in mind it is time to evolve them for whole team
How it aplies to polish market?
1. See principle #5 - B2B
FAQ
1. Q: “These principles sound like Agile things. Why are they called out separately in MT principles ?“
  1. A: Agile is more fundamental than MT and addressees generic issues. MT principles are more focused on addressing testing issues
2. Q: Most of them sound more like managers' principles than testers
  1. There is no MT specialist but MT activities are part of unified development, MT practitioner can take PM hat from time to time if needed
    1. He is optimizer
    2. Reducing waste and increasing value
    3. Follower of MT notices patterns
    4. unified development defintion
      1. https://testastic.wordpress.com/2016/01/03/the-combined-engineering-software-model/
AB Testing episodes to listen
1. basics
  1. 77
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-77-the-conception-of-the-modern-testing-principles/
    2. general: who is customer of the principles?
  2. 78
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-78-digging-in-on-modern-testing-principles/
2. Principle 1
  1. 81
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-81-business-rulz/
3. Principle 2
  1. 83
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-83-accelerating-the-team/
4. Principle 3
  1. 84
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-84-stories-about-changes-and-nets/
5. Principle 4
  1. 85
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-85-the-community-principle/
6. Principle 5
  1. 86
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-86-not-the-customers-champion/
  2. 92
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-92-customers-bugs-and-triangles/
7. Principle 6
  1. 87
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-87-the-one-about-data/
  2. 82
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-82-brent-talks-to-alan-about-data/
8. principle 7
  1. 88
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-88-testing-isnt-dead-testers-otoh/
9. Modern testing vs Context driven
  1. 94
    1. https://www.angryweasel.com/ABTesting/ab-testing-episode-94-modern-testing-meets-context-driven-testing/
10. https://testastic.wordpress.com/2016/09/05/whats-so-special-about-specialists/
11. Transition guide
  1. https://www.angryweasel.com/ABTesting/ab-testing-episode-93-the-quality-culture-transition-guide/
12. slack
13. traditional vs modern debate
  1. https://www.angryweasel.com/ABTesting/ab-testing-episode-67/
14. https://www.ministryoftesting.com/dojo/lessons/experiences-in-modern-testing-alan-page
15. https://ministryoftesting.com/dojo/lessons/ask-me-anything-alan-page-modern-testing
16. Disusion on Revison of MTP
  1. episode 148
    1. https://anchor.fm/abtesting/episodes/Episode-148-Us--Them--and-the-new--MT-Principles-e18ijn0
PARKING LOT - Place for notes without home
1. in martial art black belt is begining up to that point you just learn what you need to start
2. Modern testing principles not modern testers principles
3. Specialists is bottleneck
4. ...automation especially UI black hole of payroll