1. How did the CBT help us?
   1. App under real user conditions
      1. interaction with real third party services
   2. Understanding the importance of the functionality to achieve the business goals of the clients
      1. initially, some of these scenarios were considered as non-priority
      2. understanding of real user business scenarios
      3. feedback from customers: how to improve our product
   3. The first client bugs
      1. unique user platforms and configurations
      2. initial orientation to a specific client group and a platform related to business promises
   4. Check UX Hypothesis
      1. UX gaps and omiisions
         1. corrected / improved a number of scenarios / functions
      2. creating a new customer experience
      3. change generally accepted
   5. Built a process of interaction with the Help Desk service
      1. Employees training
      2. Solving the real users problems
   6. Team became ready for release
      1. It is IMPORTANT when Teams of professionals work on the project, united by a common goal and understanding of the product
         1. product development team
         2. project managers
         3. team of promoters
         4. support team
   7. Understanding what customers want now from CBT
      1. fully working product with a part of unimplemented functionality that does not affect on the main business scenarios
         1. disagreement with the work of the functions / partial non-implementation is resulted in negative feedback, up to the decision of refusal of the product
   8. We are ready for release!
2. How we could improve testing
   1. Adaptation and improvement of regression testing scenarios
   2. Expanded the test data
      1. accounts
         1. according to updated user groups
      2. platforms
      3. new data types
   3. Update cards for HD
3. The main problems we faced
   1. The unwillingness of the Team to simultaneously develop new functionality and lead customer support
      1. creation of regulations
   2. Unexpected using of the system
      1. new scenarios
   3. Difficult localization of client bugs related to local cryptographic problems
   4. Legacy client platforms
      1. small screens, outdated OS, crypto-providers
   5. Inconsistency of received business requirements and reality
      1. the expected types of clients / client environments and reality
   6. Interaction with product managers and promoters
   7. Not all clients participating in the PTA were loyal
4. How CBT affected on new iterations
   1. T2M increased
      1. It was necessary to perform work both on product support, problem solving of real customers, and to continue work on creating new product functionality.
         1. The scope of iterations on the development of new functionality has not been truly reduced.
   2. High loaded prerelease / release days
      1. Regression systems + work on client bugs, some of which required hot-fixes Team loss of focus
   3. Team defocused
5. If CBT would start now, then
   1. follow the recipe for beta testing: https://www.xmind.net/m/JvrB/
      1. metrics system
         1. the creation of complex goals in Ya.Metrika
      2. alerting system
      3. feedback form
      4. testing on real data
      5. security issues