-
Context
-
Mission / Drivers
-
Clients
- Kids
- Adults
- Old People
-
Problem it solves
- Seamless Entertainment
- Home Experience
- Multi platform experience
- Documentaries, Movies, Series
-
Multiple
- Genres
- Languages
-
Value
- Curated Content
- Effective Pricing Model
-
User Convenience
- No fixed show timings
-
Image of Product
- Reliable
- No Downtime
-
Premium Quality - Videos
- HD
- 4K
- Secure
-
Legal Obligations / Contracts
- Country wise restrictions for content
- Local laws retrictions
- Delayed in certain areas
-
Detailed Legal Information
- Click Here
-
Competitiors
-
Current
- Prime Video
- Hotstar
- Sony LIV, etc.
- Regional Competitors
-
Business Opportunity
- Rule the entertainment industry
-
Current Presence
- 190 Countries
- 167 Million Subscriber
-
Why now?
-
Rise of over the top entertainment
- Trigger: Covid
- Lot of uncurated content creating noise
-
Market Forces
- https://www.porteranalysis.com/porters-five-forces-of-netflix/
-
Usage
-
Personas
- Hacker
- Binge Watcher
- The Fast Forward User
- The Slow Speed User
- The Subtitle Only User
-
Normal User
- Happy Path
-
Quick Exit User
- Closes Netflix
- Comes Back
- Watches Little
- Loop
- Kids
-
Needs
-
Documentaries
- Knowledge Worthy Stuff
- History
-
Comedy
- Movies
- Shows
-
Series
- Binge
- Weekly Available
- One Time Binge
-
Movies
- Bollywood
- Hollywood, etc.
-
Kids
- Cartoon Shows
- Movies
- Multilingual
-
Situations
- Low Network
- Wifi
-
Mobile Network
- Different Providers
-
Fluctuating Network
- Wifi
- Mobile Network
- Airplane / No Network
-
Different Device Types
- Normal
- Lower Device Resolution
- Low end devices
- Space restrictions
- Memory restrictions
-
Operate via
- Touch
- Keyboard
- Remote
- Smart Watch
- Alexa Device / Google Home, etc.
-
Use
- Mobile App
- Smart TV
- Desktop
- Tablet
- Game Consoles
- Set-top boxes
- Blu-ray players
-
Misuse
- Record and Sell
- Piracy
-
By Mistake Events
- Selecting Wrong Subtitles
- Exit
- Entry to unintended / wrong profile
- Wrong Video Play
- Change password from another device
-
Malicious User Tasks
-
Hack
- Watch for Free
- Exploit User Data
- Leak Pre-release data
-
Pirate & Sell
- Ex: Youtube Live Stream
-
Clone Netflix UI
- Ex: Pirate Netflix from Tunnels
-
Usage Pattern
- One Account, One User
- One Account, Multiple User
- Recurring License, Inactive User
- Multiple Account, Single User
-
Factors affecting Testing
-
Location
- GPS Manipulation
- Emotional Analytics
- AI Suggestions
-
Technology
-
Language
- Java
-
Open Source Solutions
- MySQL
- Gluster
- Apache
- Tomcat
- Hive
- Cassandra
- Hadoop
-
Server
- Amazon EC2 S3
- Automation Strategy
-
Tools
-
Selenium
- Web
-
JMeter
- Performance
-
Appium
- Mobile
- TVs
-
TestNG
- Unit Testing
-
Team
-
Members
-
Apoorva Ram
- Test Specialist
-
Documentation
- Test Report
- Test Notes
- PCO
- Risk Report
- Traceability Matrix
-
Need to interact with
- Marketing Team
- Product Team
- Sales Team
- Development Team
- Product Management Team
- Support Team
- Video Production Team
-
Cultural Differences
- APAC, EMEA, America
-
Stake Holders
- Product Management Team
-
Ideas
- Profile for different people
- Security Risk while maintaining subscription
- Provision for removing recommendations
- To Do: Interview people / conduct surveys on likes & dislikes on netflix
-
Risks
-
Power Outage of Data Center
- Happened in 2012
-
HTSM
-
Project Environment
-
Mission
-
Customers
- Kids
- Adults
- Old People
-
Expectations
- Reliable
- No Downtime
- User Convenience
- No fixed show timings
- Multi Platform Support
- Pause / Play / FF / RW
- Premium Quality - Videos
- HD
- 4K
- Secure
- No Ads
-
Information
-
Consult them
- Marketing Team
- Product Team
- Sales Team
- Development Team
- Product Management Team
- Support Team
- Video Production Team
- Customers
- Likes
- Dislikes
-
Engineering Documents
- https://abhivaikar.github.io/howtheytest/#/
Refer Netflix Section
-
Customer Complaint History
- Support Logs
- App Reviews
- Play Store
- App Store
- Social Media Reviews
-
Source for new information
- Email Triggers
- New Requirements
- Customers
- Internet > News Feed > OTT
-
Comparable Products
- Prime Video
- Disney Hotstar
- Hulu Entertainment
-
Developer Relations
-
Feedback Loop
- Passing Through
- Standups
- Team Meetings
-
Feedback on Test Strategy
- Need to get this reviewed
-
Test Team
-
Strengths
- Exploratory Testing
- Automation Checking
- Mobile
- Web
- API
- DB
- Digital Testing
- Remote Team
-
Equipments & Tools
-
Network
- Wifi
- Mobile Data
- 3G
- 4G
- 5G
- LAN Network
-
Devices
- Mobile App
- Android
- iOS
- Different Resolutions
- Old Devices (Old OS)
- Smart TV
- Android
- Apple TV
- Others
- Desktop
- Linux
- Mac
- Windows
- Tablet
- iOS
- Android
- Game Consoles
- XBox
- PS4
- Set-top boxes
- Blu-ray players
-
Tools
- Selenium
- Web
- JMeter
- Performance
- Appium
- Mobile
- TVs
- TestNG
- Unit Testing
- JIRA
- TM4J
- Github
- Github Actions
-
Schedule
-
CI
- After each commit
- Nightly runs
- Merge Requests
- Weekly runs
- Exclusive tests
- Environment Upgrade Testing
- App Upgrade Testing
-
Documentation Testing
- 1 Week before Release
-
Test Items
-
Scope
- Unit Testing
- Integration Testing
- Component Testing
- Compatibility Testing
- Claims Testing
- App Size
- App Permissions
- App Footprint
- Scenario Testing
- Functional Testing
- Compliance Testing
- Regression Testing
- Performance Testing
- Security Testing
- Risk Testing
- UI Testing
- Visual Validations
- UX Testing
- Accessibility Testing
- Sanity Testing
-
Out of Scope
- Netflix Servers Testing
- Third Party Storage Clouds
- Payment Gateways
-
Volatility
- Yes: 15 Days - New Build
-
Retesting
- Bugs
- Change Requests
- Customer Defects
-
Testability
- Need automation ids
- Need access to DBs, APIs, etc.
- Need test accounts
- All possible models
- Use All Pairs Tool
-
Deliverable
- Test Reports
-
Recordings
- Video
- Image
-
JIRA
- Test Plans
- Test Cycles
-
Test Notes
- MCOASTER
- Risk Report
-
Release Notes
- With Apps
- Available via Web
- User Documentation
- Apps
-
Product Elements
-
Structure
-
Code Structure
- https://netflixtechblog.com/
- Proprietry
- House Brew
-
Hardware
- Devices running the software
- TV
- Smartphones
- Laptop
- Monitors Resolution
-
Non Executable Files
- Release Notes
- Help Section
- Pricing Section
-
Collateral
- App Store
- PlayStore
- Website
- All Include
- Legal Section
- Privacy
- Cookie preferences
- Terms of Use
-
Function
-
Application
- Playback Menu
- Pause / Play
- Speed
- ++
- --
- Subtitles
- Back Button
- Preview / Trailers
- Categories / Generes
- Documentaries
- Movies
- Series
- Netflix Original
- News & Popular
- User Recommendations
- Profiles
- Subscriptions
- Payments
- User Profile
-
Calculation
- Rating
- Top 10 List
- Billing Cycle
- Pending Movie Time
- Number of Episodes
- Progress Percentage
- Recommended Content
-
Time Related
- Time of Billing Cycle in UTC
- Actual Duration of Video vs Displayed Duration on Video Landing Page
-
Transformations
- Videos
- Different Quality
- Audio
- Language
- Subtitle
- Transcripts
- Home Screen Layout
- Region Wise
- Links
- Region Wise
-
StartUp / Shutdown
- Launch New Content
- Withdraw existing content
- New Account Created
- Account Locked
- Account Delete
- Subscription Plan
- Subscription Plan Expiry
- New Device Add
- Existng Device Remove
-
Multimedia
- Brightness
- Sound Levels
- Video Quality Levels
- Speed Levels
-
Error Handling
- Subscription Over
- Payment Failed
- Poor Network
- You are on Airplane Mode
- You are NOT 18+
- Your account is already logged in from another device.
- User Profile Details
- Form Validation
- Field Validation
- Payment Details
- Form Validation
- Field Validation
- Unsupported Device OS
- Unsupported Device Resolution
- Wrong Password
-
Interactions
- Video & Subtitle & Audio
- Multiple User - Simultaneous Login from Multi Devices
- Test Boundaries
- Test Exceeding Boundaries
- Browser Extensions
- Sync between website & app across platform retaining the logged in / out states
- App Space with Device Space
- Shared App Memory
- External Display & Sound Devices
- Control from
- Remote
- Mouse
- Keyboard
- Smart Devices
- Alexa
- Google Home
-
Testabililty
- Log Files
- DB Access
- Content
- User Data
- Activity Logs
- Locators: Automation Ids
- API Access
-
Failure Modes?
- What if Server goes down?
- What if JS is disabled on the client side?
- What if a power outage happens in master servers
- What if everyone logs in at a time?
- Load Limits?
- Performance Slowness?
- What if upcoming ios / android version restrict some accesses that we require?
- What if payment is done but is not getting reflected?
- What if some government laws ban netflix?
- What if an 18+ content gets mislisted in a kids account?
- What if someone's account gets hacked / locked?
- What if user profile is disabled?
- What if user's recovery email & password is also lost?
- What if the bank payment fails due to no balance?
-
Data
-
Input
- Network Speed
- Mouse
- Keyboard
- Extensions
- Smart Devices
- Remote
-
Output
- Video
- Audio
- Subtitiles
- Speed
- Error Messages
-
Preset
- Netflix Top 10
- Default Language
- Netflix Trending
- New Arrivals
- Theme - UI Mock Containers
- Legal
- T&C
- Help
- Support
-
Persistent
- User Profile
- User Preferences
- Payment Details
- My List
- History
- Progress %
-
Sequences / Combinations
- Genres
- Trending
- Top 10
- New Arrival
-
Cardinality
- Video Title
- Video Preview
- Description
- Number of Episodes / Seasons for a Series
- Cast
- Actor
- Producer
- Director
- Music
- Web URLs
-
Big / Little
- Restriction on size
- Video Size
- Preview Size
- Restriction on Length
- Video Length
- Title Length
- Subtitle Length
- Description
-
Noise
- Manipulated Cookie
- HTTP Access
- Certificate Expiry
- Existing Noise Data?
- Maybe due to non availability of restrictions in initial versions?
- Password change from other device
- Subscription cancelled / expiry / failed renewal
-
Lifecycle
- User Profile
- Created
- Accessed
- Modified
- Deleted
- Content
- Staging
- Release
- Modifications
- Delete
- Archive
-
Interface
-
User Interfaces
- Display Cards
- Buttons
- Voice Interaction
- Input Fields
-
System Interfaces
- Web
- Mobile Apps
- TV
-
API / SDK
- TV Device
- XBox Console
- Other Devices???
-
Import / Export
- User Data
-
Platform
-
External Hardware
- Servers
- Devices
- Peripherals
- Sound
- Microphone
- Remote
- Mouse
- Keyboard
-
External Software
- Browser
- Browser Versions > Min Supported Versions
- OS
- Android > Min Version (7)
- iOS > Min Version (13.0)
-
Internal Components
- Talk to Developer
-
Operations
-
Users
- Payment Type
- Plan Details
- User Roles
- Adminstrator
- Editor
- Viewer
- Compliance
- Service / Support
-
Environment
- Lighting Conditions > Brightness Update
-
Common Use
- Click
- Tap
- Remote
- Voice
- Different Personas
-
Disfavoured
- Piracy
- Legal Issues
- Factual Errors
-
Extreme Use
- Broadcast Netflix on another streaming platform
- Performance Testing
- Try multiple logins at a time point
-
Time
-
Input / Output
- Streaming Rate
- Sync
- Video
- Audio
- Subtitles
- Search
-
Fast / Slow
- Multiple Taps
- Monkey Tapping
-
Changing Rates
- Video Speed
- Max Limit
- Min Limit
- Network Speed
- Split screen
- Interrupt Scenario
- Call
- Swap between apps
- Network Switch
-
Concurrency
- Screen Mirroring
- Users
- Platform
- Devices
-
Actual Time
- Publishing Year
-
Quality Critera
-
Capability
- Video Streaming
- Flexibility
- User Interactions
-
Autotuning to conditions
- Network
- Display (Resolution)
- Brightness
-
Reliability
-
Servers are robust
- AWS S3
- High Availability
- Load Balancing
- Secure
- Restore States
-
Cloud Storage
- Multi Backups
- Sharding
-
Installability
- Minimum OS
- Minimum Browser Version
- Minimum Space
- Minimum Memory
-
Updates
- In Background
(Auto-Update)
- App Store
- Data storage?
- Settings storage?
-
OS Registry
- Installation
- Uninstallation
- Updates / Patch
-
Is the uninstall clean?
- Data Clean
- Re-install - Settings?
-
Scalability
- AWS S3
-
Performance
- Latency
- Page Load Test
- First colorful load
- GPU Overdraw testing
-
Development
- Ask Project Management
-
Usability
- Keyboard Only Usage
- Mouse Only Usage
- Voice + Tap Only Usage
- Tap only Usage
-
Charisma
- Web Design Follow
- Meet Accessibility Standards
-
Uniform Theme
- Throughout Pages
- Consistent Branding
-
Uniqueness
- HD Quality
- Latest Movies
-
Designed for easing User's common functions
- Skip Intro
- FF: 10 secs
- Reverse: 10 secs
- Compatibility
-
Security
-
Authentication
- Logged In
- Logged Out
- Hacker
-
Privacy Issues
- Informing about data safety.
- Lock User
- Password Recovery
- Account Recovery
- Payment Gateway
- Payment Details
-
Test Techiques
- Functional Testing
- Domain Testing
- Stress Testing
- Flow Testing
- Scenario Testing
- Claims Testing
- User Testing
- Risk Testing
- Automated Checking
- Exploratory Testing
-
Questions
- Kids profile provision?
- Possible Misuse?