Regression Testing
- The regression is run before any major release and is comprised of three main stages:
- Sanity Test: These high-priority tests are designed to identify obvious or critical errors that could render a build non-viable.
- Exploratory testing: This utilises a testers expertise and innovation while rapidly testing features in combination to provide a statement of quality.
- Regression Script: These are on TestRails that confirm functionality.
Note
We can use any device(Android or iOS) to perform regression testing for the test cases present on TestRail. However, it is necessary to test payment and web-view related scenarios on both devices for all regions.
To facilitate payment testing on the production environment, we require assistance from Barkha Gupta and Xurmira Farro for the EU region, while Jamie will provide support for the UK.
What build to test
- Ideally, We should run regression testing on the production environment to minimize configuration issues and closely reflect the actual release. However, if that is not feasible, we can resort to using pre-production builds.
iOS
Ask developers to add your email address to Testflight groups, production builds will be distributed through there.
Android
Ask developers to add your email to the test group or ask for a link to join the group for the relevant build, the build can then be downloaded through google play when it propagates, it can be obtained earlier through the development console.
Regression sizing
- We will dictate how much effort a regression will be by taking on the following variables:
- Amount of change, time and risk introduced since last release.
- Timescales and deadlines
- Confidence in the change and stability of the build (not quantifiable)
Examples
| Change Introduced | Type of Change | Devices(iOS + Android) | OS(iOS + Android) |
|---|---|---|---|
| 3+ months since last release | Fundamental changes to payments | Full regression performed on popular phone & 2 other screen sizes in popular device list have exploratory testing carried out | Full Regression on most popular OS version & 2nd and 3rd most popular OS version have exploratory testing carried out |
| ~2 months since last release | New feature within the UI (example: Message centre) | Critical tests performed on 2 devices & Exploratory on 2 more popular devices | Critical tests on most popular OS & Next most popular OS has exploratory testing |
| ~1 month since last release | UI improvements and/or refactoring | Critical tests performed on popular phone | Critical tests performed on a popular OS version & 1 more popular OS version has exploratory testing carried out |