QA in parallel with development, from planning to launch
As Jira-abiding QA citizens, our role is to ensure that gameplay mechanics are engaging, intuitive, and enjoyable for all players. To accomplish this, it's important to be involved in every step of the software development cycle.
In this article, I'll walk through a typical QA test cycle for player-facing features, following the journey from development through launch. I'll also explore how happy path and unhappy path testing, along with low-order and high-order testing, build confidence in a feature before it reaches players.
The complete timeline
Design > Development
Typically a kick-off meeting occurs where design communicates the new feature by outlining it's goals, mechanics, and scope.
[QA] Test Planning
Begin analyzing requirements, anticipate issues, give feedback, and writing test cases.
Playable > Testing
Dev has completed an early implementation of the new feature in a build that is ready for QA.
[QA] Test Execution
First functional pass, limited scope. Focus is on promptly identifying issues and quick turnaround of bug fixes.
[QA] Test Reporting
Provide reporting on testing status, progress, and known issues.
Feature Complete > Hardening
- Feature testing is complete
- All major bugs are fixed or at minimum entered and tracked
- The feature branch has been merged into the main branch
- Fewer and fewer commits are allowed to allow the build to stabilize
[QA] Regression & RC Testing
A Full Regression pass is conducted on the Release Candidate build.
The purpose of the Regression pass is to ensure the new features are fully functional and do not break any of the existing features.
Release Ready > Launch
The RC candidate has passed regression and is deployed to the live production environment where players will access the new feature for the first time.
[QA] Post-Launch Verification
Production environment pass is performed to ensure the live environment matches the dev environment.
Test Analyst micro view
QA responsibilities from kick-off to release
Start of milestone
-
- Read and internalize the Design Doc for your owned feature
- Create a comprehensive Test Plan
- Map out and outline the entire feature scope
- Begin with core functions before moving to smaller ones
- Ensure test cases cover both happy and unhappy paths
- Collaborate with Dev to review and approve the Test Plan
Midpoint
-
- Execute test cycles across all happy and unhappy paths incorporating
- Low-order and high-order testing
- Negative testing and edge case testing
- Rinse & Repeat test cycle loop (Testing > Bug Reporting > Verifying) until feature is fully functional, stable, and performant
- Execute test cycles across all happy and unhappy paths incorporating
End of milestone
-
- Update test cases as needed and conduct regression passes
- Perform Release Candidate (RC) verification to ensure all critical issues are resolved and quality standards are met
- Conduct Post-Launch verification and provide ongoing support in the live production environment
Two modes of testing: Economy vs. Performance
While executing these milestone responsibilities, a Test Analyst's effectiveness comes down to choosing the right cognitive mode for each task. To test efficiently, we must balance between two primary modes of operation: low-order execution and high-order exploration.
Low-order execution & the happy path
Low-order testing operates on economy mode. Think structured checklists, maximum efficiency, and baseline stability.
When writing test cases, we commonly begin with mapping out the happy path, or expected user behavior, first. In this early stage, edge cases largely remain unknown unless they are immediately obvious. With that said, when executing foundational test cases, it is of utmost importance to remain diligent in order to test effectively.
Areas that are known to be stable with relatively few historic issues would be areas where you operate in low-order testing.
High-order exploration & the unhappy path
High-order testing operates on performance mode. Think analytical problem-solving, systematic exploration, and deep creative thinking.
The unhappy path is like exploring new, uncharted territory. We can approach it by utilizing methods like negative testing and edge case exploration. Edge cases are any issues that occur when a user strays from the normal flow or "happy path". It's essential to test the unhappy path thoroughly since real players don't always follow the happy path. This is typically high-order testing because it requires you to think outside of the box.
Negative testing and edge cases are similar concepts in that they both relate to unexpected user behavior. The difference is that negative testing is typically designed and accounted for. By conducting negative testing, we ensure that a system fails and recovers gracefully (e.g. proper error handling). On the other hand, edge cases by definition are difficult to predict; they are issues that arise from rare, unusual behaviors.
If a system or feature is known to be fickle or unstable we can employ high-order testing to deliberately push it to its breaking point.
Take a look at this example
A basic test case may have you verify a button.
Low-order testing would simply suggest you to tap the button and make sure it triggers a correct action as an expected behavior.
High-order testing urges you dig deeper:
-
- How does this button function when scrolling away?
- What about if I change tab views?
- What happens if there are overlapping elements?
- Let me try tapping around the button edges
- Tapping two buttons simultaneously
- Tapping and changing tab views in quick succession
While these are essentially edge cases, a highly reproducible issue that leads to a stuck state quickly escalates into a major blocker. At that point, the potential negative impact on the player experience needs to be carefully weighed and prioritized. It is our job to uncover these issues before they reach the player so that risk can be properly assessed.
A great negative testing example is being able to change the URL in an in-game browser. That not only breaks the normal, expected user flow but exposes a critical security vulnerability.
The path to quality
In summary, the happy path is mapped during the test planning phase as the normal, expected actions a user would take, while the test execution phase is where we'll chart the unhappy path through negative testing and edge case exploration. Additionally, understanding the balance between low-order and high-order testing allows us to allocate our efforts efficiently. Exploring both the predictable happy path and the uncharted unhappy path ultimately creates a robust framework for delivering quality.
