Keeping everyone in the loop on test progress and overall stability
Test management happens at all levels, spanning from the Junior and Mid-level Analysts up to Test Leads and Managers. Every team member plays their role with their own unique scope. Each milestone is a team effort to decode the test matrix to provide a clear, real-time, snapshot of a release's overall quality before it goes live.
Throughout the QA lifecycle, three core questions help keep quality in focus:
-
- How is testing going?
- What are we not catching?
- Where should we allocate resources next?
Answering them starts with two fundamental metrics: Bug Severity and Bug Priority. Let's look at how QA and Production determine these metrics and use them to guide what happens next.
Bug Severity vs. Bug Priority: Whats the difference?
Bug Severity: Impact (QA)
When writing bugs, every good analyst will form an instinct of how severe the issue is. In order to determine the severity we consider how negatively this issue impacts the player experience.
- Blocker / Critical: Crashes, completely unresponsive elements, or blocks player progression
- High: Major functional bugs that break a feature or disrupt flow
- Medium: Moderate bugs that cause minor disruption but don't impair core functionality
- Low: Cosmetic flaws or minor text errors that have zero impact on functionality but impact surface-level polish
Bug Priority: Urgency (Production)
Our chosen severity will help inform Production and Development leads of how to prioritize the bugs in order to get them addressed.
Priorities are as classified:
- P1 (Blocker): Urgent, must fix issues that block testing or crash the build.
- P2 (High): - Important defects that must be fixed before release, but not an "all-hands-on-deck" emergency
- P3 (Medium): Standard bugs that sit in the middle; they should be addressed but not urgently
- P4 (Low): Nice-to-have fixes that the team will address only if time permits
- P5 (Backlog): The lowest priority tier, add it straight to the backlog
Additionally, any bug that is a clear P1/Blocker (e.g. crashes or blockers affecting multiple parties) requires immediate escalation to the proper team channels for remediation.
Now that we understand how both Bug Severity and Bug Priority are determined, we can start connecting the pieces of test management. Let's explore the three core questions that help bring the bigger picture into focus.
1. How is testing going?
Test reporting
When testing is in progress, it's vital for Senior Analysts to keep track of all issues that have been written up and compile them into a daily or weekly report. This snapshot gives stakeholders and devs a clear glimpse of testing progress, open issues, and the overall stability of a feature.
Daily QA Progress Report Template:
[Feature Name] QA Progress Report
QA Status: In Testing/Testing Complete
Bugs Found:
Numbered list, sorted by highest priority first
Include bug status: Open/In Progress/In QA/Fixed
P1/Blockers are called out in bold, red text
Note: You can use emojis to help denote bug severity/priority.
Test coverage & tracking
At a higher level, Test Leads will utilize this reporting data to update and maintain a Test Coverage Matrix. This is a high-level document that outlines each milestone feature and tracks its testing status. Staying on top of test coverage allows leads to make data-driven decisions regarding scheduling, risk assessment, and test gap analysis at any given moment.
| Feature | Owner | QA Readiness | Execution | Defects | Overall Status |
| PvP 2.0 | T. Chau | Ready to Test | 18/40 (45%) | 2 | At Risk |
| Dungeons | J. Snow | In Testing | 36/40 (90%) | 1 | In Progress |
| Matchmaking | Y. Camp | Blocked | 12/30 (40%) | 4 | Blocked |
The matrix provides a high-level view of testing progress and risk. For detailed information, the feature name and number of defects can be linked to the individual test plan and issue filter respectively.
2. What are we not catching?
Test gap analysis
By comparing test results against your original test plan or design requirements, you can spot where potential coverage gaps exist. These blind spots are often easy to miss when teams only validate the happy path, miss a corresponding test case for a requirement, or overlook complex interactions between interconnected systems. A test plan may reach 100% execution while still leaving significant coverage gaps.
Pinpointing these gaps allows QA teams to redirect testing resources toward high-risk or unmapped areas of the build. Rather than treating every test scenario equally, risk helps prioritize where additional coverage is most valuable. Risk acts as the compass, helping QA identify which gaps could have the greatest potential impact on quality and where additional testing is most likely to reduce that risk.
The goal isn't always 100% test coverage. It's knowing where your coverage ends and what risks exist beyond it. By understanding what has and hasn't been tested, QA can make informed decisions about where to focus testing effort next.
3. Where should we allocate resources next?
Risk assessment
Risk assessment is the method of identifying and analyzing potential issues that could damage release quality. By analyzing risk, QA can flood the highest risk areas of a feature with testing efforts. For example, if a specific system was notoriously bug-ridden during early development, it may be smart to assign more eyes to it.
To calculate risk, consider balancing two factors:
- Likelihood: Evaluates feature complexity and historical stability because defects tend to cluster in previously problematic code.
- Impact: Focuses on business-critical factors and player retention. Examples include payment issues, security vulnerabilities, or hard progression blockers.
Once you have proper insights into high-risk areas or systems, you can develop mitigation strategies to prioritize resources accordingly. Managing risk is a continuous process that evolves alongside every new feature and code change.
The bigger picture
Quality is a collective responsibility and everyone plays their part. Junior Analysts detect and determine their immediate severity, while Production and Development Leads prioritize them for resolution. Senior Analysts provide consistent reporting to keep stakeholders in the loop of test status and known issues. Meanwhile, Test Leads track overall coverage by maintaining a test coverage matrix, collaborating with Test Managers to balance testing resources utilizing methods like test gap analysis and risk assessment. In the end, everyone owns quality at their respective level and the matrix brings those pieces together into the bigger picture.
