Creating a risk-based Quality Assurance Framework

Before implementing a QA framework, you must identify and understand the potential risks in your environment. Risk is “the level of uncertainty in archiving your organisation’s objectives”. All organisations face internal and external factors and influence that lead to different levels of uncertainty, as shown in Figure 1. 

Applying a standardised risk management process identifies the effect of uncertainty on an organisation. Risk management techniques are used across the entire organisation, specific business functions, business and IT projects, and daily operational activities. 

Here are a few key areas that should be considered in your risk assessment: 

1. What are your internal stakeholders’ needs and concerns? Ask the question, “What keeps you awake at night?” Each stakeholder will apply a different lens across their area of the organisation due to their specific role and responsibilities. These other lenses provide a holistic understanding of your organisation’s risk context. 
2. What are the essential product and delivery risks in your organisation and industry? Each industry sector may be subject to government legislation and regulations or comply with specific standards, e.g., medical devices.
3. What risks do your products or services present to your customers that could affect the quality of your products or services or have life-threatening outcomes? 

Implementing ISO/IEEE 9001 Quality Management Systems (QMS) with a supporting Quality Assurance (QA) framework, such as ISO/IEEE 29119 – International Software Testing Standard, provides a business and technology mitigation strategy that works with your IT systems delivery. 

Developing your QA processes to address the identified risks can be daunting and is frequently project-specific and only sometimes organisationally driven. One way to get started is to leverage the effort already captured in ISO/IEEE 29119 – International Software Testing Standard. The standard provides a high-level framework for what processes must be considered and defined. It provides a standardised software testing vocabulary and details about how to “do” software testing.   

Like many activities related to testing, how much of this framework you employ will depend on the level of risk you need to mitigate. Figure 2 describes the hierarchical relationship between the organisation’s test policy and the primary components of the software testing standard. 

Whether you have an organisational test policy, a master test plan, or simply a test strategy, these artifacts MUST align with the corporate vision, mission, goals, and objectives. Most organisations have a variety of policies to make explicit, specific expectations of their employees and governance under which employees and the organisation’s operations must comply. 

An organisational test policy is a formal way of aligning quality assurance with all the essential business policies and critical business processes, such as IT project management and governance frameworks. The primary objective for defining a policy is to mitigate significant business risks. An example of such a policy is one that highlights the need for employee vigilance when using company IT assets. The policy also specifies what usage of these assets is appropriate.

The international software testing standard can be contextually framed to support agile, hybrid, or waterfall delivery methodologies. It assists with identifying the guidelines for testing, monitoring, and reporting issues and then aligning roles and responsibilities for different team members involved in the QA process that need to perform the process activities.  

The test policy defines the organisation’s Quality Assurance objectives, which then directs the specific project test objectives contextualised to the project’s needs when creating the test strategy.  

The critical activity of defining a test strategy is the thinking behind what actions we must complete to ensure that we have mitigated the quality risks presented by the project’s scope. Quality Control (QC) activities are defined, which fall into either verification or validation, as shown in Figure 3. 

The strategy defines the QC activities we want to apply based on the project’s risk profile. The cost of fixing defects late in development will determine the importance you place on prevention-style QC activities versus using only detection QC activities. The latter generally being performed late in the project life cycle, predominantly as testing activities. 

Placing controls on the process to monitor the identified risks are crucial in driving where in the delivery model we need to add activities to ensure that we are addressing, monitoring, and controlling the risks the organisation cares about. These controls could include the following: 

1. Implementing automated testing activities across multiple areas of the delivery model, e.g., Improving unit and integration testing by implementing CI/CD practices. 
2. Utilise static code analysis tools to detect “code smell” earlier in the development process, which can introduce security vulnerabilities into the system. 
3. Improve the quality control activities used for requirements review and walk-throughs. 
4. Tracking issues, typically defects, and broadening the types of information the process collects to provide insights into the process’s efficiency and effectiveness, e.g., understanding the root cause of the defects enables the better deployment of resources into areas that prevent defects instead the depending on only detecting them. 

Training your staff on the QA processes and controls ensures they understand the importance of following the established procedures and, most importantly, what the impact can be if the processes are not performed. Skilled staff can make better decisions because:

• They understand the risk implications the decision may have.
• They are empowered to identify opportunities for significant improvement.

For continuous improvement to be meaningful, the correct metrics must be defined to monitor the QA processes. The metrics must also support the delivery of the project to ensure they effectively manage the identified risks.  

A meaningful metric enables you to monitor your definitions of success which are typically linked to one or more of the following: 

1. Critical Success Factors (CSF) 
2. Key Performance Indicators (KPI)  
3. Agile Objectives and Key Results (OKR)  

A metric is a “force for good” as much as a “force for evil”. Any measurement will result in a behaviour change, some of which will be counterproductive. 

The objective of a metric is to measure the process, not the person, to assess the effectiveness of your QA framework and identify areas that need improvement. Every QA process must embody continuous improvement.   

Analysing the data collected can identify areas for improvement, such as understanding what areas of the delivery process introduce the most errors.  The feedback from customers and stakeholders using retrospectives allows your key working relationship partners and stakeholders to identify what went well and what they think could be improved.  

Compliance in some industry sectors is significantly more critical than in others, e.g., medical device and gaming software development activities have more stringent compliance requirements and are subject to more scrutiny by regulators. 

Basing your QA framework on an international standard can ensure that your QA activities comply with the relevant industry standards and regulations that govern your industry.  

Auditing is the most common compliance activity, but finding non-compliance during an audit may be reputationally damaging. Early detection of non-compliance, small or big, is much easy to address early than when highlighted by an external party. If you are subject to external audits, you would be well served by regularly performing internal audits and assessing your QA processes.  

With the constant churn of employee and organisation restructures, it doesn’t take long before all previous QA processes are a mere memory of ‘something we used to have’! We then spend a significant chunk of time recreating what has been lost.  

Adopting an international standard as the basis of your QA Framework ensures that risk mitigation and increasing the level of delivered quality while improving delivery efficiency are at the front of your mind. The standard provides: 

1. An industry-recognised approach to implementing software testing. 
2. It only needs tuning and continual refinement to fit your delivery context, i.e., delivery methodology and risk profile.  
3. It allows you to continually assess the QA framework’s key activities and deliverables against a published international standard. 

More importantly, no more ‘reinventing the wheel’!

If you think you may need to implement a QA framework in a risk-aware environment, reach out to the luvo testing team for guidance or support via [email protected]