Using metrics in testing to speed up the release of a digital product

Metrics in testing - a waste of time or an objective indicator of the quality and speed of development? When are they applied and what is given to a digital product? Together with the head of the Fusion Tech QA department, we will try to understand these issues and find out which metrics and how affect the development process.

QA metrics are tools that help in measuring the quality of a software product and in finding problem areas during project implementation.

WHEN TO USE METRICS?

The need to implement and apply metrics should be determined by each individual team on a particular project. To understand whether this tool is needed and which one, decide on the result that you want to get in the end. Often in articles on the topic of testing, you can see similar phrases about metrics: “control the QA process”, “give objective indicators of product readiness”, “show the effectiveness of the team and the individual employee”, and so on. One might get the impression that the metric is a universal silver bullet for all problems, which, perhaps, only does not kill vampires and werewolves. But is it really so?

The purpose of the metric is to increase the speed of project implementation at all stages, streamlining the development process and reducing the number and duration of bug fixes.

Imagine the following situation: in the previous month, the percentage of project returns (from developer to tester and vice versa) fluctuated around 3%, and in this month it increased to 15%. You might think that the development team began to work 5 times worse and it's time to get the dusty whip from the closet. But in real life, it may turn out that in the previous month there were simple layout tasks, and in the next month a component with complex logic is being formed, or the requirements for tasks are not fully described. This is an example that the end result should be considered based on the analysis of metric data in order to understand where the drawdown occurred and why.

HOW DO METRICS HELP THE PROJECT?

Metrics allow you to notice deviations of the system from the intended result in terms of such parameters as speed, quality, productivity of the digital product’s development, and connect in time to fix problems. The use of qualitative and quantitative indicators in software testing and development in general reduces the release rate of the product and guarantees its viability.

Goals for collecting metrics:

Analysis of the system operation (assessment of a set of metrics, how indicators change over time, monitoring of new processes / ideas).

Search for problem areas:

• accumulation of tasks awaiting testing;

• the number of tasks in the testing process;

• the time spent on the regression;

• the number of bugs found and closed;

• the number of rejected bugs;

• average time to fix bugs.

Analysis of the found problems.

WHAT PROBLEMS CAN METRICS LOOK AT?

Low throughput*

* Throughput is the number of tasks that go through the development team in a certain period of time from the requirements stage to the release of the product.

Let's say a developer is able to solve 10 problems in 2 weeks, then the testing team should have the ability and resources to process them. If QA is not enough, and one employee can only test 3 tasks out of 10, then the release throughput will be 3. And the opposite situation - if QA is able to test 10 tasks, and the developer can close only 3, then the efficiency (efficiency factor) will remain also at mark 3. As in the army on a forced march: the time for which the unit ran the distance will be recorded by the last soldier, that is, the unit (as a single system) as a whole will run at the speed of the slowest soldier.

In such a situation, metrics help us fix the fact that we have a “jam” somewhere or, conversely, have an excess resource, and as a result, come to a systemic balance: we receive 10 tasks within the sprint (1 development cycle) and 10 tasks arrive at the end in production.

A large number of task returns

Returns of the project from the tester to the developer due to clarification of requirements and found errors can be one of the reasons for insufficient throughput. What happens if QA returns to the previous stage (product creation) too many tasks with bugs? To say the least, the load on the testing department will increase in the future, since in addition to new tasks, you will have to look at redone ones.

Let's take a case from our practice: on a small project, a QA engineer stopped coping with the load due to a large number of tasks. Another one was added to the project, but this eased the situation for a while. Soon the problem returned, “jams” appeared again, as already two QAs were diligently submitting even more tasks, which, after correction, were returned for testing. It became clear that we should think about optimizing processes, and not about expanding the staff.

Metrics highlight the problem in the early stages, allow you to understand what is going wrong, and quickly get involved in the implementation of the project in order to solve it as soon as possible. Without them, the team misses precious time and instead of developing new features, only the fixing of identified bugs occurs.

Return loop is too long

Returns to previous steps must also be considered in terms of how long the loop is. That is, at which stage of development the problem was discovered. If we have identified bugs at the stage of requirements formation, then fixing a couple of lines of text in the brief will cost less than redoing the code and re-testing (short return loop). Finding a problem in the business logic after development, on the other hand, makes the return loop too long. It will first lead us to a business analyst to fix the requirements, and then to a systems analyst and further along the chain until we again reach the testing stage.

Similar to the previous point, the shorter the return loop, the less time it takes to fix defects, and the cheaper it is to develop a product.

Lack of balance between quality and throughput

For better or worse, we live in a world where the market decides everything. You can spend a lot of time on developing a product that is perfect to the pixel, but do not forget that competitors are on the alert. While you spend time perfecting everything, they release products with new features. Yes, their projects may have minor flaws, but they are already the first ones, the attention of the target audience is riveted to their development. Therefore, getting carried away with excessive testing is an unaffordable luxury.

In this case, metrics help to allocate and plan time for tests more efficiently for the release of new features without compromising quality.

WHAT ARE THE METRICS?

On the Internet, you can find an impressive number of different metrics. Each team uses their own depending on the project, upcoming tasks, and previous experience. The selection of the metrics necessary for work is usually carried out at the stage of testing planning and control.

Tools that are more often used in our company:

• “Queue” metric: the time of accumulation of tasks waiting for testing.

Helps to plan resources and prevent or clear project congestion.

• Metric “Density of defects”: number of defects in the module / total number of defects.

Helps to understand what requires the most attention during development.

• Metric “Reopened bugs”: number of reopened bugs / total number of bugs.

Helps to identify gaps in QA and add additional test scenarios.

• Metric “Average lifetime of a defect”: time spent on fixing defects / number of defects.

Helps in planning releases (we can roughly understand how long it will take to fix defects), as well as in assessing how effectively the development team responds to bugs.

• Metric “Number of bugs found in production”: number of defects after release / number of defects in iteration.

Allows you to understand how stable the product is after the release.

BRIEFLY ABOUT THE MAIN:

Metrics are useful, but only an additional tool. They can highlight the problem, fix where something went wrong, but they will not point directly to the cause.

The selected metrics should be brought to a single form, so that it is easier to focus on past indicators and track future progress.

For the same purpose (monitoring the effectiveness of actions), it is necessary to analyze metrics once a sprint, a month, and so on.

You should not use this tool to evaluate the performance of specific people or implement metrics for the sake of metrics without asking the question: “What do I want to measure, and how will this help me in my work?”

Try to choose those metrics that will be most useful for the project, or trust a team of professionals who will test the product at all stages of development and bring it to release faster.