ArticlesBeta Testingtesting metrics

When you’re developing an app and you have limited time and resources, the beta testing phase is usually a tough one. You want your app to be as perfect as possible and executed in the most efficient way to release it as fast as possible. Beta testing is usually a tricky part of the development cycle. You can spend a lot of money and get all the beta testers in the world that you can, but at the end of the day, this will all be completely useless and redundant if your beta test was a failure.

How do you know if your beta test was successful? One of the first and most crucial steps of the testing process is defining your testing metrics — your standards of measurements based on your goals for the test. Whether it’s a private beta, where the main focus is usually on usability and quality assurance, or a public beta, which is more marketing-focused in terms of awareness and feedback regarding points like pricing and features, beta testing varies from one mobile app to another.

Below are key testing metrics to consider while setting your testing goals.



1. Stability


beta test metrics


The basic goal of any beta test for a mobile app is to simply make sure that it’s working as it should be. Not only do you need to make sure that all the features are working perfectly, but that the app is easy to use. This is your chance to identify all the problems and their causes, including bugs, crashes, timeouts, and any bottlenecks disrupting the user experience at any point.

That’s why the first metric you need to track is the number of bugs in your app. But that’s not enough. Not all bugs are created equal, so you need to categorize those bugs into critical, high-level, and low-level ones so that you have a basic understanding of where your app stands. Obviously, you need that number to be as low as possible, if not zero. You would also need to track the evolution of those bugs over time, and how that number changes through subsequent versions of your app.


beta test metrics


However, as much as you need your app to be perfect, you shouldn’t obsess over it for too long. A 2015 survey by Evans Data conducted on more than 500 active mobile app developers showed that almost three-quarters of mobile apps are released with an average of one to ten bugs. It’s okay to release your app with a few unnoticed bugs, but you need to be on top of your game and constantly update it and fix any problems found. Another study by Equation Reach found that only 16% of users will try a failing app more than twice. So the number of bugs in your app should diminish to zero in subsequent versions.


“To have just a small handful of bugs is really quite an achievement, but the real advancement is in the constant updating of apps to fix bugs or add additional features. This virtually continuous release of software has changed the dynamic of how developers create and maintain their software.”

– Janel Garven, CEO of Evans Data



2. Churn Rate


Back in the day, companies used to focus on user acquisition as the most important mobile app metric. Everyone focused on the number of app installs, but didn’t quite pay attention to the behavior of those users. As of May 2016, a study showed that 23% of users abandon an app after just one use, and the average retention rate for apps is around 20% after only three months. So it’s pointless to focus on building a user base if you won’t be able to keep them.


app user retention


Your beta testers are a sample of your target users, so you should try to keep your testers engaged over time. Keep in mind that your beta testers are early adopters of your app and in most cases are excited to try out something new that they have early access to before it’s open to the public. So expect your users to have lower retention rates than your testers, and always aim for higher retention rates for your beta testers.

Joel Spolsky, creator of project management software Trello, suggests that the minimum number of testers you need is about 100 testers. These 100, however, need to be your reliable and dedicated testers who will provide you with really detailed and informative feedback. If you happen to have a larger team or multiple beta managers, it would be best to have 100 serious testers for every one beta manager.

In terms of churn, Spolsky also says that on average, only one out of every five testers will stick around and provide you with the kind of feedback you want, which means you’ll need to get 500 testers to finally reach those 100 quality testers. Also, don’t expect all of your testers to participate each time you release a new build. One workaround for this is to split your testers into multiple groups for every version. Of course, if you have a high quality app and select testers who are as close to your likely end users as possible, they should be more likely to stick around.



3. Meaningful Participation


in app feedback


Did you know that on average only one out of every five beta testers provide feedback? So you need to make it count. Having quality beta testers is key for your app development, but how could you possibly measure that?

One way is to track the amount of quality, actionable feedback that you receive compared to your overall feedback. Most testers will leave you generic statements like, “This didn’t work”, or even positive feedback like, “I loved it”, but these don’t really give you much insight.

In many cases, beta testers are regular end-users who are not trained on how to properly test your app or write useful detailed reports. They’re just trying out the app as any normal person would, which is why many developers end up with feedback that is unconstructive, which defeats the purpose of the test.

There are many ways to push for quality feedback with your testers, such as by rewarding them and acknowledging their work. This helps motivate them, increases their sense of ownership, and pushes them to put more time and effort into your app. Another way is to implement efficient feedback systems like in-app tools, which can help you get up to 750% more feedback from your beta testers.

These also open an immediate channel of communication between you and your testers. You can take advantage of that by always trying to actively respond to your testers, and don’t be afraid to ask them for clarification or more details or to simply just thank them, which would motivate them to participate further.