From the search engine’s first beta offering to the now-famous Gmail beta label that looked like it was never going to come off, Google has always been a disciple and pioneer of beta testing. And ever since Google released Play Store’s beta testing features back in 2013, they have been constantly iterating and improving on them. Today, Google Play’s beta testing capabilities have come a long way from their initial release.
This is not a step-by-step guide to uploading your app, adding testers, and releasing your first beta. In this post, we are going to take an overview of Google Play’s beta testing prowess and how you can use it to set up effective beta tests for your app. We will also examine the platform’s limitations and list a few tips and tricks to make the most out of your beta test.
Google Play’s Take on Beta Testing
To avoid confusion among testers and developers alike, Google Play tries to adopt a simplified approach to beta testing. It abandons the traditional “alpha” and “beta” test convention in favor of internal, closed, and open “testing tracks”.
This seems to be an effort to resolve the argument over the difference between alpha and beta apps, and whether open beta tests were “real” beta tests. The approach is quite straightforward and does work well, but it does introduce a few restrictions that you might need to work around.
Internal Test Track
- Limited to 100 testers
- Add testers with their Gmail or G-suit e-mail
- No beta review required
- No payment is required for paid apps
The internal test track gives you a lot of flexibility to iterate as you please, but it does limit you to 100 internal testers. This might sound generous compared to TestFlight’s 25 internal tester limit, but unlike iOS, Android is an open platform that does offer alternatives.
Regardless, this track is meant for QA testing and internal testing or your dogfooding program. The app builds you upload to the internal testing track avoid the delay of beta reviews, allowing you to push nightly builds to your team. Additionally, internal testers will not be charged if your app is paid, and are excluded from country restrictions.
If the number of your beta testers is small enough, and you want them to access your beta test for free, you will be able to use the internal track with external testers. However, this is a poor solution that doesn’t scale at all, and you’re better off using promo codes, which are limited to a more manageable 500 codes/app/quarter.
Closed Test Track
- No tester limit
- Add testers through Gmail, G-suite, Google Groups, or Google+ Communities
- No limit on the number of concurrent tests
Technically, the closed testing track allows unlimited testers only if you add testers through Google Groups or Google+ Communities. E-mails are restricted to a total of 200 lists and 50 lists per “track” (read: test), with each list, limited to 2000 testers. However, this amounts to a maximum of 100,000 testers per track, which is far from being a deal-breaker for all practical purposes.
While the closed test track scales well and can support a huge number of testers, closed beta tests are usually a much smaller and more targeted affair. It is usually best to start with a small number of technical or trusted users and gradually add more testers as you iterate. In this case, those 200 lists can be put to the much better use of segmenting your testers, or even A/B testing different variations of your app.
The trade-off here is that your initial build and any builds that introduce major changes will be subject to a beta review process before they are published. Fortunately, Google Play’s beta review process only takes a few hours and should be easily accommodated.
Open Test Track
- Your app is discoverable on Play Store
- You can set a maximum number of testers (minimum 1000)
Judging by its implementation of open beta tests, Google Play seems to take the “open” in “open test track” very seriously. By surfacing your app on the Play Store, Google puts your app in front of more than two billion users. It is the only beta app distribution tool to allow you to put your beta on the official store.
Google Play also enables you to limit the maximum number of allowed testers to avoid overwhelming your infrastructure. Another approach is to use country targeting to limit the availability of your beta app to a smaller market that is easier to handle.
Even if you have done extensive closed beta testing, an open beta test—especially when on the official Play Store—will give you a better idea of how your app will fare on launch. Since beta tester feedback is not public, it can pay to run a quick open beta to gauge the users’ reaction to your app.
Pros and Cons of Google Play Beta Testing
- Beta testing features are integrated into the developer console and add minimal overhead to your process.
- Testers don’t need to enable download from unknown sources in the phone settings. This becomes less important the more trust you have with your users.
- Any feedback submitted by your beta testers remains private and does not affect your app’s rating or reviews.
- Allows country targeting for both closed and open beta test releases.
- Supports staged rollouts for closed and open tests to minimize the impact of any issues on your testers.
- Enables A/B testing for Play Store listings to optimize conversion and increase downloads.
- Supports pre-registration campaigns to gather signups before the beta test is ready.
- Only one version of your app can be in an open test at any time.
- If your app is paid, both closed and open testers will have to purchase the beta app to gain access.
- Does not provide a feedback channel with testers except for private reviews on open beta tests.
- For apps with a live version, the beta testing opt-in prompt is not easily discoverable.
- APK version numbering must follow internal > closed > open > production. For example, if the app version of your closed beta is higher than that for the open beta, they will both receive the build with the higher number.
Tips for Beta Testing With Google Play
- Keep in mind that, for closed tracks, only the first one is eligible for country targeting.
- Consider soft-launching your app in a country that resembles your main market.
- Add testers to your license testers list if you want them to test in-app purchases for free.
- You can use promo codes to give your paid app to testers for free, but you can also give away promo codes for in-app purchases to incentivize testers.
- Provide a dedicated channel for your testers to submit their feedback and communicate with you.
- Use in-app analytics to gather data about your users and their behavior within the app.
- Remember that beta testing can be used for much more than bugs and crashes. Involve other teams in your beta test to maximize your benefit.
How Can Instabug Help?
Instabug is the top beta testing tool for bug reporting and user feedback in mobile apps. It provides the most useful metadata on the market, exceptional customer support, and an in-app communication channel to chat with your beta testers.
Bug and Crash Reporting
With each report, you automatically receive comprehensive data to help fix issues faster, including steps to reproduce errors, network request and console logs, and environment details. For bug reporting, your beta testers can also send screen recordings and annotatable screenshots to provide further context.
In-App Surveys and Feature Request Management
Collect user feedback from your beta testers right inside your app to minimize interruptions and boost participation rates. Get powerful insights to enhance your product roadmap with surveys that you can target to specific tester segments and feature request voting to understand user pain points and desires.
- What to ask your beta testers to get better feedback
- Comparison of the top beta app distribution tools
- What beta test legal agreements do you need for your app?
- Instabug blog category: Beta testing