How to choose devices for mobile app testing?

Testing is an integral part of the mobile app development process. During the testing process, you make sure that the app works equally well on all selected devices.

But how do you choose devices for testing, considering how many available options are out there? Read below to learn SoftTeco’s process of selecting devices for mobile testing.

Narrow the scope

The first step would be narrowing down the scope and coming up with a list of the devices that you are 100% sure you want to use. Let’s see how we create this list step by step.

Collect the requirements 

It may happen that you have zero ideas about the devices that you want to use – or the situation may be the opposite. In fact, many clients already have certain devices in mind when they come to a software development company. So step number one would be to collect all the requirements for the product and see if there is any device (or devices) that you 100% want to include. Maybe you want to cover the iOS market only: in this case, the situation is really easy. Or you may want to cover specific Android devices – think about it when talking with a business analytic or a project manager.

Identify the market share for different platforms

The next step is to analyze the market and your potential users in order to identify the market share for different platforms. 

Every country will have a different market share for different devices and platforms and your task is to identify this share. For that, you can use the available statistics from trusted sources.  In this way, you will know what devices your target audience is most interested in and you will be able to target the users more efficiently. 

Establish testing goals and constraints

One more aspect of testing to keep in mind includes your testing goals and possible constraints. Let’s see how it relates to the device choice.

Testing goals usually include:

• Usability testing
• Performance testing
• Security testing
• Network connectivity
• Functionality testing
• Compatibility testing

It is obligatory to define your testing goals in advance so you can base your further work on them. And once you define the goals and prioritize them, you can assess how many devices you will need for testing.

This is how it works. Say, you have a mobile banking application. In this case, the security of the app will be your top priority. Thus, you will need just a few devices to test the app on since the app’s security does not depend much on the device’s hardware/firmware.

However, if the performance of the app is your top priority, things will be different. Since the performance of the app will differ drastically based on the device’s hardware, it will be important to test the app on as many devices as possible.

As for the constraints,  the biggest and the most common one would be budget. If you want to test the app on many devices, it would take much more time, and hence, it would result in a big number of working hours for the QAs. So any budget limitations will directly affect the number of devices that you can test on. If this is the case, we recommend going with the most popular devices for your target audience and get back to the second picks later (if needed).

Consider using simulators or emulators

Simulators and emulators are used when you, for some reason, do not have access to the real device. You can think of them as virtual testing devices that mimic the software and the behavior of a device on your PC. For Android you use emulators and for iOS it will be simulators, correspondingly.

Simulators and emulators are a suitable alternative in case you have a limited budget. However, we do not recommend using them because the testing results on emulators and simulators are less accurate than the ones on real devices. As well, real devices have a much higher processing speed and are much more reliable.

Identify parameters for choosing the devices

Now that you know what devices you are interested in, it’s time to identify the parameters for choosing them.

Identify the needed OS versions

With Apple products and their unified hardware and software, things are relatively simple. If you decide that you want your app to run on an Apple product, you normally consider the latest and the second latest versions to support.

With Android, things get more complicated. In 2020, the most popular Android OS versions were:

As you see, there are not one or two but six popular versions and each of them must be considered in accordance with your target audience and the desired market to serve.

So how do you decide which OS versions do you need? For that, you need to identify the following:

• Your target audience – who will be using your app?
• The countries of residence of your potential users.
• The share of different OS versions in these countries.

By identifying who your users are and what devices they use the most you will clearly see which OS versions you need to focus on.

Choose the needed screen sizes and resolutions

If you want your app to seamlessly run on different devices, you need to ensure that its UI is compatible with different screen sizes and resolutions.

Things are relatively simple with Apple products. Since they have a somewhat unified UI, you will only need to consider a few devices aka different versions of iPhones and iPads. 

With Android, things get complicated. Due to the big variety of available devices with all sorts of screen sizes and resolutions, we highly recommend checking the statistics to see users’ devices’ DPIs. Such data is vital when choosing the devices for testing as it reflects the preferences of real people who will be using your app.

Consider the device’s hardware

It may happen that an application requires specific hardware in order to function properly. In this case, you will need to choose specific devices that can provide such hardware to be tested.

To illustrate this point, we’ll use the RoadLab project by SoftTeco as an example. The product is used to assess the quality of the road surface on the International Roughness Index (IRI) based on the gyroscope and accelerometer data. Hence, when testing the RoadLab application, the SoftTeco QA team needed to test it on the devices with a built-in gyroscope.

When you develop a mobile application with specific requirements for the device’s hardware, it is obligatory to test it on real devices that have this type of hardware. 

What also relates to considering the device’s hardware is the processor architecture. There are three Android processor architectures: ARM, ARM64, x86. When you test your app, you need to ensure that it works equally well on all processor architectures.

Consider the device’s software

Some apps may require the devices to have specific built-in features: support of biometrics, 5G, etc. If your app is built around these features (or at least uses them on a regular basis), it is a must to find the devices with the needed functionality. Otherwise, it may turn out that the app does not perform as intended and you might be in trouble.

Create a device coverage matrix

After you’ve created a list of the required devices to use during testing, it’s time to prioritize them. For that, you will need to create a device coverage matrix.

There are multiple ways to create this matrix: you can categorize and sort the devices by popularity, dpi, platform version, screen size, etc. A step-by-step process would be the following:

1. Create tables for the OS of choice and include all important parameters to consider.  If we take Android as an example, table 1 may include the OS versions (Pie, Marshmallow, KitKat) and their market distribution by percentage. The most popular dpi with their market distribution will go to table 2, correspondingly.

Table 1

Table 2

Combine both tables into one. That means the vertical row will have the most commonly used OS versions and the horizontal row will have the most common dpi. 

And in the empty cells, you will place the devices that correspond to both the version and the dpi requirements. 

Once the table is filled, you can create your matrix. There are usually three tiers: 

• Tier 1: most popular devices that support the most popular OS version and the most preferred dpi. 
• Tier 2: less important devices that still can be tested (or used in regression or sanity testing).
• Tier 3: devices that you don’t need to test (you can though).

By using this matrix, you can easily see the device (or devices) that are your top priority. In this way, you can efficiently distribute your testing time and resources.

Other things to consider

The steps listed above are the most common and well-known methods to identify the devices for mobile testing. However, there are a few minor things that are still worth considering as they may significantly impact the choice of devices.

Remember about particular qualities of different brands

You’ve probably heard of the Huawei Google ban that happened back in 2019. The ban implies that Google cuts Huawei’s license from their digital products, meaning that Huawei does not have access to such popular apps as Gmail, Google Drive, and even YouTube.

But what does it have to do with testing? In fact, quite a lot. Huawei does not use Google API and has its own instead. And obviously, this has to be considered when testing a Huawei device.

This is an example of a peculiarity that a device may have. Though seemingly minor, such things have a big impact on the app’s performance and thus must be remembered and tested.

The annual release of new OS versions

Both Apple and Google release a new OS version every year and both the development and testing team should be ready for that. The new versions usually come with open and closed betas of the new versions and the brands announce the release at the beginning of summer. So if you have an approximate release date for your app, you might consider aligning it with the new OS version releases.

While the testing of new iOS versions usually goes smoothly, there is a limited choice of devices for testing when it comes to Android beta testing. They normally include Google Pixel, Samsung, Xiaomi, and Nokia smartphones so the QA team should keep that in mind.

Summing up

We can go on and on about the choice of devices for mobile testing. But in the end, it all comes down to two main points: the budget and market share. You want to make sure that you select the devices that are most popular with your target audience but you also want to adhere to your budget and not go over the top too much.

However, we highly recommend not to cut the costs of testing since it’s vital for further success and distribution of an application. Even the smallest malfunction or error can significantly impact users’ satisfaction and lead to the app uninstallation. And this is the complete opposite of what you aim for, isn’t it?