Specificity of Mobile App Testing: Things to Keep in Mind
Since a big part of our lives now revolves around mobile devices (interaction with people, food ordering, entertainment, and even business), mobile users have become highly sensitive to the performance of the apps they use. In case of a minor bug or malfunction, most users will uninstall the app and switch to the app’s competitors. Hence, mobile app testing is now as important as ever.
It comes with many challenges though. Because mobile devices are quite intricate, there is a lot to test: Bluetooth connections, built-in gyroscopes, how the app reacts to the orientation change, and many other things. In this article, we tried to cover the biggest “smallest” things to consider when performing 360-degree testing of a mobile application.
Mobile and non-mobile testing: the main differences
Mobile app testing is very different from testing other platforms simply because of the conditions under which a mobile app performs. If we take web testing, for example, you will have to check your app for browser and platform compatibility - but with mobile, the number of possible OS + platform combinations is much bigger. Let’s have a look at other major differences between non-mobile and mobile that are to be considered before planning your testing. For easier understanding, we will be comparing mobile app testing to website testing.
Capacity and storage
While some flagship models are becoming as powerful as PCs and laptops in terms of RAM, for most mobile devices, their capacity and storage are still very limited. Therefore, an app’s performance directly depends on the phone’s RAM, and cumbersome, feature-packed apps may not perform as intended. Another important thing to note is that a cross-platform app will perform differently on web and mobile devices (which some developers forget to consider).
Connection and connectivity
Web apps can’t work in an offline mode while mobile apps (well, some of them) can. Hence, when performing mobile app testing, you are not only testing online/offline modes but also different connectivity types (Wi-Fi, 3G, 4G, 5G), the performance of an app in case of lost connection, and what features are available in an offline mode.
Variety of platforms and devices
One of the distinctive features of mobile devices is their vast availability of OS versions, platforms, and devices. Sure, web apps have to be compatible with a number of platforms, but this number can hardly be compared to a number of possible mobile platforms and devices.
While it’s relatively easy for iOS apps (since Apple promotes the unity of functionality and design), Android is a whole different story. The latest version of Android OS is Android 12 but there are many devices out there that support 11, 10, or even earlier versions. All that has to be considered when testing a mobile application.
Users interact with mobile apps much more regularly than with web apps and all forms of interaction have to be kept in mind. Whether it’s multitasking, performing in-app operations (i.e. payments), using biometrics, or interacting with UI elements, every process has to be thoroughly tested. This is more complex than testing user interactions with web apps and requires slightly more attention and focus.
The similarities between web and mobile app testing: main testing types
It’s worth noting that the main testing types are the same for both web and mobile. They include:
- Compatibility testing
- Security testing
- Performance testing
- Load and stress testing
- Functional testing
- Internationalization and localization testing
- Usability testing
Note though that every testing type (both for mobile and web) will have a set of specific things to test and monitor and there can be dozens of them. These things will also depend on whether an app interacts with smart devices, what kind of access it needs, and so on. This leads us to particular qualities of mobile devices to consider.
Special aspects of mobile devices to understand and remember
When you design an application for a mobile device, you need to remember that its performance can be impacted by things that are common for mobile devices. An example would be a phone call in the middle of your next CandyCrush level. In this case, you need to think about how an app would react to such an interruption. Below we listed the biggest aspects to pay attention to when talking about mobile.
Apart from all their wonderful capabilities, smartphones are phones and phones tend to make and receive calls. Hence, it’s a common thing that you can receive a phone call unexpectedly - including moments when you browse an app like Pinterest or Tiktok.
Now, the worst-case scenario would be an app crashing but we don’t want that, do we? QA engineers and developers have to make sure an app pauses (i.e. pauses a video on TikTok) and does not shut down when you receive a phone call. This is one of the primary things to consider during testing.
Another feature that’s common in most smartphones is automatic screen rotation. To be honest, we don’t expect it from a PC or a laptop but we do expect some apps to automatically adapt to a screen if we twist and turn it.
An app should continue working if the screen rotates and it should automatically roll back to its default size once we turn the screen back. This brings us to another challenge: the app should not lose in quality and performance when expanding.
A critical aspect for most smartphone owners is the battery consumption of their devices. It’s a well-known fact that heavy and feature-loaded apps make the battery drain much faster than it normally would.
The battery consumption depends on multiple factors. For instance, most apps these days use geopositioning to provide users with accurate or real-time data. However, the use of geopositioning equals increased battery consumption. Therefore, you need to test whether an app needs to use geopositioning and what permissions a user can configure for the app (use geopositioning once, all the time, or only when an app is running).
Another thing to test is how the app works in the background (i.e. sends and receives the data) and how that impacts the battery life. To test the battery consumption of an app, QA engineers first check the battery status before testing, then perform various actions within the app (i.e. location services enablement, data sync launch), and see how these actions impact the battery of the device.
Variety of screen sizes and resolutions
If you build an app for iOS, things remain relatively simple - but Android development is a whole new story. There is a huge variety of possible screen sizes and resolutions so before testing, you will need to list down all screen dimensions and resolutions that the app will support. You then test against those criteria to make sure your app performs equally well on all needed devices. This puts additional effort into testing as you will need to test an app on all intended devices and watch for the design to remain consistent.
Bluetooth and NFC interactions with other devices
One more important thing to remember about mobile devices is that they often use Bluetooth (or NFC) to interact with other devices. So if you design an app that’s intended to manage a third-party device via Bluetooth, you’ll need to test how well the interaction goes and what kind of issues might arise along the way.
Testing applications on mobile devices is a complex process that requires special knowledge, usage of numerous instruments, and understanding of different tricky aspects. Sometimes it is not possible to check some nuances even having a bunch of the devices in your hand and simulating some circumstances.
As an example, when we tested the integration flow between IVR and SIM-card dependent phone numbers of the customer, we had to use the person in that country, as otherwise, we could not test the full flow. Such things occur from time to time in international mobile projects.
Alex, Co-founder at SoftTeco
The intricacies of mobile app testing
While it’s common knowledge to test an app for security, network connectivity, and usability, sometimes developers tend to forget the small things that make a big difference. Let’s see the most common examples below.
Developers overlook the permissions settings surprisingly frequently and that may lead to certain issues with the app use. Let us elaborate on this one.
A mobile application in general, be it iOS or Android, functions in a limited-access sandbox. So when an app needs the resources of your mobile device, it has to make a request aka “ask for permission”. Sure, you’ve seen it multiple times: you open a new app and it asks “Provide access to storage?” This is a great example of permissions settings.
Now, what can go wrong here at the mobile app testing stage? Simple: QA engineers may not test all combinations of permissions. As a result, the app will not be able to function properly as it won’t be able to take a crucial step. For example, if you are working with a messenger and test all its file-sharing functions, you might miss the access to the storage set to “no”. Resulting in the app not being able to perform to its fullest potential while all its functions seem to be working.
Another good example of permissions testing is from SoftTeco’s own testing experience. We once tested a native app and we opened a Web View in it. In general, a Web View is supposed to have access to the device’s camera and files but the app’s permissions were not set correctly. So the app crashed the moment we tried to launch the Web View.
An app can store its files in internal storage, external storage, or cloud storage. So when downloading a new file, an app needs to be told where to store it. This is another area of mobile app testing that often gets overlooked.
There are several aspects of storage testing to consider:
- How will an app act if the storage is full?
- Where does the app store its files and how does it access them?
- What are the right permissions for accessing app storage?
- What are security measures needed to keep the files safe?
All these things are highly important as they directly impact the security of the sensitive information that an app processes.
Modern mobile apps make the best use of all features of a mobile device, including its sensors. An example would be playing Pokemon Go or racing games - by rotating the device, you get access to the special features of these apps, such as geolocation and different dimensions.
An average smartphone has the following sensors which can be used by third-party apps:
- Proximity Sensor
- Light Sensor.
While most of them are not critical for the functioning of an app, some of them are really important and need to be tested properly. It may also happen that an app is based on one or several sensors and if a sensor does not work properly, the whole app will shut down.
An example is an application developed by SoftTeco. The product is used by road engineers to calculate road roughness on the International Roughness Index (IRI) and it uses an accelerometer and gyroscope heavily. Needless to say, without the proper functioning of these sensors, an app would be useless as it won’t be able to perform its core functions.
A separate mobile app testing type that needs to be mentioned is testing the app’s performance when the device’s memory is full. In this case, the system will provide a limited amount of memory to the app and this might lead to certain issues, like time-out errors or app crashes. A QA engineer must make sure an app does not stop functioning if the device’s memory is full and he should test all the possible options of data processing in such a case.
It often happens that QA engineers test an app at medium or high traffic speeds, calling it real-time conditions. Indeed, in reality, the app will most probably work with high traffic - but what if it slows down all of a sudden?
One of the most frequent mistakes that QA engineers make is forgetting to test the app at low speed, simulating slow traffic. This is needed to see how exactly the app will perform in such conditions and what possible issues might occur. Sometimes, an app performs great in normal conditions but displays horrible performance if the traffic slows down. This is unacceptable and should be taken into consideration during testing.
Mobile app testing can be rather challenging since there are so many intricacies to keep in mind. When testing a mobile app, we recommend taking your time and creating a detailed checklist that outlines the process. This will help ensure that nothing is missed and that the app won’t crash after an unexpected action from a user. have any questions left, leave a comment below - we’ll gladly answer them.
SoftTecoView all articles by this author.