How to choose the right devices for mobile testing

When it comes to mobile testing, selecting the ideal devices is very dependent on your requirements. The functionality, user interfaces, performance, and other factors will vary based on the device itself so trying to choose the most appropriate devices can be challenging.

Testing mobile devices still require the application of functional, non-functionality, and real-world user testing. This article will discuss how to select devices for your mobile testing and some of the information you need to consider when deciding what devices to target.

Testing every single device in the market is impractical due to limited time and cost constraints. Therefore, picking the most suitable mobile devices for testing will ensure you deliver a well-tested application that covers the primary devices used by your customers.

In this article, we provide some basic guidelines about how to select the most appropriate mobile devices to test your application.

Mobile device testing coverage needs to consider the following areas:
• Your target users – what devices do your target audience use the most
• Your current customers – what devices are your existing customers using
• The region the software is expected to operate within

The use of application usage statistics and analytics quantifies what type of device configurations are accessing your applications. By considering the following points, along with your analytics data, you will be able to make a more accurate decision when it comes to choosing the devices.

It is a must to figure out which devices are preferred among your target audiences. Choosing devices based on their popularity in your targeted geographical areas and audience is the best place to start.

Which OS you target will be driven by the application scope, i.e. will the application need to run on IOS and Android? Test your software on several operating systems and versions to verify that it works on various platforms. Android and IOS update their statistics to the latest version in the documentation to show the usage of each OS.

For example, Android has many brands like Samsung, Nexus, etc. You can prepare an application testing device matrix to determine which devices to use to gain the most benefits in terms of the operating system, version, and brand.

It’s vital to evaluate the UI design and responsiveness across different screen sizes, resolutions, and device families like tablets and iPhones.

The properties and characteristics of the devices will vary depending on the make and model, so testing must consider the hardware characteristics of screen size, PPI, resolution, and the amount of disk space and data your mobile application uses.

The computational power of the different mobile devices also impacts your software, e.g. the CPU design in the latest iPhone performs significantly faster than a model from three years ago. If your software states it supports older mobile device models, your application should operate as effectively on these older models as it does on newer devices.

The devices you use to perform your application testing against are also influenced by how you approach the testing execution. Manual testing is time-consuming and has a one-to-one ratio between a tester and a physical mobile device. The amount of testing you can complete then is a product of the number of testers and the coverage of the “must-test” devices.

Automated testing is not limited by physical people using the devices; therefore, the number of devices you use with automation is usually constrained by the cost of the physical devices. The use of automation can increase the number of mobile devices you can cover in your testing as the approach has significantly more scalability. Adding device emulators to your automated strategy can increase your automated testing coverage.

Preparing a test plan that defines your required mobile device configurations based on operating systems, their versions, and device models. The device configurations will be determined by your current user’s device statistics, taking into consideration any new targeted audiences and their preferred mobile devices.

A basic approach for picking mobile devices for manual testing is to take the top 5 devices that access your application or website based on usage statistics and analytics. The same consideration can be applied to automated testing, but your device coverage should consist of more devices with various OS and hardware configurations. This is where device farms and emulators help increase your device testing coverage.

The market for mobile devices is constantly changing. So, it’s vital to consider the past and future releases and the release frequency when planning the testing coverage. It would help if you re-evaluated your testing strategy every quarter or twice yearly to add new devices as they become prominent while lowering the importance of (or eliminating) devices that are losing favour. Several websites can help compare mobile phones to identify different specifications of phones.

It’s essential to consider your competitor’s applications and their supported devices in the market. Based on the findings, you should also ensure that your application supports those devices. It helps capture more users as well as become a challenging competitor. Most users don’t think twice about uninstalling an application that doesn’t perform as expected.

Emulators are used in the early stages of the development process as you don’t need a physical device to emulate real devices with different OS and hardware specifications. This approach is the application of the “shift-left” quality strategy, moving key testing activities to earlier in the development process. Emulators allow the development and QA Engineers to test your application across multiple device configurations without the need to buy the physical devices.

A mobile application emulator mimics the device on your PC or Mac, allowing you to evaluate mobile applications quickly during development activities.

A device farm is a hosted testing service that allows you to test and engage with your iOS, Android, and web apps on actual, physical devices and different web browsers. The use of device farms enables you to test across more devices, without needing the own the physical mobile device. Device farms allow you to perform automated application testing, utilising a range of testing frameworks or using manual approaches that load, execute, and communicate with applications on a real-time basis on devices using remote connections.

Some of the pros and cons of using these virtual devices for mobile application testing are as follows:

• Extensive tests need to be done during the early stages of development. Therefore virtual devices are the best option to cater to this need rather than buying multiple devices for each developer, especially if it’s a large firm.
• Emulators can virtualise a wide range of devices and operating systems. It lets users quickly validate numerous platforms, including circumstances requiring unique device/OS combinations.
• Virtual devices allow you to always start from the same device state. Yet, actual devices may require a factory reset, which can be more time and effort-consuming.

• While virtual devices are helpful for performing fast functional verifications and speeding up automation, they do not replace the need to perform real-world testing on a physical device.
• An emulator will not enable the detection of usability issues. These types of errors can only be found when testing on an actual device.
• There are some testing scenarios that are difficult to virtualise. For instance, the interaction with physical devices and sensors, like cameras and biometrics devices.

A test plan covering the current customer base and integrating automated and manual testing is critical for effective mobile application testing. The test plan must include the type of device configurations you need to cover based on your application and website’s current usage data. If you are looking to increase your application’s market share, consideration of the devices that are supported by other competitor applications should be targeted.

Adopting “shift-left” quality assurance activities through the use of emulators in the development cycles and in test automation approaches provides “fast-feedback” loops earlier in development and testing. While “shift-left” can detect errors early, it will not detect all classes of errors that a mobile application can experience. It is critical that you test your application on the targeted physical mobile device configurations to ensure the end-user experience.

Applying these considerations during the planning stages will ensure that your quality assurance and testing approach is thorough and cost-effective testing.

If you are about to embark on a mobile testing project or need help deciding which tests to select, reach out to the luvo testing team via [email protected]