This week, we launched a four part series that covers simple steps you can take to improve your apps and make your release cycles faster. Our advice is based on real-world challenges and rewards from Niels Frydenholm, Senior iOS Developer at eBay Classified, who spoke at Xamarin Evolve 2014.
Today we’ll look at how to expand your testing program beyond simulators to real devices.
Would you fly with a pilot who had only used flight simulators?
“Don’t trust only emulators and simulators. You have to run on real devices, as well.”
Simulators provide quick, useful feedback, which can be an important component of efforts such as eBay’s commit tests – short tests run by developers on their own code, designed to validate basic functionality. Still, simulators are only an approximation of deployment environments, running on very different, much more powerful hardware. As such, they can mask a number of real problems while indicating others that simply do not exist. For example, several Xamarin Test Cloud customers have noted false returns from simulators in performance-dependent areas such as timing and graphic rendering.
It’s also important to remember that simulators are software applications, and they may carry their own inconsistencies and bugs. eBay noticed that a number of its intermittent, non-reproducible test failures were actually driven by problems with their simulators. The company was ultimately able to code around these failures by automatically rerunning failed tests to eliminate false negatives, but the differences between a true live device and a software simulation are significant.
Ultimately, your app will be deployed on real-world devices, each with its own set of design quirks, limitations, and bugs, and the only way to guarantee performance and quality is to test against real-world targeted platforms. This is not unique to mobile, as QA departments have been testing against a range of supported systems since the beginning of software development. What is unique to mobile is the vast device and OS fragmentation that exists, making on-device testing far more critical while also more difficult to achieve.
How to Test on Real Devices
- Don’t manage it all yourself. eBay noted the overhead of testing on even their relatively small number of target iOS platforms. “We don’t really have time to install and test on that many devices,” said Niels, so they chose to test on real devices in Xamarin Test Cloud. He also notes that for developers supporting even a handful of Android platforms, this fragmentation is far more complicated.
- Have your tests in order. Before you deploy tests to Xamarin’s device cloud, make sure you actually have a stable test suite on your local machine.
- Results reporting matters. Testing on a large number of devices will create a large volume of results. eBay receives detailed reporting from nightly test runs that feed directly into its CI system. The team has access to detailed test data, including screenshots and video from different devices that can help in pinpoint the exact reasons for failures.
Just as in aviation, there are real-world conditions for mobile apps that cannot be simulated and that are only be surfaced and dealt with by experience in real environments. Xamarin Test Cloud makes that easy.