This week, we launched a four part series that covers simple steps you can take to improve your apps and make your release cycles faster. Our advice is based on real-world challenges and rewards from Niels Frydenholm, Senior iOS Developer at eBay Classified, who spoke at Xamarin Evolve 2014.
Today, we’ll look at how integrating automated tests with your release cycles can cut the total time to release by 50% or more.
Optimizing Release Cycles
“By the end of the year, we were releasing on all supported platforms monthly.”
With multiple developers working in overlapping areas of code, eBay uses Jenkins to manage a variety of automated tests to ensure that its CI efforts run smoothly without slowing down the development process. By automating not just test execution, but also test setup, eBay removed the overhead and potential for error from test management, streamlining their entire development process.
As a result, eBay doubled the number of supported app platforms and pushed more than twice as many releases in 2014 as they did at the start of 2013.
Tips for Getting Started
Niels offered three tips for optimizing releases:
- Optimize for speed. To minimize negative impact, isolate developers from as much testing overhead as possible, and provide rapid feedback after each change, eBay uses different test suites for different jobs.
- Ten minute commit tests: eBay’s commit tests during the day are limited to a roughly ten minute subset of the full test suite. This provides an adequate level of assurance to the developer without becoming disruptive.
- Daily full suite: At the end of the day, they run a full one-hour test suite to ensure complete coverage.
- Trim the fat. Niels suggested writing around UI-based overhead for test set-up and tear-down. For example, if a UI test is designed to verify that a logged-in user can modify data through a form, the process of creating the user account, logging in, and populating the initial data set can often be automated through a faster, script-based approach without invalidating the test itself, as long as the scripted functions are tested in full elsewhere.
- Automate job creation. eBay creates branches for new features under development in order to keep its master branch clean for an App Store upload. Unfortunately, this creates a number of opportunities for human error. “We got kind of lazy. When we would make a new branch, we would have to copy all the jobs in Jenkins, and that wasn’t much fun, so developers would start skipping that.”
The company has now automated test setup and tear-down on the build server, which creates the jobs necessary to run UI tests on all platforms whenever it sees a branch beginning with the word “feature.” Once the feature branch has been merged to the master, the build server automatically deletes the jobs it created.
“Prior to automating job creation we found bugs way too late. Whether developers branch or not, we always want to find bugs as fast as possible.”
If you missed the beginning of the series, you can find Part I here. In Part III we look at which target platforms fit your testing needs and in Part IV, we explore how test automation helps teams do more important work.