TestPlant, the maker of the eggPlant range of software test automation tools, today announced that its data indicates software quality departments in the banking sector are facing scalability challenges when testing applications due to the ever-growing range of devices and browsers. Conflicts between the need to roll out new applications and updates quickly, and the need to adequately test these applications, present an insurmountable challenge if manual testing practices continue.

TestPlant counts many large financial services institutions among its customers, and recently investigated the testing challenges faced by companies in the sector. Technology fragmentation is one the biggest software testing issues in banking today as companies’ online and mobile interactions with customers are increasingly seen as a way to differentiate competitively.

TestPlant’s data indicates that a typical banking application (web or mobile) today will require approximately 200 test scripts to be created and run for each release. In today’s highly fragmented consumer technology landscape, banks’ quality assurance teams are generally choosing to test 15-30 of the most popular device/OS/browser combinations – for example, Internet Explorer on Windows 8, or Safari on an iPhone 6. Assuming just 15 combinations are tested, the time taken to manually test a typical release is more than 380 tester-days.

Subsequent releases typically add 10% more functionality, but in a purely manual testing environment these have to be fully tested again. This increases the time required to manually test each new release to 422 tester-days.

“In the past, companies were only testing on very small numbers of device and browser combinations, which would take a much smaller number of tester-days for each release,” said Antony Edwards, CTO, TestPlant. “This meant that banks could reasonably expect that a fully manual testing approach was enough, but that is no longer the case.”

As fragmentation increases, all of these numbers will also increase and it doesn’t take long to realize that testing departments will find it impossible to scale up manual testing processes. In five years’ time, TestPlant estimates the number of testing combinations to approach 60. This would see the manual testing effort for an initial release rise to more than 750 tester-days.  

“The problem is linear – for every hour it takes to manually test an application on one phone or browser, then testing more phones/browsers multiplies the time required,” said Edwards. “Banks IT teams are struggling with this because they can't just keep hiring more people to absorb the impact of fragmentation – but their marketing and customer service departments can’t afford delays.”