Every industry has some degree of automation. So why is automated testing of a digital health environment so rare? Why do we constantly see the testing phase of every EMR, EHR, or HIS implementation planned with thousands of hours of manual testing by clinicians, business analysts and IT professionals?
Among the concerns we hear are:
- Our Software vendor does not support automated testing
- Automated testing will add cost
- I already have my project plan done and can’t fit in automated testing
These are all myths.
Aversan’s test automation and test execution services can help free up your clinicians and business analysts so they can do the work they are best suited for. Aversan creates a tailored environment using our Automated Test Framework (ATF) to automate testing of the manual test scripts that you now use, saving hours of cumbersome manual test labor. We also create variants of those scripts to broaden and deepen the test coverage which gives you even more confidence your digital health systems and connected medical devices are functioning as expected.
How do you make the decision to invest in test automation? It is the correct choice for some projects, not every project. Also, it is the preferred approach for certain aspects of a project, not every aspect. We believe there needs to be a business case for test automation, in other words a return on investment (ROI). One needs to make the case for the initial investment in the test automation methodology, tools and setup, which then generates a payback based on use.
In this article we will look at two types of ROI.One is the traditional financial ROI. The other ROI is more subjective, but likely much more important because of the human element. This ROI is based on an assurance that the system meets a level of acceptable quality- in other words patient safety.
The return on investment of test automation is very high. Several case studies from past and current clients support this claim. Very often the initial cost to create the automation environment is recovered after 6 to 7 full test runs. A typical organization will have dozens of test runs over a 2 to 3 year period. Numerous test runs are necessary once an organization has procured a new software system as they incrementally configure it and migrate in legacy data. Each incremental release must not only test the new functionality but also re-test what was previously developed and validated. Then one needs to factor in releases that integrate ancillary systems and medical devices as well. : There will be multiple iterations of these release and test cycles as well. Finally, before and after promoting the solution into production, the various vendors involved may have software upgrades and patches that must be applied. Best practice requires performing a full regression test once again. Doing all those regression tests manually is just not feasible from cost, timeline or quality perspectives.
Patient Safety ROI
Let’s look at a patient safety ROI case study. British Columbia deployed a lab reporting solution across the province connecting dozens of systems.Diagnostic test reporting is not something you want to get wrong. Whether it is an error in results transmission, an error in transformation of the raw data into a standardized format, a report associated with the wrong patient or a report sent to the wrong clinician -any of these will affect patient safety.
Aversan performed component, integration, performance, user interface and workflow verification services, 80% of which used automated testing. New releases were being introduced as often as every week during development the phase. There were nearly 1700 test scenarios automated during this phase (currently this number has grown to 2100 due to changes and enhancements to the application). Each major release of the system during development requires a pre- and post-candidate test confirmation run. This equates to roughly 3000 hours of manual testing for each release (as each run takes approximately 1500 man hours). Assuming the average fully loaded cost of testers (a blend of clinicians, business analysts, IT professionals) is $100/hour, manual validation of the system would cost about $300,000 per release.
The upfront investment required to implement the Automated Test Framework and regression test suite for this program was $1,500,000. The cost to execute the automated test suite is $2500 per run ($5000 per release). Using a very modest number of 20 test runs, the cost to manually test would have been $6.0M but was only $1.6M with automation, roughly one quarter the cost. The upfront investment in the test automation suite was break even by the sixth test run. The complete case study is here.
Financial ROI
Another case study involves Manitoba Blue Cross and is a fantastic illustration of a financial ROI. The claims processing system processes approximately 250,000 claims per day, at an estimated value of $10-$50 million per day. These metrics show the magnitude and complexity of the system and its impact on the business and its clients. Prior to Aversan being engaged, the project team had only 120 test cases. Aversan broadened and deepened the test coverage to 5,500 test cases, and given the use of automation, for virtually no additional cost to each regression test. Defects and shortcomings in the implementation that were otherwise not detectable could now be identified. Thorough testing of all the test cases would be about 5,500 hours of people time. With the implementation of the ATF, the execution time for the complete suite of 5,500 test cases is 24 hours.Therefore, there was a significant savings in labour costs associated with manual testing. There was an increase in solution quality, reduced and predictable project timelines, and reduced costs and timelines for testing. The complete case study is here.
Besides the immediate results, the ROI continues even after the system is live. The Aversan ATF and automated test cases are left with the client along with training and support on its use.
In summary, test automation does have some upfront cost. But there is little or no impact on project timelines given the setup runs in parallel with other project streams. There is clearly an ROI after 6 to 7 complete regression tests. Furthermore, there is an undeniable increase in the quality of the system due to broader and deeper test coverage. The business case is easy to calculate financially. It is a bit more subjective when one factors in the patient safety ROI. But we can all agree that avoiding human harm or death due to software errors in the digital health system is priceless.
Here are some quick links to more information:
Overview of Automated Software Testing
Testing as a Service Catalogue