When Test Automation Frameworks Collapse

You need NLP test authoring for CI/CD pipelines

By SpotQA,

2019/10/18

For years DevOps teams have been waiting for a testing solution that can keep pace with the speed of development.

Download Case Study PDF

Ever since The Agile Manifesto was published in February 2001 we’ve seen increasingly innovative and efficient approaches to developing software. Now, in the age of CI/CD and DevOps, we have a whole host of tools - Jenkins, JFrog, Docker etc. that help us release quality software multiple times each day.

Alongside these tools there has been an ongoing effort to further reduce or remove the barriers between dev and ops, integrating these teams with new cultural approaches and bringing them closer together through increased communication and collaboration.

But testing has remained the stubborn bottleneck in our otherwise smooth SDLC. As development cycles continue to speed up more and more work is piled onto test teams who simply don’t have the same level of automation in their tooling to deal with the volume. Until now.

The framework collapses under its own weight

The limits of existing test automation and frameworks became all to apparent when the customer tried to automate testing across their three main applications. They spent a year - a whole year! - building a custom framework and scripting end-to-end scenarios only to find that their work was immediately obsolete due to the amount of maintenance required to keep the tests running.

Their automation was always at least one sprint behind and pressure on the test team was intensifying because the dev team continued to expand while they stood still with outdated tools and methodologies. When they attempted to build the framework for a second time and ran into exactly the same problems they knew they needed a smarter, more automated approach. Virtuoso had the solution they needed.

Test Case Explorations


NLP Test Authoring

Virtuoso’s NLP test authoring feature allowed the customer to write automated tests in plain English that could be executed immediately. The NLP engine is easy to use and widely accessible, even to personnel with limited technical or engineering ability.

The NLP engine meant that tests could be authored fast enough to enable in-sprint automation. It also allowed the QA team to build out the entire regression pack for all three applications without increasing their head count.

The issue with test automation frameworks is that they need the application’s code to exist before any testing can begin, and this was the big problem faced by the customer. Virtuosos’ NLP engine enabled an effective shift left which meant they could test much earlier in the SDLC, even before they’d written a single line of code. NLP also meant that in-sprint automation on UAT and LIVE environments was both quick and easy, dramatically reducing the release testing timeline.

Virtuoso’s NLP engine solved the big problem posed by the customers automation framework, but it also delivered a series of additional benefits for the team, like 8x faster test execution. Here are a few of the highlights:

  • Cumulative, monthly saving in testing time of 30%
  • Test execution is now 8x faster
  • Over 70% of in sprint testing is now automated
  • Pre and post-release testing reduced to a few minutes for each sprint

Unplanned releases and backend changes can be done safely because the product and dev teams have complete confidence in Virtuoso’s regression suite.

When Test Automation Frameworks Collapse - Numbers

Evolve or die

In today’s SDLCs, automation frameworks are no longer fit for purpose. They’re too unwieldy for today’s dynamic applications and you need a testing platform that can keep pace with your development. With Virtuoso you can automate more than ever before, from exploration and mapping, to rules-based testing in post-production, and everything in between. Empower your team with a testing solution that can deal with the demands of today’s software development pipelines.

Book a free demo with one of our experts to see Virtuoso in action.

Download Case Study PDF

Subscribe for update on SpotQA news, Virtuoso developments and releases.