Test Automation Tools: Build or Buy?

I will set my stall out right now and say on most occasions I would rather build something bespoke in the face of most demands for automation. A lot of my reasoning in choosing that approach is because I can, build something that is, and that I understand what components make up such a tool or framework.

In my earlier post If ‘All Else Fails, try Sikuli’, on the subject of using a bought test automation tool, I boldly stated:

“I personally just find them bloated, expensive and difficult to integrate with other frameworks. And if I have a problem to solve now, I want to solve it now, not engage with a sales team.”

That is my genuine view and I will try to expand on it below.

This is not a proposal of building everything from scratch though. There is a plentiful supply of open and free libraries, niche frameworks and tools out there that can be used to address specific problems. A sign of a good one is that it can be integrated and extended in bespoke solutions. This is more to advocate the building of bespoke solutions, using open technologies alongside hand-crafted components.

Cost

Whilst cost might not be an issue for certain budgets, it should be realised that in most cases it is not just a one off “buy the tool” cost. There will usually also be on-going maintenance and support costs to ensure it is kept up-to-date and working. Particular, cheaper, licencing structures may prevent certain parallel test execution scenarios from being possible. Finally, there might be that one critical, but obscure, interface you need to automate to that is not supported by the tool without a special plugin from the vendor.

Building something though need not cost anything. Over a above such resources required for hosting tools and the time required to build out any automation, their should be no cost. If I can build something from raw materials that are freely available, why would I want to spend any money? Any cost associated with building an automation tool compared to buying an automation tool ends right there, the hosting and time invested.

In a typical test automation solution I utilise a number of free and open technologies. Most of my automation builds have been written in either pure Java or Python. If anything extra is required I can turn to various open source projects. These tend to be more around handling specific interfaces or message protocols.

Trials, Sales and Support Team Engagement

With so much promotional material published by automation tool vendors on their product’s capabilities, it might be easy to trust that of course their tool will provide what you need. You should however trial the product first though, either the full product for a limited time or the reduced feature version for some other period decided by the vendor. During this time you may even get some way to delivering you test automation goals. Great!

This takes significant time though and there is no guarantee that one particular product on trial will make the final selection. From the effort gone into building assets for one tool on trial there is also no guarantee they will be compatible with others.

The trial will inevitably throw up some challenges as well for which you’ll need the sage advice of the vendor’s support team. Then perhaps a sales representative will recommend one of their consultants visit to help out…

This is not an account of all vendor engagements. It is more an account of several experiences rolled in to one to more specifically make the point that this is a commitment of time and energy in which the way forward to actually delivering on the test automation is as yet still an unknown.

Again, if I am able to build something from freely available raw materials, there is no need to trial anything. I know how to use the tools I work with. I know the materials I work with. There is no need to engage sales teams. I can get stuck in and start building out the solution from day one. All effort spent is productive with anything produced re-usable in the future.

Approach

Adopting any tool means adopting that tool’s, or its vendor’s, preferred way of working, when applied to any problem. This can’t be helped, and is not necessarily a bad thing. It would, for example, enforce certain restrictions on how you approach problems and within those perhaps inspire a little creativity to overcome hurdles. However, you would be stuck with an approach that might not be the best, most efficient or practical approach in all situations.

You may over time find yourself implementing various hacks and workarounds to the tool’s impositions. It depends on how you feel about that, possibly either breaking best practices or extending the tool’s capability, but one danger of this would be reducing future maintainability of the automation by working outside the parameters of the tool.

With a bespoke, built solution, there is no restriction on how you approach automation. That said, having built a number of test automation tools and frameworks I have found there is often a collection of problems that need to be solved over and over again. They may be found in different contexts, but having solved that problem once there is likely already a predefined and proven solution for the next time it is encountered.

Having the freedom to be able to build something allows for being able to adopt any relevant approach to get the job done. I would like to think that building something, whilst offering many freedoms, also imposes a responsibility to employ standard development best practices. After all, building something, anything, implies some form software development. The test automation tool, and any tests written against it, is a product that needs to be maintained and possibly handed over to others in the future.

Future Support and Maintainability

There is a concern that always comes with anything that is built. As the built automation is bespoke to the client and their problems, it is unlikely there will be many others outside of that client that can support it. Compared to any bought tool, there will likely be many others who are familiar with it and can support it. From my perspective, any built solution will have source code available and should have been built to a standard that anyone with experience and familiarity with the language of that code can adopt and further maintain it. So rather than be narrowly restricted by the tool, we would be free and liberated to change and extend the tool to match changing client and project demands.

This has been a personal justification of why I would unlikely propose buying a test automation tool. I have not yet really encountered a situation where, given the points above, it has been all that necessary. I don’t have any outright objection to them (however it might sound above), but I believe the case is strong enough to build something first and if a tool really, really needs to be purchased, at least ensure it can be integrated with what’s been built; it really should be.


Source: ministry of testing
Test Automation Tools: Build or Buy?

Share This Post

Show Buttons
Hide Buttons