Test automation can be an incredibly useful and valuable tool for your development team, however, building and maintaining certain types of automated tests can be cumbersome and expensive. Sometimes it isn't readily apparent which tests should be automated and which should remain a manual test. We reached out to our QA experts to get their take on when and why to automate tests.
That's really a multi-headed question. If you are talking about unit tests, any areas of code that apply more than a basic level of logic should probably be automated. If you are branching out further into something like integration tests, I would say that you would want to automate key points of interaction between the pieces. On the subject of something like UI tests, my answer would be any regression test that you are sick of doing that can be easily automated and won't be so fragile that you spend more time fixing it than actually testing.
To answer this one, I am going to take the question and flip it: What tests should NOT be automated? New and/or rapidly changing functionality: If the product is changing so quickly, automating the testing of it might be more trouble than it's worth as it changes. Any tests involving subjectivity, such as look and feel, workflow, or other user experience type testing. The extremely complex, if it is relatively easy to manually test, but incredibly difficult to automate, then it is probably best left as a manual test.
What does that leave for automation? Plenty! Including (but not limited to): repetitive testing, data driven testing, load testing, smoke testing, and regression testing.
My rule of thumb is, if something sucks to test manually and you know you are going to have to test this thing at least three times in the future. Why not automate it? If you can’t automate the whole scenario, can you automate part of it? It is okay to take an incremental approach to creating your automated tests and make your manual tests suck a little less.
The short answer is that it depends. Sometimes there are tests that can't be emulated on a server for automated testing because the server doesn't have the hardware to support it (like bluetooth testing for mobile devices). Though where possible I usually like to automate tests that are repetitive that run on multiple builds/hardware platforms, tests that can be confusing and have many opportunities for introduced human error, and tests that can take a long time and effort to run manually. This then allows a tester to make sure they cover the scenarios that are impossible to test through automation. But, you do also have to consider the maintainability when writing automated tests, because if they are incredibly fragile there will be more maintenance headaches than there was in the original question/problem that was being solved.