There are two ways to automate web testing. The goal is to test the functionality of a web application. One way is to write automation that drives a browser. The second is to use a library that imitates a browser session and communicate directly with the server.
The other method is to cut the browser out of the equation. Tools such as HTTPUnit, HTMLUnit, Canoo Web Test, WebDriver, and TestMaker use this method. A similar method is used by other tools including JMeter, LibWWW, and curl. The obvious disadvantage to using this method is browser compatibility issues.
Tools that use macros to drive a browser would fall into the first category. Tools that have an “expect” like dialog would fall into the latter.
I usually advocate the first method, because nothing beats having a real browser excercise your application.
The often overlooked advantage of the second method is that you can more easily run browserless tests as integration and smoke tests. Because it doesn’t have the overhead of the full browser, it is lightweight, and client independent. It runs faster and can be more easily run concurrently (which makes it useful for performance testing.) Browser timing issues (and crashes) are less of a problem.
I’d actually recommend type 2 tools for testing links, page flow, content (text/images), and principal functionality. But if user interface testing is needed, I’d use type 1 tools. However, the truth is that UI and ajax timing related issues are much easier to find with manual testing. I’d guess 3/4 of what automation buys you can be done with a browserless tool.
The advantage that browser-driven testing buys you is with recording tools. However, the penalty is with stability and speed of your tests.
5 thoughts on “two ways to automate web testing”
Hi Aaron: Nice blog entry. Thanks for including TestMaker.
You should know that TestMaker bridges the first and second categories of test. For instance, we ship with TestGen4Web (and soon Selenium) as plug-ins to Firefox. Playback there is through the browser. When you need to run the recorded test as a load and performance test or a service monitor then TestMaker runs the TestGen4Web test (or Selenese test) using HTMLUnit.
Details are at http://selenium.pushtotest.com.
Keep up the good blogging.
@Frank: is a plug worth ignoring all issues of this blog entry? ;-)
What tool do you work on, Marc? I’d like to know more about recording with non-browser driven tools.
Just a test as it seems that the comment I’ve posted doesn’t appear
Ok, comment are working, it was probably my fault.
Yes, HtmlUnit is based on HttpClient. JS support is provided by HtmlUnit whereas SSL support is provided directly by HTTPClient.
I’m a bit surprised of your next post, should I have given you kudos for all inaccuracies of this post like Frank did?