Python is getting to be quite popular as the language of choice for testers. One nice thing about Python is the philosophy that there should be “only one way to do it” — the exact opposite of Perl’s mantra “there’s more than one way to do it.”
This lends precision to Python scripts which might not be the case with other tools. Of course, above the bare language level, there are idioms to learn and a variety of tools that can be used.
Here are a number of tools written in or using python for testing:
unittest – this is a typical xUnit framework, and will probably be the base of most tests. There are the expected methods setUp() and tearDown(), as well as asserts. Tests extend TestCase, are grouped into TestSuites, and executed with TestRunners the default being TextTestRunner. Nothing surprising here for xUnit users, and a clear, concise way to get started for those new to writing code tests. A couple niceties for running unit tests in Python that show maturity (over JUnit, for example) are the TestLoader and TestResult classes, for organizing tests into suites and gathering results, respectively.
Selenium – Selenium RC is written in Java, but has a python driver. A good way to ease into Python test development is through the Selenium IDE which can convert recorded tests into Python. Not only will it help you learn the Selenium API and Python syntax, but it will also introduce you to unittest.
Windmill – Windmill is very similar to Selenium, but written in Python. It uses a web proxy, is javascript driven, and includes a test recorder that works with Internet Explorer. This seems to be a popular feature and is helping Windmill gain converts. For me, however, the interactive shell is much more interesting and more useful than the Selenium interactive experience. If you prefer, you can write your tests in Javascript as well as Python.
twill – twill is “a simple scripting language for Web browsing”. Think of it as a selenese-style interface for the grandmother of all web testing techniques, the venerable (and painful) telnet port 80. Another analogy would be using htmlunit from a command line. It looks very interesting for non-browser-based testing.
Pamie – Pamie is a port of Samie, a Perl framework for automating IE via COM. Watir (Watij, Watin) is a similar framework written in Ruby which is probably more mature. I don’t know if Pamie is still active.
Mechanize – There is also a port of Perl’s WWW::Mechanize to Python for those who like it.
WebUnit – WebUnit is an extension of unittest that provides additional features for web testing. A WebTestCase drives an HTTP Request and has additional helpers. It looks fairly simple and straightforward, and would fall into the same category of non-browser driven tests with Mechanize, Twill, HTMLUnit, and Webrat, though probably not as featureful. (But sometimes you don’t need all the features.)
Fitnesse – Fitnesse, like Selenium is written in Java, but fixtures can also be written in Python via Jython. There is also a port of Fit called PyFit, but it may be outdated. Personally, I like the idea of wiki documentation tied to user-editable test data and developer written test fixtures as code. I’d like to see updated Fit style testing frameworks with a better wiki than Fitnesse.
webtest – Not to be confused with Canoo Web Test, webtest “is an extraction of paste.fixture.TestApp" In a nutshell, it’s for testing at the WSGI application layer.
Saving the best for last…
doctest – Write literate tests using docstrings. Ability to execute tests from the Python shell. It’s like fitnesse in your source code.
nose – “nose extends unittest to make testing easier.” It’s the way the cool kids are writing their tests in Python these days. I don’t know much about it.
py.test – py.test is an alternate testing framework, that is also compatible with unittest (and nose). It’s quite different, emphasising simplicity (in code) with “no boilerplate test functions”. I haven’t quite wrapped my head around it, but the website says it is a “mature command line testing tool” that “automatically collects and executes tests” and “supports many testing practises and methods”. Perhaps it’s Python’s answer to Rspec, but going in a completely different direction. One that looks much more interesting to me.
See also the following:
http://pycheesecake.org/wiki/AgileTestingArticlesAndTutorials
http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy
http://www.simplistix.co.uk/presentations/testing04/testing04.pdf
My strategy would be to start with what i know, Selenium, and then investigate unittest and python practices more thoroughly. Twill looks interesting, and I’d probably research py.test over nose.
TextTest is like RSpec or JBehavior
twill can use wsgi_intercept to reroute httplib requests directly to a WSGI application (like using webtest) and eliminate needing server startup.
coverage.py (by Gareth Rees & Ned Batchelder) uses sys.settrace for code coverage by your tests
doctest is “literate testing” or “executable documentation” — alternative to unittest ~ rspec.
see also epydoc
buildbot – python continuous integration tool
A good presentation http://www.simplistix.co.uk/presentations/testing04/testing04.pdf
figleaf is another coverage tool
another link to notes on a presentation by Grig Gheorghiu and Titus Brown
http://jjinux.blogspot.com/2007/02/pycon-testing-tools-in-python.html
Robot Framework is a generic keyword-driven test automation framework for acceptance level testing
http://code.google.com/p/robotframework/
http://sourceforge.net/projects/pywinauto/
Windows Automation via COM