Weekly Wednesday Webinar on Selenium & Sauce Labs

I’ve been working at Sauce Labs for a while now, helping enterprise users build test automation frameworks and implement continuous integration using Selenium & Sauce Labs.

In order to reach a larger audience — and to learn more about people’s challenges developing test automation — I’m going to be hosting a weekly webinar on using Selenium with Sauce Labs for test automation.

So, starting this week, each Wednesday during lunch (12:30pm Mountain Time) I’ll host a webinar / office hours.  I’ll begin with a brief presentation introducing the topic, followed by a demo (live coding — what could go wrong?), and then open it up for questions & comments.

The first webinar will be tomorrow at 12:30pm MST.  The topic is DesiredCapabilities.

I’ll talk about what desired capabilities are, how to use desired capabilities with Sauce Labs, and show how you can use the  Sauce Labs platform configurator to generate desired capabilities.  I’ll also talk about Sauce Labs specific capabilities used to report on tests and builds.

Register on EventBrite here: Selenium & Sauce Labs Webinar: Desired Capabilities

Join the WebEx directly: Selenium & Sauce Labs Webinar: Desired Capabilities

Contact me if you’d like a calendar invite or more info.

Test Management

A well organized article on IBM developerworks about test management:


The takeway I like is:

Testing tasks:

  • Organization
  • Planning
  • Authoring
  • Executing
  • Reporting

Really not that specific to testing, but helpful to think about when “planning” or “managing” or whatever it is you call it.

I don’t know if I endorse the conclusions or descriptions of the article, but this is a good outline.  I’ll try to elaborate on my “interpretation” of it.

and looking for work

After a month, I should be in pretty good shape, if I haven’t taken a job already if the pay and position is right.  I think I’d really like to help build a new and growing QA department up.  I don’t really have any management experience, but I’ve found in the last six months that I have a lot of ideas I’d like to put into action, and work on a lightweight but effective and thorough process using open source tools.

It’s a really exciting time for open source QA tools, and while Selenium shows the way technically, its things like FitNesse and the “wiki way” that are really exciting.  Not using wiki’s specifically, but the idea of leveraging hypertext (linking documents) — and the web (server-side applications) to create QA/PM/Development “mashups.”

I know it’s a silly web 2.0 type word, but the idea is great.  Chicago Crime was a perfect illustration for what you can do with online data, AJAX, web services, google maps, and an MVC framework.  It’s the killer app behind Django.

There are some great QA tools out there.  Bugzilla, Jira, Selenium, Watir, FitNesse, Ant, Luntbuild, etc. can all be put together into something more powerful than their parts.  Maybe the tools need re-written, but it’d suck to throw away the years of development, and especially all the experience and data that businesses alread have.

Whether businesses use a “QA site”, something else like it, or implement their own  solution doesn’t really matter.  The time is ripe for tools integration to make QA, Dev, and PM teams more productive, but especially to make the “work” part of their jobs more enjoyable.

That’s where I want to be.  The idea of marketing a QA site (0r PM site) as a product is really just the way to get the idea in the door.

Dev tools seem to be migrating towards IDE plugins (with bastard children like mylyn crossing over to QA and PM) — but I don’t really think developers really love eclipse that much that they would never want to leave it (but there was this think called emacs…)

The PM trend has been to lighten up and use hosted apps like basecamp and be nimble or agile or (if you prefer expensive products) LEAN.  I think PM has gone a little too far, or rather that PM is too much being relegated to the developers or managers.

QA is still a mishmash of tools.  There are a lot of new tools and some old ones that keep getting better.  The real excitement is in the fairly recent tools like Selenium, Fit, WATiR, Canoo, etc. that are now maturing and being accepted — though I worry about people losing interest in them, or not really seeing the potential as they go into maintenance mode like Bugzilla did for a long while.

I want to unlock the potential of these tools (and others) and now that we have the potential with open source, to realize the gains, do our jobs better, and make them less tedious.

Testing tutorials to write

Here are the testing tutorials I’m going to work on:

Fit & Fitnesse
Customizing Bugzilla
Bugzilla features: web services and email_in
TestNG vs. Junit
PHP SimpleTest
Perl Test::Class & WWW::Mechanize
Test Planning, Test Cases & Defects

Bugzilla and the Summer of Code

I learned from the Bugzilla-dev email list that the Google Summer of Code is taking applications starting tomorrow.

Of course, Mozilla is a sponsor, and Bugzilla is a Mozilla project, and we’d like be involved. Some of the top Bugzilla developers are interested in mentoring (and I would be too, if I were any kind of a mentor — I could be a cheerleader.) The Bugzilla summer of code page is here if you’d like to apply.

I know I would if I were a student. I’m going to be working on some Bugzilla integration with QA Site, really soon now.

Which reminds me, I’m supposed to be working on Bugzilla QA for the 3.0.4 release, but I’m swamped right now with the new One Shore design which, of course, uses Bugzilla.

By the way, an interesting project that I might check out when I have the time is Tikal, which has a new UI for Bugzilla that I like the look of, as well as support for tasks, etc..

Anyway, here’s my ideas for Summer of Code projects for Bugzilla:

Here’s some ideas for projects:

  1. More charts and graphs (the kids love the graphics)
  2. Outlook (or Evolution) plugin
  3. Firefox extension (I saw a request for this on Guru.com recently)
  4. Automatic bug submissions from build, test tools (e.g, Ant, Selenium)
  5. Bugzilla based social network (Assignees == Friends/?)
  6. Template UI loader/switcher (like in a CMS)

Any other ideas? Let me know or go to the Bugzilla Summer of Code 2008 Community page.

Getting out of the way – requirement #1 of the ideal framework

The framework that manages best, manages least”  or something like that.
Even in the original it didn’t necessarily mean the least amount of government, but that because of the structure, less intervention was required.

This doesn’t directly boil down to simplicity,  but simplicity is the easiest way to reduce management overhead.  That’s why things like wikis, agile programming, and lightweight project management tools like Basecamp are so popular.  But what do you do when complexity is needed?  Fine grained authorization, detailed workflow, complex views, localization, etc. (More on this later.)

A good framework should not have complex configuration, obscure conventions, or rigid patterns.

On the one hand, Rails is very good at this.  You can build sites with Rails without really understand Ruby or Rails.  On the other, it’s very difficult to go outside the Rails box.  Rails has improved, or rather techniques have been learned that make this doable.  For the most common 80% of tasks, though, I’d say Rails sets the standard for getting out of the way.

Rails’ philosophy of “convention over configuration” works in Rails because for the most part, it gets the intuitive part of convention right.  It also benefits from being an early comer (in at least the current round) among frameworks.  Many imitators who diverge do so at their own peril, first because Rails did an exceptionally good job, and second, because “The Rails Way” (good book by the way, by Obie Fernandez.  Don’t think I’m plugging for him though, I don’t know him, but my uninformed personal opinion of him isn’t positive.) has become a de facto standard of comparison with most other frameworks that came after it.

ActiveRecord, however, does a great job at first –it was arguably the defining feature that made Rails popular, but then falls down when you want to go outside the class-is-a-table pattern.  Not saying it blocks you, but it gets in the way.  ActiveRecord is so valuable for prototyping new applications, however, it’s almost a must for a good framework to support it.

I’d argue a good framework should allow things like ActiveRecord, not necessarily support them.  I haven’t yet seen the tool, method, pattern that makes transitioning from ActiveRecord smoothe.

When convention falls down and configuration is needed, then things can get in the way, and perhaps one of Rails’ stumbling blocks is the advanced beginner has had it so easy so far with sensible defaults, that transitioning to manual configuration is as like hitting a wall as hitting the water at high speed can be.  If you weren’t going so fast to start with, you’d slip right into the configuration.  And after something like Struts or Spring IOC, Rails configuration is like slipping into a jacuzzi after a long traffic jam.

Similar story with URLs.  Rails’ page-controller default routing is a thing of beauty, and a true aid to site design all around.  But when the page-is-a-controller metaphor breaks down, it’s a pain.  Probably more because you tend to think that way and choose that as the wrong pattern when you shouldn’t have.  Also, without an easily customizable front controller, pages classes tend to have too much “baked” in, and the complexity is in the wrong place.

The thing I like most about Rails is how well it does “get out of the way”, and the thing I like about CodeIgniter is that it accomplishes it almost as well,  but transparently, and could benefit from some of Rails’ magic.  Which leads to my next requirement for the ideal framework:

Being obvious about what it’s doing.

Framework fetish and rants on SolarPHP, Django, and Python

I’ve been looking at more frameworks — I know, it’s a problem. Including an in depth look at Joomla and a peek at SolarPHP, including a look at YAWP, Savant, PDO.I’m not going to use Solar, but a day spent looking at it is a day wasted. A quick criticism: It’s too “magical”, requires too much memorization, has tons of errors in it’s documentation.

I wish there was an easy way to submit errata or make changes. The SolarPHP manual is a wiki (which supposedly gives them an excuse to prevent page navigation), but normal users can’t edit it (which is fine), but there should be a workflow to propose changes — that’s a problem with wikis in general, no workflow, but it goes counter to the wiki “philosophy” which is silly.

Solar has a very obtuse system. While someone familiar with Rails-like applications can figure it out (with the help of the documentation and some guesswork — because the documentation includes things like mis-named files that cause opaque error messages. Every error you make will produce an exception with the exact same text:

“Exception encountered in the fetch() method. in Solar.php line 363”

and the stack trace doesn’t even mention your application code at all.

While Rails had an unusual configuration system, it had reasonable defaults, that were fairly easy to learn. Every framework that imitates Rails and adopts convention over configuration has the problem that it’s not Rails. The further you stray from Rails the more users will be frustrated when it diverges. Corollarily(?), the closer you hew to Rails, the more frustrated you’ll be when it (often unexpectedly — read: undocumentedly) diverges. I don’t want to learn a new “convention over configuration” for every framework.

SolarPHP’s ORM sucks, too. While I didn’t look into enough to say how much it sucks, it’s a rule that anything that copies hibernate sucks, and suffers the same problem as Rails imitators. The more it diverges from Hibernate’s syntax (even if it’s an improvement), the worse it is. And I’ve a hunch that Solar’s ORM isn’t nearly powerful or flexible enough to justify it. I categorically refuse to learn another ORM syntax.

SolarPHP’s templating is unimpressive, it’s MVC file layout is uninspired (and non-standard), it’s authentication is inadequate, it’s localization is simplistic (but works — on the otherhand, how hard is a string substitution?) In short, it didn’t really bring anything that appealed to me, while giving enough frustrations (mainly with the documentation) and annoyances (mainly because of differences with my knowledge of existing frameworks) to put me off.

I can’t believe it, but it seems to not even have a scaffolding script to help you learn (or not need to learn) it’s file layout structure.

Anyway, I’ve distilled some requirements out of the experience. I think a good framework should:

  1. Get out of the way
  2. Be obvious about what it’s doing
  3. Leverage existing knowledge and techniques
  4. Help write better code by:
    1. Making code simpler, less verbose
    2. Help me stay organized
    3. Help me avoid shooting myself in the foot
  5. Allow hacks when needed

I’ll post more on that later.

I’m not trying to hate on SolarPHP. It’s bearing the brunt of my accumulated dissatisfaction with frameworks. I mentioned before that the only ones that have really excited me (since Rails) are Wicket and CodeIgniter. The former runs on Java, which in my book makes it only suitable for intranets or large (read: managed by a full time sysadmin) sites. The latter is too simplistic by just a little bit. It could actually benefit from a bit more magic, and things like authentication and ajax left as an exercise for the user. Which is fine, especially if you were going to build everything from scratch (or from Zend / eZ) yourself anyway.

One framework I haven’t looked at, but (to spread the hate around) I’m sure I wouldn’t like is Django. That’s just from the general impression I’ve gotten by crowdsurfing the intermemeoshere. I’m sure Lawrence, Kansas is a nice town and all, but building a CMS for a small community newspaper isn’t exactly brain surgery. Chicago crime is way cool, but that’s just showing off Google web services with a neat idea. It could’ve been done in VBscript and it would have been just as cool.

My impression is that Django consists of the core team marketing it plus a bunch of people who thing Python is cooler than Ruby. The one thing python has over ruby is a halfway decent Apache module. And I do mean halfway. At that end of the spectrum (near the very bottom) speed, is not a deciding factor. I think python’s syntax is silly, though like almost everyone else, I thought tuples as a type were cool. You can get the practical benefits of that in PHP nowadays with list() — which I think is prettier syntax.

To all the PHP haters out there whose only arguments are “it’s possible to write code that has a SQL injection vulnerabily” and “I don’t like (or understand) some of the function names”, I don’t want to hear it. PHP isn’t the best language in the world, get over it. But it’s got a better syntax (and object model) than Python.

Do you know what “def” means? It means “define function”. Defining a function with part of the word “define” is empirically uglier than with the word “function” and less fun than abbreviating it “fun.” Mandatory underscores, bad keywords and yes that horrible white space delimiter are enough to make me not want to use python.

If you don’t like PHP function names, it’s probably because you never learned C. Hint: the 3 or so example functions that hurt people’s tender sensibilities in every “I hate PHP” rant are direct transcriptions of functions used to write your favorite scripting language, which was written in C and I almost guarantee it has the dreaded strpos(), strlen(), et al, buried in it’s source code.

The fact that every rant on PHP contains complaints about the same functions as a usenet post from 1997 shows that they don’t really have their own opinion on it.

And don’t get me started on “self”. Python’s bolted on objects may be better than Perl’s but that’s not saying anything at all. One thing that appealed to me about Python was that you could take more control of your request lifecyle — what I really liked about mod_perl, but it turns out that building it all into index.php is both faster and more stable.

But I’ll probably check out Django, and learn from it too. I’m sure I’ll find things I like and things I hate, and things I hadn’t thought of. Do I think it’ll have that “killer feature” that will make me want to switch? No, but I’ll still enjoy it. I enjoy Python, I just don’t like it’s syntax. And all other things being equal (which they practically are), I’d rather use PHP’s syntax, and put up with it’s shortcomings.

I did like my time studying SolarPHP, even if I don’t want to use it. I learned some interesting things with dependency injection in PHP that I could never pull off myself. And as much as I maligned it’s ORM, I could never build something like it — and even if I could, I’m sure someone would hate my API.

Whew! This turned out to be quite a rant, and with any luck, I’ll draw vultures, both for and against. Which is something I definitely don’t want. I’m well aware that your average script kiddie can kick techno-sand in my puny face, and the only reason I feel comfortable ranting is because I’m sure no one more nerd-savvy than my mom will read this post. But with the way search engines work these days, my little flicker of ill-informed self-commentary will probably draw moths and rangers with buckets of techno-water to douse my fragile ego.

Maybe I should get some adwords up before the bonfire starts.

I’ll post what I really wanted to, next.

Me on Marc Fleury on Sun purchase of MySQL

The last post was supposed to be this one, but I got distracted on how I found it. Needless to say, I’ll probably be haunting marcf.blogspot.com more, because he has some interesting commentary, as does Mark Shuttleworth of Thawte and Ubuntu.

Unlike Marc Andressen, who, while once genius, has become a trivial marketing guru and vaccuous political dilettante. Which I admit I’d probably become if I got rich too. Although, if Ning were better, judging by the postings on Guru.com, he’d be rich. Maybe its because Dolphin is eating his lunch. But if I had $100 (the typical asking rate on Guru for a MySpace or Facebook clone) for everyone who wants to build a social network site, I’d be pretty rich too.

I’m risking getting completely off track again, so here’s what I said about the purchase of MySQL:

Reward buyout.

Sun has already made a fortune off of MySQL. So has Oracle, so have Microsoft, Redhat, IBM, and Cisco.

Sun isn’t really trying to make money off of MySQL, they’re trying to make money off of the *next* MySQL. I don’t know what that is. But it will be open source, and it will increase productivity immensely. It will, as Bob Young says “grow the pie.”

Before MySQL there were two types of databases, Oracle and Access. Oracle was really good and Access had a lot of users. MySQL currently has *way* more users than Access did when MySQL came on the scene, and MySQL now is probably a better database than Oracle was then.

That’s not saying MySQL is so great, but that technology has come a long way. Did Oracle in 1995 have some features that MySQL doesn’t? Sure.

Whether you’re using Postgres, SQLite, Oracle, Microsoft SQL Server, HSQLD, Firebird, Derby, or whatever, you’re benefiting from the popularity of MySQL. If you’re still using Progress, dBase, or Sybase, maybe not.

My point is that MySQL did for databases what Netscape did for the internet, what Apache did for web servers, what Star Office did for alternate word processors, what Sendmail did for email servers, and what JBoss did for J2EE.

In the startup investment averse climate these days, acquisition is the way the market rewards innovation. You of all people should be aware of that, Marc. But perhaps you’re a little too close.

Despite the real value that JBoss brings to Redhat, (and perhaps MySQL will bring to Sun) the biggest reason for its acquisition is making you rich.

Because when the next Marc Fleury comes along with a silly idea to quit his job and move into his in-laws’ garage for a year to work on an idea he has for some open source application that will vastly increase the market size of technology X, the RedHats and Suns (and Ciscos and Dells) of the world want him to succeed. Or at least believe he’s got a shot.

Redhat itself was probably the biggest reward acquisition of all time. Only it was an IPO climate then. People wanted to reward Bob Young, and Alan Cox, and Linus Torvards, Tim O’reilly, and many open source folks they knew (or hoped) were in on the IPO.

That’s not to say the market always does what it wants or is always rational. Not everyone with a great (and successful) idea is always rewarded. Eric Allman, by all rights, should be almost as rich as Marc Andressen. While I doubt he’s doing too badly, I’d say email was as at least as big of a productivity gain as the browser. Maybe its because sendmail wasn’t as revolutionary as Netscape, or because it was open source, or because he had more modest ambitions, or greater risk aversion.

While I worry about JBoss “stagnating” at Redhat or MySQL “floundering” at Sun or Tangosol getting “buried” at Oracle, I’m not too worried. Because these products have already made huge productivity gains, and even if they died instantly on acquisition, they’ve already done the damage of 1) releasing their ideas and 2) showing other bright young entrepeneurs it can be done.

If only my in-laws had a garage.

Of course I’m spending 6 months off in Ecuador, but I haven’t made nearly the progress I’d like, due to lack of focus and perhaps biting off more than I can chew. I think QA Site is the way to go and CuencaTravel (somewhat of a social network site) is a good product to test it with.

I’m lacking in some areas of expertise, such as email server administration, graphic design, and sales.

A simple junit rant

I’m going to make this very simple. My last junit rant (online — I think on theserverside.com) resulted in a nice email from Cedric Beust about this new thing he was working on called testNG. Oh how I wish I could use it at work. Luckily, that will be an option in 3 days, as I will retire and work for myself full time. JUnit 4 has been a blessing over 3, though dependencies and useful annotations are still sorely missed. @BeforeClass is also mostly verboten, but is allowed in special cases when it shouldn’t be used!

Anyway, this is a very simple rant, though I’m very curious about how I uncovered it.

No assertNotEquals() yet!?!

I’ve wasted time trying to find a rationale. Of course, I have found the junit addons package, but I’m not going to force my employer into that dependency. Maybe I’m missing something, but this is probably the most valid unit test. The likelihood of two things being equal when you expect them to is pretty high, and the likelihood of something being true when you expect it to is even higher. What you really want to know is when something isn’t true (not difficult, even without assertFalse(), which for some reason is religiously implemented) or more especially when two values aren’t equal.

Roundoff errors immediately come to mind, and are an ideal candidate for the architypical unit test. Date handling is another example, and string munging also makes sense. So would image manipulation, though Java isn’t a good language for that.

Anyway, how I came across this, and I’m probably just stupid, is with temperature conversions. Here’s my test that I would expect to fail:

Temperature t = new Temperature(-40.0, ‘F’);
assertEquals(-40.1f, t.convert());

And here’s the code that is being exercised:

public float convert()
if (scale == ‘F’) { return 5 * (temp – 32) / 9; }
else if (scale == ‘C’) { return (9 * temp / 5) + 32; }
else { throw new RuntimeException (“unknown temperature scale”);

Maybe I’ve spent too much time in scripting languages that do a better job of casting or something. Anyway, it’s not a big issue what’s causing my bug, but I want my test to uncover it.


1 – It looks like there is no assertEquals(float, float.) So it is calling assertEquals(Object, Object). Behind the scenes is it treating them as doubles, integers, or strings?

1.5 – It looks like assertNotSame() will do the trick for this particular case. Is this the answer to checking round off errors? Still, it wouldn’t work on more complex arrangements, or on bit arrays (images), right? I guess it’s fair to write your own assert for comparing images, but surely someone’s done it before.

2 – It also looks like TestNG doesn’t have assertNotEquals either. I’ll probably get another note from Cedric that it was added to and and released yesterday or something.

I admit I was a bit star struck and felt ahead of the curve too that someone with reputation responded to my message. I should start a geek celebrity sighting scrapbook. I still remember when Havoc Pennington responded to an simple question about Metacity code. He dismissed it at the time as a pet project he’d abandoned and then a year or two later it got adopted as Redhat’s default X window manager. I seriously doubt my asinine patch was ever included. But I’ve got (or had) code in docuwiki. I wanted prettier table layouts and hacked a few features into it. I think that was the point I realized that I thought I could really code, submitting a patch and getting it listed on the release notes.

Nobody expects the QA inquistion

Our primary area of focus in testing is usability. Usability and security. Security and usability. Our two main areas of focus are usability and security…and functionality… Our three areas of focus — don’t forget stability. Among our areas of focus — Amongst our many areas of focus are included usability, stability, functionality, and security…and performance.

I’ll come in again.

The Quality Assurance TEAM