A few months ago I was mulling over the best way to approach a proposed automation initiative I had proposed when I came across a well written and highly informative blog by Elisabeth Hendrickson that helped me focus on my objectives and narrow my search criteria. Our situation is largely similar to a lot of other organisations in that we have a web-based product with a plethora of checks that are ripe for automation. We’re understaffed and overworked (aren’t we all?), so my primary aims included:
- A way of executing automated functional checks on existing features quickly for each candidate build
- Transparency across platforms and browsers (Windows, MacOS, Firefox, Chrome, IE, etc.)
- Allow the sapient testers to focus on testing and discovering new information about the product
- An efficient way of communicating automation results to all stakeholders without overwhelming them with data
- A way of involving the non-technical stakeholders so they feel part of the technical side of things (without having to be technical)
- Providing a way of capturing screenshots that allow manual inspection of aesthetics (also useful for demos, presentations, bug reports, etc)
- Build some bridges between technology-business, hopefully stimulating further collaboration through visibility
I’m a big fan of scripted languages and tend to go for Perl more often than not, but Ruby was catching my eye for this particular adventure. I love the ability to simply “write’n’run” code without having to deal with compilation, linking, etc or having to invest in expensive IDEs to bring projects to life. Ruby has elegant syntax and a wealth of support in the online community, plus the inclusion of IRB (Interactive Ruby Shell) which was marvellously easy to use when experimenting with new code.
I just wanted an intelligent editor and a command line. As far as editors go, vim was always an option but as I was going to implement this on a Mac, I plumped for BBEdit which, as Bare Bones Software suggests, “it doesn’t suck”. This afforded me the ability to switch between multiple files very quickly (you can do this with vim too, but I liked the presentation and the reviews were favourable).
The other obvious choice was Watir which seemed like a natural fit to me, but after reading a series of posts from Alister Scott, I chose to go with Watir-Webdriver instead. I particularly enjoyed his ‘meaty example‘ post, comparing Ruby automated APIs. The syntax looked elegant as well as affording me the chance to make the code maintainable (as well as readable). In principle, this choice would support IE, Firefox and Chrome across Windows and MacOS (with the possibility of Opera too). It also allowed me to capture ‘full screen’ screenshots (as opposed to ‘visible area’ screenshots) – something which my previous endeavours with Selenium caused more pain by not working as I’d hoped.
There are a number of frameworks to choose from but the first couple I looked at were a bit too ‘codey’ for my non-technical stakeholders, namely Test::Unit and RSpec. I even bought a copy of The Rspec Book but in doing so ended up reading and digesting the chapters within that talked about Cucumber. A soon as I saw this I knew it would give me the best chance of drawing in the non-technical stakeholders. I ended up buying The Cucumber Book as well, but not until I was most of the way through implementation – thankfully there are lots of great online references, tutorials, and examples to learn from on the web. The key things about Cucumber that won me over were:
- Plain text descriptions
- Nifty results presentation with the default Cucumber HTML formatter
- Even niftier results presentation with a customised Cucumber HTML formatter
- Ability slice and dice the automation suite using Cucumber tags – a very powerful feature
Now, granted we don’t do any sort of BDD at our place, but that didn’t stop me from choosing Cucumber (nor should it, IMHO)
Move to the current day, the initiative is complete and has been well received so far.
I’ll elaborate more on the details in later blog posts.
We used Watir in my first workplace, it was pretty decent but then we somehow got caught up in the Selenium buzz, and opted for that before really giving Watir a chance. To be fair though, that was probably more a team skills choice than anything else.
What attracted me to Watir was it’s ability to do screen shot comparison, to pin point cosmetic flaws introduced with changes to stylesheets or broken functionality. Sadly we never got to pursue that though.
Looking forward to hearing more. Our agency is only starting to think about automation now, so it’ll be good to hear how you get on. Are you running via a continuous integration server, or are the test just in place to reduce regression test cycle times and run seldom?
[Del’s Reply: The setup sits on a dedicated machine, pointing to our system test environment and utilised every time we get a delivery. It’s earning its crust at the moment and I’ve found it very easy to maintain so far]