Quantitative vs Qualitative Reporting (Part 1)

I used to work in structured environments for many years where in the field of (test) reporting, numbers were definitely king. The upper echelons of the management team demanded a quantitative style report and that’s what I delivered. I had suggested that an alternative style of reporting would be beneficial so that they could actually see and understand what was going on but they loved their graphs too much to let go of them.

While this was somewhat disappointing I never let the reporting style they demanded dictate my thought process when dealing with the unique set of circumstances and considerations that inevitably come with each and every project. In other words, for instance, I never concerned myself whether the progress graph inched another percentage point towards the exalted 100% each week or not, but rather focused on learning as much as we could within the time we had.

I’ve always had a firm belief that there’s always more than one way to do a job. Whatever worked for you at Company A would almost certainly not work at Company B as the context would be very different and I’ve prided myself as being very adaptable in that respect. My current role is at a comparatively smaller company where reporting (mercifully) isn’t governed by the digit. That said, management did ask me when I came on board to produce some reporting stats which they considered ‘normal’ – sadly, their definition of ‘normal’ turned out to be quantitative.

The management here however, are open to suggestion so with that in mind, I conducted a test on some management staff using different reporting styles, primarily quantitative and qualitative, and asked them for some honest feedback about what was really useful. I have the Rapid Software Testing course from last November to thank for so much inspiration in this area.

(Note to Shmuel Gershon – this was slightly different to the one I told you about earlier 🙂 )

I took a hypothetical project, Project 007 (Yesh, Mish Moneypenny!) and began providing them with snippets of what would constitute a potential quantitative report. We started with the progress graph:

They felt comfortable with this graph.

“Ah, you’re making good progress I see!” they said “although you’ve stalled a bit. Anything we should be worried about?”
“Would you be less concerned if the last entry had advanced past 70%?” I queried
“Yes!” came the resolute reply
“Ok, so any progress is good”, I pressed
“Definitely”

So I decided to add another metric to the fray. [sarcasm] But not just any metric, oh no, this one is the daddy of (test) metrics, the top dog, the big kahuna. The metric that spawned a million high fives, popped champagne corks, and congratulation speeches on a job well done in that everything is now ‘fully’ tested. Ladies and gentlemen, I give you the test case status graph. [/sarcasm]

“Nice, looks like you’ll finish on time!” they quipped after a cursory glance. I must admit, this took me by surprise.
“How would you define ‘finished’”, I asked
“Well, the majority of test cases pass.” they replied. A few heads nodded sagely in agreement
“Did you check the legend at the top of the graph?” I asked
“No, wait….what does red mean….oh no, wait, this isn’t good at all. Oh dear!”
“Still think we’ll ‘finish’ on time? I pressed
“No chance!” came the resolute reply

The discussion then quickly turned towards defects – how many – what severity, so I added another metric to the ensemble. Defect arrival rate (which simply means the number of (project) defects that were created after each day).

There was around thirty seconds of synchronised head twisting, much like an audience watching a very rapid tennis match as they cross referenced the defect data with the test case data. It made me a little dizzy.

“Why weren’t there more defects being raised around the time the majority of the test cases were run?” they asked
(When asked for clarification they meant between the 7/12/10 and 15/12/10).
“I think you’ll find there are around the same number of defects raised in that time frame” I replied.
They considered this for a moment, “So you failed thirty test cases on a set of Minor defects?”
“Yes” I confirmed. “Would you rather I had passed the test cases despite the defects?”
“No, of course not” came the reply.
They looked puzzled “But what about all those Major and Critical defects created between the 1st and the 7th?”
“Ah, the majority of those were encountered while not actually running any test cases” I stated
They took a moment once again to consider this.
“You are fully testing this product, aren’t you?” that asked
“Can you define ‘fully tested’, please?” I countered
[Lots of mumbling…]
“Well, fully tested obviously means testing all the requirements” they agreed
“Ah, I suspected you may say that – All requirements are ‘checked’ as part of the test cases” I assured.
[More mumbling…]
“Would it help if I provided one more metric?” I queried
“Yes!”

Enter the cumulative defect count…….

[Even more mumbling…]
“This doesn’t help us” they retorted.
(I didn’t expect it would, but wanted to see their reaction)

Now obviously I was leading them up the garden path and I was deliberately playing devil’s advocate. Why? To illustrate to them that misuse of numbers as part of reporting can be dangerous and misleading. Although I’ve paraphrased a lot of the dialogue here, I don’t want to portray my management in a bad light, or for you to have a poor opinion of them. To their credit, once they saw the failure of this particular reporting style (in this particular context) they were open to alternatives but I wanted to dispel any expectations that numeric based reports would be a way forward for them.

Enter the ‘Low-Tech Dashboard’ (and their response) – coming in part 2 – watch this space.