The top-ten tenets of software quality assurance, part six: testing

Testers have to be destroyers, argues Mark Wilson. But too often testers aren't trying too hard to bust the software. Rather, they are trying to show it working

"So. Can you do it?"

I looked up at the public-sector project manager, now looking me right in the eyes, man-to-man.

I'd been implementing ISO 9001 (the old BS 5750) at a university company; I'd gone freelance to do the same at a big energy utility, and somehow I'd blagged my way into service change management for Compaq at a central government department yet, and now… I was about to take the role of test manager! Not only responsible for detecting defects, but he'd also thrown in the vacant build/release manager responsibilities for good measure.

I set my jaw defiantly.

"Steve, you have my word as a gentleman. I can do it."

And I looked into his eyes as we shook on it. There then followed two months of ‘fun and games', as the dev manager put it, as we struggled to fix dozens of defects so severe that we had to sub-classify ‘severity ones' into real showstoppers that would lose the department large quantities of money, and the rest, which were just whole functions that didn't work.

The irony is that while this and several more contract roles as test manager gave me a lot of confidence in doing the job, it wasn't until I began assuring the testing of lots of projects that I really learned what testing was about.

Test manager's grind

The problem was that each time I did the job in hot blood I was simply given a baptism of fire and expected to hit the ground running (which, after all, is what contractors are supposed to do). This usually consisted of writing the test plan document, fighting for test environments, getting the testers to write the scripts, getting the releases built into the test environment, executing the tests as fast as we could, and being ready to re-test as soon as the fixes came back.

The rest of the time was being bollocked by the project manager for not being on plan. I knew nothing about coverage (how many tests, what sort of tests), nothing about priority, and had no idea if we were finding enough defects or the right sort of defects.

What I did know was that all the stuff in the original BS 5750 Part One/ISO 9001 about product identification and traceability was right. Half of the bugs seemed to have been fixed in previous releases; sometimes they were things that had been fixed once already in our release. Trying to find what had been changed from one release to another was a nightmare.

Computing to the rescue

I needed something that made sense of it all; a model that worked. I found it, strangely enough, in this very magazine, Computing. I kept the article for years. It was about the V-model, that paradigm of the good old waterfall method. But I found that it worked every time.

On major integration projects, even if the components have been developed using Agile, the testing in ‘the large' usually follows the V-model. For the uninitiated, the V-model maps the level of testing with the level of decomposition.

The V-model maps the level of testing with the level of decomposition

Once you have worked out what each activity one the right-hand of the V needs to do and produce, you are halfway to producing a test strategy or master test plan. But forget about the designs on the left-hand side. What about the test design?

Analysis, analysis, analysis

I believe that the key activity in the testing life-cycle is test design. And the most important part of test design is the analysis part.

In safety-critical software, every single path that the tester's analysis uncovers must be tested. One of those paths may lead to a plane crash or a nuclear reactor exploding or a missile detonating in the wrong place or time. That's obviously not needed in business or business-process software, but the analysis is. Even if you are only going to run the high-likelihood/high value tests, you still have to know what they all are, in order to prioritise them.

The top-ten tenets of software quality assurance, part six: testing

Testers have to be destroyers, argues Mark Wilson. But too often testers aren't trying too hard to bust the software. Rather, they are trying to show it working

Once when I was QA-ing the end-to-end testing of some very big, complex releases, I looked closely at the end-to-end (E2E) tests in the noble test director (now Micro Focus ALM). The testers were giving the software such an easy ride, I could hear it laughing at them. I remembered a book I had once been shown by a Chartered IEEE software engineer called The Art of Software Testing by Glenford Myers.

The title made you think; maybe testing required insight and a ‘gift', and wasn't just going "1. Login. 2. Open an account. 3. Select option A etc."

I got hold of a copy. It was a game-changer for me. Chapter 2 contains a very amusing exercise you can try on your friends and relatives. It finds out if you'd be any good as a tester, especially the test analysis that I think is the root of effective testing. Anyway, I did the test, I got the E2E testers to do it, got my boss to do it, got some programmers to do it and I got my wife to do it.

The exercise asks you to come up with test cases to cover as many logical conditions as you can find present in a simple program. The program simply draws a triangle when you supply the three dimensions, and then says whether the triangle is scalene, isosceles, or equilateral. Old Glenford reckons there are at least 20 test cases/conditions.

The results of the exercise were a shock.

Do the test yourself! [PDF]

We quickly set in train recruitment to beef-up the E2E testing, did some reviews of the existing tests and enforced reviews of the test steps. But it was three more years before we came up with a test condition matrix for all requirements. This not only isolated all the possible paths, but awarded a business-priority to them, and mapped them to the test specification.

Some of our suppliers at that time were much more rigorous. Doing an audit at a big defence contractor supplier, I asked to see how the test analysis was done. There was a top-down command-and-control structure in operation, and the work of a grunt couldn't proceed without review and sign-off by the team leader. And no test design was allowed until the analysis stage had been done, and all the conditions neatly laid out.

The less this analysis takes place, the greater the risk that the miserable defects lurking in the software won't appear until the customer is using it.

All this reinforces my view (stated in my earlier article on the Formal Review) that the majority of serious defects are to be found at the analysis phase of anything. In a forthcoming article I'll show how simple metrics can tell you what's going wrong with the way you run large-scale system testing.

The mind of a test manager

Some test managers think it's enough to simply be doing the activity, like a worker who thinks they've earned a day's pay by merely turning up; that having some testers and some tests and confirming to the project manager that testing is underway does the business.

In reality, they should feel like they are locked in mortal combat with a release that's probably stuffed with juicy defects, but which they stand little chance of detecting. It's a bit like my first Ford Mustang. It was built in 1966 and had a 5.4 litre V8 that delivered vast amounts of lazy long-stroke low-revving torque, just by pressing the accelerator, which was really a ‘V8-things-happen button'. When you pressed it, massive V8 things happened. So it is with unfocused testing activity. Stuff happens; tests get executed, but it's baggy and loose and simply not directed at smashing holes in the release you're testing.

A deep desire for destruction

I think the motivation for testing is important. A tester is often a frustrated developer; someone who happened to do some testing and found they liked the arse-about-face mindset needed to deconstruct something that someone else has lovingly crafted - a programming poacher turned digital gamekeeper.

Developers are creators; testers have to be destroyers. Their objective has to be de-composing what's been carefully composed. Testers test by looking at the exact requirements, and then working out the combinations of conditions (or paths) that the requirement logically facilitates. They can be decisions, or ranges of data.

Too often, the tests I see aren't looking to bust the software, they're trying to show it working, like some sort of demo with FAQs. You can see this with those all-in-one faux-test cases: the conflation or concatenation of all the myriad logical conditions into one suspiciously handy sequential test script. A sort of everything-in-the-garden's-rosy hike from start to finish, calling at each data in and data out, with stops for the odd expected result.

QA The Barbarian

Sometimes when I've reviewed tests like these, I've asked the tester if they're really trying to break it. For I think there ought to be a real tension between the testers and the developers; the test team of testers really ought to be seen as the enemy by the developers.

One way of doing this is to simply go off-piste. In 2001 I used to dread the arrival on a large government testing project of the business's user-acceptance testing super-user; a person who knew the business processes backwards (I think he wrote most of them).

We'd hit the end of the day, having got the numbers of defects open in System Test to almost the acceptable side of abysmal, and then this person would turn up as my coat went on, flex his wrists and then spend the next four hours absolutely pulling the release to bits with completely unstructured exploratory tests.

I could do a convincing impression of him and got my own back later ordering various unwanted items of food for him, but for a while he had us on the ropes. Ninety per cent of what he found was regressive behaviour; he wasn't even looking at the requirements for the release under test.

Cautionary tales

The money's all gone

This isn't a cautionary tale, but the person who said it was certainly ‘a caution'.

A little handyman once came into a sweaty, window-less room where I was managing the migration testing of a building society's data onto a replacement database.

As he replaced the lightbulb (or did something with caulk or mastic, I forget), he asked, jocularly, if we'd tested it all okay. When I said not yet, he sniggered "But they're still there", pointing at the PCs on our desks.

In fact, his mockery wasn't far wide of the mark. What the testers were doing was manually checking a huge sample of the migrated data on the target test environment, and cleansing the data as they went along. This took forever, but the intensely risk-averse, very old-fashioned and prudent management wanted it that way. These were people's savings, you know; it was the society's money being lent. I found it hard to be cheerful about this dunder-headed way of testing. Not so the head of savings at the board meeting the morning after the big overnight migration of all the millions of pounds' worth of savings:

"So Phil," boomed an old board member to the head of savings, "How did the overnight migration go?"

With a faint wink at me, Phil said, in a low voice, "Not well. All the money's gone".

Now I like a laugh, but these were very dour, serious, upstanding grey-haired people. Just when the board's jaws couldn't drop any lower and I started grinning, Phil smacked his thigh and roared, "Only kidding! It's fine!"

Both he and I left shortly afterwards.

Ten grand a defect

Due to poor test analysis, poor coverage and a lack of financial incentive, I found that end-to-end testing was costing a major systems integrator £10,000 to find each defect. At one of the integrator's partners, they found no defects at all during one quarter, and they both actually said the systems under test must be nearly perfect (despite the frequent emergency patches the live systems required).

A million quid a month

An enormous sum, anyway. This is what a government department (the one I referred to in the opening paragraph) risked over-paying training providers in the first month of go-live due to an embarrassing defect in a little C program. Risk-based testing? This was simply a bit of arithmetic that had escaped unit testing, system testing and user-acceptance testing. It was only spotted when the C-tester came back from holiday.

This column will be taking a brief break, but will return shortly, starting with number seven, configuration management.

Mark Wilson is a Quality Assurance consultant with over 30 years' experience in quality management, quality assurance, test management and configuration management. He can be contacted by emailing [email protected]

Missed the previous instalments?

Part One, the contract review

Part Two, documentation

Part Three, the formal review

Part Four, methods

Part Five, design