The top-ten tenets of software quality assurance, part five: design
Design is all about applying the fundamental principles of engineering, believes Mark Wilson, in the latest in his ten-part series
I can't design software; I'm not a software architect, I'm a QA manager. But I've knocked about with people who are good designers, and I learned to read designs and code to see if they complied with the principles of software design (which themselves derive from well-established engineering design), which hopefully are contained in design standards.
I do a love a good design. I like the way that the words ‘ingenious' and ‘ingenuity' come from the same Latin root as the word ‘engineer'. In Germany they're called Ingenieurs, a professional title protected by law. So on your passport or on a form, as well as Mr, Mrs, Ms, doctor they have an option for Eng.; if you're lucky enough to be one, you will be formally addressed as such.
Nice to know
I got shown what a well-designed program looked like in my first term at Polytechnic (they were called that then because they were designed to get you into a technical career, not to educate you). I was excited. There was this thing in the Pascal programming language called ‘Procedures'. You defined this routine once, and you could call it in every time you wanted it.
This re-usability immediately appealed to me. Then they introduced us to If, Then, Else, While.. and Case statements; Records and Arrays. While I was able to write working examples of all these things, and was shown how to lay-out and nest code properly - and also how to comment properly - I knew I wasn't up to being a programmer or software engineer.
That's why I got into QA.
At my first proper job at a University of Leeds-owned software consultancy, I was introduced to proper software engineering by about 20 gifted professors, doctors and chartered engineers from the world of CAD-CAM. Everything about these people shouted "Thorough! Clear! Measured! Clever!".
They had even picked the programming language Ada for their government-funded endeavour because it
a) Had a validated compiler and was an ANSI/MIL-STD;
b) Was strongly-typed; and,
c) Had separately-compilable packages.
It was also very easy to understand; even I could read a program and tell what was happening. There was a wonderful atmosphere in this semi-academic/semi-industrial enclave; they encouraged you to ask questions and were happy to show you how things worked. They even sat me down in front of a video of a Californian weird-beard called Grady Booch who was showing what rip-roaring things could be done with Ada (I later had him to thank, too, for the Unified Process, which I was pleased to use on several big projects, and which I used to devise various quality gates).
I learned how standard Ada Packages could drive code re-use to great heights. The CAD-CAM consultancy ended up releasing commercial products that achieved as much as 80% re-use.
Big bad world
Being around good designs became the norm, so when I was eventually cast adrift into the big, bad world of commercial software, it came as a shock.
I began seeing huge legacy programs running on ICL mainframes; raggedy, arcane programs running on Vax mainframes. I saw C-code and I remembered how I'd loathed it, compared to Pascal. Pascal was like the George Orwell of code - as clear as a pane of glass. C, Kernigan and Ritchie style, was like the spidery scrawl of a crazed adolescent holed up in a garret.
For the first time I began to understand how utterly horrible it must be for software maintenance staff, when there is either no documentation or ancient documentation, tasked with deciphering the accretion of years of hurried, badly-thought out kludges.
Y2K came, date-fields got changed, but on it goes and still the stuff runs on mainframes used by nearly all the major banks. In fact, some estimates put the total number of lines of COBOL code at 200 billion - yes, even today. I got a good insight into design ten years later from one of those old-timers who helped write those zillions of lines of COBOL (see my Cautionary Tales below).
An antidote to chaos
So to me, design isn't about how ingenious or flash it is: it's about something that's understandable and decipherable; it's something that can be explained in English. Indeed, something that's clearly and consistently documented. As software guru Martin Fowler said in Refactoring: Improving the Design of Existing Code, 1999, "Good programmers write code that humans can understand." In other words, something that's maintainable.
Maintainability
I may have never have written a functioning program for money, but as a QA person I am a design stakeholder. And my watchword for design is maintainability: enhance-ability, portability, adaptability. Just as it's imperative - to save money - to discover defects at the requirements stage, it's also imperative to save money by making software easy to maintain. I'm not talking about something that's done for a one-off; I mean stuff that's going to be around and will need re-visiting. And people should never think that this isn't going to happen (95 per cent of all US ATMs run legacy COBOL, and most of the people that wrote it are retired).
People save time, in the short term, by not commenting code properly, not laying it out to make it more readable, not addressing compiler warnings, not identifying the reasons for changes in code control databases and sneering at documentation. The modern phrase for storing up horrible these problems like a time-bomb is ‘technical debt'. Well, like real debt, technical debt gathers interest and always has to be paid back.
As I hope I'm making clear in these articles, poor quality is short-termism and costs more in the long run. Hard to maintain software costs four times as much to maintain as good quality software (ref: the Systems Sciences Institute at IBM, 2017).
Humility
This may strike you as odd, but I have found that successful projects often have people on the design side who are honest about their profession's limitations and dismissive of hubristic designs. Most developers and programmers aren't wizards or magicians: it's less a case of Dungeons and Dragons; more digestives and back-of-a-fag-packets.
They will readily acknowledge that a database is really just a big structured list of things, and things about those things. And that in the business world, there's not much real-time or safety-critical or complex linear algebra stuff going on, not much, er, rocket-science.
The top-ten tenets of software quality assurance, part five: design
Design is all about applying the fundamental principles of engineering, believes Mark Wilson, in the latest in his ten-part series
Cautionary tales
All COTS together
I was once lucky enough to work on a massive programme for the NHS. It's still going, but was brutally re-architected a few years ago, reducing the size of the behemoth it had become by 80 per cent. Some 15,000 person-hours alone had been expended pursuing some of the worst designs ever to come out of Rational Requisite Pro, a requirements management tool developed by Rational Software (now owned by IBM).
The core of the system was a program that would field incoming messages and route them to the correct destination, based on their definition in a message data standard. There were one million users. The details of, err, 60 million patients would be held in a very big database. There would be ancillary things like role-based access and accountants' access to data.
For the prototype, a simple design using a piece of commercial off-the-shelf software (COTS) was knocked together, and this worked well enough to win the contract to develop the full system.
I was full of enthusiasm - absolutely pumped-up. I couldn't wait to see the design, but was struggling to understand why I never got to see a simple depiction or description of what the system would be. Everything I saw just made it seem more complex and esoteric.
Service (i.e. ops and support) architects architected massive diagrams. As QA manager, I started asking for an overall design, but was told there wasn't really one - no code would be written and it was just a simple integration of sophisticated COTS products. Despite these airy promises, the lack of a feasible, coherent design meant that in a couple of years, an Indian subcontractor would nevertheless go on to write almost a million lines of code.
Get real, Mark
Still, it was an exciting opportunity to put everything we'd all been learning and doing for the past decade or so, surely? Hundreds of eager analysts, architects, designers, coders, testers, all now working on the biggest civilian IT project in the world… I thought so, until one day, checking for design and coding standards, I asked to see one of the programs that implemented role-based access.
"Er, it's a very long program, this, isn't it? And are they global variables? Is this just one long program? Doesn't call any sub-routines?" I noted a tab-key frenzy of nested if-then-else statements.
"Look, Mark, this is the real world. You do what you have to do."
I was astonished. This would have failed one of the earliest exercises we did on Pascal programming.
I got used to hearing this. Instead of seizing this once-in-a-lifetime opportunity to write great software, they were feverishly trying to cram anything that would work into the time they had. I also realised that the reason there were so many hugely expensive COTS products being shoe-horned together was to justify the price that had been won.
The limitations of the designs became horribly clear when the company was tripped up by one of the worst software practices there is: hard-coding values rather than reading them in from a table. Every single message value (of hundreds taken from a published Message Information Manual) was hard-coded into the very long transactional messaging program. And it had eventually to handle five million transactions a day.
One day, very near the go-live date, the NHS decided to issue a new version of the Manual. No amount of punch-balls, fake stand-ups and break-out areas was gonna be up to it. If they had designed this thing with a table containing the values, it would have been a comparatively short job to make the changes. But it was a nightmare. An old sweat programmer told me later he reckoned he could have written the glorified look-up table in COBOL in an afternoon.
Too big? Cut it in half
The design issues continued to beggar belief. The patient database was to be implemented in Oracle. Because there were 60.5 million people to go onto it, it would be the biggest database in the world. Oracle therefore refused to support it. So they all decided a good way to deal with this was…cut it in half.
So that operation meant that on day one half the database was used, then the next day the other half was used. On the third day (I forget how) it all got copied over so that, in effect, to meet the contract requirement, the database was as one. Except that this design didn't specify the performance of the cut-in-half rigmarole. And so, it stopped working when live-like data was loaded onto it.
Catch-22 mid-air disaster
On my first contract, I had to get the software department of an energy utility the ISO 9001 badge.
I asked the leader of a team adding functions to a billing system to show me one of his designs. Standing there with a Nokia 909 mobile phone in a leather holster on his hip, he confirmed he really was a cowboy by declaring his team "Didn't need no designs".
"So what is it, John...just one big program, or does it have sub-routines and things?"
"We just add this bit on to what's there. We don't want need telling how to program."
But this attitude to endlessly kludging a past-it system was common.
After about eight years of constantly finding fault with a client's loan-servicing system and its quality problems, pushing for better version and change control, a patient but harassed project leader eventually told me why the thing was always out-of-whack and was always regressing following changes and always needed re-work.
It wasn't software source control; it wasn't the changes my company were specifying and testing for the client. These were mere band-aids. Like the scene in Catch-22 when Captain Yossarian thinks he has done a good job dressing the light wound on Snowden's leg, only to find that he's missed the lad's entrails spilling out of his flight-jacket, I learned the grisly truth: hundreds and hundreds of changes had been applied to an unfit and out-of-date system over many years, which was nonetheless business-critical and implemented a very profitable model.
Each change was unique to each operational country (about 20 of them) and was applied in a baffling sequence in order to match key business schedules. The build-path, the dependencies between components, and the back-and-forward compatibility was so complex that no-one actually knew what versions of what worked with other versions.
I almost did what Yossarian does with the dying airman and said, "There, there...".
Anyway, more QA incorrigibilities next time!
Mark Wilson is a Quality Assurance consultant with over 30 years' experience in quality management, quality assurance, test management and configuration management. He can be contacted by emailing [email protected]
Missed the previous instalments?
Part One, the contract review
Part Two, documentation
Part Three, the formal review
Part Four, methods