Banking Review

The question of quality

Quality is not about the production and uniformity of your product.
Nor is it about putting the stamp of approval on repeatable ‘crap’
BY STEPHEN WILSON

In the early 1990s, I worked for the first medical device company to be certified under the then new ISO 9000 quality management standard. We were hugely proud, and mounted an expensive sales and marketing campaign to capitalise on our quality mark.

Tragically, within two years, product failures emerged leading to the deaths of several patients. Our customers and sales force were incredulous. What good was ‘quality’ certification if our products were fatally flawed?

We were not alone. By the end of the 1990s, many thought that quality was dead. Famed quality prize winners like Florida Power & Light abandoned their ISO 9000 programs because they weren’t realising measureable financial benefits. A half-hearted defence was sometimes proffered: quality management was really all about process and uniformity; it did not actually guarantee what laypeople would call ‘quality’. Cynics in turn countered that if a product spec was crap, then ISO 9000 would ensure the manufacturer turned out repeatable crap.

There were deep debates about whether ISO 9000 really constituted a ‘standard’ in the sense that engineers recognise. And in the wake of legal action over product failures, there was much hand wringing over what quality audits really mean.

It is said that those who cannot remember the past are condemned to repeat it. I defy any info-sec practitioner to read the 2002 paper Is ISO 9000 really a standard  and not see the parallels with security practice today. Note especially this quote: “We have bred a generation of quality managers, consultants and auditors which believes that ISO 9000 is mainly about conforming to the requirements”.

I’m afraid security professionals are increasingly falling into the same trap: management by rote.

I had a telling argument with a senior PCI auditor only this month about those infamous and perplexing cases where PCI compliant organisations have been badly breached. He tried to explain the contradiction by saying, “Of course, they were only compliant at a particular point in time”. To which I responded, “But what else can a PCI audit mean?” That is, if a security audit tells us nothing about how an organisation normally operates in between audits, then what good is it really?

The backlash against formulaic security has started. In March 2009, the US congressional Homeland Security Committee held a hearing entitled Do the PCI Data Standards Reduce Cybercrime? The committee’s internal investigations showed that: “the PCI Standards are of questionable strength and effectiveness”. The chair summed up the situation thus: “The essential flaw with the PCI Standard is that it allows companies to check boxes, but not necessarily be secure. Compliance does not equal security. We have to get beyond check box security”.

And it seems that company boards at large are losing interest in info-sec, contrary to the popular belief that security and risk have never been more important. Carnegie Mellon’s Cylab undertakes an annual survey of corporate governance of digital assets. Their 2010 report, Governance of Enterprise Security  found that: “When asked to identify their boards’ three top priorities, ‘improving computer and data security’ was not selected by any respondent” (emphasis in original).

Security management by rote
The real trouble is that we’ve tried to mechanise security management, as if it were just like generic manufacturing. We use essentially the same processes and policy templates for all businesses. Don’t get me wrong: process is important, and we do want our security responses to be repeatable and uniform. But not robotic. For the truth is, there is no algorithm for doing the right thing.

An algorithm is a repeatable set of instructions or recipe that can be followed to automatically perform some task or solve some structured problem. Given the same conditions and the same inputs, an algorithm will always produce the same results. This is a good thing in a predictable world, but no algorithm can cope with unexpected events; an algorithm’s designer needs to have a complete view of all circumstances in advance.

Mathematicians have long known that some very simple tasks cannot be done algorithmically. The classic ‘travelling salesman’ problem, of how to plot the shortest course through multiple connected towns, has no single recipe for success. There is no way to trisect an angle using a compass and a ruler. There is no consistent way to tell if any given computer program is ever going to stop.

So what makes us think that auditing a real-life information system with a check list, no matter how long, will tell us if it is secure?
What’s the alternative? It’s challenging and it’s ill-defined, but we need to think outside the checkbox.

Like any complex management field, security is all about problem solving. There’s never going to be a formula for it. Rather, we need to put smart people on the job and let them get on with it, using their experience and their wits. Good security like good design frankly involves a bit of magic. We can foster security excellence through expertise, teamwork, research, innovation and agility. We need security leaders who have the courage to treat each new threat on its merits, trust their professional instincts, try new things and break the mould. And recognise management fads for what they are.

Stephen Wilson, founder of the Lockstep Group, is an analyst, consultant and innovator in digital identity and privacy.