Tuesday, April 29, 2008

Because Hackers Don't Care... (Why Metrics Don't Work)

Lets start with some statistics:

99% of all workstations with up-to-date antivirus
Antivirus blocks over 99% of all malware.

That is amazing! That is great stuff to show the IT Director, CIO, CSO, mom and to put on the wall. But, yet, a company I know (not the one I work for) still managed to get a virus which brought about some painful downtime.

The virus was one of the 1% that the antivirus doesn't block and it spread through the organisation like wildfire. Essentially the saving grace was that it infected a small part of the network, brought that down and didn't spread from there. Luck. It was also non-destructive other than network downtime. Luck.

The metrics lied.

You could say that there was residual risk but it really looks quite small. What is 1% between friends? But that 1% is precisely what any hacker (or virus writer etc) worth his salt is targeting.

So, where to from here?

I won't throw the baby out with the bathwater. 99% of PCs with antivirus is certainly safer than 50% or 0%. 99% of PCs fully patched is safer then 70% or even 100% of PCs almost fully patched. But 99% of PCs with antivirus is not a guarantee that no virus will find its way to destroying your network. It is important that your boss(es) know this and more important is that you know this.

And have plans in place when the 1% risk becomes reality.


Unknown said...

Metrics don't lie, they're just interpreted badly.

Richard Catto said...

I have had this happen to me once before. I got a virus even though I have up to date anti-virus software installed on my PC.

I caught the virus via a web page with malware on it.

I had to discover how to rid my machine of the virus manually. I found instructions online, but it took a while before I managed that feat.

The virus blocked my access to certain known anti-virus web sites.

Unknown said...

I look at this differently. To me, metrics are useful when the context is understood in a way that allows a story to be told.

In this case, the metrics seem to be flawed - but I would argue the wrong things are being measured. Stated differently, the challenge is knowing what to measure and how to find it.

Unknown said...

This is just a simple example and I'm sure a lot of InfoSec departments measure % machines with up to date virus signatures.

The fact is that 90% is better than 80% but 99% is not nearly as good as 100%.

Anonymous said...

The metrics are not lying, just the understanding of the quantitative methods used to determine them is lacking.

The methods and means of deteriming valid metrics are perception based and are not scientific.

There are many risk methodologies that are valid and which have been deployed in finance and operational engineering for decades.

Come on, even marketing is jumping on the use of analytics and we in IT are being left behind.