This is a few days old, but I didn't mention it when I first saw it: Microsoft Exploit Predictions Right 40% of Time, from Slashdot. (The link is to Slashdot; here's the original article.)
Apparently Microsoft has a mechanism for predicting which parts of its code are most likely to be exploited by malevolent hackers; they made nine predictions in October, and four actually got exploited. Of course, the Slashdot commentary is filled with reflexive Microsoft-bashing, and people pointing out that that's worse than flipping a coin. But the correct comparison isn't to flipping a coin. The correct comparison is to the number of hits that would have been obtained if Microsoft had picked nine pieces of their code at random, which presumably is much less than four. There are a few people in the comments trying to put numbers on this, but nothing that really sounds informed.
Subscribe to:
Post Comments (Atom)
3 comments:
Yep. While approximately 40% of their positive predictions were correct, we have no idea how many pieces of code were exploited that were NOT predicted. If we knew that number, and if we did a comparison as you describe, THEN we'd actually be able to conclude something.
This is just a corollary of the well-known theorem: "people on Slashdot don't know anything".
Apparently Microsoft has a mechanism for predicting which parts of its code are most likely to be exploited by malevolent hackers [...]. The correct comparison is to the number of hits that would have been obtained if Microsoft had picked nine pieces of their code at random, which presumably is much less than four.
I believe this should be clarified. The Microsoft Exploitability Index assigns a number 1, 2 or 3 to known vulnerabilities (they have a CVE!) related to Microsoft software (unless a public exploit is already available for said vulnerability, then no score is assigned). Often the vulnerabilities are reported by external parties, but in any case it's not about predicting random code, it's about tristate classifying already identified bugs. According to the linked article, the predictions are valid for 30 days or until a new set of patches is released. Since the group manager at Microsoft Security Research Center agreed to do a postmortem evaluation, we may assume that the lifetimes of the October predictions have expired.
In October they had 20 such bugs: nine were tagged "1" (consistent exploit likely), seven tagged "2" (inconsistent exploit likely) and four tagged "3" (functioning exploit unlikely). Four of the nine "1":s got exploited, zero of the "2":s and zero of the "3":s. It's up to you if you consider this good or bad, but clearly they did well by not incorrectly tagging a vulnerability as a "3" -- that would probably be the worst mistake.
I'll leave it to those interested to play with the combinatorics. Finally, people at Slashdot bash Microsoft for fun: correctness has nothing to do with it. (It's been that way since the start.)
Post a Comment