Moneyball in Infosec

“It is unbelievable how much you do not know about the game you have been playing all your life.” - Mickey Mantle

‍

That line from the opening scene of Moneyball hits harder the older I get. The book and the movie were not really about baseball. They were about how an entire industry can convince itself that it understands reality while systematically ignoring the only variables that actually matter. Baseball scouts trusted their eyes, their instincts, and what they believed was their experience. They looked at a player’s face, his posture, his swing, the way his arm moved when he threw. All of it felt like useful information, and yet almost all of it turned out to be wrong. It was a series of logical fallacies.

‍

The first fallacy was an appeal to authority, which makes sense because scouts are, afterall, tasked with the job of “looking” at players and deciding if they are team material or not. Why would you not trust the authority of someone who had the exact job required to find the talent? The second fallacy was an argument from tradition, where scouts evaluated players the way they always had because that was how the game was always played. There is also the availability bias, where scouts remember the big plays but forget about the small ones. There are others too!

‍

But what Billy Beane of the Oakland A’s and a small group of heretics figured out was a brutally simple concept. Getting on base and limiting outs mattered more than anything else, no matter how ugly it looked. It wasn’t how confident the scouts felt in the player, just whether the player reliably avoided making an out and getting on base. For every dollar spent on talent, the only rational question was how many times that dollar translated into someone standing safely on a base. The Oakland A’s competed with teams that had multiples of their budget and nearly beat all of them by refusing to play the same game. The Oakland A’s had to play a very different game - a game of money and runs. In doing so, they permanently changed baseball.

‍

Infosec is still in its pre-Moneyball era.

‍

We have built an entire industry around expert opinion. We trust the Infosec equivalent of scouts. Vendors, researchers, consultants, and internal teams are all telling stories about what matters based on what they are good at finding or what they have always believed. We stare at CVSS base scores the way scouts stared at batting stances. We argue endlessly about whether every vulnerability matters. We confuse identified hacker activity with loss, and vulnerability with risk. Security programs inherit checklists, scanning best practices, and prioritization models simply because they are the way we have always operated. The fact that something has always been done that way becomes its own justification. Security pros remember the big CVEs that got them hacked, but forget about the hundreds of thousands of useless ones that their teams are pointlessly working on, that never harmed a soul. You see? It is the same set of logical fallacies!

‍

We want to create Infosec Moneyball.

‍

The game we think we are playing is vulnerability reduction. The game we are actually playing is loss prevention, or maybe breach-prevention, if you want to widen the aperture. At Root Evidence, we realized our industry was playing a very silly game. The uncomfortable truth is that the only thing that matters is how much money you are spending to reduce some quantified loss. If that proportion is less than 1, then you should do it; if it’s not, you shouldn’t. That is the equation. Dollars spent on security versus dollars not lost by the organization.

‍

There are nuances that are worth writing here because people will complain if I don’t mention them. There are attackers who pre-position quietly for future leverage. There are nation states who are not seeking immediate financial loss at all. Those are real, serious threats, and they deserve consideration. But, for the vast majority of companies, this is a straight Moneyball problem. They can choose to spend enormous sums chasing the theoretical universe of more than three hundred thousand CVEs, most of which have no bearing on their actual risk profile. Or they can right-size their security program to focus narrowly and ruthlessly on what prevents loss.

‍

Like the Oakland A’s, this almost always means spending less, not more. And like the Oakland A’s, it means going directly against conventional wisdom. It also means that we become the heretics, replete with the army of nay-sayers who will undoubtedly follow, until they, finally, begrudgingly realize that we’re right when it finally hits their pocketbook to be wrong.

‍

It means rejecting the idea that a mature security program must look a certain way or generate a certain volume of findings, or fix anything above some arbitrary grade or score. It means turning the entire mental model upside down and asking a question that makes many people uncomfortable. Does this reduce loss or not?

‍

I like to pick on the humble, slightly weak SSL/TLS certificate as an example because it exemplifies the absurdity of our current approach. Yes, something is technically wrong with a weak SSL/TLS cert. No one disputes that. But if no adversary is attacking it and no loss is occurring, or likely to occur, then treating it as a meaningful risk today is pure security theater. Maybe one day it will matter a great deal. When that day comes or when we see strong signals that it is being exploited elsewhere, we should fix it immediately. Until then, spending time and money on it is indistinguishable from a scout obsessing over the shape of a player’s jawline, in terms of efficacy.

‍

I am partially to blame, make no mistake. I surfaced a great many exploits and issues that never were meaningful over my 30 years in cybersecurity. Very few of those vulnerabilities that I created were ever taken seriously by adversaries. So blame me. But like Billy Beane, I know there is a better path, no matter how many mistakes I made along the way to get here.  In fact, I may not have gotten to ‘here’ without making those mistakes.

‍

Moneyball did not make baseball less rigorous by virtue of not over-spending like it had previously, it made it more honest. Infosec needs the same reckoning. We need to stop inventing new variables that have nothing to do with loss and stop rewarding systems that generate noise instead of outcomes. The goal is not to find everything. The goal is to prevent damage/loss. Once that becomes the only thing we measure, the rest of the industry will look very different, very quickly.

‍

Decades in infosec has shown me how little I actually knew about cybersecurity.

‍

‍