On "Normalizing" or "Scaling" Cybersecurity Metrics and Measuring The Right Thing For The Right Entities

Introduction

One challenge of cyber security is deciding where to prioritize one’s limited cyber security resources. Who’s doing “okay?” Who desperately needs help? Who should we just quarantine until they can get their chaotic users under control? At a minimum, can we at least rank countries according to who’s worst/best? We need to have some basis for prioritizing our limited time and resources.

Focusing On Botted Hosts Sending Spam

To make this problem concrete, consider the terrific spam-related metrics shared by the Spamhaus Composite Blocking List (CBL).¹ On Wednesday, March 9th, 2016, the CBL knew about 9,121,043 IP addresses associated with spam-sending botted hosts. That’s a LOT of malware-infected systems!

As shown in table 1, just three countries – India, Vietnam and China – account for roughly 1/3rd of all CBL listings, and a total of just 10% of all countries (20 out of 200) collectively accounted for ~3/4ths of all CBL listings:

Surely we can all agree that those 20 countries represent the “worst of the worst” when it comes to malware infected systems used to send spam? Surprisingly, no.

Normalizing By Country Population

Representatives of some of those “top” countries, perhaps feeling a bit self-defensive, may be quick to point out that they’ve got huge populations, so it really isn’t “fair” to just compare “raw counts” between countries. E.g., India has 1,182,291 CBL listings, but spread over a population of over 1.25 billion Indians, that’s a rate of just (1,182,291 / 1,250,000,000) * 100 = 0.0945%

By comparison, Italy has 178,898 CBL listings, but a population of only 60 million, which yields a rate of (178,898 / 60,000,000) * 100 = 0.2981%.

Dividing Italy’s 0.2981% by India’s 0.0945%, we can see that Italy is currently 3.155 times “more infested” than India on a per-capita basis.

Arguably, then, Italy should “obviously” be prioritized ahead of India when it comes to any hypothetical anti-bot “clean up campaign,” right? Actually, no. Malware infections per capita represent a measure of infection density. Infection density is important if you’re thinking about efficient infection cleanup, but largely irrelevant if your goal is to reduce the impact of the bots on mail servers and their admins.

Ranking According to Actual Pain Delivered Toward A Target

Mail admins running mail servers under siege from spam really don’t care about “infection rates per capita.” They care about the spam traffic they’re seeing. Fortunately, the CBL has data about that as well. Focusing on spam that’s being delivered (vs. botted hosts potentially able to send spam) changes the picture dramatically:

Table 2. Spam Sent To One CBL Spamtrap, By Top 20 Origin Countries, Past Three Days

Now the problem is clearly not the huge number of botted hosts in India, but the huge volumes of spam coming from botted systems in the United States. That is, while there are a relatively small number of botted hosts in the United States, those botted hosts are typically particularly aggressive, sending an average of 231 spam/bot, while by comparison, the botted hosts in India send a measly 4 spam/bot. Treating all botted hosts as if they were essentially equivalent is clearly unwarranted. Some of those infected systems are heavy artillery, while others are mere cap guns.

[We must also remember to take the above numbers with a “grain of salt” since that the statistics in the above table are based on a single CBL spamtrap site, which although they are among the best data currently available, may not be reflective of global spam flows overall]

Teasing Apart US Spam Traffic Sources

Let’s now drill down on spam traffic from US ISPs. Where does the majority of US-origin spam traffic actually come from?

For the purpose of this analysis, ISPs are identified by their Autonomous System Number, or “ASN.” Many ISPs use only a single ASN, but some (such as rr.com) are associated with multiple ASNs, often as a result of legacy networks getting acquired as a result of mergers and acquisitions. When multiple ASNs are associated with the same ISP, we’ve aggregated them for the purposes of this analysis.

Table 3. US ASNs, Ranked By Traffic/ASN, Last Three Days, 500K+ spam/ASN

Unfiltered Pain

There is one other reality that we must remember: the spam that the CBL sees all gets blocked (at least if you’re using the CBL as part of your spam filtering, as many sites do).

Thus, ironically, if we were to prioritize working on the ISPs that are most broadly represented in the CBL, you’d (in some ways) be wasting your time: the spam from those hosts is already getting blocked, at least at sites that use the CBL.

So now you can see the problem. We need to identify the hosts that are successfully delivering spam IN SPITE of block list entries and other anti-spam heuristics.

Documenting the “false negatives” that get through filtering is a hard and largely thankless job, and one that relies on inherently error-prone mechanisms such as users pushing a “this is spam” button, or perhaps the processing of mail streams by multiple categorization engines.

Author: Joe St Sauver, Ph.D., Scientist, Farsight Security, Inc., Member of CyberGreen’s Statistics Experts Group

¹ http://www.abuseat.org/