Head: The inexorable rise of threat intelligence

Watching anomalies is no longer enough. Increasingly, they must be understood

Tweet

These are boom times for threat intelligence, an evolving innovation that has turned network security on its head. Hitherto, defences were static, designed as generalized barriers against attacks whose scale and form, it was assumed, could be predicted with some certainty. Threat intelligence works on the model that attacks are multi-faceted, evolve rapidly, and might even be unique: this requires a constant feed of new data points so that defences can be continuously adjusted.

Despite its success, the term has acquired different meanings, which can cause confusion. For some vendors, threat intelligence is primarily the orthodox alerting layer used to warn defenders of an anomaly in a security system such as an inline firewall or intrusion detection system, or an out-of-band authentication or privilege management system.

For others, it represents an external or even crowdsourced view of Internet threats detected and pooled between vendors, possibly using an agreed sharing protocol, for example news of an exploit targeting a new software vulnerability. More recently it has even come to mean threat research where a company is paid to search for targeted malware campaigns from known cybercriminal groups.

For a growing band of organisations, it often means all of these at once. The important point is that each approach has benefits for given cost. How far a customer goes to expand its threat intelligence feeds is a business decision.

Anomaly detection

At the core of conventional threat intelligence systems is the idea that networks have a baseline of normal operation against which anomalies will stand out. Network security’s simplest layer is a mesh of policies that define what is allowed and what is not, for example forbidden IP sources, applications or connections through ports. But many other troubling events might appear to be perfectly normal – a valid but hijacked credential being used to access a server for instance – while being nothing of the sort.

Threat intelligence is the layer that sifts these ‘permitted’ events so as to make a decision about them. Indeed, this is another way of defining what threat intelligence means in conceptual terms: it is a support system for making decisions, either by an engineer or an automated system. The question is what happens next. Is this event good, bad or worthy of further investigation? Typically, this is done though manual intervention in which a human operator drills down to check on a range of parameters guided by some form of risk scoring.

All major security platforms use this principle in their design but it does (even they would admit), have drawbacks. Although alerting beats static policies – it allows some events to fall into a grey area – it still depends on an idea of normality that can be too easily bypassed by clever attackers.

The best example of this is the way cybercriminals have used credential theft in documented, targeted attacks. An anomaly detection system might not spot stolen credentials as anomalous because, as far as its policy engine is concerned, the credentials are legitimate for the assets being accessed.

And even the best-trained anomaly-detection system will struggle to get it right all the time, which risks generating productivity-sapping false positives. The risk, then, is that the threat intelligence feed is set to be overly permissive.

A further problem is that organisations will have alert data coming at them from multiple systems at once. Loading IT staff with more date feeds data doesn’t tell them which ones are the most important or should be addressed first. Nor does it distinguish between types of threat, for example tactical threats faced every day from longer-term strategic ones. Ultimately, threat intelligence as a purely technical exercise doesn’t give defenders any idea of who is attacking them, which weaknesses they are probing for and whether they’re likely to succeed.

External intelligence

These days, internal live threat data is being complemented with external data feeds, from vendors but also industry bodies and even government CERTs. Once seen as optional, in recent years these have become an essential part of the threat intelligence landscape.

One that every network will use is vulnerability data cross-referenced to news of real-world exploits. Industry systems such as the Common Vulnerabilities and Exposures (CVE) and the Common Vulnerability Scoring System (CVSS) are ostensibly ways of referencing software flaws across vendors and working out which ones are the most serious. But every defender knows they offer an early-warning system for likely incursions. If an unpatched or zero-day flaw is assigned a CVE and high-severity CVSS rating, that is the one attackers will reach for.

Crowdsourcing

Building on this, a second trend in threat intelligence is that it should be based on sharing data rather than monopolizing it. When external threat intelligence feeds first came along, it was in the form of single-sourced streams designed to service one set of proprietary equipment. Each vendor had their own, in some cases aided by anonymized collection from its own customer base.

This was a start but it struggled as cyberattacks rose in complexity, volume and targeting. These were just too many attacks, many crafted to defeat individual defences. It was realized that combining attack and threat data across multiple sources, ideally through an API, offered a better model and so industry bodies and alliances were formed to allow that to happen.

So is threat intelligence just better monitoring? The pay-off with better intelligence is that organisations not only have a better chance of deflecting attacks but of detecting the ones that manage to bypass their defences. In a sense, it becomes not simply about anticipating attacks but responding to them. Speed of response has become so important to modern cybersecurity that, arguably, it is becoming the metric that separates success from failure. The challenge remains integration and assimilation; acquiring more threat sources doesn’t guarantee they will be used effectively.

The future of threat intelligence is that these core processes will become automated. For the foreseeable future humans will design the policies, but increasingly it will be machines processing the data points and making the decisions.