EQL for the masses

EQL, or the Event Query Language, is an elegant, powerful, and extensible language built in-house at Endgame to express relationships between security-relevant events. We designed it from the ground up to be generic, apply to multiple use cases, and avoid reliance on any particular architecture.

EQL is part of the core technology that drives our endpoint security product. It powers high confidence detections that run on customer endpoints, it is used to perform basic searching, and it gives hunt teams the tooling necessary to sift through massive amounts of data across their endpoint data and detect intrusions. But, there’s nothing inherently tied to Endgame with the language itself.

Up until now, the only way to use EQL was via the Endgame product. That changed today with the public release of EQL. This release includes the core EQL language, a schema mapping to Sysmon, and a set of analytics initially focused on Atomic Blue.

Why did we build and open source this?

It’s becoming a cliché, but it’s true: looking for signatures of known malware or infrastructure IOCs is helpful, but it’s not enough. Security practitioners must assume that adversaries have breached defenses and are conducting operations inside their networks. Because of this, security teams must not only worry about prevention, but also about how they can detect activity post-compromise.

The security community has rallied around the MITRE ATT&CK framework as an important knowledge base of post-compromise adversary activity. We’ve been thrilled to see rapid growth in the number of security teams and researchers working to understand the detection opportunities and challenges in the post-compromise space. However, we note (as do many others) that the tools available to our community for universal expression of post-compromise analytics have some serious limitations which we’ll describe further below.

Arguably the most significant challenge has been the coupling between everyone’s unique data sources and their analytics built on top of that data and corresponding schema(s). This has made it difficult to share actionable analytics between teams. After reading our introductory EQL blogpost, taking our product for a spin, or watching Endgame researchers talk about EQL, a number of researchers began talking to us about how EQL could fill this gap. This matches internal observations we’ve had for over a year - that if we release it to the community, EQL will improve the community’s collective ability to express detection logic and share amongst teams.

While EQL has been linked to the Endgame product, there are no inherent ties or dependencies. We of course natively support a mapping to our rich endpoint-centric dataset, robust query support, and automated detections via EQL in our powerful architecture. However, mappings are possible to any security dataset, and we believe now is the time to allow people outside our customer base to use EQL.

With this release, we are providing the core language, a sysmon integration, and a python-based EQL engine, which includes CLI functionality to run EQL queries over JSON. We’re also providing example analytics which are described later in this post. With this toolkit, users can immediately begin prototyping analytics in EQL.

EQL advantages

We are not here to criticize other technologies and projects which have sought to achieve goals like common schemas, querying structure, and other things similar to what we’ve done with EQL. EQL has some significant advantages which we hope cause people in the security community to take a closer look, even if other technologies are being used or considered. These include:

• Supporting the necessary logic to express relationships between security-relevant events. EQL can compare values and fields with exact or wildcard matching and supports basic AND, OR, NOT boolean search operations. Fields can be compared to strings, integers, decimal values, and against other fields. Most importantly, matching is enabled across a series of events including different types of events (e.g. process, file, and registry) rather than matching on only a single event.
• Minimal learning curve to write analytics. EQL looks like many other query languages. It is intended to search across structured data in an intuitive manner which is highly conducive to quickly writing behavioral analytics. This also leads to excellent readability of each analytic. It also supports traditional IOC searching, but EQL makes it easy to accurately describe activity and behavior beyond simple IOCs.
• No dependence on particular data sources or schema. Other technologies are tied closely to a given data source, and those writing analytics need to focus heavily on that particular data source and schema to write an analytic instead of just focusing on the logic they’re trying to express. EQL’s method of abstracting data sources via extensible schema mappings is powerful and allows for easy use without any need to pre-normalize data.
• Built to hunt. EQL includes strong native post-processing capabilities such as sorting, filtering, and stacking, which allow a user to easily filter out noise. Its schema translation capability also makes it straightforward to extend across multiple data sources without a need for data normalization. Data normalization into the universal schema is supported for users who want to eliminate the need for query-time normalization.

EQLLib and Atomic Blue

We’re providing robust documentation on the many interesting EQL primitives and operators which are part of this release. We wanted to go further than just unleashing EQL on people in the form of a tool and documentation alone. To that end, we have provided a rich set of analytics called EQLLib to help people become familiar with the language and try things out.

We’re huge fans of our partner Red Canary’s Atomic Red Team project. It is the most well-known and expansive test framework of adversary techniques out there. Adversary simulation projects are generally great in that they allow teams interested in detection aligned to the MITRE ATT&CK matrix to easily generate artifacts on systems.

Atomic Red does a great job generating artifacts for the majority of techniques described by ATT&CK, but there’s no expansive mapping of Atomic Red into data source-agnostic language that a defender can action. Atomic Blue starts to fill this gap. Atomic Blue is a curated set of EQL logic which describes how to find endpoint artifacts associated with execution of a significant number of techniques covered by Atomic Red Team.

You can find Atomic Blue within the EQLLib repository. This initial analytics repository is significant, but we’ve only scratched the surface of what’s possible, and we look to the community to help us expand even further.

What’s next for this project?

This initial release is the beginning of the journey, not the final destination. Next week, you’ll see an expanded “how-to” guide along with some data to make it even easier to use EQL. To expand the applicability of EQL, we plan to release support for additional data sources and technologies in the coming months. You’ll also see us releasing additional analytics on a regular basis as the technology expands and evolves.

We’re looking forward to feedback and contributions from others. We strongly believe that we’re collectively lacking a good way to describe detection logic universally across datasets, and EQL is a great way to address this and other limitations. Please try it out and let us know what you think.