Loading

MISP Integration

Serverless Observability Serverless Security Stack 9.0.0

Version 1.41.0 (View all)
Subscription level
What's this?
Basic
Developed by
What's this?
Elastic
Ingestion method(s) API

The MISP integration uses the REST API from the running MISP instance to retrieve indicators and Threat Intelligence.

The MISP integration configuration allows to set the polling interval, how far back it should look initially, and optionally any filters used to filter the results.

The filters themselves are based on the MISP API documentation and should support all documented fields.

The MISP integration configuration allows to set the polling interval, how far back it should look initially, and optionally any filters used to filter the results. This datastream supports expiration of indicators of compromise (IOC). This data stream uses the /attributes/restSearch API endpoint which returns more granular information regarding MISP attributes and additional information such as decay_score. Using decay_score, the integration makes the attribute as decayed/expired if >= 50% of the decaying models consider the attribute to be decayed. Inside the document, the field decayed is set to true if the attribute is considered decayed. More information on decaying models can be found here.

The ingested IOCs expire after certain duration which is indicated by the decayed field. An Elastic Transform is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named logs-ti_misp_latest.dest_threat_attributes-* which only contains active and unexpired IOCs. The latest destination index also has an alias named logs-ti_misp_latest.threat_attributes. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for Threat Attributes datastream are also pointing to the latest destination indices containing active IoCs. Please read ILM Policy below which is added to avoid unbounded growth on source datastream .ds-logs-ti_misp.threat_attributes-* indices.

By default, the integration uses incremental updates, only fetching attributes that have been modified since the last poll (tracked via an internal cursor). However, MISP's decay scores are dynamic and decrease over time, which means an attribute's decay status may change without the attribute itself being modified. In such cases, incremental updates would not capture the updated decay state.

To address this, users can enable the Enable Daily Refetch toggle. When enabled, the integration will:

  1. Perform a daily full refetch: Every 24 hours, the cursor is reset and all attributes from the configured Initial Interval are re-fetched from MISP.
  2. Update decay states: Thanks to the re-ingest of all attributes with their current decay scores from MISP, it removes any that have since been marked as decayed from destination indices.

This approach ensures that:

  • The destination indices stay aligned with MISP's current view of valid indicators
  • Attributes that become decayed in MISP are automatically removed in the next refetch cycle from destination indices
Note

This mode will re-ingest all attributes within the Initial Interval window, which may result in higher data volume during the refetch period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion will be removed immediately.

Some IOCs may never get decayed/expired and will continue to stay in the latest destination indices logs-ti_misp_latest.dest_threat_attributes-*. To avoid any false positives from such orphaned IOCs, users are allowed to configure IOC Expiration Duration parameter while setting up the integration. This parameter deletes all data inside the destination indices logs-ti_misp_latest.dest_threat_attributes-* after this specified duration is reached, defaults to 90d after attribute's max(last_seen, timestamp). Note that IOC Expiration Duration parameter only exists to add a fail-safe default expiration in case IOCs never expire.

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_misp.threat_attributes-* are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.