Elastic accelerates SIEM data onboarding with Automatic Import powered by Search AI

Elastic is accelerating the adoption of AI-driven security analytics by automating SIEM data onboarding with Automatic Import. This new feature — the only one of its kind for a security analytics or SIEM solution — automates the development of custom data integrations. Elastic Security now adds custom data sources faster than any competing security analytics solution, facilitating broader visibility and easier SIEM implementation. 

Establishing visibility across an enterprise IT environment is inherently difficult, but no matter how the attack surface changes — applications created, systems added, infrastructure moved to the cloud — security teams can’t afford to fly blind. Unfortunately, onboarding custom data has remained costly and complex — until now.

Automatic Import automates the development of custom data integrations with generative AI, cutting the effort needed to create and validate custom integrations — from up to several days to less than 10 minutes — and significantly lowering the learning curve for onboarding data. The feature is powered by the Elastic Search AI Platform, which provides model-agnostic access to harness the knowledge from large language models (LLMs) and the ability to ground answers in proprietary data using retrieval augmented generation (RAG). It is also made possible by our rich expertise in enabling security teams to leverage data of any kind and the flexibility of our Search AI Lake.

Automatic Import arrives at a critical moment as organizations explore replacement options for their legacy SIEM tools. Collecting and normalizing data is among the first phases of any migration plan, starting with leveraging prebuilt data integrations. Technologies that require custom connectors typically come next, but the manual nature of building each such integration can slow adoption of the new SIEM and retirement of the old solution. Automatic Import addresses these challenges.

Elastic ships with 400+ prebuilt data integrations and counting, and Automatic Import makes it practical to extend visibility beyond these to an evolving array of security-relevant technologies and applications. These integrations normalize data to Elastic Common Schema (ECS), enabling uniform analysis with dashboards, search, alerting, machine learning, and more. Public LLMs can readily process and analyze data in ECS format because it is a popular open source data specification.

Automatic Import is easy to use and available to everyone with an Enterprise license. The user specifies some settings and uploads sample data from which the feature will extrapolate what to expect from the data source. These log samples are paired with LLM prompts that have been honed by Elastic engineers to reliably produce conformant Elasticsearch ingest pipelines. Automatic Import then iteratively builds, tests, and tweaks a custom ingest pipeline until it meets Elastic integration requirements.

In just minutes, the feature generates and validates a custom integration that accurately maps raw data into ECS and custom fields, populates contextual information (such as related.* fields), and categorizes events.

Elastic Automatic Import supports a variety of structured and unstructured log formats, including JSON, NDJSON, Syslog, and CSV. Popular large language model (LLM) providers, such as Amazon Bedrock, Google Gemini, and OpenAI, are supported by Automatic Import for integration generation.

Provide a name and description for the new data source.

Next, fill in other details and provide some sample data, anonymized as you see fit.

Click “Analyze logs” to submit integration details, sample logs, and expert-written instructions from Elastic to the specified LLM, which builds the integration package using generative AI. Automatic Import then fine-tunes the integration in an automated feedback loop until it is validated to meet Elastic requirements.

Automatic Import presents recommended mappings to ECS fields and custom fields. You can easily adjust these settings if necessary.

After finalizing the integration, add it to Elastic Agent or view it in Kibana. It is now available alongside your other integrations and follows the same workflows as prebuilt integrations. 

Upon deployment, you can begin analyzing newly ingested data immediately.

Automatic Import lowers the time required to build and test custom data integrations from days to minutes, accelerating the switch to AI-driven security analytics. The feature arrives during a time of change in the SIEM market with many longtime customers of legacy SIEMs now migrating to modern technologies.

Elastic pairs the unique power of Automatic Import with Elastic’s deep library of prebuilt data integrations, enabling wider visibility and fast data onboarding. In conjunction with Elastic AI Assistant for rule conversion, the feature substantially simplifies SIEM migration.

Interested in our Express Migration program to level up to Elastic? Contact Elastic to learn more.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.