Indexing data into Elasticsearch
editIndexing data into Elasticsearch
editBy now you’ve probably spun up a deployment and might be wondering what’s next. Congratulations on completing that first big step! Now let’s help you do something with it. You likely have data that you want to add, known as ingesting or indexing, to Elasticsearch, so let’s explore some options.
Best practices for managing your data
editThis section provides some best practices for managing your data to help you set up a production environment that matches your workloads, policies, and deployment needs.
Plan your data structure, availability, and formatting
edit- Build a data architecture that best fits your needs. Your Elastic Cloud Enterprise deployment comes with default hot tier Elasticsearch nodes that store your most frequently accessed data. Based on your own access and retention policies, you can add warm, cold, frozen data tiers, and automated deletion of old data.
- Make your data highly available for production environments or otherwise critical data stores, and take regular backup snapshots.
- Normalize event data to better analyze, visualize, and correlate your events by adopting the Elastic Common Schema (ECS). Elastic integrations use ECS out-of-the-box. If you are writing your own integrations, ECS is recommended.
Add your data
edit- Migrate and upload existing data into your deployment.
-
Add inbound integrations for new data sources. You can either use Elastic provided integrations, or create your own:
- To use Elastic provided integrations, check the Elastic integrations page.
- To integrate with Cloud Service Provider log and metric services, check these tutorials: AWS, GCP, Azure.
- To write your own, choose an ingestion method.
Optimize data storage and retention
editOnce you have your data tiers deployed and you have data flowing, you can manage the index lifecycle.
Elastic integrations provide default index lifecycle policies, and you can build your own policies for your custom integrations.