Monitor Elasticsearch Service costs with billing integration and alerts

Monitoring and managing your Elasticsearch Service (ESS) usage and costs is crucial for efficient operations. The Elasticsearch Service Billing integration offers a streamlined approach to collect, visualize, and act upon your billing data. In this article, I'll guide you through deploying this integration, customizing it to align with your organizational structure, and setting up alerts to monitor team-specific expenditures.

Fill in the requested information, which you collected in the previous step. You can also customize how far back the billing data should be collected during the first run (one year by default) and how often the collection should occur. Please note, however, that the billing data is collected with a daily granularity, so changing the default frequency of 24 hours will bring little benefit.

Also note that the second dataset (Elasticsearch Service Credits) requires an active commercial commitment with Elastic to collect data properly.

Finally, deploy the integration to the agent policy of your choice. With version 1.2.0 of the integration, and underlying stack version 8.17.0+, it is also possible to deploy this integration agentless.

Once deployed, the integration will start polling Elastic billing APIs until your configured lookbehind period is fully onboarded. This can take a few minutes — be patient after the initial install, or change the time picker to whatever lookbehind value you used during install to see the progress in real time.

Now that you are collecting billing information for the entire organization, you will want to attribute the costs related to specific deployments to the teams that own them.

For this purpose, create an @custom pipeline for the integration, where you can define your logic.

For my example, I will use the following logic:

1. All deployments with “test” in their name will be attributed to the QA team.

2. All deployments with “security” in their name will be attributed to the Security team.

3. All other deployments will be attributed to the Development team.

You could also use the enrich processor to create a “join” with a reference base, thereby linking each deployment to its owner.

You can now analyse the spending trends of your teams, thanks to this freshly created field. Let’s use some ES|QL for this purpose:

FROM metrics-ess_billing.billing-*

| STATS `Team spending`=SUM(ess.billing.total_ecu)
    BY BUCKET(@timestamp,"1 month"), team

It looks like the trend is fairly stable, except for this large bill in November. Maybe I should configure some alerting to identify the spending trend faster next time.

Monitoring team expenditures is vital for budget management. With Kibana's alerting features, you can configure notifications to alert you when a team's spending trend surpasses a predefined threshold — let’s prevent this November fluke from happening again.

Since I already queried for the data and wrote the ES|QL query, most of the work is already done! I can simply click on the Alerts button at the top of my screen and change my configuration and threshold to match my goals.

In my case, I will refine the granularity to daily spending and set a per-team budget of 1,000 ECUs:

Notice how I can live-test my query and confirm that this would indeed have triggered in November, when my Development team blew the budget.

The final step is to configure the action you want Kibana to run on alert trigger. In my case, this will certainly be an email, but you can refer to the documentation for more details about all available connector types.

By integrating the Elasticsearch Service Billing data into your observability pipeline, you gain real-time visibility into your cloud spending, ensuring financial accountability across teams. With custom tagging and alerting in Kibana, you can track expenses, enforce budget policies, and prevent unexpected cost overruns.

Ready to take control of your Elasticsearch Service costs? Deploy the integration today, customize it to match your team structure, and set up alerts to stay ahead of budget surprises.