« Deploy Fleet Server on-premises and Elasticsearch on Cloud Fleet Server scalability »
Elastic Docs ›Fleet and Elastic Agent Guide [8.17] ›Deployment models

Deploy Fleet Server on Kubernetes

edit

Deploy Fleet Server on Kubernetes

edit

If your Elastic Stack is orchestrated by ECK, we recommend to deploy the Fleet Server through the operator. That simplifies the process, as the operator automatically handles most of the resources configuration and setup steps.

Refer to Run Fleet-managed Elastic Agent on ECK for more information.

This guide assumes familiarity with Kubernetes concepts and resources, such as Deployments, Pods, Secrets, or Services, as well as configuring applications in Kubernetes environments.

To use Fleet for central management, a Fleet Server must be running and accessible to your hosts.

You can deploy Fleet Server on Kubernetes and manage it yourself. In this deployment model, you are responsible for high-availability, fault-tolerance, and lifecycle management of the Fleet Server.

To deploy a Fleet Server on Kubernetes and register it into Fleet you will need the following details:

  • The Policy ID of a Fleet policy configured with the Fleet Server integration.
  • A Service token, used to authenticate Fleet Server with Elasticsearch.

  • For outgoing traffic:

    • The Elasticsearch endpoint URL where the Fleet Server should connect to, configured also in the Elasticsearch output associated to the policy.
    • When a private or intermediate Certificate Authority (CA) is used to sign the Elasticsearch certificate, the Elasticsearch CA file or the CA fingerprint, configured also in the Elasticsearch output associated to the policy.

  • For incoming connections:

    • A TLS/SSL certificate and key for the Fleet Server HTTPS endpoint, used to encrypt the traffic from the Elastic Agents. This certificate has to be valid for the Fleet Server Host URL that Elastic Agents use when connecting to the Fleet Server.
  • Extra TLS/SSL certificates and configuration parameters in case of requiring mutual TLS (not covered in this document).

This document walks you through the complete setup process, organized into the following sections:

Compatibility

edit

Fleet Server is compatible with the following Elastic products:

  • Elastic Stack 7.13 or later.

    • For version compatibility, Elasticsearch must be at the same or a later version than Fleet Server, and Fleet Server needs to be at the same or a later version than Elastic Agent (not including patch releases).
    • Kibana should be on the same minor version as Elasticsearch.

Prerequisites

edit

Before deploying Fleet Server, you need to:

  • Prepare the SSL/TLS configuration, server certificate, Fleet Server host settings, and needed Certificate Authorities (CAs).
  • Ensure components have access to the ports needed for communication.
Fleet Server and SSL/TLS certificates considerations
edit

This section shows the minimum requirements in terms of Transport Layer Security (TLS) certificates for the Fleet Server, assuming no mutual TLS (mTLS) is needed. Refer to One-way and mutual TLS certifications flow and Elastic Agent deployment models with mutual TLS for more information about the configuration needs of both approaches.

There are two main traffic flows for Fleet Server, each with different TLS requirements:

[Elastic Agent → Fleet Server] inbound traffic flow
edit

In this flow Fleet Server acts as the server and Elastic Agent acts as the client. Therefore, Fleet Server requires a TLS certificate and key, and Elastic Agent will need to trust the CA certificate used to sign the Fleet Server certificate.

A Fleet Server certificate is not required when installing the server using the Quick start mode, but should always be used for production deployments. In Quick start mode, the Fleet Server uses a self-signed certificate and the Elastic Agents have to be enrolled with the --insecure option.

If your organization already uses the Elastic Stack, you may have a CA certificate that could be used to generate the new cert for the Fleet Server. If you do not have a CA certificate, refer to Generate a custom certificate and private key for Fleet Server for an example to generate a CA and a server certificate using the elasticsearch-certutil tool.

Before creating the certificate, you need to know and plan in advance the hostname / URL that the Elastic Agent clients will use to access the Fleet Server. This is important because the hostname part of the URL needs to be included in the server certificate as an x.509 Subject Alternative Name (SAN). If you plan to make your Fleet Server accessible through multiple hostnames or FQDNs, add all of them to the server certificate, and take in mind that the Fleet Server also needs to access the Fleet URL during its bootstrap process.

[Fleet Server → Elasticsearch output] outbound traffic flow
edit

In this flow, Fleet Server acts as the client and Elasticsearch acts as the HTTPS server. For the communication to succeed, Fleet Server needs to trust the CA certificate used to sign the Elasticsearch certificate. If your Elasticsearch cluster uses certificates signed by a corporate CA or multiple intermediate CAs you will need to use them during the Fleet Server setup.

If your Elasticsearch cluster is on Elastic Cloud or if it uses a certificate signed by a public and known CA, you won’t need the Elasticsearch CA during the setup.

In summary, you need:

  • A server certificate and key, valid for the Fleet Server URL. The CA used to sign this certificate will be needed by the Elastic Agent clients and the Fleet Server itself.
  • The CA certificate (or certificates) associated to your Elasticsearch cluster, except if you are sure your Elasticsearch certificate is fully trusted publicly.
Default port assignments
edit

When Elasticsearch or Fleet Server are deployed, components communicate over well-defined, pre-allocated ports. You may need to allow access to these ports. Refer to the following table for default port assignments:

Component communication

Default port

Elastic Agent → Fleet Server

8220

Fleet Server → Elasticsearch

9200

Fleet Server → Kibana (optional, for Fleet setup)

5601

Elastic Agent → Elasticsearch

9200

Elastic Agent → Logstash

5044

Elastic Agent → Kibana (optional, for Fleet setup)

5601

In Kubernetes environments, you can adapt these ports without modifying the listening ports of the Fleet Server or other applications, as traffic is managed by Kubernetes Services. This guide includes an example where Elastic Agents connect to the Fleet Server through port 443 instead of the default 8220.

Add Fleet Server

edit

A Fleet Server is an Elastic Agent that is enrolled in a Fleet Server policy. The policy configures the agent to operate in a special mode to serve as a Fleet Server in your deployment.

Fleet preparations
edit

If you already have a Fleet policy with the Fleet Server integration, you know its ID, and you know how to generate an Elasticsearch service token for the Fleet Server, skip directly to Fleet Server installation.

Also note that the service token required by the Fleet Server is different from the enrollment tokens used by Elastic Agents to enroll to Fleet.

  1. In Kibana, open Fleet → Settings and ensure the Elasticsearch output that will be used by the Fleet Server policy is correctly configured, paying special attention that:

    • The hosts field includes a valid URL that will be reachable by the Fleet Server Pod(s).

    • If your Elasticsearch cluster uses certificates signed by private or intermediate CAs not publicly trusted, you have added the trust information in the Elasticsearch CA trusted fingerprint field or in the advanced configuration section through the ssl.certificate_authorities setting. For an example, refer to Secure Connections documentation.

      This validation step is critical. The Elasticsearch host URL and CA information has to be added in both the Elasticsearch output and the environment variables provided to the Fleet Server. It’s a common mistake to ignore the output settings believing that the environment variables will prevail, when the environment variables are only used during the bootstrap of the Fleet Server.

      If the URL that Fleet Server will use to access Elasticsearch is different from the Elasticsearch URL used by other clients, you may want to create a dedicated Elasticsearch output for Fleet Server.

  2. Go to Fleet → Agent Policies and select Create agent policy to create a policy for the Fleet Server:

    • Set a name for the policy, for example Fleet Server Policy Kubernetes.
    • Do not select the option Collect system logs and metrics. This option adds the System integration to the Elastic Agent policy. Because Fleet Server will run as a Kubernetes Pod without any visibility to the Kubernetes node, there won’t be a system to monitor.
    • Select the output that the Fleet Server needs to use to contact Elasticsearch. This should be the output that you verified in the previous step.
    • Optionally, you can set the inactivity timeout and inactive agent unenrollment timeout parameters to automatically unenroll and invalidate API keys after the Fleet Server agents become inactive. This is especially useful in Kubernetes environments, where Fleet Server Pods are ephemeral, and new Elastic Agents appear in Fleet UI after Pod recreations.

  3. Open the created policy, and from the Integrations tab select Add integration:

    • Search for and select the Fleet Server integration.

    • Select Add Fleet Server to add the integration to the Elastic Agent policy.

      At this point you can configure the integration settings per Fleet Server scalability.

    • When done, select Save and continue. Do not add an Elastic Agent at this stage.

  4. Open the configured policy, which now includes the Fleet Server integration, and select Actions → Add Fleet Server. In the next dialog:

    • Confirm that the policy for Fleet Server is properly selected.

    • Choose a deployment mode for security:

      • If you select Quick start, the Fleet Server generates a self-signed TLS certificate, and subsequent agents should be enrolled using the --insecure flag.
      • If you select Production, you provide a TLS certificate, key and CA to the Fleet Server during the deployment, and subsequent agents will need to trust the certificate’s CA.

    • Add your Fleet Server Host information. This is the URL that clients (Elastic Agents) will use to connect to the Fleet Server:

      • In Production mode, the Fleet Server certificate must include the hostname part of the URL as an x509 SAN, and the Fleet Server itself will need to access that URL during its bootstrap process.
      • On Kubernetes environments this could be the name of the Kubernetes service or reverse proxy that exposes the Fleet Server Pods.
      • In the provided example we use https://fleet-svc.<namespace> as the URL, which corresponds to the Kubernetes service DNS resolution.
    • Select generate service token to create a token for the Fleet Server.

    • From Install Fleet Server to a centralized host → Linux, take note of the values of the following settings that will be needed for the Fleet Server installation:

      • Service token(specified by --fleet-server-service-token parameter).
      • Fleet policy ID (specified by --fleet-server-policy parameter).
      • Elasticsearch URL (specified by --fleet-server-es parameter).

  5. Keep the Kibana browser window open and continue with the Fleet Server installation.

    When the Fleet Server installation has succeeded, the Confirm Connection UI will show a Connected status.

Fleet Server installation
edit
Installation overview
edit

To deploy Fleet Server on Kubernetes and enroll it into Fleet you need the following details:

  • Policy ID of the Fleet policy configured with the Fleet Server integration.
  • Service token, that you can generate following the Fleet preparations or manually using the Elasticsearch-service-tokens command.
  • Elasticsearch endpoint URL, configured in both the Elasticsearch output associated to the policy and in the Fleet Server as an environment variable.
  • Elasticsearch CA certificate file, configured in both the Elasticsearch output associated to the policy and in the Fleet Server.
  • Fleet Server certificate and key (for Production deployment mode only).
  • Fleet Server CA certificate file (for Production deployment mode only).
  • Fleet Server URL (for Production deployment mode only).

If you followed the Fleet Server and SSL/TLS certificates considerations and Fleet preparations you should have everything ready to proceed with the Fleet Server installation.

The suggested deployment method for the Fleet Server consists of:

  • A Kubernetes Deployment manifest that relies on two Secrets for its configuration:

    • A Secret named fleet-server-config with the main configuration parameters, such as the service token, the Elasticsearch URL and the policy ID.
    • A Secret named fleet-server-ssl with all needed certificate files and the Fleet Server URL.
  • A Kubernetes ClusterIP Service named fleet-svc that exposes the Fleet Server on port 443, making it available at URLs like https://fleet-svc, https://fleet-svc.<namespace> and https://fleet-svc.<namespace>.svc.

Adapt and change the suggested manifests and deployment strategy to your needs, ensuring you feed the Fleet Server with the needed configuration and certificates. For example, you can customize:

  • CPU and memory requests and limits. Refer to Fleet Server scalability for more information about Fleet Server resources utilization.
  • Scheduling configuration, such as affinity rules or tolerations, if needed in your environment.
  • Number of replicas, to scale the Fleet Server horizontally.
  • Use an Elasticsearch CA fingerprint instead of a CA file.
  • Configure other Environment variables.
Installation Steps
edit

  1. Create the Secret for the Fleet Server configuration.

    kubectl create secret generic fleet-server-config \
--from-literal=elastic_endpoint='<ELASTICSEARCH_HOST_URL>' \
--from-literal=elastic_service_token='<SERVICE_TOKEN>' \
--from-literal=fleet_policy_id='<POLICY_ID>'

    When running the command, substitute the following values:

    • <ELASTICSEARCH_HOST_URL>: Replace this with the URL of your Elasticsearch host, for example 'https://monitoring-es-http.default.svc:9200'.
    • <SERVICE_TOKEN>: Use the service token provided by Kibana in the Fleet UI.
    • <POLICY_ID>: Replace this with the ID of the created policy, for example 'dee949ac-403c-4c83-a489-0122281e4253'.

    If you prefer to obtain a yaml manifest of the Secret to create, append --dry-run=client -o=yaml to the command and save the output to a file.

  2. Create the Secret for the TLS/SSL configuration:

    The following command assumes you have the Elasticsearch CA available as a local file.

    kubectl create secret generic fleet-server-ssl \
  --from-file=es-ca.crt=<PATH_TO_ES_CA_CERT_FILE>

    When running the command, substitute the following values:

    • <PATH_TO_ES_CA_CERT_FILE> with your local file containing the Elasticsearch CA(s).

    If you prefer to obtain a yaml manifest of the Secret to create, append --dry-run=client -o=yaml to the command and save the output to a file.

    If your Elasticsearch cluster runs on Elastic Cloud or if it uses a publicly trusted CA, remove the es-ca.crt key from the proposed secret.

  3. Save the proposed Deployment manifest locally, for example as fleet-server-dep.yaml, and adapt it to your needs:

    apiVersion: v1
kind: Service
metadata:
  name: fleet-svc
spec:
  type: ClusterIP
  selector:
    app: fleet-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8220
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fleet-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fleet-server
  template:
    metadata:
      labels:
        app: fleet-server
    spec:
      automountServiceAccountToken: false
      containers:
      - name: elastic-agent
        image: docker.elastic.co/beats/elastic-agent:8.17.1
        env:
          - name: FLEET_SERVER_ENABLE
            value: "true"
          - name: FLEET_SERVER_ELASTICSEARCH_HOST
            valueFrom:
              secretKeyRef:
                name: fleet-server-config
                key: elastic_endpoint
          - name: FLEET_SERVER_SERVICE_TOKEN
            valueFrom:
              secretKeyRef:
                name: fleet-server-config
                key: elastic_service_token
          - name: FLEET_SERVER_POLICY_ID
            valueFrom:
              secretKeyRef:
                name: fleet-server-config
                key: fleet_policy_id
          - name: ELASTICSEARCH_CA
            value: /mnt/certs/es-ca.crt
        ports:
        - containerPort: 8220
          protocol: TCP
        resources: {}
        volumeMounts:
        - name: certs
          mountPath: /mnt/certs
          readOnly: true
      volumes:
      - name: certs
        secret:
          defaultMode: 420
          optional: false
          secretName: fleet-server-ssl

    Manifest considerations:

    • If your Elasticsearch cluster runs on Elastic Cloud or if it uses a publicly trusted CA, remove the ELASTICSEARCH_CA environment variable from the manifest.
    • Check the image version to ensure its aligned with the rest of your Elastic Stack.
    • Keep automountServiceAccountToken set to false to disable the Kubernetes Provider.
    • Consider configuring requests and limits always as a best practice. Refer to Fleet Server scalability for more information about resources utilization of the Fleet Server.
    • You can change the listening port of the service to any port of your choice, but do not change the targetPort, as the Fleet Server Pods will listen on port 8220.
    • If you want to expose the Fleet Server externally, consider changing the service type to LoadBalancer.

  4. Deploy the configured manifest to create the Fleet Server and service:

    kubectl apply -f fleet-server-dep.yaml

    Ensure the Service, the Deployment and all the referenced Secrets are created in the same Namespace.

  5. Check the Fleet Server Pod logs for errors and confirm in Kibana that the Fleet Server agent appears as Connected and Healthy in Kibana → Fleet.

    kubectl logs fleet-server-69499449c7-blwjg

    It can take a couple of minutes for Fleet Server to fully start. If you left the Kibana browser window open during Fleet preparations it will show Connected when everything has gone well.

    In Production mode, during Fleet Server bootstrap process, the Fleet Server might be unable to access its own FLEET_URL. This is usually a temporary issue caused by the Kubernetes Service not forwarding traffic to the Pod(s).

    If the issue persists consider using https://localhost:8220 as the FLEET_URL for the Fleet Server configuration, and ensure that localhost is included in the certificate’s SAN.

Expose the Fleet Server to Elastic Agents

edit

This may include the creation of a Kubernetes service, an ingress resource, and / or DNS registers for FQDNs resolution. There are multiple ways to expose applications in Kubernetes.

Considerations when exposing Fleet Server:

  • If your environment requires the Fleet Server to be reachable through multiple hostnames or URLs, you can create multiple Fleet Server Hosts in Fleet → Settings, and create different policies for different groups of agents.
  • Remember that in Production mode, the hostnames used to access the Fleet Server must be part of the Fleet Server certificate as x.509 Subject Alternative Names.
  • Align always the service listening port to the URL. If you configure the service to listen in port 8220 use a URL like https://service-name:8220, and if it listens in 443 use a URL like https://service-name.

Below is an end to end example of how to expose the server to external and internal clients using a LoadBalancer service. For this example we assume the following:

  • The Fleet Server runs in a namespace called elastic.
  • External clients will access Fleet Server using a URL like https://fleet.example.com, which will be resolved in DNS to the external IP of the Load Balancer.
  • Internal clients will access Fleet Server using the Kubernetes service directly https://fleet-svc-lb.elastic.
  • The server certificate has both hostnames (fleet.example.com and fleet-svc-lb.elastic) in its SAN list.

  1. Create the LoadBalancer Service

    kubectl expose deployment fleet-server --name fleet-svc-lb --type LoadBalancer --port 443 --target-port 8220

    That command creates a service named fleet-svc-lb, listening on port 443 and forwarding the traffic to the fleet-server deployment’s Pods on port 8220. The listening --port (and the consequent URL) of the service can be customized, but the --target-port must remain on the default port (8220), because it’s the port used by the Fleet Server application.

  2. Add https://fleet-server.example.com and https://fleet-svc-lb.elastic as a new Fleet Server Hosts in Fleet → Settings. Align the port of the URLs if you configured something different from 443 in the Load Balancer.
  3. Create a Fleet policy for external clients using the https://fleet-server.example.com Fleet Server URL.
  4. Create a Fleet policy for internal clients using the https://fleet-svc-lb.elastic Fleet Server URL.
  5. You are ready now to enroll external and internal agents to the relevant policies. Refer to Next steps for more details.

Troubleshoot Fleet Server

edit
Common Problems
edit

The following issues may occur when Fleet Server settings are missing or configured incorrectly:

  • Fleet Server is trying to access Elasticsearch at localhost:9200 even though the FLEET_SERVER_ELASTICSEARCH_HOST environment variable is properly set.

    This problem occurs when the output of the policy associated to the Fleet Server is not correctly configured.

  • TLS certificate trust issues occur even when the ELASTICSEARCH_CA environment variable is properly set during deployment.

    This problem occurs when the output of the policy associated to the Fleet Server is not correctly configured. Add the CA certificate or CA trusted fingerprint to the Elasticsearch output associated to the Fleet Server policy.

  • In Production mode, Fleet Server enrollment fails due to FLEET_URL not being accessible, showing something similar to:

    Starting enrollment to URL: https://fleet-svc/
1st enrollment attempt failed, retrying enrolling to URL: https://fleet-svc/ with exponential backoff (init 1s, max 10s)
Error: fail to enroll: fail to execute request to fleet-server: dial tcp 34.118.226.212:443: connect: connection refused
Error: enrollment failed: exit status 1

    If the service and URL are correctly configured, this is usually a temporary issue caused by the Kubernetes Service not forwarding traffic to the Pod, and it should be cleared in a couple of restarts.

    As a workaround, consider using https://localhost:8220 as the FLEET_URL for the Fleet Server configuration, and ensure that localhost is included in the certificate’s SAN.

Next steps

edit

Now you’re ready to add Elastic Agents to your host systems. To learn how, refer to Install Fleet-managed Elastic Agents, or Run Elastic Agent on Kubernetes managed by Fleet if your Elastic Agents will also run on Kubernetes.

When you connect Elastic Agents to Fleet Server, remember to use the --insecure flag if the quick start mode was used, or to provide to the Elastic Agents the CA certificate associated to the Fleet Server certificate if production mode was used.

« Deploy Fleet Server on-premises and Elasticsearch on Cloud Fleet Server scalability »