Air-gapped environments

When running Elastic Agents in a restricted or closed network, you need to take extra steps to make sure:

The Elastic Package Registry must therefore be accessible from Kibana via an HTTP Proxy and/or self-hosted.

The Elastic Artifact Registry must therefore be accessible from Kibana via an HTTP Proxy and/or self-hosted.

When upgrading all the components in an air-gapped environment, it is recommended that you upgrade in the following order:

Set the following property in Kibana to enable air-gapped mode in Fleet. This allows Fleet to intelligently skip certain requests or operations that shouldn’t be attempted in air-gapped environments.

xpack.fleet.isAirGapped: true

Starting from version 8.9.0, when Elastic Agent tries to perform an upgrade, it first verifies the binary signature with the key bundled in the agent. This process has a backup mechanism that will use the key coming from https://artifacts.elastic.co/GPG-KEY-elastic-agent instead of the one it already has.

In an air-gapped environment, an Elastic Agent which doesn’t have access to a PGP/GPG key from https://artifacts.elastic.co/GPG-KEY-elastic-agent would fail to be upgraded. For versions 8.9.0 to 8.10.3, you can resolve this problem following the steps described in the associated known issue documentation.

Starting in version 8.10.4, you can resolve this problem by configuring Elastic Agents to download the PGP/GPG key from Fleet Server.

Starting in version 8.10.4, Elastic Agent will:

By default, Fleet Server serves Elastic Agents with the key located in FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp. The key is served through the Fleet Server endpoint GET /api/agents/upgrades/{major}.{minor}.{patch}/pgp-public-key.

If there isn’t a default.pgp key in the FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp directory, Fleet Server instead will attempt to retrieve a PGP/GPG key from the URL that you can specify with the server.pgp.upstream_url setting.

You can prevent Fleet from downloading the PGP/GPG key from server.pgp.upstream_url by manually downloading it from https://artifacts.elastic.co/GPG-KEY-elastic-agent and storing it at FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp.

To set a custom URL for Fleet Server to access a PGP/GPG key and make it available to Elastic Agents:

server.pgp.upstream_url: <http://my-web-server:8080/default.pgp>

The setting server.pgp.upstream_url must point to a web server hosting the PGP/GPG key, which must be reachable by the host where Fleet Server is installed.

Note that:

By default Kibana downloads package metadata and content from the public Elastic Package Registry at epr.elastic.co.

If you can route traffic to the public endpoint of the Elastic Package Registry through a network gateway, set the following property in Kibana to use a proxy server:

xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net

For more information, refer to Using a proxy server with Elastic Agent and Fleet.

If routing traffic through a proxy server is not an option, you can host your own Elastic Package Registry.

The Elastic Package Registry can be deployed and hosted onsite using one of the available Docker images. These docker images include the Elastic Package Registry and a selection of packages.

There are different distributions available:

To update the distribution image, re-pull the image and then restart the docker container.

Every distribution contains packages that can be used by different versions of the Elastic Stack. The Elastic Package Registry API exposes a Kibana version constraint that allows for filtering packages that are compatible with a particular version.

Use the xpack.fleet.registryUrl property in the Kibana config to set the URL of your hosted package registry. For example:

xpack.fleet.registryUrl: "http://package-registry.corp.net:8080"

You can configure the Elastic Package Registry to listen on a secure HTTPS port using TLS.

For example, given a key and a certificate pair available in /etc/ssl, you can start the Elastic Package Registry listening on the 443 port using the following command:

docker run -it -p 443:443 \
  -v /etc/ssl/package-registry.key:/etc/ssl/package-registry.key:ro \
  -v /etc/ssl/package-registry.crt:/etc/ssl/package-registry.crt:ro \
  -e EPR_ADDRESS=0.0.0.0:443 \
  -e EPR_TLS_KEY=/etc/ssl/package-registry.key \
  -e EPR_TLS_CERT=/etc/ssl/package-registry.crt \
  docker.elastic.co/package-registry/distribution:9.0.0-beta1

The Elastic Package Registry supports TLS versions from 1.0 to 1.3. The minimum version accepted can be configured with EPR_TLS_MIN_VERSION, it defaults to 1.0. If you want to restrict the supported versions from 1.2 to 1.3, you can use EPR_TLS_MIN_VERSION=1.2.

If you are using self-signed certificates or certificates issued by a custom Certificate Authority (CA), you need to set the file path to your CA in the NODE_EXTRA_CA_CERTS environment variable in the Kibana startup files.

NODE_EXTRA_CA_CERTS="/etc/kibana/certs/ca-cert.pem"

Elastic Agents must be able to access the Elastic Artifact Registry to download binaries during upgrades. By default Elastic Agents download artifacts from https://artifacts.elastic.co/downloads/.

To make binaries available in an air-gapped environment, you can host your own custom artifact registry, and then configure Elastic Agents to download binaries from it.

Not using Fleet? For standalone Elastic Agents, you can set the binary download location under agent.download.sourceURI in the elastic-agent.yml file, or run the elastic-agent upgrade command with the --source-uri flag specified.