Connect to Elasticsearch clusters in a different Elastic Cloud on Kubernetes environment

These steps describe how to configure a remote cluster connection between Elasticsearch clusters handled by different Elastic Cloud on Kubernetes (ECK) operators.

After the connection is established, you’ll be able to run CCS queries from Elasticsearch or set up CCR.

In this scenario, most of the configuration must be performed manually, as Elastic Cloud on Kubernetes cannot orchestrate the setup across both clusters. For fully automated configuration between ECK-managed clusters, refer to Connect to Elasticsearch clusters in the same ECK environment.

For other remote cluster scenarios with ECK, refer to Remote clusters on ECK.

Before you start, consider the security model that you would prefer to use for authenticating remote connections between clusters, and follow the corresponding steps.

API key

For deployments based on Elastic Stack 8.14 or later, you can use an API key to authenticate and authorize cross-cluster operations to a remote cluster. This model uses a dedicated service endpoint, on port 9443 by default, and gives administrators fine-grained control over remote access. The API key is created on the remote cluster and defines the permissions available to all cross-cluster requests, while local user roles can further restrict, but not extend, those permissions.

TLS certificate (deprecated in Elastic Stack 9.0.0)

This model uses mutual TLS authentication over the Elasticsearch transport interface for cross-cluster operations. User authentication is performed on the local cluster and a user's role names are passed to the remote cluster for authorization. Because a superuser on the local cluster automatically gains full read access to the remote cluster, this model is only suitable for clusters within the same security domain.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: <cluster-name>
  namespace: <namespace>
spec:
  version: 9.2.4
  remoteClusterServer:
    enabled: true
  nodeSets:
    - name: default
      count: 3
      ...
      ...
		

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: <cluster-name>
  namespace: <namespace>
spec:
  version: 9.2.1
  remoteClusterServer:
    enabled: true
    service:
      spec:
        type: LoadBalancer
  nodeSets:
    - name: default
      count: 3
      ...
      ...
		

kubectl expose service quickstart-es-remote-cluster \
  --name=quickstart-es-remote-cluster-lb \
  --type=LoadBalancer \
  --port=9443 --target-port=9443
		

kubectl get secret quickstart-es-transport-certs-public \
-o go-template='{{index .data "ca.crt" | base64decode}}' > eck_transport_ca.crt
		

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: <remote-cluster-name>
spec:
  transport:
    service:
      spec:
        type: LoadBalancer
		

On the local cluster, use Kibana or the Elasticsearch API to add the remote ECK cluster with the following connection settings:

• Remote address: Use the FQDN or IP address of the LoadBalancer service or alternative resource that you created to expose the remote cluster.

  • For API key-based authentication, use the server interface address and port.
  • For TLS certificate-based authentication, use the transport interface address and port.

  If you haven't changed the external listening port of the kubernetes service, the port should be 9443 for API key-based authentication, or 9300 for TLS certificate-based authentication.

• TLS server name: You can try leaving this field empty first. If the connection fails, and your environment is presenting the ECK-managed certificates during the TLS handshake, use <cluster-name>-es-remote-cluster.<namespace>.svc as the server name. For example, for a cluster named quickstart in the default namespace, use quickstart-es-remote-cluster.default.svc.

To add a remote cluster in Kibana:

1. Go to the Remote Clusters management page in the navigation menu or use the global search field.

2. Select Add a remote cluster.

3. In Select connection type, choose the authentication mechanism you prepared earlier (API keys or Certificates), and click Next.

4. Set the Remote cluster name: This cluster alias is a unique identifier that represents the connection to the remote cluster and is used to distinguish local and remote indices.

   When using API key authentication, this alias must match the Remote cluster name you configured when adding the API key.

5. In Connection mode, select Manually enter proxy address and server name to enable the proxy mode and fill in the following fields:

   • Proxy address: Enter the endpoint of the remote cluster, including the hostname, FQDN, or IP address, and the port:

     Make sure you use the correct port for your authentication method:

     • API keys: Use the port configured in the remote cluster interface of the remote cluster (defaults to 9443).
     • TLS certificates: Use the Elasticsearch transport port (defaults to 9300).

     Starting with Kibana 9.2, this field also supports IPv6 addresses. When using an IPv6 address, enclose it in square brackets followed by the port number. For example: [2001:db8::1]:9443.

   • Server name (optional): Specify a value if the TLS certificate presented by the remote cluster is signed for a different name than the remote address.

6. Click Next.

7. In Confirm setup, click Add remote cluster (you have already established trust in a previous step).

To add a remote cluster, use the cluster update settings API. Configure the following fields:

• Remote cluster alias: When using API key authentication, the cluster alias must match the one you configured when adding the API key.

• mode: proxy

• proxy_address: Enter the endpoint of the remote cluster, including the hostname, FQDN, or IP address, and the port. Both IPv4 and IPv6 addresses are supported.

  Make sure you use the correct port for your authentication method:

  • API keys: Use the port configured in the remote cluster interface of the remote cluster (defaults to 9443).
  • TLS Certificates: Use the Elasticsearch transport port (defaults to 9300).

  When using an IPv6 address, enclose it in square brackets followed by the port number. For example: [2001:db8::1]:9443.

• server_name: Specify a value if the certificate presented by the remote cluster is signed for a different name than the proxy_address.

This is an example of the API call to add or update a remote cluster:

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "alias-for-my-remote-cluster": {
          "mode":"proxy",
          "proxy_address": "<REMOTE_CLUSTER_ADDRESS>:9443",
          "server_name": "<REMOTE_CLUSTER_SERVER_NAME>"
        }
      }
    }
  }
}
		

1. Align the alias with the remote cluster name used when adding the API key.

For a full list of available client connection settings in proxy mode, refer to the remote cluster settings reference.

From the local cluster, check the status of the connection to the remote cluster. If you encounter issues, refer to the Troubleshooting guide.

				GET _remote/info
		

In the response, verify that connected is true:

{
  "<remote-alias>": {
    "connected": true,
    "mode": "proxy",
    "proxy_address": "<REMOTE_CLUSTER_ADDRESS>:9443",
    "server_name": "<REMOTE_CLUSTER_SERVER_NAME>",
    "num_proxy_sockets_connected": 18,
    "max_proxy_socket_connections": 18,
    "initial_connect_timeout": "30s",
    "skip_unavailable": true,
    "cluster_credentials": "::es_redacted::"
  }
}
		

If you're using the API key–based security model for cross-cluster replication or cross-cluster search, you can define user roles with remote indices privileges on the local cluster to further restrict the permissions granted by the API key. For more details, refer to Configure roles and users.