Next Steps

edit

Now that you have provisioned your first Elastic Cloud Enterprise cluster, you can try out a few things in the Cloud UI:

Enable Kibana

edit

Kibana is an open source analytics and visualization platform designed to work with Elasticsearch that makes it easy to perform advanced data analysis and to visualize your data in a variety of charts, tables, and maps. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

Every Elasticsearch cluster can use Kibana, but you must enable it first.

To enable Kibana on your cluster:

  1. Log into the Cloud UI.
  2. On the Clusters page, select the cluster where you want to enable Kibana.
  3. Go to the Kibana tab. If the cluster does not already have a running Kibana instance, click Enable Kibana.

    The new Kibana instance takes a few moments to provision.

After provisioning Kibana is complete, the Kibana tab shows you the endpoint URLs where you can access Kibana.

You can log into Kibana with the elastic superuser. The password was provided when you created your cluster or can be reset. On AWS and not able to access Kibana? Check if you need to update your endpoint URL first.

Enable Monitoring (formerly Marvel)

edit

The X-Pack monitoring features let you monitor Elasticsearch through Kibana. You can view your cluster’s health and performance in real time and analyze past cluster, index, and node metrics. In Elasticsearch versions before 5.0, Marvel provides similar monitoring functionality.

In Elasticsearch 5.0, the monitoring features of Marvel became part of X-Pack. If you are using an Elasticsearch version before 5.0, think Marvel whenever you read about the X-Pack monitoring features.

Monitoring consists of two components:

  • A Monitoring agent that is installed on each node in your cluster. The Monitoring agent collects and indexes metrics from Elasticsearch, either on the same cluster or by sending metrics to an external monitoring cluster. Elastic Cloud Enterprise manages the installation and configuration of the monitoring agent for you, and you should not modify any of the settings.
  • The Monitoring (formerly Marvel) application plugin in Kibana that visualizes the monitoring metrics through a dashboard.

The steps in this section cover only the enablement of Monitoring. For more information on how to use Monitoring itself, see the X-Pack Monitoring documentation or the Marvel documentation if you are using an Elasticsearch version before 5.0.

Monitoring for Production Use

edit

For production use, you should log metrics for clusters to a dedicated monitoring cluster (but never to the logging-and-metrics cluster that is used by ECE). Monitoring indexes metrics into Elasticsearch and these indexes consume storage, memory and CPU cycles like any other index. By using a separate monitoring cluster, you avoid affecting your production clusters.

You should also create a dedicated user for the clusters sending metrics and the monitoring cluster receiving them. For more information on creating a user with the right privileges, see Monitoring and Security (for version 5.0 and later) and Using Marvel with Shield (for versions before 5.0).

How many monitoring clusters you use depends on your requirements:

  • You can ship metrics for many clusters to a single monitoring cluster if your business requirements permit it.
  • While monitoring will work with a cluster running a single node, you need a minimum of three monitoring nodes to make monitoring highly available.
  • You might need to create dedicated monitoring clusters for isolation purposes in some cases. For example:

    • If you have many clusters and some of them are much larger than others, creating separate monitoring clusters prevents a large cluster from potentially affecting monitoring performance for smaller clusters.
    • If you need to silo Elasticsearch data for different business departments. Clusters that have been configured to ship metrics to a target monitoring cluster have access to indexing data and can manage monitoring index templates, which is addressed by creating separate monitoring clusters.

Monitoring indices that get sent to a monitoring cluster are not cleaned up automatically. You can use Curator to clean up these monitoring indices, like any other time-based index.

To avoid compatibility issues between versions, the cluster sending monitoring metrics and the monitoring cluster receiving them should be at the same Elasticsearch version. If using the same version is not feasible, check for breaking changes in the X-Pack Release Notes or the Marvel Release Notes to make sure that your versions are compatible.

Configure Where Monitoring Data Is Sent

edit

When you enable monitoring on a cluster, you are configuring where the monitoring agent for your current cluster should send its metrics.

There are some prerequisites to keep in mind:

  • Both the cluster that is sending monitoring metrics and the monitoring cluster that receives the metrics must be configured to use Security (formerly Shield).
  • Only monitoring clusters that are at the same version or a later version than your current cluster are shown in the Cloud UI. To make sure that monitoring data continues to be sent, you should always keep the monitoring cluster at a version that is ahead or equal to the production cluster.

To enable monitoring, you need to:

  1. Log into the Cloud UI.
  2. On the Clusters page, select the cluster where you want to enable monitoring and go to one of the following tabs:

    • For version 5.0 and later*: Click Monitoring.
    • For versions before 5.0*: Click Marvel.
  3. Let the user interface guide you through the steps for specifying the monitoring cluster that will receive the metrics.

    If a cluster is not listed, make sure that it is running a compatible version and is configured to use Security.

    Remember to send metrics for production clusters to a dedicated monitoring cluster (but never to the logging-and-metrics cluster that is used by ECE).

To work with the monitoring metrics, access the Monitoring application in Kibana.

Enable Graph (Versions before 5.0)

edit

Graph capabilities let you discover how items in an Elasticsearch index are related. You can explore the connections between indexed terms and see which connections are the most meaningful. This can be useful in a variety of applications, from fraud detection to recommendation engines.

In version 5.0 and later, Graph is automatically enabled, as X-Pack is installed along with Elasticsearch and Kibana.

For versions before 5.0: Graph is a plugin that needs to be added to your cluster. To enable Graph:

  1. Log into the Cloud UI.
  2. On the Clusters page, select the cluster where you want to enable Graph. Edit the cluster configuration by clicking Edit icon.
  3. In the Plugins section, select the graph plugin, if it is not installed already.
  4. Click Save changes.

After the plugin is installed, Graph can be accessed from Kibana.

Connect to Your Cluster

edit

To connect to your cluster, you need to look up the the cluster Endpoint URL:

  1. Log into the Cloud UI, if you aren’t logged in already.
  2. On the Clusters page, select one of your clusters.
  3. Under Endpoints, the endpoint link for Elasticsearch is listed. If you already enabled Kibana, the endpoint where you can access Kibana is listed as well. Click the Elasticsearch endpoint to connect to the cluster in your browser. You should get see a standard message like this:

    {
      "name" : "instance-0000000000",
      "cluster_name" : "85943ce00a934471cb971009e73d2d39",
      "cluster_uuid" : "0z2PsOX1TCGSk7PKgB9ajg",
      "version" : {
        "number" : "2.4.1",
        "build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
        "build_timestamp" : "2016-09-27T18:57:55Z",
        "build_snapshot" : false,
        "lucene_version" : "5.5.2"
      },
      "tagline" : "You Know, for Search"
    }

    If you are prompted for authentication credentials, you are trying to connect to a cluster that already has Shield enabled or that uses the X-Pack security features. Specify the credentials of a user like the default elastic user to connect.

Currently, we support the following ways of connecting to an Elasticsearch cluster:

RESTful API with JSON over HTTP and HTTPS

Used by the curl command and most programming languages that aren’t Java. To interact with your cluster, use your Elasticsearch cluster endpoint information from the Overview page in the Cloud UI. Port 9200 is used for HTTP connections, ports 9243 and 443 are used for HTTPS. Using HTTPS is generally recommended, as it is more secure.

If this is your first time using Elasticsearch, you can try out some curl commands to become familiar with the basics. If you’re on an operating system like macOS or Linux, you probably already have the curl command installed. For example, to connect to your cluster from the command line over HTTPS with the curl command:

curl -k https://45e366dc3a4142e9a4d6bbe3c7eedee7.192.168.43.10.ip.es.io:9243
{
  "name" : "instance-0000000000",
  "cluster_name" : "45e366dc3a4142e9a4d6bbe3c7eedee7",
  "version" : {
    "number" : "2.3.5",
    "build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
    "build_timestamp" : "2016-07-27T10:36:52Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

To make this curl command work with your cluster, you need to replace the endpoint URL with your own. +TIP: If you created a cluster on Elasticsearch 5.0 or later or if you already enabled the security features, you must include authentication details with the -u parameter. For example: curl -u elastic:W0UN0Rh9WX8eKeN69grVk3bX https://85943ce00a934471cb971009e73d2d39.us-east-1.aws.found.io:9243.

Java transport client

A good choice if your applications are using Java. This lighter-weight transport client forwards requests to a remote cluster over port 9300, using the native Elasticsearch transport protocol.

To learn more about how you can use the Java transport client, typically in conjunction with X-Pack security features or Shield, see Configure the Java Transport Client.