- Elastic Cloud Enterprise - Elastic Cloud on your Infrastructure: other versions:
- Introducing Elastic Cloud Enterprise
- Preparing your installation
- Installing Elastic Cloud Enterprise
- Identify the deployment scenario
- Install ECE with Ansible
- Install ECE on a public cloud
- Install ECE on your own premises
- Log into the Cloud UI
- Install ECE on additional hosts
- Post-installation steps
- Configuring your installation
- System clusters configuration
- Configure deployment templates
- Tag your allocators
- Edit instance configurations
- Create instance configurations
- Create deployment templates
- Configure system deployment templates
- Configure index management for templates
- Updating custom templates to support
node_roles
and autoscaling - Updating custom templates to support Integrations Server
- Default instance configurations
- Include additional Kibana plugins
- Manage snapshot repositories
- Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 6.4 - 6.8
- Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 7.0 and later
- Snapshotting to Azure Storage for Elasticsearch 5.x and 6.4 - 6.8
- Snapshotting to Azure Storage for Elasticsearch 7.0 and later
- Snapshotting to Minio on-premise storage
- Manage licenses
- Change the ECE API URL
- Change endpoint URLs
- Enable custom endpoint aliases
- Configure allocator affinity
- Securing your installation
- Monitoring your installation
- Administering your installation
- Getting started with deployments
- Create your deployment
- Access Kibana
- Manage data from the command line
- Configure the Java transport client (deprecated)
- Next steps
- Migrate your Elasticsearch data
- Indexing data into Elasticsearch
- Ingest data with Node.js on Elastic Cloud Enterprise
- Ingest data with Python on Elastic Cloud Enterprise
- Ingest data from Beats to Elastic Cloud Enterprise with Logstash as a proxy
- Ingest data from a relational database into Elastic Cloud Enterprise
- Ingest logs from a Python application using Filebeat
- Ingest logs from a Node.js web application using Filebeat
- Administering deployments
- Change your deployment configuration
- Stop routing requests or pause nodes
- Terminate a deployment
- Restart a deployment
- Restore a deployment
- Delete a deployment
- Migrate to index lifecycle management
- Disable a data tier
- Access the Elasticsearch API console
- Work with snapshots
- Upgrade versions
- Upgrade App Search to Enterprise Search
- Editing your user settings
- Deployment autoscaling
- Configure Beats and Logstash with Cloud ID
- Keep your clusters healthy
- Secure your clusters
- Reset the password for the
elastic
user - Secure your settings
- Secure your 7.x clusters with LDAP
- Secure your 7.x clusters with Active Directory
- Secure your 5.x and 6.x clusters with LDAP
- Secure your 5.x and 6.x clusters with Active Directory
- Secure your clusters with SAML
- Secure your clusters with OpenID Connect
- Secure your clusters with Kerberos
- Reset the password for the
- Trust management
- Deployment heap dumps
- Deployment thread dumps
- Traffic Filtering
- Connect to your cluster
- Manage your Kibana instance
- Manage your APM & Fleet Server
- Manage your APM Server (versions before 7.13)
- Manage your Integrations Server
- Enable logging and monitoring
- Enable cross-cluster search and cross-cluster replication
- Enable App Search
- Enable Enterprise Search
- Enable Graph (versions before 5.0)
- Troubleshooting
- RESTful API
- Authentication
- API calls
- How to access the API
- API examples
- Setting up your environment
- A first API call: What deployments are there?
- Create a first Deployment: Elasticsearch and Kibana
- Applying a new plan: Resize and add high availability
- Updating a deployment: Checking on progress
- Applying a new deployment configuration: Upgrade
- Enable more stack features: Add Enterprise Search to a deployment
- Dipping a toe into platform automation: Generate a roles token
- Customize your deployment
- Remove unwanted deployment templates and instance configurations
- Secure your settings
- API reference
- API - Base Url
- APM routes
- Accounts
- App Search - Read-only
- Authentication
- User authentication information
- Disable elevated permissions
- Enable elevated permissions
- Login to ECE
- Logout from ECE
- Refresh authentication token
- Delete API keys
- Get all API keys
- Create API key
- Delete API keys of multiple users
- Get all API keys for all users
- Delete API key
- Get API key
- Available authentication methods
- SAML callback
- Initiate SAML protocol
- Delete API keys for a user
- Get API key metadata for all keys created by the user
- Delete an API key for a user
- Get a user API key
- Billing Costs Analysis
- Get costs overview for the organization. Currently unavailable in self-hosted ECE.
- Get deployments costs for the organization. Currently unavailable in self-hosted ECE.
- Get itemized costs by deployments. Currently unavailable in self-hosted ECE.
- Get itemized costs for the organization. Currently unavailable in self-hosted ECE.
- Comments
- Deployment - CRUD
- Deployment - Commands
- Search Deployments
- Migrate Elasticsearch and associated Kibana resources to enable CCR
- Migrate Elasticsearch resource to use ILM
- Migrate Elasticsearch resource to use SLM
- Reset elastic user password
- Restart Deployment Elasticsearch Resource
- Shutdown Deployment Elasticsearch Resource
- Move Elasticsearch resource instances
- Get details about heap dumps for a Deployment
- Start all instances
- Stop all instances
- Start maintenance mode (all instances)
- Stop maintenance mode (all instances)
- Set size overrides
- Start instances
- Stop instances
- Start maintenance mode
- Stop maintenance mode
- Set size overrides
- Capture a new on-demand heap dump for the given instance
- Download the given instance’s heap dump
- Capture a new thread dump for the given instance
- Restart Deployment Stateless Resource
- Shutdown Deployment Stateless Resource
- Upgrade Kibana, APM, Integrations Server, AppSearch, Enterprise Search inside Deployment
- Deployment - Diagnostics
- Deployment - Info
- Deployment - Remote Clusters
- Deployment - Upgrade Assistant
- Deployment - resync
- Deployment Resources - CRUD
- Deployment Resources - Proxy
- Deployments - Extensions
- Deployments - Notes
- Deployments - Templates
- Deployments - Traffic Filter - Associations
- Deployments - Traffic Filter - CRUD
- Integrations Server routes
- Networking - Deployment Domain Names
- Organizations
- Platform
- Platform - Adminconsoles
- Platform - Allocators
- Get allocators
- Resynchronize allocators
- Search allocators
- Delete allocator
- Get allocator
- Resynchronize allocator
- Move clusters
- Move clusters by type
- Delete allocator logging settings
- Get allocator logging settings
- Update allocator logging settings
- Set allocator logging settings
- Start maintenance mode
- Stop maintenance mode
- Get allocator metadata
- Set allocator metadata
- Delete allocator metadata item
- Set allocator metadata item
- Get allocator settings
- Update allocator settings
- Set allocator settings
- Platform - Configuration - Instances - CRUD
- Platform - Configuration - Security
- Platform - Configuration - Security - Extra Certificates - CRUD
- Platform - Configuration - Security Deployment
- Platform - Configuration - Security Realms
- List security realm configurations
- Reorder security realms
- Create Active Directory configuration
- Delete Active Directory configuration
- Get Active Directory configuration
- Update Active Directory configuration
- Create LDAP configuration
- Delete LDAP configuration
- Get LDAP configuration
- Update LDAP configuration
- Create SAML configuration
- Delete SAML configuration
- Get SAML configuration
- Update SAML configuration
- Platform - Configuration - TLS
- Platform - Configuration - Trust relationships
- Platform - Constructors
- Platform - License
- Platform - Repository - CRUD
- Platform - Runners
- Platform - configuration - Store
- Platform - proxies
- Roles
- Stack - Instance Types - CRUD
- Stack - Versions - CRUD
- Telemetry
- Templates - Deployments - Global
- Users
- Definitions
AbsoluteRefId
AccountResponse
AccountTrustRelationship
AccountTrustSettings
AccountUpdateRequest
ActiveDirectoryGroupSearch
ActiveDirectorySecurityRealmLoadBalance
ActiveDirectorySecurityRealmRoleMappingRule
ActiveDirectorySecurityRealmRoleMappingRules
ActiveDirectorySettings
ActiveDirectoryUserSearch
AdminconsoleInfo
AdminconsolesOverview
AllocatedInstancePlansInfo
AllocatedInstanceStatus
AllocatorBuildInfo
AllocatorCapacity
AllocatorCapacityMemory
AllocatorHealthStatus
AllocatorInfo
AllocatorMoveRequest
AllocatorOverview
AllocatorSettings
AllocatorZoneInfo
AllocatorsSummary
AllocatorsZoneSummary
ApiBaseUrlData
ApiKeyResponse
ApiKeysResponse
Apm
ApmConfiguration
ApmCrudResponse
ApmInfo
ApmPayload
ApmPlan
ApmPlanControlConfiguration
ApmPlanInfo
ApmPlansInfo
ApmResourceInfo
ApmSettings
ApmSubInfo
ApmSystemSettings
ApmTopologyElement
AppSearch
AppSearchConfiguration
AppSearchInfo
AppSearchNodeTypes
AppSearchPayload
AppSearchPlan
AppSearchPlanControlConfiguration
AppSearchPlanInfo
AppSearchPlansInfo
AppSearchResourceInfo
AppSearchSettings
AppSearchSubInfo
AppSearchSystemSettings
AppSearchTopologyElement
AuthenticationInfo
AutodetectStrategyConfig
AvailableAuthenticationMethods
Balance
BasicFailedReply
BasicFailedReplyElement
Blessing
Blessings
BlessingsWithMeta
BoolQuery
CapacityConstraintsResource
ChainStatus
ChangeSourceInfo
ClusterCredentials
ClusterCurationSettings
ClusterCurationSpec
ClusterInstanceConfigurationInfo
ClusterInstanceDiskInfo
ClusterInstanceInfo
ClusterInstanceMemoryInfo
ClusterLicenseInfo
ClusterMetadataCpuResourcesSettings
ClusterMetadataInfo
ClusterMetadataPortInfo
ClusterMetadataResourcesSettings
ClusterMetadataSettings
ClusterPlanStepInfo
ClusterPlanStepLogMessageInfo
ClusterPlanWarning
ClusterSnapshotRepositoryDefault
ClusterSnapshotRepositoryInfo
ClusterSnapshotRepositoryReference
ClusterSnapshotRepositoryStatic
ClusterSnapshotRetention
ClusterSnapshotSettings
ClusterSystemAlert
ClusterTopologyInfo
Comment
CommentCreateRequest
CommentUpdateRequest
CommentWithMeta
CommentsWithMetas
CompatibleNodeTypesResource
CompatibleVersionResource
ConfigStoreOption
ConfigStoreOptionData
ConfigStoreOptionList
ConstructorHealthStatus
ConstructorInfo
ConstructorOverview
ContainerConfigHostConfig
ContainerSetSummary
ContainerSetsSummary
ContainersEntry
ContainersEntryOptions
ContainersEntryOptionsACL
ContainersEntryOptionsAuth
ContainersEntryOptionsContainerConfig
ContainersEntryOptionsOverrides
CoordinatorSummary
CoordinatorsSummary
Costs
CostsOverview
CreateApiKeyRequest
CreateExtensionRequest
Creates
DeleteApiKeysRequest
DeleteUsersApiKeysRequest
DeploymentCosts
DeploymentCreateMetadata
DeploymentCreateRequest
DeploymentCreateResources
DeploymentCreateResponse
DeploymentCreateSettings
DeploymentDeleteResponse
DeploymentDiagnostics
DeploymentDomainName
DeploymentGetResponse
DeploymentHeapDumps
DeploymentLogging
DeploymentLoggingSettings
DeploymentMetadata
DeploymentMetrics
DeploymentMetricsSettings
DeploymentObservability
DeploymentObservabilitySettings
DeploymentResource
DeploymentResourceCommandResponse
DeploymentResourceCrudResponse
DeploymentResourceUpgradeResponse
DeploymentResources
DeploymentRestoreResponse
DeploymentSearchResponse
DeploymentSettings
DeploymentShutdownResponse
DeploymentTemplateInfoV2
DeploymentTemplateReference
DeploymentTemplateRequestBody
DeploymentUpdateMetadata
DeploymentUpdateRequest
DeploymentUpdateResources
DeploymentUpdateResponse
DeploymentUpdateSettings
DeploymentUpgradeAssistantStatusResponse
DeploymentsCosts
DeploymentsListResponse
DeploymentsListingData
DeploymentsSearchResponse
Dimension
DiscreteSizes
DtsDimensionCosts
DtsQuantity
DtsRate
Elasticsearch
ElasticsearchClusterBlockingIssueElement
ElasticsearchClusterBlockingIssues
ElasticsearchClusterInfo
ElasticsearchClusterPlan
ElasticsearchClusterPlanInfo
ElasticsearchClusterPlansInfo
ElasticsearchClusterRole
ElasticsearchClusterSecurityInfo
ElasticsearchClusterSettings
ElasticsearchClusterTopologyElement
ElasticsearchClusterTrustSettings
ElasticsearchClusterUser
ElasticsearchConfiguration
ElasticsearchCuration
ElasticsearchDependant
ElasticsearchElasticUserPasswordResetResponse
ElasticsearchInfo
ElasticsearchMasterElement
ElasticsearchMasterInfo
ElasticsearchMonitoringInfo
ElasticsearchNodeType
ElasticsearchPayload
ElasticsearchPlanControlConfiguration
ElasticsearchReplicaElement
ElasticsearchResourceInfo
ElasticsearchScriptTypeSettings
ElasticsearchScriptingUserSettings
ElasticsearchShardElement
ElasticsearchShardsInfo
ElasticsearchSystemSettings
ElasticsearchUserBundle
ElasticsearchUserPlugin
ElevatePermissionsRequest
ElevatedPermissions
EmptyResponse
EnableIlmRequest
EnrollmentTokenRequest
EnterpriseSearch
EnterpriseSearchConfiguration
EnterpriseSearchInfo
EnterpriseSearchNodeTypes
EnterpriseSearchPayload
EnterpriseSearchPlan
EnterpriseSearchPlanControlConfiguration
EnterpriseSearchPlanInfo
EnterpriseSearchPlansInfo
EnterpriseSearchResourceInfo
EnterpriseSearchSettings
EnterpriseSearchSubInfo
EnterpriseSearchSystemSettings
EnterpriseSearchTopologyElement
ExistsQuery
Extension
ExtensionFileMetadata
Extensions
ExternalHyperlink
ExternalTrustRelationship
FilterAssociation
GenericResponse
GlobalDeploymentTemplateInfo
GlobalDeploymentTemplateRegion
GrowShrinkStrategyConfig
HeapDump
Hyperlink
IdResponse
IndexPattern
IndexSynchronizationResults
InstanceConfiguration
InstanceConfigurationInfo
InstanceMoveRequest
InstanceOverrides
InstanceTypeResource
IntegrationsServerConfiguration
IntegrationsServerCrudResponse
IntegrationsServerInfo
IntegrationsServerPayload
IntegrationsServerPlan
IntegrationsServerPlanControlConfiguration
IntegrationsServerPlanInfo
IntegrationsServerPlansInfo
IntegrationsServerResourceInfo
IntegrationsServerSettings
IntegrationsServerSystemSettings
IntegrationsServerTopologyElement
ItemCosts
ItemsCosts
KeystoreContents
KeystoreSecret
Kibana
KibanaClusterInfo
KibanaClusterPlan
KibanaClusterPlanInfo
KibanaClusterPlansInfo
KibanaClusterSettings
KibanaClusterTopologyElement
KibanaConfiguration
KibanaDeeplink
KibanaPayload
KibanaPlanControlConfiguration
KibanaResourceInfo
KibanaSubClusterInfo
KibanaSystemSettings
LdapGroupSearch
LdapSecurityRealmLoadBalance
LdapSecurityRealmRoleMappingRule
LdapSecurityRealmRoleMappingRules
LdapSettings
LdapUserSearch
LicenseInfo
LicenseObject
ListEnrollmentTokenElement
ListEnrollmentTokenReply
LoggingSettings
LoginRequest
LoginState
ManagedMonitoringSettings
MatchAllQuery
MatchNoneQuery
MatchQuery
Metadata
MetadataItem
MetadataItemValue
MetadataItems
ModelVersionIndexSynchronizationResults
MoveApmClusterConfiguration
MoveApmClusterDetails
MoveAppSearchConfiguration
MoveAppSearchDetails
MoveClustersCommandResponse
MoveClustersDetails
MoveClustersRequest
MoveElasticsearchClusterConfiguration
MoveElasticsearchClusterDetails
MoveEnterpriseSearchConfiguration
MoveEnterpriseSearchDetails
MoveKibanaClusterConfiguration
MoveKibanaClusterDetails
NestedQuery
NodeTypeResource
Note
Notes
ObservabilityIssue
Organization
OrganizationInvitation
OrganizationInvitationRequest
OrganizationInvitations
OrganizationList
OrganizationMembership
OrganizationMemberships
OrganizationRequest
Orphaned
OrphanedElasticsearch
PendingState
PendingStates
PendingStatesWithMeta
Period
PlanStrategy
PlatformInfo
PlatformResourcesSummary
PlatformServiceImageInfo
PlatformServiceInfo
PortBinding
PrefixQuery
ProxiesAllocationsInfo
ProxiesFilter
ProxiesFilteredGroup
ProxiesFilteredGroupHealth
ProxiesHealth
ProxiesHttpSettings
ProxiesSSOSettings
ProxiesSettings
ProxiesSummary
ProxyAllocationCounts
ProxyAllocationInfo
ProxyInfo
ProxyOverview
ProxySummary
PublicCertChain
PublicCertChainCollection
QueryContainer
QueryStringQuery
RangeQuery
ReadOnlyRequest
ReadOnlyResponse
RegionInfo
RemoteResourceInfo
RemoteResourceRef
RemoteResources
ReplyWarning
RepositoryConfig
RepositoryConfigs
RequestEnrollmentTokenReply
ResourceHeapDumps
RestartPolicy
RestoreSnapshotApiConfiguration
RestoreSnapshotConfiguration
RestoreSnapshotRepoConfiguration
Role
RoleAggregate
RoleAggregateCreateData
RoleAggregates
RoleWithMeta
RollingGrowShrinkStrategyConfig
RollingStrategyConfig
RulesetAssociations
RunnerBuildInfo
RunnerContainerInfo
RunnerInfo
RunnerOverview
RunnerRoleInfo
RunnerRolesInfo
RunnersSummary
SamlAttributeSettings
SamlIdpSettings
SamlSecurityRealmRoleMappingRule
SamlSecurityRealmRoleMappingRules
SamlSettings
SamlSpSettings
SearchRequest
SecurityDeployment
SecurityDeploymentCreateRequest
SecurityDeploymentTopology
SecurityDeploymentUpdateRequest
SecurityRealmInfo
SecurityRealmInfoList
SecurityRealmsReorderRequest
ServiceUrl
SimplifiedLineItem
SnapshotRepositoryConfiguration
SnapshotStatusInfo
SsoAuthenticationMethodInfo
StackVersionApmConfig
StackVersionAppSearchConfig
StackVersionArchiveProcessingError
StackVersionArchiveProcessingResult
StackVersionConfig
StackVersionConfigPost
StackVersionConfigs
StackVersionElasticsearchConfig
StackVersionEnterpriseSearchConfig
StackVersionInstanceCapacityConstraint
StackVersionIntegrationsServerConfig
StackVersionKibanaConfig
StackVersionMetadata
StackVersionNodeType
StackVersionTemplateFileHash
StackVersionTemplateInfo
TargetElasticsearchCluster
TelemetryConfig
TelemetryConfigRequest
TermQuery
TlsPublicCertChain
TokenResponse
TopologyElementControl
TopologySize
TrafficFilterEgressRule
TrafficFilterRule
TrafficFilterRulesetInfo
TrafficFilterRulesetRequest
TrafficFilterRulesetResponse
TrafficFilterRulesets
TrafficFilterSettings
TransientApmPlanConfiguration
TransientAppSearchPlanConfiguration
TransientElasticsearchPlanConfiguration
TransientEnterpriseSearchPlanConfiguration
TransientIntegrationsServerPlanConfiguration
TransientKibanaPlanConfiguration
TrustRelationshipCreateRequest
TrustRelationshipCreateResponse
TrustRelationshipGetResponse
TrustRelationshipUpdateRequest
TrustRelationshipUpdateResponse
TrustRelationshipsListResponse
UnreachableRegionInfo
UpdateExtensionRequest
UpdatedTlsChain
Updates
UsageStats
User
UserApiKey
UserList
UserMetadata
UserSecurity
UserSecurityRealm
ZookeeperNodeInfo
ZookeeperSummary
- Changes to index allocation and API
- Script reference
- Glossary
- Release notes
- Elastic Cloud Enterprise 3.1.1
- Elastic Cloud Enterprise 3.1.0
- Elastic Cloud Enterprise 3.0.0
- Elastic Cloud Enterprise 2.13.4
- Elastic Cloud Enterprise 2.13.3
- Elastic Cloud Enterprise 2.13.2
- Elastic Cloud Enterprise 2.13.1
- Elastic Cloud Enterprise 2.13.0
- Elastic Cloud Enterprise 2.12.4
- Elastic Cloud Enterprise 2.12.3
- Elastic Cloud Enterprise 2.12.2
- Elastic Cloud Enterprise 2.12.1
- Elastic Cloud Enterprise 2.12.0
- Elastic Cloud Enterprise 2.11.2
- Elastic Cloud Enterprise 2.11.1
- Elastic Cloud Enterprise 2.11.0
- Elastic Cloud Enterprise 2.10.1
- Elastic Cloud Enterprise 2.10.0
- Elastic Cloud Enterprise 2.9.2
- Elastic Cloud Enterprise 2.9.1
- Elastic Cloud Enterprise 2.9.0
- Elastic Cloud Enterprise 2.8.1
- Elastic Cloud Enterprise 2.8.0
- Elastic Cloud Enterprise 2.7.2
- Elastic Cloud Enterprise 2.7.1
- Elastic Cloud Enterprise 2.7.0
- Elastic Cloud Enterprise 2.6.2
- Elastic Cloud Enterprise 2.6.1
- Elastic Cloud Enterprise 2.6.0
- Elastic Cloud Enterprise 2.5.1
- Elastic Cloud Enterprise 2.5.0
- Elastic Cloud Enterprise 2.4.3
- Elastic Cloud Enterprise 2.4.2
- Elastic Cloud Enterprise 2.4.1
- Elastic Cloud Enterprise 2.4.0
- Elastic Cloud Enterprise 2.3.2
- Elastic Cloud Enterprise 2.3.1
- Elastic Cloud Enterprise 2.3.0
- Elastic Cloud Enterprise 2.2.3
- Elastic Cloud Enterprise 2.2.2
- Elastic Cloud Enterprise 2.2.1
- Elastic Cloud Enterprise 2.2.0
- Elastic Cloud Enterprise 2.1.1
- Elastic Cloud Enterprise 2.1.0
- Elastic Cloud Enterprise 2.0.1
- Elastic Cloud Enterprise 2.0.0
- Elastic Cloud Enterprise 1.1.5
- Elastic Cloud Enterprise 1.1.4
- Elastic Cloud Enterprise 1.1.3
- Elastic Cloud Enterprise 1.1.2
- Elastic Cloud Enterprise 1.1.1
- Elastic Cloud Enterprise 1.1.0
- Elastic Cloud Enterprise 1.0.2
- Elastic Cloud Enterprise 1.0.1
- Elastic Cloud Enterprise 1.0.0
- What’s new with the Elastic Stack
- About this product
Ingest data with Python on Elastic Cloud Enterprise
editIngest data with Python on Elastic Cloud Enterprise
editThis guide tells you how to get started with:
- Securely connecting to Elastic Cloud Enterprise with Python
- Ingesting data into your deployment from your application
- Searching and modifying your data on Elastic Cloud Enterprise
If you are an Python application programmer who is new to the Elastic Stack, this content can help you get started more easily.
Time required: 45 minutes
Prerequisites
editThese steps are applicable to your existing application. If you don’t have one, you can use the example included here to create one.
Get the elasticsearch
packages
editpython -m pip install elasticsearch python -m pip install elasticsearch-async
Create the setup.py
file
edit# Elasticsearch 7.x elasticsearch>=7.0.0,<8.0.0
Create a deployment
edit- Log into the Elastic Cloud Enterprise admin console.
- Select Create deployment.
- Give your deployment a name. You can leave all other settings at their default values.
- Select Create deployment and save your Elastic deployment credentials. You will need these credentials later on.
-
You also need the Cloud ID later on, as it simplifies sending data to Elastic Cloud Enterprise. Select the deployment name from the Elastic Cloud Enterprise portal or the Deployments page and copy down the information under Cloud ID:
Connect securely
editWhen connecting to Elastic Cloud Enterprise you can use a Cloud ID to specify the connection details. You must pass the Cloud ID that you can find in the cloud console.
To connect to, stream data to, and issue queries with Elastic Cloud Enterprise, you need to think about authentication. Two authentication mechanisms are supported, API key and basic authentication. Here, to get you started quickly, we’ll show you how to use basic authentication, but you can also generate API keys as shown later on. API keys are safer and preferred for production environments.
Basic authentication
editFor basic authentication, use the same deployment credentials (username
and password
parameters) and Cloud ID you copied down earlier when you created your deployment. (If you did not save the password, you can
reset the password.
.)
You first need to create and edit an example.ini
file with your deployment details:
[ELASTIC] cloud_id = DEPLOYMENT_NAME:CLOUD_ID_DETAILS user = elastic password = LONGPASSWORD ca = /path/to/your/elastic-ece-ca-cert.pem
This line is only used when you have a self signed certificate for your Elastic Cloud Enterprise proxy. If needed, specify the full path to the PEM formatted root cetificate (Root CA) used for the Elastic Cloud Enterprise proxy. You can retrieve the certificate chain from your ECE system by following the instructions in Get existing ECE security certificates. Save the final certificate in the chain to a file. In this example, the file is named |
The following examples are to be typed into the Python interpreter in interactive mode. The prompts have been removed to make it easier for you to copy the samples, the output from the interpreter is shown unmodified.
Import libraries and read in the configuration
edit❯ python3 Python 3.9.6 (default, Jun 29 2021, 05:25:02) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin Type "help", "copyright", "credits" or "license" for more information. from elasticsearch import Elasticsearch, helpers import configparser from ssl import create_default_context config = configparser.ConfigParser() config.read('example.ini')
Output
edit['example.ini'] >>>
Instantiate the Elasticsearch connection
editcontext = create_default_context(cafile=config['ELASTIC']['ca']) es = Elasticsearch( cloud_id=config['ELASTIC']['cloud_id'], scheme="https", use_ssl=True, verify_certs=False, ssl_show_warn=False, http_auth=(config['ELASTIC']['user'], config['ELASTIC']['password']) )
You can now confirm that you have connected to the deployment by returning some information about the deployment:
es.info()
Output
edit{'name': 'instance-0000000000', 'cluster_name': '747ab208fb70403dbe3155af102aef56', 'cluster_uuid': 'IpgjkPkVQ5efJY-M9ilG7g', 'version': {'number': '7.15.0', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': '79d65f6e357953a5b3cbcc5e2c7c21073d89aa29', 'build_date': '2021-09-16T03:05:29.143308416Z', 'build_snapshot': False, 'lucene_version': '8.9.0', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}
Ingest data
editAfter connecting to your deployment, you are ready to index and search data. Let’s create a new index, insert some quotes from our favorite characters, and then refresh the index so that it is ready to be searched. A refresh makes all operations performed on an index since the last refresh available for search.
Index a document
edites.index( index='lord-of-the-rings', document={ 'character': 'Aragon', 'quote': 'It is not this day.' })
Output
edit{'_index': 'lord-of-the-rings', '_type': '_doc', '_id': 'IanWEnwBg_mH2XweqDqg', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 34, '_primary_term': 1}
Index another record
edites.index( index='lord-of-the-rings', document={ 'character': 'Gandalf', 'quote': 'A wizard is never late, nor is he early.' })
Output
edit{'_index': 'lord-of-the-rings', '_type': '_doc', '_id': 'IqnWEnwBg_mH2Xwezjpj', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 35, '_primary_term': 1}
Index a third record
edites.index( index='lord-of-the-rings', document={ 'character': 'Frodo Baggins', 'quote': 'You are late' })
Output
edit{'_index': 'lord-of-the-rings', '_type': '_doc', '_id': 'I6nWEnwBg_mH2Xwe_Tre', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 36, '_primary_term': 1}
Refresh the index
edites.indices.refresh(index='lord-of-the-rings')
Output
edit{'_shards': {'total': 2, 'successful': 1, 'failed': 0}}
When using the es.index
API, the request automatically creates the lord-of-the-rings
index, if it doesn’t exist already, as well as document IDs for each indexed document if they are not explicitly specified.
Search and modify data
editAfter creating a new index and ingesting some data, you are now ready to search. Let’s find what different characters have said things about being late:
result = es.search( index='lord-of-the-rings', query={ 'match': {'quote': 'late'} } ) result['hits']['hits']
Output
edit[{'_index': 'lord-of-the-rings', '_type': '_doc', '_id': '2EkAzngB_pyHD3p65UMt', '_score': 0.5820575, '_source': {'character': 'Frodo Baggins', 'quote': 'You are late'}}, {'_index': 'lord-of-the-rings', '_type': '_doc', '_id': '10kAzngB_pyHD3p65EPR', '_score': 0.37883914, '_source': {'character': 'Gandalf', 'quote': 'A wizard is never late, nor is he early.'}}]
The search request returns content of documents containing late
in the quote field, including document IDs that were automatically generated.
You can make updates to specific documents using document IDs. Let’s add a birthplace for our character:
This update example uses the field |
Output
edites.get(index='lord-of-the-rings', id='2EkAzngB_pyHD3p65UMt') {'_index': 'lord-of-the-rings', '_type': '_doc', '_id': '2EkAzngB_pyHD3p65UMt', '_version': 2, '_seq_no': 3, '_primary_term': 1, 'found': True, '_source': {'character': 'Frodo Baggins', 'quote': 'You are late', 'birthplace': 'The Shire'}}
For frequently used API calls with the Python client, check Examples.
Switch to API key authentication
editTo get started, authentication to Elasticsearch used the elastic
superuser and password, but an API key is much safer and a best practice for production.
In the example that follows, an API key is created with the cluster monitor
privilege which gives read-only access for determining the cluster state. Some additional privileges also allow create_index
, write
, read
, and manage
operations for the specified index. The index manage
privilege is added to enable index refreshes.
The easiest way to create this key is in the API console for your deployment. Select the deployment name and go to Elasticsearch > API console:
POST /_security/api_key { "name": "python_example", "role_descriptors": { "python_read_write": { "cluster": ["monitor"], "index": [ { "names": ["test-index"], "privileges": ["create_index", "write", "read", "manage"] } ] } } }
The output is:
edit{ "id" : "API_KEY_ID", "name" : "python_example", "api_key" : "API_KEY_DETAILS" }
Edit the example.ini
file you created earlier and add the id
and api_key
you just created. You should also remove the lines for user
and password
you added earlier after you have tested the api_key
, and consider changing the elastic
password using the Cloud UI.
[DEFAULT] cloud_id = DEPLOYMENT_NAME:CLOUD_ID_DETAILS apikey_id = API_KEY_ID apikey_key = API_KEY_DETAILS
You can now use the API key in place of a username and password. The client connection becomes:
es = Elasticsearch( cloud_id=config['DEFAULT']['cloud_id'], api_key=(config['DEFAULT']['apikey_id'], config['DEFAULT']['apikey_key']), )
Check Create API key API to learn more about API Keys and Security privileges to understand which privileges are needed. If you are not sure what the right combination of privileges for your custom application is, you can enable audit logging on Elasticsearch to find out what privileges are being used. To learn more about how logging works on Elastic Cloud Enterprise, check Monitoring Elastic Cloud deployment logs and metrics.
For more information on refreshing an index, searching, updating, and deleting, check the elasticsearch-py examples.
Best practices
edit- Security
-
When connecting to Elastic Cloud Enterprise, the client automatically enables both request and response compression by default, since it yields significant throughput improvements. Moreover, the client also sets the SSL option
secureProtocol
toTLSv1_2_method
unless specified otherwise. You can still override this option by configuring it.Do not enable sniffing when using Elastic Cloud Enterprise, since the nodes are behind a load balancer. Elastic Cloud Enterprise takes care of everything for you. Take a look at Elasticsearch sniffing best practices: What, when, why, how if you want to know more.
- Schema
- When the example code is run, an index mapping is created automatically. The field types are selected by Elasticsearch based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you design the schema for your production use cases.
- Ingest
-
For more advanced scenarios, Bulk helpers gives examples for the
bulk
API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually.
On this page
- Prerequisites
- Get the
elasticsearch
packages - Create the
setup.py
file - Create a deployment
- Connect securely
- Basic authentication
- Import libraries and read in the configuration
- Output
- Instantiate the Elasticsearch connection
- Output
- Ingest data
- Index a document
- Output
- Index another record
- Output
- Index a third record
- Output
- Refresh the index
- Output
- Search and modify data
- Output
- Output
- Switch to API key authentication
- The output is:
- Best practices