- Kibana Guide: other versions:
- What is Kibana?
- What’s new in 8.3
- Kibana concepts
- Quick start
- Set up
- Install Kibana
- Configure Kibana
- Alerting and action settings
- APM settings
- Banners settings
- Enterprise Search settings
- Fleet settings
- i18n settings
- Logging settings
- Logs settings
- Metrics settings
- Monitoring settings
- Reporting settings
- Search sessions settings
- Secure settings
- Security settings
- Spaces settings
- Task Manager settings
- Telemetry settings
- URL drilldown settings
- Start and stop Kibana
- Access Kibana
- Securing access to Kibana
- Add data
- Upgrade Kibana
- Configure security
- Configure reporting
- Configure logging
- Configure monitoring
- Command line tools
- Production considerations
- Discover
- Dashboard and visualizations
- Canvas
- Maps
- Build a map to compare metrics by country or region
- Track, visualize, and alert on assets in real time
- Map custom regions with reverse geocoding
- Heat map layer
- Tile layer
- Vector layer
- Plot big data
- Search geographic data
- Configure map settings
- Connect to Elastic Maps Service
- Import geospatial data
- Troubleshoot
- Reporting and sharing
- Machine learning
- Graph
- Alerting
- Observability
- APM
- Security
- Dev Tools
- Fleet
- Osquery
- Stack Monitoring
- Stack Management
- REST API
- Get features API
- Kibana spaces APIs
- Kibana role management APIs
- User session management APIs
- Saved objects APIs
- Data views API
- Index patterns APIs
- Alerting APIs
- Action and connector APIs
- Cases APIs
- Import and export dashboard APIs
- Logstash configuration management APIs
- Machine learning APIs
- Short URLs APIs
- Get Task Manager health
- Upgrade assistant APIs
- Kibana plugins
- Troubleshooting
- Accessibility
- Release notes
- Developer guide
Daily Elasticsearch Snapshots
editDaily Elasticsearch Snapshots
editFor local development and CI, Kibana, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).
A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of Elasticsearch, and a JSON-based manifest file containing metadata about the distributions.
A dashboard is available that shows the current status and compatibility of the latest Elasticsearch snapshots.
Process Overview
edit- Elasticsearch snapshots are built for each current tracked branch of Kibana.
-
Each snapshot is uploaded to a public Google Cloud Storage bucket,
kibana-ci-es-snapshots-daily
.- At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first.
- Each snapshot is tested with the latest commit of the corresponding Kibana branch, using the full CI suite. 3a. If a test fails during snapshot verification the Kibana Operations team will skip it and create an issue for the team to fix the test, or work with the Elasticsearch team to get a fix implemented there. Once the fix is ready a Kibana PR can be opened to unskip the test.
-
After CI
- If the snapshot passes, it is promoted and automatically used in CI and local development.
- If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and Kibana.
Using the latest snapshot
editWhen developing locally, you may wish to use the most recent Elasticsearch snapshot, even if it’s failing CI. To do so, prefix your commands with the follow environment variable:
KBN_ES_SNAPSHOT_USE_UNVERIFIED=true
You can use this flag with any command that downloads and runs Elasticsearch snapshots, such as scripts/es
or the FTR.
For example, to run functional tests with the latest snapshot:
KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server
For Pull Requests
editCurrently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change.
-
Edit
Jenkinsfile
in the root of the Kibana repo -
Add
env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'
at the top of the file. - Commit the change
Your pull request should then use the latest snapshot the next time that it runs. Just don’t merge the change to Jenkinsfile
!
Google Cloud Storage buckets
editkibana-ci-es-snapshots-daily
editThis bucket stores snapshots that are created on a daily basis, and is the primary location used by kbn-es
to download snapshots.
Snapshots are automatically deleted after 10 days.
The file structure for this bucket looks like this:
-
<version>/manifest-latest.json
-
<version>/manifest-latest-verified.json
-
<version>/archives/<unique id>/*.tar.gz
-
<version>/archives/<unique id>/*.tar.gz.sha512
-
<version>/archives/<unique id>/manifest.json
kibana-ci-es-snapshots-permanent
editThis bucket stores only the most recently promoted snapshot for each version. Old snapshots are only deleted when new ones are uploaded.
This bucket serves as permanent snapshot storage for old branches/versions that are no longer being built. kbn-es
checks the daily bucket first, followed by this one if no snapshots were found.
The file structure for this bucket looks like this:
-
<version>/*.tar.gz
-
<version>/*.tar.gz.sha512
-
<version>/manifest.json
How snapshots are built, tested, and promoted
editEach day, a Jenkins job runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of Kibana.
Build
editThis Jenkins job builds the Elasticsearch snapshots and uploads them to GCS.
The Jenkins job pipeline definition is in the Kibana repo.
- Checkout Elasticsearch repo for the given branch/version.
-
Run
./gradlew -p distribution/archives assemble --parallel
to create all of the Elasticsearch distributions. - Create a tarball for each distribution.
- Create a manifest JSON file containing info about the distribution, as well as its download URL.
-
Upload the tarballs and manifest to a unique location in the GCS bucket
kibana-ci-es-snapshots-daily
.-
e.g.
<version>/archives/<unique id>
-
e.g.
-
Replace
<version>/manifest-latest.json
in GCS with this newest manifest.-
This allows the
KBN_ES_SNAPSHOT_USE_UNVERIFIED
flag to work.
-
This allows the
- Trigger the verification job, to run the full Kibana CI test suite with this snapshot.
Verification and Promotion
editThis Jenkins job tests the latest Elasticsearch snapshot with the full Kibana CI pipeline, and promotes if it there are no test failures.
The Jenkins job pipeline definition is in the Kibana repo.
- Checkout Kibana and set up CI environment as normal.
-
Set the
ES_SNAPSHOT_MANIFEST
env var to point to the latest snapshot manifest. - Run CI (functional tests, integration tests, etc).
-
After CI
- If there was a test failure or other build error, send out an e-mail notification and stop.
- If there were no errors, promote the snapshot.
Promotion is done as part of the same pipeline:
-
Replace the manifest at
kibana-ci-es-snapshots-daily/<version>/manifest-latest-verified.json
with the manifest from the tested snapshot.- At this point, the snapshot has been promoted and will automatically be used in CI and in local development.
-
Replace the snapshot at
kibana-ci-es-snapshots-permanent/<version>/
with the tested snapshot by copying all of the tarballs and the manifest file.
On this page