Product

Introducing a new architecture for Kibana

By
Josh Dover

We’re excited to share more details today about a long-running project that the Kibana team has been working on for the past couple of years: the Kibana development platform. 

If you’re a Kibana plugin developer or enjoy lurking around our GitHub repo, you’ve probably seen mentions of “the new platform.” This is a re-architecture project we’re close to completing that enables the future of Kibana plugin development for both developers at Elastic and the wider Kibana community. So why did we build this? Let’s start with a bit of history.

Kibana 4.0, a modest JavaScript app

The first “architecture” of Kibana was put together all the way back in 2015, starting with Kibana 4.0. At the time, we had just three apps: Discover, Visualize, and Dashboard. Even more critically, we only had four full-time developers.

Kibana 4.0: Not too shabby for only having four full-time developers

With such a small team, we reached for familiar libraries and built Kibana around them: Angular.js, Express, and RequireJS. We added Hapi.js and used Hapi’s plugin system to build server-side plugins. On the front end, each application was a separate Angular.js app built on some shared code. Using familiar tools allowed us to iterate quickly, avoid reinventing the wheel, and ship features at a whiplash pace. It worked well!

Fast forward a few years and Kibana comprised a whopping 97 plugins and was actively worked on by over 100 Elastic developers. Kibana had become one of the largest open source JavaScript apps in the world. The simple architecture we had built was starting to bow under this weight. Writing cross-application integrations was difficult, unintended breakages were common, development performance was poor, and ramping up new developers on Kibana was difficult.

Our patchwork system of solutions had finally taken us as far as it could and it was clear we needed to think again from first principles. A group of Kibana engineers set out to design a new system that could scale with Kibana, our team, and the community: the Kibana development platform.

The Kibana development platform, a grown-up application framework

Goals

The primary goal of the Kibana platform project was to increase both the velocity and stability of adding new features. To fulfill this goal, we needed to make Kibana’s codebase easy to change, easy to learn, and easy to thoroughly test.

To achieve these goals we produced a list of features that this new architecture should have:

  • Consistent architecture across client and server code
  • A small, explicit set of foundational APIs that is separate from plugin code
  • Simple and well-defined plugin runtime and execution flow
  • Isolated plugins with explicit APIs and dependencies
  • Framework-agnostic and futureproof APIs
  • Full test coverage
  • Type safety

Microservices

One of the key problems with our legacy code was how much it was slowing down individual teams from iterating quickly.

The popular microservices trend is often the recommended pattern to reach for when re-architecting a system to solve this problem. By breaking up responsibilities into independent systems with clear boundaries, each team can work within their domains with fewer cross-cutting concerns.

Microservices proper aren’t an option for Kibana, which is deployed as a single artifact on millions of our customers’ own machines. Simple deployment and configuration is a goal of all products at Elastic. Breaking up Kibana into smaller, separate deployments would complicate that.

However, similar benefits can be realized by enabling strict encapsulation between components. The original Kibana architecture allowed for many components to affect the entire system, often in unexpected ways. By facilitating and enforcing encapsulation with explicit boundaries and dependencies between subsystems, we focused on achieving many of the same benefits without complicating our deployment story.

We see the Kibana platform as a hybrid approach: a single artifact that you can download and run on your laptop, but built with explicit boundaries between its pluggable systems.

With this design, we avoid the complexity of deploying microservices and handling network failures, while still promoting the isolation between systems that enables teams to move quickly.

Key aspects of the Kibana platform

Core

The Kibana core is the root system that boots Kibana, validates configuration, loads plugins, and provides the primitive APIs needed to build a plugin in Kibana. When you run ./bin/kibana or load Kibana in your browser this is the system that starts executing first.

The core is made up of a set of services, each of which provides APIs to plugins at different points of the system’s lifecycle. Core services are always present and cannot be disabled. Anything we consider to be essential to building a Kibana plugin exists as a core service.

The underlying implementation details of core services are encapsulated and hidden from plugins. This yields a number of benefits, not the least of which is high test coverage. By only allowing plugins access to purpose-built features, we can ensure that the core’s small API surface is incredibly robust.

The core also aims to be framework and technology agnostic. We want this platform to grow with the project for years to come. Locking the entire project into one technology has bitten us in the past (Angular.js, Hapi.js) and is a decision we plan to avoid going forward. When possible, we prefer to provide primitives or well-formed abstractions over specific frameworks. For instance, applications in the Kibana platform can be written using virtually any UI framework (though we prefer React and our EUI library) and our HTTP interface is a very generic abstraction over Hapi.js. This allows plugins to experiment with and adopt new technologies and allows the core to remove underlying frameworks without breaking the interface provided to plugins.

Plugins

While the core provides the foundation of Kibana, plugins are where the magic happens. Virtually every feature you use in Kibana is built inside of a plugin. In general, a plugin is a group of functionalities that can be toggled on or off to provide features and apps in Kibana.

Plugins have access to all of the APIs exposed by core services. But the really neat thing about plugins is that they may have dependencies on other plugins. This allows plugins to integrate at runtime via explicit contracts that these plugins expose to one another. Plugins can expose something as simple as a small integration API or provide a feature-rich service to other plugins. This allows us to keep adding new services to Kibana without expanding the API surface of the core. By baking this feature into the platform’s design, we can make sure that changes to these interfaces are well understood and communicated.

So what can a plugin do? Well, they can register HTTP endpoints and UI applications, query and create data in Elasticsearch, and provide generic services to other plugins. In other words, quite a bit.

Lifecycles

All core services and plugins are organized and executed in the same set of lifecycle stages: setup, start, and stop. Both the client and server execute these lifecycle stages sequentially when starting Kibana.

Different sets of functionality are available during each lifecycle stage. It is up to each service and plugin to return the APIs it wishes to expose during these lifecycle stages for other services and plugins to consume.

By organizing all of Kibana around these stages, we make reasoning about when code will execute much simpler and have tight control over which features are available to other components at different points in time.

Server and browser

Up until this point we’ve talked about the core as if it’s a single system. However, that isn’t quite true. In reality, we have a server-side and client-side core. Each follows a similar design but is made up of a slightly different set of services.

As you might expect, the server-side services provide typical backend functionality, like HTTP routes, while the client-side services provide frontend functionality, like UI mounting. But while the set of APIs is different, both systems share the same design, patterns, and lifecycle stages.

This makes learning how to build a Kibana plugin much simpler. In legacy Kibana, developers had to learn about Hapi’s plugin system on the server and Angular’s module system in the browser. Developers today can learn a single pattern for building plugins that is portable between the two environments.

Results

Leveraging this design, the Kibana teams have been incrementally migrating their plugins to the Kibana platform APIs over the past several minor versions. Today, we’re happy to report that all applications will be running on the platform as of version 7.9.0 (coming soon).

One of the biggest improvements users of Kibana will notice with this change is a much faster navigation experience between applications. We have removed the “loading screen of death” (as some call it) when switching between applications, enabling you to quickly find, analyze, and share your data. In 7.9, you will be able to flip between Dashboard, Maps, Canvas, APM, and all your other favorite Kibana apps nearly instantly! Gone are the days of the loading screen breaking your flow.

But this is just the beginning. Developers at Elastic are now able to build features faster, more efficiently, and produce better code. For Kibana users, this translates to more features in every release! No matter which solutions you are using — Elastic Enterprise Search, Observability, or Security — expect the capabilities to grow even faster.

Further reading for plugin developers

As we’ve rolled out these changes, we’ve introduced significant changes to our experimental plugin API available to non-Elastic plugin developers. If you are a developer that maintains a Kibana plugin, you will need to update your plugin to work with future versions of Kibana. Our incremental approach has led to churn in the experimental API and we generally recommend developing your Kibana platform plugin against 7.9 or higher to reduce the amount of work to upgrade. With this in mind, we will be disabling the legacy plugin API in 7.10.

Editor's Note — September 3, 2020: The version of Kibana in which the legacy plugin API will be disabled has been updated from 7.11 to 7.10.

The good news is that the Kibana platform is ready for you to test out today! See our migration guide for upgrading your plugin. We will be releasing additional documentation (including a full API reference) in the coming months. Keep in mind that the plugin API remains experimental and is subject to change. We’re excited to see what you build! If you have any questions, reach out to us in our Kibana forums. And as with all releases, as soon as 7.9 is out it will be available on Elastic Cloud. So keep an eye out and try it free.

Acknowledgments

This project has been years in the making and owes thanks to nearly every Kibana developer at Elastic. Thanks to Pierre Gayvallet, Rudolf Meijering, Jin Mu, and Mikhail Shustov and for taking it from concept to a hard-won reality. The Elastic Community thanks you!