Testing

We use functional tests to make sure the Kibana UI works as expected. It replaces hours of manual testing by automating user interaction. To have better control over our functional test environment, and to make it more accessible to plugin authors, Kibana uses a tool called the FunctionalTestRunner.

The FunctionalTestRunner is very bare bones and gets most of its functionality from its config file, located at {kib-repo}blob/7.9/test/functional/config.js[test/functional/config.js]. If you’re writing a plugin outside the Kibana repo, you will have your own config file. See Functional Tests for Plugins outside the Kibana repo for more info.

There are three ways to run the tests depending on your goals:

When run without any arguments the FunctionalTestRunner automatically loads the configuration in the standard location, but you can override that behavior with the --config flag. List configs with multiple --config arguments.

There are also command line flags for --bail and --grep, which behave just like their mocha counterparts. For instance, use --grep=foo to run only tests that match a regular expression.

Logging can also be customized with --quiet, --debug, or --verbose flags.

Use the --help flag for more options.

The tests are written in mocha using @kbn/expect for assertions.

We use WebDriver Protocol to run tests in both Chrome and Firefox with the help of chromedriver and geckodriver. When the FunctionalTestRunner launches, remote service creates a new webdriver session, which starts the driver and a stripped-down browser instance. We use browser service and webElementWrapper class to wrap up Webdriver API.

The FunctionalTestRunner automatically transpiles functional tests using babel, so that tests can use the same ECMAScript features that Kibana source code uses. See {kib-repo}blob/7.9/style_guides/js_style_guide.md[style_guides/js_style_guide.md].

Provider:

Code run by the FunctionalTestRunner is wrapped in a function so it can be passed around via config files and be parameterized. Any of these Provider functions may be asynchronous and should return/resolve-to the value they are meant to provide. Provider functions will always be called with a single argument: a provider API (see the Provider API Section).

A config provider:

// config and test files use `export default`
export default function (/* { providerAPI } */) {
  return {
    // ...
  }
}

// on CI test suite will be run twice: in Chrome and Firefox
describe('My Cross-browser Test Suite', function () {
  this.tags('includeFirefox');

  it('My First Test');
}

If the tests do not apply to Firefox, assign the skipFirefox tag.

To run tests on Firefox locally, use config.firefox.js:

node scripts/functional_test_runner --config test/functional/config.firefox.js

Tests should run at the positive security boundry condition, meaning that they should be run with the mimimum privileges required (and documented) and not as the superuser. This prevents the type of regression where additional privleges accidentally become required to perform the same action.

The functional UI tests now default to logging in with a user named test_user and the roles of this user can be changed dynamically without logging in and out.

In order to achieve this a new service was introduced called createTestUserService (see test/common/services/security/test_user.ts). The purpose of this test user service is to create roles defined in the test config files and setRoles() or restoreDefaults().

An example of how to set the role like how its defined below:

await security.testUser.setRoles(['kibana_user', 'kibana_date_nanos']);

Here we are setting the test_user to have the kibana_user role and also role access to a specific data index (kibana_date_nanos).

Tests should normally setRoles() in the before() and restoreDefaults() in the after().

This annotated example file shows the basic structure every test suite uses. It starts by importing @kbn/expect and defining its default export: an anonymous Test Provider. The test provider then destructures the Provider API for the getService() and getPageObjects() functions. It uses these functions to collect the dependencies of this suite. The rest of the test file will look pretty normal to mocha.js users. describe(), it(), before() and the lot are used to define suites that happen to automate a browser via services and objects of type PageObject.

import expect from '@kbn/expect';
// test files must `export default` a function that defines a test suite
export default function ({ getService, getPageObject }) {

  // most test files will start off by loading some services
  const retry = getService('retry');
  const testSubjects = getService('testSubjects');
  const esArchiver = getService('esArchiver');

  // for historical reasons, PageObjects are loaded in a single API call
  // and returned on an object with a key/value for each requested PageObject
  const PageObjects = getPageObjects(['common', 'visualize']);

  // every file must define a top-level suite before defining hooks/tests
  describe('My Test Suite', () => {

    // most suites start with a before hook that navigates to a specific
    // app/page and restores some archives into {es} with esArchiver
    before(async () => {
      await Promise.all([
        // start with an empty .kibana index
        esArchiver.load('empty_kibana'),
        // load some basic log data only if the index doesn't exist
        esArchiver.loadIfNeeded('makelogs')
      ]);
      // go to the page described by `apps.visualize` in the config
      await PageObjects.common.navigateTo('visualize');
    });

    // right after the before() hook definition, add the teardown steps
    // that will tidy up {es} for other test suites
    after(async () => {
      // we unload the empty_kibana archive but not the makelogs
      // archive because we don't make any changes to it, and subsequent
      // suites could use it if they call `.loadIfNeeded()`.
      await esArchiver.unload('empty_kibana');
    });

    // This series of tests illustrate how tests generally verify
    // one step of a larger process and then move on to the next in
    // a new test, each step building on top of the previous
    it('Vis Listing Page is empty');
    it('Create a new vis');
    it('Shows new vis in listing page');
    it('Opens the saved vis');
    it('Respects time filter changes');
    it(...
  });

}

The first and only argument to all providers is a Provider API Object. This object can be used to load service/page objects and config/test files.

Within config files the API has the following properties

Within service and PageObject Providers the API is:

Within a test Provider the API is exactly the same as the service providers API but with an additional method:

The FunctionalTestRunner comes with three built-in services:

The Kibana functional tests define the vast majority of the actual functionality used by tests.

Full list of services that are used in functional tests can be found here: {kib-repo}blob/7.9/test/functional/services[test/functional/services]

Services are intentionally generic. They can be literally anything (even nothing). Some services have helpers for interacting with a specific types of UI elements, like pointSeriesVis, and others are more foundational, like log or config. Whenever you want to provide some functionality in a reusable package, consider making a custom service.

To create a custom service somethingUseful:

The purpose for each PageObject is pretty self-explanatory. The visualize PageObject provides helpers for interacting with the visualize app, dashboard is the same for the dashboard app, and so on.

One exception is the "common" PageObject. A holdover from the intern implementation, the common PageObject is a collection of helpers useful across pages. Now that we have shareable services, and those services can be shared with other FunctionalTestRunner configurations, we will continue to move functionality out of the common PageObject and into services.

Please add new methods to existing or new services rather than further expanding the CommonPage class.

Remember that you can’t run an individual test in the file (it block) because the whole describe needs to be run in order. There should only be one top level describe in a file.

Another important gotcha is writing stable tests by being mindful of timing. All methods on remote run asynchronously. It’s better to write interactions that wait for changes on the UI to appear before moving onto the next step.

For example, rather than writing an interaction that simply clicks a button, write an interaction with the a higher-level purpose in mind:

Bad example: PageObjects.app.clickButton()

class AppPage {
  // what can people who call this method expect from the
  // UI after the promise resolves? Since the reaction to most
  // clicks is asynchronous the behavior is dependant on timing
  // and likely to cause test that fail unexpectedly
  async clickButton () {
    await testSubjects.click(‘menuButton’);
  }
}

Good example: PageObjects.app.openMenu()

class AppPage {
  // unlike `clickButton()`, callers of `openMenu()` know
  // the state that the UI will be in before they move on to
  // the next step
  async openMenu () {
    await testSubjects.click(‘menuButton’);
    await testSubjects.exists(‘menu’);
  }
}

Writing in this way will ensure your test timings are not flaky or based on assumptions about UI updates after interactions.

From the command line run:

node --debug-brk --inspect scripts/functional_test_runner

This prints out a URL that you can visit in Chrome and debug your functional tests in the browser.

You can also see additional logs in the terminal by running the FunctionalTestRunner with the --debug or --verbose flag. Add more logs with statements in your tests like

// load the log service
const log = getService(‘log’);

// log.debug only writes when using the `--debug` or `--verbose` flag.
log.debug(‘done clicking menu’);

macOS users on a machine with a discrete graphics card may see significant speedups (up to 2x) when running tests by changing your terminal emulator’s GPU settings. In iTerm2: * Open Preferences (Command + ,) * In the General tab, under the "Magic" section, ensure "GPU rendering" is checked * Open "Advanced GPU Settings…​" * Uncheck the "Prefer integrated to discrete GPU" option * Restart iTerm

Kibana is migrating unit testing from Mocha to Jest. Legacy unit tests still exist in Mocha but all new unit tests should be written in Jest.

Mocha tests are contained in __tests__ directories.

Running Mocha Unit Tests

yarn test:mocha

Jest tests are stored in the same directory as source code files with the .test.{js,mjs,ts,tsx} suffix.

Running Jest Unit Tests

yarn test:jest

In order to write those tests there are two main things you need to be aware of. The first one is the different between jest.mock and jest.doMock and the second one our jest mocks file pattern. As we are running js and ts test files with babel-jest both techniques are needed specially for the tests implemented on Typescript in order to benefit from the auto-inference types feature.

Both methods are essentially the same on their roots however the jest.mock calls will get hoisted to the top of the file and can only reference variables prefixed with mock. On the other hand, jest.doMock won’t be hoisted and can reference pretty much any variable we want, however we have to assure those referenced variables are instantiated at the time we need them which lead us to the next section where we’ll talk about our jest mock files pattern.

Specially on typescript it is pretty common to have in unit tests jest.doMock calls which reference for example imported types. Any error will thrown from doing that however the test will fail. The reason behind that is because despite the jest.doMock isn’t being hoisted by babel-jest the import with the types we are referencing will be hoisted to the top and at the time we’ll call the function that variable would not be defined.

In order to prevent that we develop a protocol that should be followed:

The standard yarn test task runs several sub tasks and can take several minutes to complete, making debugging failures pretty painful. In order to ease the pain specialized tasks provide alternate methods for running the tests.

You could also add the --debug option so that node is run using the --debug-brk flag. You’ll need to connect a remote debugger such as node-inspector to proceed in this mode.

node scripts/mocha --debug <file>

With yarn test:karma, you can run only the browser tests. Coverage reports are available for browser tests by running yarn test:coverage. You can find the results under the coverage/ directory that will be created upon completion.

yarn test:karma

Using yarn test:karma:debug initializes an environment for debugging the browser tests. Includes an dedicated instance of the Kibana server for building the test bundle, and a karma server. When running this task the build is optimized for the first time and then a karma-owned instance of the browser is opened. Click the "debug" button to open a new tab that executes the unit tests.

yarn test:karma:debug

In the screenshot below, you’ll notice the URL is localhost:9876/debug.html. You can append a grep query parameter to this URL and set it to a string value which will be used to exclude tests which don’t match. For example, if you changed the URL to localhost:9876/debug.html?query=my test and then refreshed the browser, you’d only see tests run which contain "my test" in the test description.

This should work super if you’re using the Kibana plugin generator. If you’re not using the generator, well, you’re on your own. We suggest you look at how the generator works.

To run the tests for just your particular plugin run the following command from your plugin:

yarn test:mocha
yarn test:karma:debug # remove the debug flag to run them once and close

To run the tests locally:

To run the x-pack tests, swap the config file out for x-pack/test/accessibility/config.ts.

After the server is up, you can go to this instance of Kibana at localhost:5620.

The testing is done using axe. The same thing that runs in CI, can be run locally using their browser plugins: