Configure the validating webhook

ECK can be configured to provide a validating webhook that validates Elastic custom resources (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, and Elastic Maps Server) before they are created or updated. Validating webhooks provide immediate feedback if a submitted manifest contains invalid or illegal configuration — which can help you catch errors early and save time that would otherwise be spent on troubleshooting.

Validating webhooks are defined using a ValidatingWebhookConfiguration object that defines the following:

When using the default operator.yaml manifest, ECK is installed with a ValidatingWebhookConfiguration configured as follows:

If you installed ECK without the webhook and want to enable it later on, or if you want to customise the configuration such as providing your own certificates, this section describes the options available to you.

You can customise almost all aspects of the webhook setup by changing the operator configuration.

This section describes how you can use your own certificates for the webhook instead of letting the operator manage them automatically. There are a few important things to be aware of when going down this route:

helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace \
  --set=webhook.manageCerts=false \
  --set=webhook.certsSecret=elastic-webhook-server-custom-cert \
  --set=webhook.caBundle=$(base64 -w 0 ca.crt)

This section describes how to use cert-manager to manage the webhook certificate. It assumes that there is a ClusterIssuer named self-signing-issuer available.

helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace \
  --set=webhook.manageCerts=false \
  --set=webhook.certsSecret=elastic-webhook-server-cert \
  --set=webhook.certManagerCert=elastic-webhook-server-cert

To disable the webhook, set the enable-webhook operator configuration flag to false and remove the ValidatingWebhookConfiguration named elastic-webhook.k8s.elastic.co:

kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io elastic-webhook.k8s.elastic.co

You might see errors in your Kubernetes API server logs indicating that it cannot reach the operator service (elastic-webhook-server). This could be because no operator pods are available to handle request or because a network policy or a firewall rule is preventing the control plane from accessing the service. To help with troubleshooting, you can change the failurePolicy of the webhook configuration to Fail. This will cause create or update operations to fail if there is an error contacting the webhook. Usually the error message will contain helpful information about the failure that will allow you to diagnose the root cause.

Webhooks require network connectivity between the Kubernetes API server and the operator. If the creation of an Elasticsearch resource times out with an error message similar to the following, then the Kubernetes API server might be unable to connect to the webhook to validate the manifest.

Error from server (Timeout): error when creating "elasticsearch.yaml": Timeout: request did not complete within requested timeout 30s

If you get this error, try re-running the command with a higher request timeout as follows:

kubectl --request-timeout=1m apply -f elasticsearch.yaml

As the default failurePolicy of the webhook is Ignore, the above command should succeed after about 30 seconds. This is an indication that the API server cannot contact the webhook server and has foregone validation when creating the resource.

On GKE private clusters, you may have to add a firewall rule allowing access to port 9443 from the API server so that it can contact the webhook. See the GKE documentation on firewall rules and the Kubernetes issue for more details.

It is possible that a network policy is blocking any incoming requests to the webhook server. Consult your system administrator to determine whether that is the case, and create an appropriate policy to allow communication between the Kubernetes API server and the webhook server. For example, the following network policy simply opens up the webhook port to the world:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-webhook-access-from-any
  namespace: elastic-system
spec:
  podSelector:
    matchLabels:
      control-plane: elastic-operator
  ingress:
  - from: []
    ports:
      - port: 9443

If you want to restrict the webhook access only to the Kubernetes API server, you must know the IP address of the API server, that you can obtain through this command:

kubectl cluster-info | grep master

Assuming that the API server IP address is 10.1.0.1, the following policy restricts webhook access to just the API server.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-webhook-access-from-apiserver
  namespace: elastic-system
spec:
  podSelector:
    matchLabels:
      control-plane: elastic-operator
  ingress:
  - from:
      - ipBlock:
          cidr: 10.1.0.1/32
    ports:
      - port: 9443

If your attempts to update a resource fail with an error message similar to the following, you can force the webhook to ignore it by removing the kubectl.kubernetes.io/last-applied-configuration annotation from your resource.

admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "quickstart" is invalid: some-misspelled-field: Invalid value: "some-misspelled-field": some-misspelled-field field found in the kubectl.kubernetes.io/last-applied-configuration annotation is unknown