Red Hat Enterprise Linux (RHEL) and CentOS

edit

This section provides you instructions for preparing your hosts with the following OS distributions:

Red Hat Enterprise Linux 8 (RHEL 8) with Podman
edit

The following instructions show you how to prepare your hosts on Red Hat Enterprise Linux 8 (RHEL 8 ) with Podman. Podman is recommended when using RHEL 8 distributions.

Prerequisites
edit

Create a RHEL 8 VM (the version must be >= 8.5, but <9). Follow your internal guidelines to add a vanilla RHEL 8 VM to your environment. Note that the version must be >= 8.5, but <9.

Verify that required traffic is allowed. Check the Networking prerequisites and Google Cloud Platform (GCP) guidelines for a list of ports that need to be open. The technical configuration highly depends on the underlying infrastructure.

+ Example: For AWS, allowing traffic between hosts is implemented using security groups.

Configure the RHEL 8 Host
edit
  1. Install the OS packages lvm2, iptables, sysstat, and net-tools by executing

    sudo dnf install lvm2 iptables sysstat net-tools 

    The ECE diagnostic script requires net-tools.

  2. Remove docker and previously installed podman packages (if previously installed)

    sudo dnf remove docker docker-ce podman podman-remote containerd.io
  3. As a sudoers user, disable SELinux by adding the following parameter to /etc/selinux/config

    SELINUX=disabled
  4. Install podman version 4.2.*.

    sudo dnf install podman-4.2.* podman-remote-4.2.*
  5. If podman requires a proxy in your infrastructure setup, modify the /usr/share/containers/containers.conf file and add the HTTP_PROXY and HTTPS_PROXY environment variables in the [engine] section. Please note that multiple env variables in that configuration file exists — use the one in the [engine] section.

    Example:

    [engine]
    env = ["HTTP_PROXY=http://{proxy-ip}:{proxy-port}", "HTTPS_PROXY=http://{proxy-ip}:{proxy-port}"]
  6. Reload systemd configuration

    sudo systemctl daemon-reload
  7. Create OS groups, if they do not exist yet

    Reference: Users and permissions

    sudo groupadd elastic
    sudo groupadd podman
  8. Add user elastic to the podman group

    Reference: Users and permissions

    sudo useradd -g "elastic" -G "podman" elastic
  9. As a sudoers user, add the following line to /etc/sudoers.d/99-ece-users

    Reference: Users and permissions

    elastic ALL=(ALL) NOPASSWD:ALL
  10. Add the required options to the kernel boot arguments

    sudo /sbin/grubby --update-kernel=ALL --args='cgroup_enable=memory cgroup.memory=nokmem swapaccount=1'
  11. Create the directory

    sudo mkdir -p /etc/systemd/system/podman.socket.d
  12. As a sudoers user, create the file /etc/systemd/system/podman.socket.d/podman.conf with the following content. Set the correct ownership and permission.

    Both ListenStream= and ListenStream=/var/run/docker.sock parameters are required!

    File content:

    [Socket]
    ListenStream=
    ListenStream=/var/run/docker.sock
    SocketMode=770
    SocketUser=elastic
    SocketGroup=podman

    File ownership and permission:

    sudo chown root:root /etc/systemd/system/podman.socket.d/podman.conf
    sudo chmod 0644 /etc/systemd/system/podman.socket.d/podman.conf
  13. As a sudoers user, create the (text) file /usr/bin/docker with the following content. Verify that the regular double quotes in the text file are used (ASCII code Hex 22)

    #!/bin/bash
    podman-remote --url unix:///var/run/docker.sock "$@"
  14. Set the file permissions on /usr/bin/docker

    sudo chmod 0755 /usr/bin/docker
  15. As a sudoers user, add the following two lines to section [storage] in the file /etc/containers/storage.conf. Verify that those parameters are only defined once. Either remove or comment out potentially existing parameters.

    runroot = "/mnt/data/docker/runroot/"
    graphroot = "/mnt/data/docker"
  16. Enable podman so that itself and running containers start automatically after a reboot

    sudo systemctl enable podman.service
    sudo systemctl enable podman-restart.service
  17. Enable the overlay kernel module (check Use the OverlayFS storage driver) that the Podman overlay storage driver uses (check Working with the Container Storage library and tools in Red Hat Enterprise Linux).

    In Docker world there are two overlay drivers, overlay and overlay2, today most users use the overlay2 driver, so we just use that one, and called it overlay. Refer also to Use the OverlayFS storage driver.

    echo "overlay" | sudo tee -a /etc/modules-load.d/overlay.conf
  18. Format the additional data partition

    sudo mkfs.xfs /dev/nvme1n1
  19. Create the /mnt/data/ directory used as a mount point

    sudo install -o elastic -g elastic -d -m 700 /mnt/data
  20. As a sudoers user, modify the entry for the XFS volume in the /etc/fstab file to add pquota,prjquota. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    Replace /dev/nvme1n1 in the following example with the corresponding device on your host, and add this example configuration as a single line to /etc/fstab.

    /dev/nvme1n1	/mnt/data	xfs	defaults,nofail,x-systemd.automount,prjquota,pquota  0 2
  21. Restart the local-fs target

    sudo systemctl daemon-reload
    sudo systemctl restart local-fs.target
  22. Set the permissions on the newly mounted device

    ls /mnt/data
    sudo chown elastic:elastic /mnt/data
  23. Create the /mnt/data/docker directory for the Docker service storage

    sudo install -o elastic -g elastic -d -m 700 /mnt/data/docker
  24. Disable the firewalld service. The service is not compatible with Podman and interferes with the installation of ECE. You must disable firewalld before installing or reinstalling ECE.

    If firewalld does not exist on your VM, you can skip this step.

    sudo systemctl disable firewalld
  25. Configure kernel parameters

    cat <<EOF | sudo tee -a /etc/sysctl.conf
    # Required by Elasticsearch 5.0 and later
    vm.max_map_count=262144
    # enable forwarding so the Docker networking works as expected
    net.ipv4.ip_forward=1
    # Decrease the maximum number of TCP retransmissions to 5 as recommended for Elasticsearch TCP retransmission timeout.
    # See https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config-tcpretries.html
    net.ipv4.tcp_retries2=5
    # Make sure the host doesn't swap too early
    vm.swappiness=1
    EOF
  26. Apply the new sysctl settings

    sudo sysctl -p
    sudo systemctl restart NetworkManager
  27. As a sudoers user, adjust the system limits. Add the following configuration values to the /etc/security/limits.conf file.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    elastic          soft    nproc          unlimited
    elastic          hard    nproc          unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
  28. NOTE: This step is optional if the docker registry doesn’t require authentication.

    Authenticate the elastic user to pull images from the docker registry you use, by creating the file /home/elastic/.docker/config.json. This file needs to be owned by the elastic user. If you are using a user name other than elastic, adjust the path accordingly.

    Example: In case you use docker.elastic.co, the file content looks like as follows:

    {
     "auths": {
       "docker.elastic.co": {
         "auth": "<auth-token>"
       }
     }
    }
  29. Restart the podman service by running this command:

    sudo systemctl daemon-reload
    sudo systemctl restart podman
  30. Reboot the RHEL host

    sudo reboot

Red Hat Enterprise Linux (RHEL) and CentOS with Docker

edit

The following instructions show you how to prepare your hosts on Red Hat Enterprise Linux (RHEL) and on CentOS with Docker.

Check the supported Linux kernel
edit

Elastic Cloud Enterprise requires the default RHEL/CentOS 3.10.0-1160.31.1 or later kernel.

  1. Use the following command to check your kernel version:

    uname -r

    Before you proceed, update the OS, and reboot the system.

  2. Update the system:

    sudo yum update
    sudo reboot
  3. Enable the overlay kernel module (check Use the OverlayFS storage driver) that the docker overlay2 storage driver uses.

    echo "overlay" | sudo tee -a /etc/modules-load.d/overlay.conf
  4. Refresh the dynamically generated grub2 configuration and configure grub to boot the newly installed kernel 3.10.0-1160.31.1 or later:

    sudo grub2-set-default 0
    sudo grub2-mkconfig -o /etc/grub2.cfg
  5. Add the required options to the kernel boot arguments:

    sudo /sbin/grubby --update-kernel=ALL --args='cgroup_enable=memory cgroup.memory=nokmem swapaccount=1'
Install Docker
edit

Which version of Docker you install depends on whether you use Red Hat Enterprise Linux (RHEL) or CentOS.

======= RHEL or CentOS 7

If you are using RHEL/CentOS 7, install Docker version 20.10.7 or later:

  1. Add the Docker repository:

    sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
    [dockerrepo]
    name=Docker Repository
    baseurl=https://download.docker.com/linux/centos/7/x86_64/stable
    enabled=1
    gpgcheck=1
    gpgkey=https://download.docker.com/linux/centos/gpg
    EOF
    
    sudo yum makecache fast

On RHEL 7, enable the rhel-7-server-extras-rpm and rhel-7-server-optional-rpms repositories.

  1. Install Docker version 20.10.7 or later:

    sudo yum install -y docker-ce-20.10* docker-ce-cli-20.10* containerd.io-1.5.*

======= RHEL 8 or CentOS 8

If you are using RHEL 8 or CentOS 8, install Docker version 20.10.7 or later:

  1. Add the Docker repository:

    # Add yum config manager
    sudo yum install -y 'dnf-command(config-manager)'
    
    # Add the docker-ce centos yum repo
    sudo yum config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
    
    # Update the yum cache
    sudo yum makecache --timer
  2. Install Docker version 20.10.7 or later:

    sudo yum install -y docker-ce-20.10* docker-ce-cli-20.10* containerd.io-1.5.*
Set up XFS quotas
edit

XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and CentOS already provide XFS as the default file system; however, quotas might be disabled. If quotas are disabled, you need to enable them.

Disk space quotas set a limit on the amount of disk space a cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space.

You can use use LVM, mdadm, or a combination of the two for block device management, but this configuration is not documented nor is it supported in Elastic Cloud Enterprise.

You must use XFS and have quotas enabled on all allocators, otherwise disk usage won’t display correctly.

To set up XFS with quotas on a single, pre-partitioned block device named /dev/xvdg1:

  1. Format the partition:

    sudo mkfs.xfs /dev/xvdg1
  2. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  3. Modify the entry for the XFS volume in the /etc/fstab file to add pquota,prjquota. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    /dev/xvdg1	/mnt/data	xfs	defaults,nofail,x-systemd.automount,prjquota,pquota  0 2
  4. Regenerate the mount files:

    sudo systemctl daemon-reload
    sudo systemctl restart local-fs.target
Update the configurations settings
edit
  1. Stop the Docker service:

    sudo systemctl stop docker
  2. Configure kernel parameters:

    cat <<EOF | sudo tee -a /etc/sysctl.conf
    # Required by Elasticsearch 5.0 and later
    vm.max_map_count=262144
    # enable forwarding so the Docker networking works as expected
    net.ipv4.ip_forward=1
    # Decrease the maximum number of TCP retransmissions to 5 as recommended for Elasticsearch TCP retransmission timeout.
    # See https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config-tcpretries.html
    net.ipv4.tcp_retries2=5
    # Make sure the host doesn't swap too early
    vm.swappiness=1
    EOF

    The net.ipv4.tcp_retries2 setting applies to all TCP connections and will affect the reliability of communication with systems other than Elasticsearch clusters too. If your clusters communicate with external systems over a low quality network then you may need to select a higher value for net.ipv4.tcp_retries2.

    1. Apply the settings:

      RHEL/CentOS 7:

      sudo sysctl -p
      sudo systemctl restart network

      RHEL/CentOS 8:

      sudo sysctl -p
      sudo systemctl restart NetworkManager
  3. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    elastic          soft    nproc          unlimited
    elastic          hard    nproc          unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited

    The default limit for number of processes is too low. Remove it and rely on the kernel limit instead (for RHEL/CentOS 7 only).

    sudo rm /etc/security/limits.d/20-nproc.conf
  4. NOTE: This step is optional if the docker registry doesn’t require authentication.

    Authenticate the elastic user to pull images from the docker registry you use, by creating the file /home/elastic/.docker/config.json. This file needs to be owned by the elastic user. If you are using a user name other than elastic, adjust the path accordingly.

    Example: In case you use docker.elastic.co, the file content looks like as follows:

    {
     "auths": {
       "docker.elastic.co": {
         "auth": "<auth-token>"
       }
     }
    }
  5. If you did not create the mount point earlier (if you did not set up XFS), create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  6. If you set up a new device with XFS earlier:

    1. Mount the block device (change the device name if you use a different device than /dev/xvdg1):

      sudo mount /dev/xvdg1
    2. Set the permissions on the newly mounted device:

      sudo chown $USER:$USER /mnt/data
  7. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
  8. Disable the firewalld service. The service is not compatible with Docker and interferes with the installation of ECE. You must disable firewalld before installing or reinstalling ECE.

    sudo systemctl disable firewalld
Configure the Docker daemon options
edit

Docker creates a bridge IP address that can conflict with IP addresses on your internal network. To avoid an IP address conflict, change the --bip=172.17.42.1/16 parameter in our examples to something that you know will work. If there is no conflict, you can omit the --bip parameter. The --bip parameter is internal to the host and can be set to the same IP for each host in the cluster. More information on Docker daemon options can be found in the dockerd command line reference.

You can specify --log-opt max-size and --log-opt max-file to define the Docker daemon containers log rotation.

  1. Update the /etc/systemd/system/docker.service.d/docker.conf file. If the file path and file do not exist, create them.

    [Unit]
    Description=Docker Service
    After=multi-user.target
    
    [Service]
    ExecStart=
    ExecStart=/usr/bin/dockerd --data-root /mnt/data/docker --storage-driver=overlay2 --bip=172.17.42.1/16 --raw-logs --log-opt max-size=500m --log-opt max-file=10 --icc=false
  2. Apply the updated Docker daemon configuration:

    1. Reload the Docker daemon configuration:

      sudo systemctl daemon-reload
    2. Restart the Docker service:

      sudo systemctl restart docker
    3. Enable Docker to start on boot:

      sudo systemctl enable docker
  3. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  4. Pin the Docker version to ensure that the docker-ce package does not get upgraded:

    echo "exclude=docker-ce docker-ce-cli containerd.io" | sudo tee -a /etc/yum.conf
  5. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    SETTINGS
  6. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  7. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, check Custom Docker daemon options in the Docker documentation.

  8. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.