Fleet and Elastic Agent 8.12.0

Details
Starting in this release, Fleet ingest pipelines can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a Fleet custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For details and investigation about the issue refer to #175254. A fix is planned for delivery in the next 8.12 minor release.

Affected ingest pipelines

APM

For APM, if you had previously defined an ingest pipeline of the form traces-apm@custom to customize the ingestion of documents ingested to the traces-apm data stream, then by nature of the new @custom hooks introduced in issue #168019, the traces-apm@custom pipeline will be called as a pipeline processor in both the traces-apm.rum and traces-apm.sampled ingest pipelines. See the following for a comparison of the relevant processors blocks for each of these pipeline before and after upgrading to 8.12.0:

// traces-apm-8.x.x
{
	"pipeline": {
		"name": "traces-apm@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm-8.12.0
{
	"pipeline": {
		"name": "global@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom", <--- Duplicate pipeline entry
		"ignore_missing_pipeline": true
	}
}

// traces-apm.rum-8.x.x
{
	"pipeline": {
		"name": "traces-apm.rum@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm.rum-8.12.0
{
	"pipeline": {
		"name": "global@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom", <--- Collides with `traces-apm@custom` that may be preexisting
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm.rum@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm.sampled-8.x.x
{
	"pipeline": {
		"name": "traces-apm.rum@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm.sampled-8.12.0
{
	"pipeline": {
		"name": "global@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom", <--- Collides with `traces-apm@custom` that may be preexisting
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm.sampled@custom",
		"ignore_missing_pipeline": true
	}
}

The immediate workaround to avoid this unwanted behavior is to edit both the traces-apm.rum and traces-apm.sampled ingest pipelines to no longer include the traces-apm@custom pipeline processor.

Please note that this is a temporary workaround, and this change will be undone if the APM integration is upgraded or reinstalled.

Elastic Agent

The elastic_agent integration is subject to the same type of breaking change as described for APM, above. The following ingest pipelines are impacted:

The behavior is similar to what’s described for APM above: pipelines such as logs-elastic_agent.filebeat will include a pipeline processor that calls logs-elastic_agent@custom. If you have custom processing logic defined in a logs-elastic_agent@custom ingest pipeline, it will be called by all of the pipelines listed above.

The workaround is the same: remove the logs-elastic_agent@custom pipeline processor from all of the ingest pipelines listed above.

Details

When you run the elastic-agent enroll command for an RPM or DEB Elastic Agent package, a Retarting agent daemon message appears in the command output, followed by a Restart attempt failed error.

Impact

The error does not mean that the enrollment failed. The enrollment actually succeeded. You can ignore the Restart attempt failed error and continue by running the following commands, after which Elastic Agent should successfully connect to Fleet:

sudo systemctl enable elastic-agent
sudo systemctl start elastic-agent

Details

In 8.12 the default memory queue flush interval was raised from 1 second to 10 seconds. In many configurations this improves performance because it allows the output to batch more events per round trip, which improves efficiency. However, the SQS input has an extra bottleneck that interacts badly with the new value.

For more details see #37754.

Impact

If you are using the Elasticsearch output, and your configuration uses a performance preset, switch it to preset: latency. If you use no preset or use preset: custom, then set queue.mem.flush.timeout: 1s in your output configuration.

If you are not using the Elasticsearch output, set queue.mem.flush.timeout: 1s in your output configuration.

To configure the output parameters for a Fleet-managed agent, see Advanced YAML configuration. For a standalone agent, see Outputs.

Details

When you set up Fleet with a very high volume of Elastic Agent policies, one thousand or more, you may encounter an error similar to the following:

[ERROR][plugins.fleet] Unknown error happened while checking Uninstall Tokens validity: 'ResponseError: all shards failed: search_phase_execution_exception
	Caused by:
		too_many_nested_clauses: Query contains too many nested clauses; maxClauseCount is set to 5173

The exact number of Elastic Agent policies required to cause the error depends in part on the size of the Elasticsearch cluster, but generally it can happen with volumes above approximately one thousand policies.

Impact

Currently there is no workaround for the issue but a fix is planned to be included in the next version 8.12 release.

Note that according to our policy scaling recommendations, the current recommended maximum number of Elastic Agent policies supported by Fleet is 500.

Details

An issue discovered in Fleet Server prevents Elastic Agents that have been upgraded to version 8.12.0 from being upgraded again, using the Fleet UI, to version 8.12.1 or higher.

This issue is planned to be fixed in versions 8.12.2 and 8.13.0.

Impact

As a workaround, we recommend you to use the Kibana Fleet API to update any documents in which upgrade_details is either null or not defined. Note that these steps must be run as a superuser.

 POST _security/role/fleet_superuser
 {
    "indices": [
        {
            "names": [".fleet*",".kibana*"],
            "privileges": ["all"],
            "allow_restricted_indices": true
        }
    ]
  }

POST _security/user/fleet_superuser
 {
    "password": "password",
    "roles": ["superuser", "fleet_superuser"]
 }

curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \
  -H'x-elastic-product-origin:fleet' \
  http://localhost:9200/.fleet-agents/_update_by_query \
  -d '{
  "script": {
    "source": "ctx._source.remove(\"upgrade_details\")",
    "lang": "painless"
  },
  "query": {
    "bool": {
        "must_not": {
          "exists": {
            "field": "upgrade_details"
          }
        }
      }
    }
}'

DELETE _security/user/fleet_superuser
DELETE _security/role/fleet_superuser

After running these API requests, wait at least 10 minutes, and then the agents should be upgradeable again.

Details

Support for a remote Elasticsearch output was introduced in this release to enable Elastic Agents to send integration or monitoring data to a remote Elasticsearch cluster. A bug has been found that causes Elastic Defend response actions to stop working when a remote Elasticsearch output is configured for an agent.

Impact

This bug is currently being investigated and is expected to be resolved in an upcoming release.