Connector known issues

OOM errors when syncing large database tables

Syncs after the initial sync can cause out-of-memory (OOM) errors when syncing large database tables. This occurs because database connectors load and store IDs in memory. For tables with millions of records, this can lead to memory exhaustion if the connector service has insufficient RAM.

To mitigate this issue, you can:

Upgrades from deployments running on versions earlier than 8.9.0 can cause sync job failures

Due to a bug, the job_type field mapping will be missing after upgrading from deployments running on versions earlier than 8.9.0. Sync jobs won’t be displayed in the Kibana UI (job history) and the connector service won’t be able to start new sync jobs. This will only occur if you have previously scheduled sync jobs.

To resolve this issue, you can manually add the missing field with the following command and trigger a sync job:

resp = client.indices.put_mapping(
    index=".elastic-connectors-sync-jobs-v1",
    properties={
        "job_type": {
            "type": "keyword"
        }
    },
)
print(resp)

const response = await client.indices.putMapping({
  index: ".elastic-connectors-sync-jobs-v1",
  properties: {
    job_type: {
      type: "keyword",
    },
  },
});
console.log(response);

PUT .elastic-connectors-sync-jobs-v1/_mapping
{
  "properties": {
    "job_type": {
      "type": "keyword"
    }
  }
}

The connector service will fail to sync when the connector tries to fetch more more than 2,147,483,647 (2^31-1) documents from a data source

A workaround is to manually partition the data to be synced using multiple search indices.

Custom scheduling might break when upgrading from version 8.6 or earlier.

If you encounter the error 'custom_schedule_triggered': undefined method 'each' for nil:NilClass (NoMethodError), it means the custom scheduling feature migration failed. You can use the following manual workaround:

resp = client.update(
    index=".elastic-connectors",
    id="connector-id",
    doc={
        "custom_scheduling": {}
    },
)
print(resp)

const response = await client.update({
  index: ".elastic-connectors",
  id: "connector-id",
  doc: {
    custom_scheduling: {},
  },
});
console.log(response);

POST /.elastic-connectors/_update/connector-id
{
  "doc": {
    "custom_scheduling": {}
  }
}

This error can appear on Connectors or Crawlers that aren’t the cause of the issue. If the error continues, try running the above command for every document in the .elastic-connectors index.

Connectors upgrading from 8.7 or earlier can be missing configuration fields

A connector that was created prior to 8.8 can sometimes be missing configuration fields. This is a known issue for the MySQL connector but could also affect other connectors.

If the self-managed connector raises the error Connector for <connector_id> has missing configuration fields: <field_a>, <field_b>..., you can resolve the error by manually adding the missing configuration fields via the Dev Tools. Only the following two field properties are required, as the rest will be autopopulated by the self-managed connector:

Python connectors that upgraded from 8.7.1 will report document volumes in gigabytes (GB) instead of megabytes (MB)

As a result, true document volume will be under-reported by a factor of 1024.

The following Elastic managed connectors will not run correctly on Elastic Cloud in 8.9.0. They are still available as self-managed connectors.