IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

› ›

Web crawler API reference

edit

Web crawler API reference

edit

App Search provides API operations for the web crawler. This document provides a reference for each API operation, as well as shared concerns:

Shared concerns

edit

All web crawler API operations share the following concerns.

Engine

edit

Most endpoints within the crawler API are scoped to a particular App Search engine. The engine is identified by the engine name value provided in the URL of the request. If an engine could not be found for any API request, an empty HTTP 404 response will be returned.

Access

edit

Unless security is disabled, you must provide credentials to access API operations.

See Authentication.

Crawler

edit

Responds with domain objects configured for a given App Search engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler

# 200 OK
{
  "domains": [
    {
      "id": "{DOMAIN_ID}",
      "name": "{DOMAIN_NAME}",
      "document_count": 0,
      "entry_points": [
        {
          "id": "6087cec06dda9bdfb4a49e39",
          "value": "/"
        }
      ],
      "crawl_rules": [],
      "default_crawl_rule": {
        "id": "-",
        "order": 0,
        "policy": "allow",
        "rule": "regex",
        "pattern": ".*"
      },
      "sitemaps": []
    }
  ]
}

Crawl requests

edit

Each crawl performed by the Enterprise Search web crawler has an associated crawl request object. The crawl requests API allows operators to create new crawl requests and to view and control the state of existing crawl requests.

Get current active crawl request

edit

Returns a crawl request object for an active crawl or returns an HTTP 404 response if there is no active crawl for a given App Search engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_requests/active

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "601b21adbeae67679b3b760a",
  "status": "running",
  "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
  "begun_at": "Wed, 03 Feb 2021 22:20:31 +0000",
  "completed_at": null
}

For cases when there is no active crawl for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "error": "There are no active crawl requests for this engine"
}

Cancel an active crawl

edit

Cancels an active crawl for a given App Search engine or returns an HTTP 404 response if there is no active crawl for a given App Search engine.

It may take some time for the crawler to detect the cancellation request and gracefully stop the crawl. During the time, the status of the crawl request will remain canceling.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_requests/active/cancel

In case of success, the response contains a single crawl request object with a canceling state:

# 200 OK
{
  "id": "601b21adbeae67679b3b760a",
  "status": "canceling",
  "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
  "begun_at": "Wed, 03 Feb 2021 22:20:31 +0000",
  "completed_at": null
}

For cases when there is no active crawl for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "error": "There are no active crawl requests for this engine"
}

List crawl requests

edit

Returns a list of crawl requests for a given engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_requests

page[current] (optional): Current page number (default: 1).
page[size] (optional): Page size (default: 25). The maximum is 100, and be will truncated if a larger size is requested.

# 200 OK
{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 3,
      "size": 25
    }
  },
  "results": [
    {
      "id": "601b21adbeae67679b3b760a",
      "status": "running",
      "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
      "begun_at": "Wed, 03 Feb 2021 22:20:31 +0000",
      "completed_at": null
    },
    {
      "id": "60147e93beae67bf7ef72e86",
      "status": "success",
      "created_at": "Fri, 29 Jan 2021 21:30:59 +0000",
      "begun_at": "Fri, 29 Jan 2021 21:31:00 +0000",
      "completed_at": "Fri, 29 Jan 2021 21:35:20 +0000"
    },
    {
      "id": "60146c07beae67f397300128",
      "status": "canceled",
      "created_at": "Fri, 29 Jan 2021 20:11:51 +0000",
      "begun_at": "Fri, 29 Jan 2021 20:11:52 +0000",
      "completed_at": "Fri, 29 Jan 2021 20:12:51 +0000"
    }
  ]
}

Create a new crawl request

edit

Requests a new crawl for a given App Search engine. If there is already an active crawl, the request returns an HTTP 400 response with an error message.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_requests

In case of success, the response contains a single crawl request object with a pending state:

# 200 OK
{
  "id": "601b21adbeae67679b3b760a",
  "status": "pending",
  "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
  "begun_at": null,
  "completed_at": null
}

When there is already an active crawl, the API returns an HTTP 400 response:

# 400 Bad Request
{
  "error": "There is an active crawl for the engine \"your-engine\", please wait for it to finish or abort it before requesting another one"
}

Create a new partial crawl request

edit

Certain properties of a crawl can be overridden at request time, allowing the user to request a crawl against a specified subset of their corpus. Such a crawl is considered a partial crawl.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_requests
{
  "overrides": {
    "max_crawl_depth": {INTEGER},
    "domain_allowlist": {ARRAY<STRING>},
    "seed_urls": {ARRAY<STRING>},
    "sitemap_urls": {ARRAY<STRING>}
  }
}

overrides (optional)

Allows the user to override a subset of crawl configuration for the requested crawl.

max_crawl_depth (optional): Maximum depth to follow links while discovering new content.
domain_allowlist (optional): Array of domain names for restricting which links to follow.
seed_urls (optional): Array of initial URLs to crawl. Defaults to the configured entrypoints for each crawler domain.
sitemap_urls (optional): Array of sitemap URLs to be used for content discovery.

Below are some examples of partial crawl requests:

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/wellness/crawler/crawl_requests
{
  "overrides": {
    "max_crawl_depth": 2,
    "seed_urls": ["http://www.awesome-recipes.com/slow-cooker"]
  }
}

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/wellness/crawler/crawl_requests
{
  "overrides": {
    "domain_allowlist": ["http://www.awesome-workouts.com"]
  }
}

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/wellness/crawler/crawl_requests
{
  "overrides": {
    "max_crawl_depth": 2,
    "sitemap_urls": ["http://www.awesome-content.com/sitemap.xml"]
  }
}

View details for a crawl request

edit

Returns details of a given crawl request. The crawl request is identified with a unique Crawl Request ID value.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_requests/{CRAWL_REQUEST_ID}

# 200 OK
{
  "id": "60147e93beae67bf7ef72e86",
  "status": "success",
  "created_at": "Fri, 29 Jan 2021 21:30:59 +0000",
  "begun_at": "Fri, 29 Jan 2021 21:31:00 +0000",
  "completed_at": "Fri, 29 Jan 2021 21:35:20 +0000"
}

Crawl schedules

edit

Each engine using the Enterprise Search web crawler has an associated crawl schedule object. The crawl schedule API allows operators to specify a frequency at which new crawls will be started. If there is an active crawl, new crawls will be skipped.

Get current crawl schedule

edit

Returns a crawl schedule object or returns an HTTP 404 response if there is no crawl schedule object for a given App Search engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_schedule

For successful calls, the response is going to look like this:

# 200 OK
{
  "engine": {ENGINE_NAME},
  "frequency": 2,
  "unit": "week"
}

For cases when there is no crawl schedule for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "errors": ["No crawl schedule found"]
}

Create or update a crawl schedule

edit

Upserts a crawl schedule for a given App Search engine.

PUT <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_schedule
{
  "frequency": {INTEGER},
  "unit": {ENUM}
}

frequency (required): A positive integer.
unit (required): Should be one of: hour, day, week, month.

In case of success, the response contains the crawl schedule object:

# 200 OK
{
  "engine": {ENGINE_NAME},
  "frequency": 2,
  "unit": "week"
}

When the parameters are invalid, the API returns an HTTP 400 response:

# 400 Bad Request
{
  "errors": [
    "Crawl schedule frequency must be an integer",
    "Crawl schedule unit must be one of hour, day, week, month"
  ]
}

Delete a crawl schedule

edit

Deletes a crawl schedule for a given App Search engine.

DELETE <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/crawl_schedule

In case of success, the response contains an object showing the crawl schedule has been deleted:

# 200 OK
{
  "deleted": true
}

For cases when there is no crawl schedule for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "errors": ["No crawl schedule found"]
}

Process crawls

edit

A process crawl is an operation that re-processes the documents in your engine using the current crawl rules, without waiting for a full re-crawl. See Process crawl.

Use the following operations to manage process crawls for a given engine.

List process crawls

edit

Returns a list of process crawls for the given engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/process_crawls

page[current] (optional): Current page number (default: 1).
page[size] (optional): Page size (default: 25). The maximum is 100, and be will truncated if a larger size is requested.

# 200 OK
{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 3,
      "size": 25
    }
  },
  "results": [
    {
      "id": "61421fe6e725695876d72d2e",
      "dry_run": true,
      "total_url_count": 167,
      "denied_url_count": 92,
      "domains": [
        "https://swiftype.com"
      ],
      "process_all_domains": false,
      "created_at": "2021-09-15T16:31:34Z",
      "begun_at": "2021-09-15T16:31:35Z",
      "completed_at": "2021-09-15T16:31:52Z"
    },
    {
      "id": "61421e15e72569a22ad6f549",
      "dry_run": false,
      "total_url_count": 7793,
      "denied_url_count": 1028,
      "domains": [
        "https://swiftype.com",
        "https://www.elastic.co"
      ],
      "process_all_domains": true,
      "created_at": "2021-09-15T16:23:49Z",
      "begun_at": "2021-09-15T16:23:49Z",
      "completed_at": "2021-09-15T16:25:48Z"
    },
    {
      "id": "61421b93e725694296d664f0",
      "dry_run": false,
      "total_url_count": 8525,
      "denied_url_count": 930,
      "domains": [
        "https://www.elastic.co"
      ],
      "process_all_domains": true,
      "created_at": "2021-09-15T16:13:07Z",
      "begun_at": "2021-09-15T16:13:07Z",
      "completed_at": "2021-09-15T16:15:29Z"
    }
  ]
}

View details for a process crawl

edit

View the details of the process crawl with the given ID.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/process_crawls/{PROCESS_CRAWL_ID}

The response includes summary information of the process crawl, identifying the total number of URLs that were re-processed along with the number of URLs identified for deletion.

{
  "id": "61421fe6e725695876d72d2e",
  "dry_run": true,
  "total_url_count": 167,
  "denied_url_count": 92,
  "domains": [
    "https://swiftype.com"
  ],
  "process_all_domains": false,
  "created_at": "2021-09-15T16:31:34Z",
  "begun_at": "2021-09-15T16:31:35Z",
  "completed_at": "2021-09-15T16:31:52Z"
}

View denied URLs for a process crawl

edit

View a sample of 100 URLs identified for deletion by a given process crawl. This API complements the dry run capability of a process crawl to test configuration before committing to deletion. See Create a new process crawl.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/process_crawls/{PROCESS_CRAWL_ID}/denied_urls

{
  "total_url_count": 167,
  "denied_url_count": 92,
  "sample_size": 100,
  "denied_urls_sample": [
    "https://swiftype.com/documentation",
    "https://swiftype.com/documentation/site-search/site_search",
    "https://swiftype.com/documentation/site-search/guides/crawler-optimization",
    "https://swiftype.com/documentation/site-search/guides/multiple-domains",
    "https://swiftype.com/documentation/site-search/guides/engine-cloning",
    ...
  ]
}

The denied URLs API reads from event logs generated during the process crawl. If the logs for a given process crawl have been deleted or are otherwise unavailable, an empty array is returned.

{
  "total_url_count": 167,
  "denied_url_count": 92,
  "sample_size": 100,
  "denied_urls_sample": []
}

Create a new process crawl

edit

Request a new process crawl for the given engine.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/process_crawls
{
  "dry_run": {BOOLEAN}
  "domains": {ARRAY}
}

dry_run (optional): If true, the process crawl identifies but does not delete documents that are no longer permitted by the current crawl configuration. Defaults to false.
domains (optional): Limits the process crawl to a subset of crawler domains. By default, the process crawl will apply to all crawler domains on the engine.

{
  "id": "61421b93e725694296d664f0",
  "dry_run": false,
  "total_url_count": 0,
  "denied_url_count": 0,
  "domains": [
    "https://www.elastic.co"
  ],
  "process_all_domains": true,
  "created_at": "2021-09-15T16:13:07Z",
  "begun_at": null,
  "completed_at": null
}

URL validation and debugging

edit

The web crawler provides several API operations to help you troubleshoot crawls. Use the following operations to validate and debug specific URLs from the perspective of the web crawler.

Validate a domain

edit

Validate the given domain from the perspective of the web crawler.

Optionally, use the checks parameter to choose the specific validations to perform. The response includes an ordered set of results. Each provides the detailed results of a specific check. If any checks fail, the crawler skips remaining checks.

Use this operation to validate a domain before adding it to the crawl configuration for an engine, or troubleshoot issues crawling a specific domain. This operation visits the domain as the web crawler, without starting a full crawl. The results reflect how the web crawler sees that domain now. It does not represent the states of previous crawls. For historical information, see Trace a URL.

You can use this operation to identify issues with a domain. The domain owner can fix the issues, and you can verfiy the fixes. Repeat this process until all checks pass. This process is more convenient than requesting a new crawl to confirm the fixes.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/crawler/validate_url
{
  "url": {URL},
  "checks": {CHECKS}
}

url (required)

The URL of the domain to validate.

checks (optional)

Checks to perform. Array including any of the following supported values. To run additional checks, validate a specific page within the context of an engine. See Validate a URL.

dns
robots_txt
tcp
url
url_content
url_request

Example response for url: https://www.elastic.co and checks: ["url", "tcp", "url_request"]:

# 200 OK
{
  "url": "https://www.elastic.co",
  "normalized_url": "https://www.elastic.co/",
  "valid": true,
  "results": [
    {
      "result": "ok",
      "name": "url",
      "details": {},
      "comment": "URL structure looks valid"
    },
    {
      "result": "ok",
      "name": "tcp",
      "details": {
        "host": "www.elastic.co",
        "port": 443
      },
      "comment": "TCP connection successful"
    },
    {
      "result": "ok",
      "name": "url_request",
      "details": {
        "status_code": 200,
        "content_type": "text/html; charset=utf-8",
        "request_time_msec": 166
      },
      "comment": "Successfully fetched https://www.elastic.co: HTTP 200."
    }
  ]
}

Validate a URL

edit

Validate the given URL from the perspective of the web crawler. The given URL is assumed to be a web page or other web content. To validate a domain, see Validate a domain.

Use this operation to troubleshoot issues crawling web content at a specific URL. This operation visits the URL as the web crawler, without starting a full crawl. The results reflect how the web crawler sees that URL now. It does not represent the states of previous crawls. For historical information, see Trace a URL.

You can use this operation to identify issues with a specific URL. The domain owner can fix the issues, and you can verfiy the fixes. Repeat this process until all checks pass. After fixing all issues, start a new crawl to discover and extract the content at the URL. See Create a new crawl request.

To troubleshoot content extraction issues at a specific URL, see Extract content from a URL. The url_content check for this operation confirms that content was extracted, but it does not include the content within the response.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/validate_url

url (required)

The URL to validate.

checks (optional)

Checks to perform. Array including any of the following supported values. This operation offers more checks than the operation to validate a domain, since this operation runs in the context of a specific engine.

crawl_rules
domain_access
dns
robots_txt
tcp
url
url_content
url_request

Example response for url: https://www.elastic.co and checks: ["url", "domain_access", "crawl_rules"]:

# 200 OK
{
  "url": "https://www.elastic.co",
  "normalized_url": "https://www.elastic.co/",
  "valid": true,
  "results": [
    {
      "result": "ok",
      "name": "url",
      "details": {},
      "comment": "URL structure looks valid"
    },
    {
      "result": "ok",
      "name": "domain_access",
      "details": {
        "domain": "https://www.elastic.co"
      },
      "comment": "The URL matches one of the domains configured for the engine"
    },
    {
      "result": "ok",
      "name": "crawl_rules",
      "details": {
        "rule": "domain=https://www.elastic.co, default allow all rule"
      },
      "comment": "The URL is allowed by one of the crawl rules"
    }
  ]
}

Extract content from a URL

edit

Use the web crawler to extract content from the given URL.

Use this operation to troubleshoot issues with content extraction at a specific URL. For general URL validation, see Validate a URL.

This operation visits the URL as the web crawler, without starting a full crawl, and extracts content. The results reflect the current crawl configuration for the engine, and reveal how the web crawler sees the content at that URL right now. However, if the content is already indexed, the response includes that information as well.

This operation does not make changes to any documents in the engine. Use this operation to identify and fix context extraction issues at a specific URL. The content owner can fix the issues, and you can verfiy the fixes. Repeat this process until the desired content is extracted. After fixing all issues, start a new crawl to discover and extract the content at the URL. See Create a new crawl request.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/extract_url
{
  "url": {URL}
}

url (required): The URL to crawl to extract content.

For successful calls, the response is going to look like this:

# 200 OK
{
  "url": "https://www.example.com",
  "normalized_url": "https://www.example.com/",
  "results": {
    "download": {
      "status_code": 200
    },
    "extraction": {
      "content_hash": "fb38982491c4a9377f8cf0c57e75e067bca65daf",
      "content_hash_fields": [
        "title",
        "body_content",
        "meta_keywords",
        "meta_description",
        "links",
        "headings"
      ],
      "content_fields": {
        "title": "Example Domain",
        "body_content": "Example Domain This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission. More information...",
        "links": [
          "https://www.iana.org/domains/example"
        ],
        "headings": [
          "Example Domain"
        ]
      }
    },
    "indexing": {
      "document_id": null,
      "document_fields": null
    },
    "deduplication": {
      "urls_count": 0,
      "urls_sample": []
    }
  }
}

Details for selected response fields:

results.deduplication: The total count of URLs with this same content and a sample of those URLs. See Duplicate document handling.
results.extraction.content_fields: The content fields extracted by the crawler—what the crawler will extract for the next crawl.
results.extraction.content_hash: The hash used to uniquely identify this content. See Duplicate document handling.
results.extraction.content_hash_fields: The document schema fields used to create the content hash. See Duplicate document handling.
results.indexing: The fields that are currently indexed—what the crawler extracted during a previous crawl (if any).

Trace a URL

edit

Trace the recent history of the given URL from the perspective of the web crawler. Determine if the web crawler saw the URL, how it discovered it, and other events specific to that URL. Use the response to see the history of a specific URL from the perspective of the web crawler, and debug any issues crawling the URL.

This operation provides a view of the web crawler events logs for a specific engine and URL. The response returns an array of recent crawl requests, where each indicates if the URL was found. This is followed by specific crawler events, grouped by type. For details on each crawler event, see Web crawler events logs reference.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/trace_url
{
  "url": {URL}
}

url (required): The URL to trace.

For successful calls, the response is going to look like this:

# 200 OK
{
  "url": "https://www.elastic.co/blog",
  "normalized_url": "https://www.elastic.co/blog",
  "crawl_requests": [
    {
      "crawl_request": {
        "id": "61240b9d2a02b14c1df12f98",
        "status": "running",
        "created_at": "2021-08-23T20:57:01Z",
        "begun_at": "2021-08-23T20:57:02Z",
        "completed_at": null
      },
      "found": true,
      "discover": [
        {
          "timestamp": "Mon, 23 Aug 2021 20:57:02 +0000",
          "event_id": "61240b9e2a02b11895f12f9e",
          "message": null,
          "event_type": "allowed",
          "deny_reason": null,
          "crawl_depth": 1,
          "source_url": null
        }
      ],
      "seed": {
        "timestamp": "Mon, 23 Aug 2021 20:57:02 +0000",
        "event_id": "61240b9e2a02b11895f12f9f",
        "message": null,
        "url_type": "content",
        "source_type": "seed-list",
        "source_url": null
      },
      "fetch": {
        "timestamp": "Mon, 23 Aug 2021 20:57:03 +0000",
        "event_id": "61240b9f2a02b1f7caf12fab",
        "message": null,
        "event_outcome": "failure",
        "duration_msec": 152.537107,
        "http_response": {
          "status_code": "301",
          "body_bytes": null
        },
        "redirect": {
          "location": "https://www.elastic.co/blog/",
          "chain": [],
          "count": 1
        }
      },
      "output": {
        "timestamp": "Mon, 23 Aug 2021 20:57:03 +0000",
        "event_id": "61240b9f2a02b1f7caf12fae",
        "message": "Successfully processed the 301 response",
        "event_outcome": "success",
        "event_module": "app_search",
        "duration_msec": 36.45282,
        "engine": {
          "id": "61240b592a02b19de7f12f8e",
          "name": "elastic-blog"
        },
        "document": {
          "id": null
        }
      }
    }
  ]
}

If the given URL was not seen during the recent crawls, the response will look like this:

# 200 OK
{
  "url": "https://github.com",
  "normalized_url": "https://github.com/",
  "crawl_requests": [
    {
      "crawl_request": {
        "id": "61240b9d2a02b14c1df12f98",
        "status": "canceled",
        "created_at": "2021-08-23T20:57:01Z",
        "begun_at": "2021-08-23T20:57:02Z",
        "completed_at": "2021-08-23T21:04:39Z"
      },
      "found": false,
      "discover": [],
      "seed": null,
      "fetch": null,
      "output": null
    }
  ]
}

User agent

edit

Responds with the User-Agent header used by the crawler.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/crawler/user_agent

# 200 OK
{
  "user_agent": "Elastic Crawler (0.0.1)"
}

Domains

edit

List domains

edit

Returns a list of domains for the given engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains

page[current] (optional): Current page number (default: 1).
page[size] (optional): Page size (default: 25). The maximum is 100, and be will truncated if a larger size is requested.

# 200 OK
{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 2,
      "size": 25
    }
  },
  "results": [
    {
      "id": "61a8eb863932dd78141fb0df",
      "name": "https://www.elastic.co",
      "document_count": 0,
      "entry_points": [
        {
          "id": "61a8a1ed3932ddd80cc4b719",
          "value": "/"
        }
      ],
      "crawl_rules": [],
      "default_crawl_rule": {
        "id": "-",
        "order": 0,
        "policy": "allow",
        "rule": "regex",
        "pattern": ".*"
      },
      "sitemaps": []
    },
    {
      "id": "61a8a1ed3932ddd80cc4b718",
      "name": "https://swiftype.com",
      "document_count": 0,
      "entry_points": [
        {
          "id": "61a8eb863932dd78141fb0e0",
          "value": "/"
        }
      ],
      "crawl_rules": [],
      "default_crawl_rule": {
        "id": "-",
        "order": 0,
        "policy": "allow",
        "rule": "regex",
        "pattern": ".*"
      },
      "sitemaps": []
    }
  ]
}

Create a new domain

edit

Create a (crawler) domain for a given App Search engine.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains
{
  "name": {STRING}
}

name (required)

The domain URL.

auth (optional)

The information required to crawl a domain protected by HTTP authentication.

For Basic auth, set type to basic, and provide the username and password:

{
  "auth": {
    "type": "basic",
    "username": "kimchy",
    "password": ":)"
  }
}

Alternatively, set the value of the Authorization header directly. Set type to raw, and provide the value:

{
  "auth": {
    "type": "raw",
    "value": "Bearer some-token"
  }
}

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{DOMAIN_ID}",
  "name": "{DOMAIN_NAME}",
  "document_count": 0,
  "entry_points": [
    {
      "id": "6087cec06dda9bdfb4a49e39",
      "value": "/"
    }
  ],
  "crawl_rules": [],
  "default_crawl_rule": {
    "id": "-",
    "order": 0,
    "policy": "allow",
    "rule": "regex",
    "pattern": ".*"
  },
  "sitemaps": []
}

View details for a domain

edit

Get domain object for a given App Search engine.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{DOMAIN_ID}",
  "name": "{DOMAIN_NAME}",
  "document_count": 0,
  "entry_points": [
    {
      "id": "6087cec06dda9bdfb4a49e39",
      "value": "/"
    }
  ],
  "crawl_rules": [],
  "default_crawl_rule": {
    "id": "-",
    "order": 0,
    "policy": "allow",
    "rule": "regex",
    "pattern": ".*"
  },
  "sitemaps": []
}

Update a domain

edit

Updates a domain for a given App Search engine.

PUT <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}
{
  "name": {STRING}
}

name

The domain URL.

auth (optional)

The information required to crawl a domain protected by HTTP authentication.

For Basic auth, set type to basic, and provide the username and password:

{
  "auth": {
    "type": "basic",
    "username": "kimchy",
    "password": ":)"
  }
}

Alternatively, set the value of the Authorization header directly. Set type to raw, and provide the value:

{
  "auth": {
    "type": "raw",
    "value": "Bearer some-token"
  }
}

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{DOMAIN_ID}",
  "name": "{DOMAIN_NAME}",
  "document_count": 0,
  "entry_points": [
    {
      "id": "6087cec06dda9bdfb4a49e39",
      "value": "/"
    }
  ],
  "crawl_rules": [],
  "default_crawl_rule": {
    "id": "-",
    "order": 0,
    "policy": "allow",
    "rule": "regex",
    "pattern": ".*"
  },
  "sitemaps": []
}

Delete a domain

edit

Deletes a domain for a given App Search engine.

DELETE <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}

In case of success, the response contains an object showing the domain has been deleted:

# 200 OK
{
  "deleted": true
}

Entry points

edit

Create a new entry point

edit

Create an entry point for a domain.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/entry_points
{
  "value": {STRING}
}

value (required): The entry point path.

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{ENTRY_POINT_ID}",
  "value": "/blog"
}

Update an entry point

edit

Updates an entry point for a domain.

PUT <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/entry_points/{ENTRY_POINT_ID}
{
  "value": {STRING}
}

value: The entry point path.

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{ENTRY_POINT_ID}",
  "value": "/blog"
}

Delete an entry point

edit

Deletes an entry point for a domain.

DELETE <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/entry_points/{ENTRY_POINT_ID}

In case of success, the response contains an object showing the entry point has been deleted:

# 200 OK
{
  "deleted": true
}

Crawl rules

edit

Create a new crawl rule

edit

Create a crawl rule for a domain.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/crawl_rules
{
  "policy": {ENUM},
  "rule": {ENUM},
  "pattern": {STRING},
  "order": {INTEGER}
}

policy (required): Accepted values are allow and deny.
rule (required): Accepted values are begins, ends, contains and regex.
pattern (required): The path pattern to match against.
order (optional): An integer representing this crawl rule’s position within the list of crawl rules for the domain. The order of crawl rules is significant.

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{CRAWL_RULE_ID}",
  "order": 0,
  "policy": "allow",
  "rule": "begins",
  "pattern": "/ignore"
}

Update a crawl rule

edit

Updates a crawl rule for a domain.

PUT <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/crawl_rules/{CRAWL_RULE_ID}
{
  "policy": {ENUM},
  "rule": {ENUM},
  "pattern": {STRING},
  "order": {INTEGER}
}

policy: Accepted values are allow and deny.
rule: Accepted values are begins, ends, contains and regex.
pattern: The path pattern to match against.
order: An integer representing this crawl rule’s position within the list of crawl rules for the domain. The order of crawl rules is significant.

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{CRAWL_RULE_ID}",
  "order": 0,
  "policy": "allow",
  "rule": "begins",
  "pattern": "/ignore"
}

Delete a crawl rule

edit

Deletes a crawl rule for a domain.

DELETE <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/crawl_rules/{CRAWL_RULE_ID}

In case of success, the response contains an object showing the crawl rule has been deleted:

# 200 OK
{
  "deleted": true
}

Sitemaps

edit

Create a new sitemap

edit

Create a sitemap for a domain.

POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/sitemaps
{
  "url": {STRING}
}

url (required): The sitemap URL.

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{SITEMAP_ID}",
  "url": "https://elastic.co/sitemap2.xml"
}

Update a sitemap

edit

Updates a sitemap for a domain.

PUT <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/sitemaps/{SITEMAP_ID}
{
  "url": {STRING}
}

url: The sitemap URL.

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "{SITEMAP_ID}",
  "url": "https://elastic.co/sitemap2.xml"
}

Delete a sitemap

edit

Deletes a sitemap for a domain.

DELETE <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/{ENGINE_NAME}/crawler/domains/{DOMAIN_ID}/sitemaps/{SITEMAP_ID}

In case of success, the response contains an object showing the sitemap has been deleted:

# 200 OK
{
  "deleted": true
}

« Synonyms API API Clients »