Using hybrid search for gopher hunting with Elasticsearch and Go

In the previous parts of this series, it was demonstrated how to use the Elasticsearch Go client for traditional keyword search and vector search. This third part covers hybrid search. We'll share examples of how you can combine both vector search and keyword search using Elasticsearch and the Elasticsearch Go client.

Prerequisites

Just like part one in this series, the following prerequisites are required for this example:

Installation of Go version 1.21 or later
Create your own Go repo using the recommended structure and package management covered in the Go documentation
Creating your own Elasticsearch cluster, populated with a set of rodent-based pages, including for our friendly Gopher, from Wikipedia:

Connecting to Elasticsearch

As a reminder, in our examples, we will make use of the Typed API offered by the Go client. Establishing a secure connection for any query requires configuring the client using either:

Cloud ID and API key if making use of Elastic Cloud
Cluster URL, username, password and the certificate

Connecting to our cluster located on Elastic Cloud would look like this:

func GetElasticsearchClient() (*elasticsearch.TypedClient, error) {
	var cloudID = os.Getenv("ELASTIC_CLOUD_ID")
	var apiKey = os.Getenv("ELASTIC_API_KEY")

	var es, err = elasticsearch.NewTypedClient(elasticsearch.Config{
		CloudID: cloudID,
		APIKey:  apiKey,
		Logger:  &elastictransport.ColorLogger{os.Stdout, true, true},
	})

	if err != nil {
		return nil, fmt.Errorf("unable to connect: %w", err)
	}

	return es, nil
}

The client connection can then be used for searching, as demonstrated in the subsequent sections.

Manual boosting for hybrid search

When combining any set of search algorithms, the traditional approach has been to manually configure constants to boost each query type. Specifically, a factor is specified for each query, and the combined results set is compared to the expected set to determine the recall of the query. Then we repeat for several sets of factors and pick the one closest to our desired state.

For example, combining a single text search query boosted by a factor of 0.8 with a knn query with a lower factor of 0.2 can be done by specifying the Boost field in both query types, as shown in the below example:

func HybridSearchWithBoost(client *elasticsearch.TypedClient, term string) ([]Rodent, error) {
	var k = 10
	var numCandidates = 10
	var knnBoost float32 = 0.2
	var queryBoost float32 = 0.8

	res, err := client.Search().
		Index("vector-search-rodents").
		Knn(types.KnnSearch{
			Field:         "text_embedding.predicted_value",
			Boost:         &knnBoost,
			K:             &k,
			NumCandidates: &numCandidates,
			QueryVectorBuilder: &types.QueryVectorBuilder{
				TextEmbedding: &types.TextEmbedding{
					ModelId:   "sentence-transformers__msmarco-minilm-l-12-v3",
					ModelText: term,
				},
			}}).
		Query(&types.Query{
			Match: map[string]types.MatchQuery{
				"title": {
					Query: term,
					Boost: &queryBoost,
				},
			},
		}).
		Do(context.Background())

	if err != nil {
		return nil, err
	}

	return getRodents(res.Hits.Hits)
}

The factor specified in the Boost option for each query is added to the document score. By increasing the score of our match query by a larger factor than the knn query, results from the keyword query are more heavily weighted.

The challenge of manual boosting, particularly if you're not a search expert, is that it requires tuning to figure out the factors that will lead to the desired result set. It's simply a case of trying out random values to see what gets you closer to your desired result set.

Reciprocal Rank Fusion in hybrid search & Go client

Reciprocal Rank Fusion, or RRF, was released under technical preview for hybrid search in Elasticsearch 8.9. It aims to reduce the learning curve associated with tuning and reduce the amount of time experimenting with factors to optimize the result set.

With RRF, the document score is recalculated by blending the scores by the below algorithm:

score := 0.0
// q is a query in the set of queries (vector and keyword search)
for _, q := range queries {
    // result(q) is the results 
    if document in result(q) {
        // k is a ranking constant (default 60)
        // rank(result(q), d) is the document's rank within result(q) 
        // range from 1 to the window_size (default 100)
        score +=  1.0 / (k + rank(result(q), d))
    }
}

return score

The advantage of using RRF is that we can make use of the sensible default values within Elasticsearch. The ranking constant k defaults to 60. To provide a tradeoff between the relevancy of returned documents and the query performance when searching over large data sets, the size of the result set for each considered query is limited to the value of window_size, which defaults to 100 as outlined in the documentation.

k and windows_size can also be configured within the Rrf configuration within the Rank method in the Go client, as per the below example:

func HybridSearchWithRRF(client *elasticsearch.TypedClient, term string) ([]Rodent, error) {
	var k = 10
	var numCandidates = 10

	// Minimum required window size for the default result size of 10
	var windowSize int64 = 10
	var rankConstant int64 = 42

	res, err := client.Search().
		Index("vector-search-rodents").
		Knn(types.KnnSearch{
			Field:         "text_embedding.predicted_value",
			K:             &k,
			NumCandidates: &numCandidates,
			QueryVectorBuilder: &types.QueryVectorBuilder{
				TextEmbedding: &types.TextEmbedding{
					ModelId:   "sentence-transformers__msmarco-minilm-l-12-v3",
					ModelText: term,
				},
			}}).
		Query(&types.Query{
			Match: map[string]types.MatchQuery{
				"title": {Query: term},
			},
		}).
		Rank(&types.RankContainer{
			Rrf: &types.RrfRank{
				WindowSize:   &windowSize,
				RankConstant: &rankConstant,
			},
		}).
		Do(context.Background())

	if err != nil {
		return nil, err
	}

	return getRodents(res.Hits.Hits)
}

Conclusion

Here we've discussed how to combine vector and keyword search in Elasticsearch using the Elasticsearch Go client.

Check out the GitHub repo for all the code in this series. If you haven't already, check out part 1 and part 2 for all the code in this series.

Happy gopher hunting!

Resources

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Start a free trial now.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Using hybrid search for gopher hunting with Elasticsearch and Go

Prerequisites

Connecting to Elasticsearch

Manual boosting for hybrid search

Reciprocal Rank Fusion in hybrid search & Go client

Conclusion

Resources

Related content

Faster integration tests with real Elasticsearch

A conversational search experience for retail: Elasticsearch Relevance Engine with Google Cloud’s generative AI

How to use Hybrid search for an e-commerce product catalog

Building a recipe search with Elasticsearch

How to ingest data from AWS S3 into Elastic Cloud - Part 3: Elastic S3 Connector

Ready to build state of the art search experiences?