Unexpected behaviour in MultiSearchAsync

[IanSymplectic]

Elastic.Clients.Elasticsearch version:
9.22
Elasticsearch version:
9.22
.NET runtime version:
.NET 10 (also observed in .NET Standard 2.0)
Operating system version:
Windows 11
Description of the problem including expected versus actual behavior:
I've been updating our integration with Elastic.Clients.Elasticsearch from 8.15.6 to 9.2.2 (aiming to make minimal changes to our existing code), and have encountered what seems to be an error in the new version. Specifically, when calling MultiSearchAsync the resulting MultiSearchResponse has Responses all representing errors where previously they successfully contained the number of hits for each query. Elasticsearch itself seems to return a successful response with the required information, but the client library doesn't surface it to my calling code. This worked in the previous version.

Steps to reproduce:

1. Use MultiSearchAsync to get counts of documents matching queries
2. Inspect the returned response, and see that it contains errors rather than matching document counts

Expected behavior
MultiSearchResponse should contain responses with the number of hits.

Here is a sample program which demonstrates the issue - it logs the apparently-successful response from Elasticsearch, and then hits an exception rather than using the results.

using Elastic.Clients.Elasticsearch;
using Elastic.Clients.Elasticsearch.Core.MSearch;
using Elastic.Clients.Elasticsearch.Core.Search;
using Elastic.Clients.Elasticsearch.QueryDsl;
using Elastic.Transport;
using Elastic.Transport.Diagnostics.Auditing;
using System.Diagnostics.CodeAnalysis;
using System.Text;

// This is a demonstration of performing multiple count queries in a single request using the Elasticsearch .NET client.
// In v9.2.2, it results in an error despite Elasticsearch seeming to respond correctly. The equivalent code worked well in v8.15.6.

// Adjust these values to match your Elasticsearch cluster and index.
const string username = "your-username";
const string password = "your-password";
const string clusterUrl = "your-url";
const string indexName = "your-index";
List<Query> queries = new List<Query> {
	new TermQuery("id", 1),
	new TermQuery("id", 2)
};

// Below this line is the code to perform the multi-count operation, and should not need to be edited to demonstrate the issue.


// Create the client, and set up logging.
ElasticsearchClientSettings connectionSettings = new ElasticsearchClientSettings(
		new SingleNodePool(new Uri(clusterUrl))
	)
	.EnableHttpCompression()
	.Authentication(new BasicAuthentication(username, password))
	.ThrowExceptions();

connectionSettings.ServerCertificateValidationCallback(CertificateValidations.AllowAll);
DevRequestLogger requestLogger = new DevRequestLogger(Console.WriteLine);
connectionSettings.EnableDebugMode(requestLogger.LogRequest);
ElasticsearchClient client = new ElasticsearchClient(connectionSettings);


// Now try to fetch and log our multi-count results.
try
{
	List<long> result = await HelperMethods.MultiCountAsync(client, indexName, queries, CancellationToken.None);
	// In v9.2.2, we never reach this line due to an exception being thrown.
	Console.WriteLine($"Counts: {string.Join(", ", result)}");
} catch (Exception ex) {
	Console.WriteLine($"An error occurred: {ex.Message}");
}


static class HelperMethods {
	/// <summary>
	/// Uses the provided client to execute the specified queries, returning the count of results for each query in the same order.
	/// </summary>
	public static async Task<List<long>> MultiCountAsync(ElasticsearchClient client, [NotNull] string indexName, [NotNull] IEnumerable<Query> queries, CancellationToken cancellationToken)
	{
		List<SearchRequestItem> searchItems = queries.Select(query => new SearchRequestItem(new MultisearchBody
		{
			Query = query,
			// We don't want any information about the results themselves - just the number of total hits
			Size = 0,
			TrackTotalHits = new TrackHits(true),
			Source = new SourceConfig(false)
		})).ToList();

		// If there are no searches to perform, we can just return an empty list
		if (searchItems.Count == 0)
		{
			return new List<long>();
		}

		MultiSearchResponse<object> multiSearchResponse = await client.MultiSearchAsync<object>(
			new MultiSearchRequest(indexName, searchItems)
			{
				// We can reduce the amount of data we have to transfer by only asking for the fields we need (hits.total.value for the result count)
				FilterPath = new[] { "responses.hits.total.value" }
			},
			cancellationToken
		).ConfigureAwait(false);

		if (multiSearchResponse.Responses == null || multiSearchResponse.Responses.Count != searchItems.Count)
		{
			throw new Exception($"The number of responses from a multi-search ({multiSearchResponse.Responses?.Count ?? 0}) did not match the number of searches sent ({searchItems.Count}). Please check that the {indexName} index exists.");
		}

		return multiSearchResponse.Responses.Select((response, index) => response.SafeMatch(
			success => success.Total,
			// In 9.2.2, we hit this branch due to an apparent error in the multi-search response handling, despite Elasticsearch returning a valid response.
			failure => throw new MultiSearchException(index, failure.Error)
		)).ToList();
	}

	/// <summary>
	/// Behaves like <see cref="Union{TUnion1, TUnion2}.Match"/>, but throws an <see cref="InvalidOperationException"/> if either value is unexpectedly null.
	/// </summary>
	public static TResult SafeMatch<TUnion1, TUnion2, TResult>(this Union<TUnion1, TUnion2> union, Func<TUnion1, TResult> first, Func<TUnion2, TResult> second)
	{
		return union.Match(
			value1 =>
			{
				if (value1 == null) { throw new InvalidOperationException("Unexpectedly-missing first value"); }
				return first(value1);
			},
			value2 =>
			{
				if (value2 == null) { throw new InvalidOperationException("Unexpectedly-missing second value"); }
				return second(value2);
			}
		);
	}
}

/// <summary>
/// Represents an exception encountered while performing a batch of searches.
/// </summary>
class MultiSearchException : Exception {
	/// <summary>
	/// The index of the search result which encountered the error.
	/// </summary>
	public int ResultIndex { get; }

	public ErrorCause? ErrorCause { get; }

	public MultiSearchException(int resultIndex, ErrorCause? errorCause)
	{
		ResultIndex = resultIndex;
		ErrorCause = errorCause;
	}

	public override string Message => $"Search result with index {ResultIndex} encountered an error: {ErrorCause?.Reason ?? "[no specified reason]"}";
}

/// <summary>
/// Allows logging Elasticsearch requests and responses.
/// </summary>
class DevRequestLogger {
	/// <summary>
	/// The action to perform when logging a request.
	/// </summary>
	Action<string> LoggingCallback { get; }

	public DevRequestLogger(Action<string> loggingCallback)
	{
		LoggingCallback = loggingCallback;
	}

	/// <summary>
	/// Builds a string representation provided <paramref name="callDetails"/>, and passes it to the <see cref="LoggingCallback"/>.
	/// </summary>
	public void LogRequest([NotNull] ApiCallDetails callDetails)
	{
		if (callDetails == null) throw new ArgumentNullException(nameof(callDetails));

		StringBuilder logBuilder = new StringBuilder();

		// Log the HTTP method and URI path
		logBuilder.AppendLine($"{callDetails.HttpMethod} {callDetails.Uri?.PathAndQuery}");

		// We'll attempt to log the timings of the request next
		Audit? auditEvent = callDetails.AuditTrail?.FirstOrDefault(a => a.Event == AuditEvent.HealthyResponse);
		if (auditEvent != null)
		{
			TimeSpan duration = auditEvent.Ended.Subtract(auditEvent.Started);
			logBuilder.AppendLine($"Initiated: {auditEvent.Started.ToLocalTime().ToString("dd MMM yyyy HH:mm:ss.fffffffzzz")}");
			logBuilder.AppendLine($"Duration: {(int)duration.TotalMilliseconds}ms");
		}

		// Then we'll include the request if we have it
		logBuilder.AppendLine("Request:");
		string requestBody = callDetails.RequestBodyInBytes == null ? "" : Encoding.UTF8.GetString(callDetails.RequestBodyInBytes);
		logBuilder.AppendLine(requestBody == "" ? "[no request body]" : requestBody);

		// We'll only have a response if there is a status code
		if (callDetails.HttpStatusCode == null)
		{
			logBuilder.AppendLine("[no response]");
		}
		else
		{
			logBuilder.AppendLine($"Response (status {callDetails.HttpStatusCode}):");
			// Include the response body if we have it
			string responseBody = callDetails.ResponseBodyInBytes == null ? "" : Encoding.UTF8.GetString(callDetails.ResponseBodyInBytes);
			logBuilder.AppendLine(responseBody == "" ? "[no response body]" : responseBody);
		}

		LoggingCallback(logBuilder.ToString());
	}
}

Please let me know if I can provide any further information!

[flobernd]

Hi @IanSymplectic , Elasticsearch returning a valid response does not mean that the individual searches can't fail.

Could you please capture the JSON request and response and double check if the request succeeds when e.g. executed in the Kibana dev console?

You can capture the JSON payloads using the .EnableDebugMode(...) switch in the ElasticsearchClientSettings. Afterwards, the JSON payloads are available in the ApiCallDetails of the response.

[IanSymplectic]

Hi @flobernd , here are the request and response as captured by the DevRequestLogger in the example:

Request:
null
{"query":{"term":{"id":{"value":1}}},"size":0,"_source":false,"track_total_hits":true}
null
{"query":{"term":{"id":{"value":2}}},"size":0,"_source":false,"track_total_hits":true}

Response (status 200):
{
  "responses" : [
    {
      "hits" : {
        "total" : {
          "value" : 1
        }
      }
    },
    {
      "hits" : {
        "total" : {
          "value" : 1
        }
      }
    }
  ]
}