Fix Elasticsearch Failing Tests

[Alirexaa]

Close: #4968

Microsoft Reviewers: Open in CodeFlow

[Alirexaa]

@eerhardt, Kinnda I'm stuck here.
WithDataShouldPersistStateBetweenUsages(useVolume :true) worked fine
but WithDataShouldPersistStateBetweenUsages(useVolume :false) failed.
using WithDataBindMount the container starts but API calls fail. There may be a problem in the container.

This is my test log:

| [2024-07-25T20:27:28] Aspire.Hosting.DistributedApplication Information: Aspire version: 8.1.1-dev
| [2024-07-25T20:27:28] Aspire.Hosting.DistributedApplication Information: Distributed application starting.
| [2024-07-25T20:27:28] Aspire.Hosting.DistributedApplication Information: Application host directory is: D:\Workspace\aspire\tests\Aspire.Hosting.Tests
| [2024-07-25T20:27:29] Aspire.Hosting.Dcp.DcpHostService Information: Starting DCP with arguments: start-apiserver --monitor 9720 --detach --kubeconfig "C:\Users\Hp\AppData\Local\Temp\aspire.ha4kzmlq.aan\kubeconfig"
| [2024-07-25T20:27:31] Aspire.Hosting.Dcp.start-apiserver.api-server Information: Starting API server...

| [2024-07-25T20:27:31] Aspire.Hosting.Dcp.start-apiserver.api-server Information: API server started	{"Address": "::1", "Port": 60833}

| [2024-07-25T20:27:31] Aspire.Hosting.Dcp.start-apiserver.dcp-host Information: Starting DCP controller host

| [2024-07-25T20:27:31] Aspire.Hosting.Dcp.start-apiserver.dcp-host Information: Started all services	{"count": 1}

| [2024-07-25T20:27:31] Aspire.Hosting.Dcp.dcpctrl Information: starting controller manager

| [2024-07-25T20:27:31] Aspire.Hosting.DistributedApplication Information: Distributed application started. Press Ctrl+C to shut down.
| [2024-07-25T20:27:36] Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler Information: service /elasticsearch-http-1a9fdce4 is now in state Ready	{"ServiceName": {"name":"elasticsearch-http-1a9fdce4"}, "Reconciliation": 7}

| [2024-07-25T20:27:36] Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler Information: service /elasticsearch-internal-1a9fdce4 is now in state Ready	{"ServiceName": {"name":"elasticsearch-internal-1a9fdce4"}, "Reconciliation": 9}

Invalid Elasticsearch response built from a unsuccessful () low level call on PUT: /people/_doc/1
 Exception: An error occurred while sending the request.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:07.5491833
# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

Invalid Elasticsearch response built from a unsuccessful () low level call on GET: /people/_doc/1
 Exception: An error occurred while sending the request.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:18.4748617
# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

Invalid Elasticsearch response built from a unsuccessful () low level call on PUT: /people/_doc/1
 Exception: An error occurred while sending the request.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:00.2007237
# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

Invalid Elasticsearch response built from a unsuccessful () low level call on GET: /people/_doc/1
 Exception: An error occurred while sending the request.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:12.0679370
# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

Invalid Elasticsearch response built from a unsuccessful () low level call on PUT: /people/_doc/1
 Exception: An error occurred while sending the request.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:03.0137229
# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

Invalid Elasticsearch response built from a successful (404) low level call on GET: /people/_doc/1

# Audit trail of this API call:
 - [1] HealthyResponse: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:03.5101853
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [people]","resource.type":"index_or_alias","resource.id":"people","index_uuid":"_na_","index":"people"}],"type":"index_not_found_exception","reason":"no such index [people]","resource.type":"index_or_alias","resource.id":"people","index_uuid":"_na_","index":"people"},"status":404}

Invalid Elasticsearch response built from a unsuccessful () low level call on PUT: /people/_doc/1
 Exception: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://elastic:redacted@localhost:60839/ Took: 00:01:00.0216966
 - [2] MaxTimeoutReached:
# OriginalException: System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

Invalid Elasticsearch response built from a unsuccessful (503) low level call on GET: /people/_doc/1
 Exception: Request failed to execute. Call: Status code 503 from: GET /people/_doc/1. ServerError: Type: no_shard_available_action_exception Reason: "No shard available for [get [people][1]: routing [null]]"

# Audit trail of this API call:
 - [1] BadResponse: Node: http://elastic:redacted@localhost:60839/ Took: 00:00:00.0142227
# OriginalException: Elastic.Transport.TransportException: Request failed to execute. Call: Status code 503 from: GET /people/_doc/1. ServerError: Type: no_shard_available_action_exception Reason: "No shard available for [get [people][1]: routing [null]]"
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"No shard available for [get [people][1]: routing [null]]"}],"type":"no_shard_available_action_exception","reason":"No shard available for [get [people][1]: routing [null]]"},"status":503}

[eerhardt]

I'm no expert in elsaticsearch. Is the container reading the saved data correctly? What else could be different between data volumes and bind mounts?

[Alirexaa]

@eerhardt PTAL, it seems worked.

[mitchdenny]

Looks like adding longer retries for resiliency helped here.