Worker hang up in an infinite loop, to be researched

[Keksoj]

In production, we have witnessed a Sōzu worker becoming irresponsive, and when investigating, we found it was stuck in an infinite loop.

To the best of our understanding, this loop is stuck:

sozu/lib/src/protocol/pipe.rs

Lines 428 to 458 in 409af59

	while res == SocketResult::Continue {
	// no more data in buffer, stop here
	if self.backend_buffer.available_data() == 0 {
	count!("bytes_out", sz as i64);
	metrics.bout += sz;
	self.backend_readiness.interest.insert(Ready::READABLE);
	self.frontend_readiness.interest.remove(Ready::WRITABLE);
	return SessionResult::Continue;
	}
	let (current_sz, current_res) = self.frontend.socket_write(self.backend_buffer.data());
	res = current_res;
	self.backend_buffer.consume(current_sz);
	sz += current_sz;
	if current_sz == 0 && res == SocketResult::Continue {
	self.frontend_status = match self.frontend_status {
	ConnectionStatus::Normal => ConnectionStatus::ReadOpen,
	ConnectionStatus::WriteOpen => ConnectionStatus::Closed,
	s => s,
	};
	}
	if !self.check_connections() {
	metrics.bout += sz;
	count!("bytes_out", sz as i64);
	self.frontend_readiness.reset();
	self.backend_readiness.reset();
	self.log_request_success(metrics);
	return SessionResult::Close;
	}
	}

The Rustls pipeline went through, but writing on the socket failed.

self.frontend.socket_write() is dispatched on a Rustls session, and the retured tuple (current_sz, current_res) has always the value (0, SocketResult::Continue).

current_sz should have non-zero value, or some retry should take place, or some status should be logged.

The result sent by Rustls may be misunderstood and should be better investigated.

[Keksoj]

solved by #1133