bug: grpc-web plugin truncates trailing headers

[RogueJin]

Current Behavior

The response from grpc server has a trailing header but APISIX grpcweb plugin truncates it and causes client error.

The related code of grpc-dotnet could be checked here:
https://github.com/grpc/grpc-dotnet/blob/46480fca2fccddb3391cf267004f281afe87571d/src/Grpc.Net.Client/Internal/GrpcProtocolHelpers.cs#L386

Expected Behavior

Trailing headers could be reserved for client.

Error Logs

No response

Steps to Reproduce

1. run APISIX via docker compose
2. setup upstream grpcweb api
3. config route

{
  "uri": "/*",
  "name": "grpcweb",
  "plugins": {
    "cors": {},
    "grpc-web": {}
  },
  "upstream_id": "479432631159620291",
  "status": 1
}

5. curl

curl http://localhost:9080/xxxxxxxxx -X OPTIONS -v

Environment

• APISIX version (run apisix version): 3.5.0
• Operating system (run uname -a):

Linux 49c5eb4d90cd 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 GNU/Linux

• OpenResty / Nginx version (run openresty -V or nginx -V):
  nginx version: openresty/1.21.4.1
  built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
  built with OpenSSL 1.1.1s 1 Nov 2022
  TLS SNI support enabled

• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):

• APISIX Dashboard version, if relevant:

• Plugin runner version, for issues related to plugin runners:

• LuaRocks version, for installation issues (run luarocks --version):

[shreemaan-abhishek]

I'm not sure if APISIX modified the response because it's clear from the code that the only (response) body modification involved in the grpc-web plugin is base64 encoding:

https://github.com/shreemaan-abhishek/apisix/blob/88da2ecdd2a331d62f4c6d9849910d164ad33ead/apisix/plugins/grpc-web.lua#L135-L150

Can encoding lead to such an issue? 🤔

[RogueJin]

The content-type is application/grpc-web and it uses binary encoding according to code.

content-type: application/grpc-web
grpc-accept-encoding: identity,gzip,deflate

https://github.com/shreemaan-abhishek/apisix/blob/88da2ecdd2a331d62f4c6d9849910d164ad33ead/apisix/plugins/grpc-web.lua#L43-L48

Just as you mentioned, the weird thing is that there is no explicit code modifying response body.

Any clue?

[Affanmir]

Facing the same issue, any update on this?

[kayx23]

See projectcontour/contour#3233

[shreemaan-abhishek]

continue discussion on #10573

[sheharyaar]

@RogueJin , can you please check this and verify if changing from grpc-web to grpc-web-text works for you : #10573 (comment)

[RogueJin]

we decided to give up grpc-web months ago since the issue, but I would check it in my spare time. thanks for the fixing anyway.

@RogueJin , can you please check this and verify if changing from grpc-web to grpc-web-text works for you : #10573 (comment)

[sheharyaar]

@RogueJin , you can also checkout #10851 , if that fixes the issue