Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

客户端与服务端payload参数不一致,服务端响应数据包长度超出客户端payload时的问题 #7021

Closed
2 tasks done
SeanHongXing opened this issue Dec 15, 2020 · 13 comments

Comments

@SeanHongXing
Copy link

SeanHongXing commented Dec 15, 2020

  • I have searched the issues of this repository and believe that this is not a duplicate.
  • I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

  • Dubbo version: 2.7.7
  • Java version: 1.8

Steps to reproduce this issue

  1. 客户端通过URL直连服务端(dubbo协议、单连接),URL中指定payload
  2. 客户端请求数据,服务端返回数据包长度大于客户端payload,小于服务端payload

server

public class DubboServer {
    public interface DownloadService {
        byte[] download(int size);
        class DownloadServiceImpl implements DownloadService {

            @Override
            public byte[] download(int size) {
                byte[] bytes = new byte[size];
                Arrays.fill(bytes, (byte) 0);
                return bytes;
            }
        }
    }
    public static void main(String[] args) throws InterruptedException {
        DownloadService downloadService = new DownloadService.DownloadServiceImpl();
        ApplicationConfig application = new ApplicationConfig();
        application.setName("dubboServer");
        ProtocolConfig protocol = new ProtocolConfig();
        protocol.setName("dubbo");
        protocol.setPort(12345);
        protocol.setThreads(20);
        protocol.setPayload(10 * 1024 * 1024);
        RegistryConfig registry = new RegistryConfig("zookeeper://xxxx:2181");
        ServiceConfig<DownloadService> service = new ServiceConfig<>();
        service.setApplication(application);
        service.setRegistry(registry);
        service.setProtocol(protocol);
        service.setInterface(DownloadService.class);
        service.setRef(downloadService);
        service.setVersion("1.0.0");
        service.export();
        Thread.currentThread().join();
    }
}

client

public class DubboClient {
    public static void main(String[] args) {
        ApplicationConfig application = new ApplicationConfig();
        application.setName("dubboClient");
        ReferenceConfig<DownloadService> reference = new ReferenceConfig<>();
        reference.setApplication(application);
        reference.setUrl("dubbo://127.0.0.1:12345?payload=1000");
        reference.setInterface(DownloadService.class);
        reference.setVersion("1.0.0");
        reference.setTimeout(60_000);
        DownloadService downloadService = reference.get();
        byte[] bytes = downloadService.download(300 * 1024);
    }
}

Expected Result

客户端直接抛出ExceedPayloadLimitException

Actual Result

客户端等待超时时间,提示RemoteTimeoutException
实际抓包结果:
image

问题定位

服务端返回响应时,协议头中声明了数据包长度,客户端接收响应通过ExchangeCodec.decode()解码,在该方法中调用了checkPayload()方法,checkPayload()直接抛出ExceedPayloadLimitException,该异常最终由netty处理,dubbo没有将服务端的响应对应到之前的请求上,且没有将异常信息反馈给服务端,服务端依旧发送响应数据包,客户端接收后直接丢弃。

影响

发生于客户端指定dubbo协议payload的情形,当客户端payload与服务端payload参数不一致时,服务端响应数据包长度大于客户端payload,客户端未正确处理数据包超出payload的异常。最终导致客户端超时等待,且服务端依旧将数据包发送完,浪费网络开销,客户端调用方被阻塞至超时。

@xiaoheng1
Copy link
Contributor

你是如何在客户端指定 payload 参数的?

@SeanHongXing
Copy link
Author

SeanHongXing commented Dec 17, 2020

你是如何在客户端指定 payload 参数的?

客户端通过serUrl(dubbo://host:port?payload=xxxx)l直连的客户端,

@xiaoheng1
Copy link
Contributor

@SeanHongXing 确实有问题,并且打印出异常后,还收到了数据,打印了这个异常:Dubbo client can not supported string message

@SeanHongXing
Copy link
Author

SeanHongXing commented Dec 17, 2020

@SeanHongXing 确实有问题,并且打印出异常后,还收到了数据,打印了这个异常:Dubbo client can not supported string message

嗯是这样的,可以稳定重现,这个我认为可能是dubbo没有处理好这种场景,实际应用中如果使用不当也是有可能发生这种问题。服务端数据包发完问题倒是不大,只是占用点带宽,但客户端等待超时这个影响就比较大了,集群配置下,还可能会多次重试

@xiaoheng1
Copy link
Contributor

我认为可以在 HeaderExchangeHandler 的 caught 方法中对 ExceedPayloadLimitException 异常进行处理,来修复此问题。

@xiaoheng1
Copy link
Contributor

@SeanHongXing 你在测的时候遇到了 Dubbo client can not supported string message 这个异常没?

@SeanHongXing
Copy link
Author

SeanHongXing commented Dec 18, 2020

你是如何在客户端指定 payload 参数的

日志里会打印,应该是服务端后续的数据包被当做没有协议头的数据了

@xiaoheng1
Copy link
Contributor

应该不是这个问题,我看了下 url 还是带有 payload 参数的

@SeanHongXing
Copy link
Author

应该不是这个问题,我看了下 url 还是带有 payload 参数的

哪个不是这个问题?

应该是服务端后续的数据包被当做没有协议头的数据了

这个吗?

@xiaoheng1
Copy link
Contributor

我看到第二次收到数据,包长度增加了,正常的包应该是 870 字节,但是第二次收到了 887 字节

@xiaoheng1
Copy link
Contributor

应该不是这个问题,我看了下 url 还是带有 payload 参数的

哪个不是这个问题?

应该是服务端后续的数据包被当做没有协议头的数据了

这个吗?

这个应该是正解,最后走 telnetCodec 了,所以会报 Dubbo client can not supported string message 这个错

@SeanHongXing
Copy link
Author

应该不是这个问题,我看了下 url 还是带有 payload 参数的

哪个不是这个问题?

应该是服务端后续的数据包被当做没有协议头的数据了

这个吗?

这个应该是正解,最后走 telnetCodec 了,所以会报 Dubbo client can not supported string message 这个错

是的,没走到ExchangeCodec,后续的数据包当成telnet数据包了,直接就丢弃了

xiaoheng1 added a commit to xiaoheng1/incubator-dubbo that referenced this issue Feb 24, 2021
…tent, and the server response packet length exceeds the client payload.
xiaoheng1 added a commit to xiaoheng1/incubator-dubbo that referenced this issue Feb 25, 2021
…tent, and the server response packet length exceeds the client payload.

(cherry picked from commit 86d00b5)
@xiaoheng1
Copy link
Contributor

This issue has been handled on #7287

&READY-TO-CLOSE&

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants