connection refused by submited flink cluster to docker Hadoop cluster on macOS.

[kalencaya]

I'm using docker-compose.yml bootstrap Hadoop cluster on macOS as my local develop environment and try to submit flink job to it.
This is my docker-compose.yml and env: https://github.com/flowerfine/scaleph/tree/dev/tools/docker/hadoop

To run successfully on macOS, I had added ${host ip} datanode namenode ... ${namenode container id} ${datanode container id} ... to my local hosts file, then I can browse HDFS from web-ui, submit flink application or per-job deploy mode job, create session cluster.
But when I try to access submited flink cluster, such as list jobs or submit flink job to created session cluster, exception encontered:

Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 7c8c951d151d/127.0.0.1:41565
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
	at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:834)

Notice that every attemp will fail on another port, not always 127.0.0.1:41565.
Can any one give me some advice? I will appreciate your help

[kalencaya]

When try to submit flink to Hadoop cluster, Hadoop will assign random port for flink cluster. On docker environment, new assigned port has not been exposed and network exception raised.
Following image is Hadoop on metal machine:

This is Hadoop on docker: