Using ice4j behind NLB in a VPC at AWS (Kubernetes)

[citosid]

First of all, thank you so much for this awesome work! it has been really nice working with this tool.

We have a problem though. Our infrastructure runs behind an Application Load Balancer inside a VPC in a Kubernetes cluster. The VPC has an external IP.

The requests are something like this:

pod:10.0.0.2
ec2Instance: 172.16.4.145
eni: 52.204.71.1 # Example IP
alb: 
  - 35.170.123.9
  - 34.203.105.22

User -> alb (35.170.123.9) -> ec2Instance -> pod

The discovery from the pod to find the external IP works, as can be seen from the logs:

INFO: Using org.ice4j.ice.harvest.StunMappingCandidateHarvester, face=/172.16.4.145, mask=/52.204.71.1
Jun 03, 2020 6:52:18 AM org.ice4j.ice.harvest.MappingCandidateHarvesters initialize

However, as mentioned before, the IPs the user is hitting, are the IP's from the ALB. And those are different from the one the pod is using to go to internet.

We cannot "hard-code" the IPs of the ALB in the configuration because those are elastic. AWS will change them as the load in our system increases or decreases.

And the IP of the ENI does not allow access from the outside, is just from within the VPC out.

Is there any way to use the DNS name to do the discovery?

Thank you for your time and hard work!

[bgrozev]

First of all, thank you so much for this awesome work! it has been really nice working with this tool.

Thank you for the kind words!

Is there any way to use the DNS name to do the discovery?

No, but it should be easy to add a new MappingCandidateHarvester similar to StunMappingCandidateHarvester. How often would you need to rediscover?

[citosid]

Thanks for your quick answer @bgrozev !

The TTL for the DNS in the ALB is 60 seconds. This is to quickly react to changes in the traffic.

https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#request-routing

[bgrozev]

You could have a timer running rediscovery every minute

[citosid]

I'm not really a java programmer. I can see the Stun harvester is done here:

https://github.com/jitsi/ice4j/blob/master/src/main/java/org/ice4j/ice/harvest/StunMappingCandidateHarvester.java#L70

Could you point me in the right direction?

[bgrozev]

All mapping harvesters are initialized here:
https://github.com/jitsi/ice4j/blob/master/src/main/java/org/ice4j/ice/harvest/MappingCandidateHarvesters.java#L118

[citosid]

Thanks @bgrozev . Will try to implement it and make a PR. Should I send it to you?

[bgrozev]

Just open it here and we'll take a look

[citosid]

@bgrozev , we're still working on this and researching our options. Question, can I add multiple IP addresses here:

NAT_HARVESTER_PUBLIC_ADDRESS

[citosid]

@bgrozev, I don't think we are going to need this. At least not for the time being. We found a different way to do it.

You can see it here.

We are still figuring how to make this more resilient to changes in the NLB. But maybe using (in our specific case) octo for Jitsi will help us to restart the video bridges without loosing the whole conference.

However, we will still be willing to write our own harvester in the future. But for the time being, this seems to be good enough four our use case.

I will not close the issue, but if you feel is better to, feel free to do it.

Thank you so much for your hard work!

[citosid]

@bgrozev , so after a few months more testing and working to do our release, we are in the staging phase. And it seems we might need to do this in the end.

Has it been any work on your side? We are looking into how to fix this (either by adding this code here or doing it externally)

[bgrozev]

@citosid there's been no work on this from our side.

[citosid]

Ok. Will keep looking into it and will let you know.

Thank you so much!!!

[jbg]

@citosid What about if we made it possible to specify a DNS name for NAT_HARVESTER_PUBLIC_ADDRESS (e.g. k8s-default-jvb-abcdef-abcdef.elb.us-east-1.amazonaws.com), and when starting ICE, the DNS name is resolved? This way you'd always have the current IP(s) for the load balancer.