Error in launching task: Trying to map ports but no network interface is available

[futuralogic]

We're in the process of evaluating nomad and trying to test the IIS driver but can't get it to register a single site.

This occurs on two separate IIS servers (we are testing with).

OS: Server 2012 R2 Standard (x64)

Is there something we're missing in our client configuration or in the job spec?

> nomad run job.nomad

Except from the nomad log:

client.driver_mgr.win_iis.exe: Error in launching task: Trying to map ports but no network interface is available: driver=win_iis @module=win_iis timestamp=2021-02-15T16:57:03.502-0600
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: panic: runtime error: invalid memory address or nil pointer dereference: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: [signal 0xc0000005 code=0x1 addr=0x18 pc=0xc23e52]: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: : driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: goroutine 49 [running]:: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: github.com/roblox/nomad-driver-iis/iis.(*taskHandle).handleError(0xc0002ff5e0, 0xe043b5, 0x52, 0xece940, 0xc0001fe880): driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: 	C:/Users/jjj/go/src/github.com/Roblox/nomad-driver-iis/iis/handle.go:170 +0x72: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: github.com/roblox/nomad-driver-iis/iis.(*taskHandle).run(0xc0002ff5e0, 0xc000321c80): driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: 	C:/Users/jjj/go/src/github.com/Roblox/nomad-driver-iis/iis/handle.go:111 +0xdfc: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: created by github.com/roblox/nomad-driver-iis/iis.(*Driver).StartTask: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: 	C:/Users/jjj/go/src/github.com/Roblox/nomad-driver-iis/iis/driver.go:313 +0x74e: driver=win_iis

Here is the job file:

job "web-test" {
	datacenters = ["dc1"]
	type = "service"

	group "web-test-group" {
		count = 1

		restart {
		  attempts  = 1
		  delay     = "30s"
		  interval	= "5m"
		  mode	= "delay"
		}

		network {
			port "web" {
				to = 10200
			}
		}

		task "web-site" {
			driver = "win_iis"

			config {
				
				path = "C:\\inetpub\\wwwroot"
				
				bindings {
					type 			= "http"
					resource_port 	        = "web"
				}
			}

			resources {
				cpu    = 100
				memory = 20
			  }

			service {
				name = "web-test"
				port = "web"
				
				tags = [
					"traefik.enable=false",
					"traefik.endpoints=web",
					"traefik.routers.cutracker.rule=Host(`something`)"
				]

				check {
					name = "web"
					port = "web"
					type = "http"					
					path = "/"
					interval = "30s"
					timeout = "10s"
				}
			}

		}
	}
}

Client config:

datacenter = "dc1"
region = "us"
data_dir   = "C:/d/nomad/data"

consul {
  address = "127.0.0.1:8500"
}

advertise {
    http = "10.43.0.5"
    rpc = "10.43.0.5"
    serf = "10.43.0.5"
}

log_level = "DEBUG"
log_file = "C:/d/nomad/logs/"
log_rotate_duration = "4h"
log_rotate_max_files = 10

client {
  enabled = true
  meta = {
	"workload_env"="dev"
	"caps_iis"="true"
  }
  
}

acl {
  enabled    = false
}

plugin "raw_exec" {
  config {
    enabled = false
  }
}

plugin "win_iis" {
  client {
    enabled = true
    stats_interval= "30s"
  }
}

[Vulfox]

Though I don't test with Windows2012R2 on the regular, the last time I did test against that OS version, I didn't come across any issues. I think the code is unable to find the network resource port that is established outside of the task stanza. If you move

network {
  port "web" {
    to = 10200
  }
}

Also the to usage indicated a bridge network, were you meaning to use a static port?

[futuralogic]

@Vulfox I have attempted specifying static and also tried mode = "host" in the network stanza.

You mention "move" of the network stanza - where would I move it to? My understanding is it can only belong in the group where it is currently located. Although the example in the project seems to use an older nomad syntax where the network resources are specified in the resources stanza inside the task. I did attempt this, of course, nomad complained.

So, I have tried:

network {
   port "web" {
      static = 10200
   }
}

And:

network {
   mode = "host"
   port "web" {
      static = 10200 # and to = 10200
   }
}

[shishir-a412ed]

Though I don't test with Windows2012R2 on the regular, the last time I did test against that OS version, I didn't come across any issues. I think the code is unable to find the network resource port that is established outside of the task stanza. If you move

network {
  port "web" {
    to = 10200
  }
}

Also the to usage indicated a bridge network, were you meaning to use a static port?

@Vulfox @futuralogic is using a dynamic port mapping where a dynamically allocated random port will be mapped to 10200 inside the job. Specifying at the group level is the right place, as the newer (current) version of Nomad allows you to specify port mappings at the group level, and that will be available to all the tasks in the group.

Looking at the error

Error in launching task: Trying to map ports but no network interface is available:

hashicorp/nomad#8488 might be helpful.
There was a similar issue in the docker driver in v0.12.0, which was fixed here

@futuralogic Which version of Nomad are you running? Just trying to check if you are not on an older version, and this might already be fixed.

@Vulfox You might wanna check with hashicorp/nomad maintainers if this issue was fixed for all custom drivers or just docker.

On a side note:

client.driver_mgr.win_iis.exe: panic: runtime error: invalid memory address or nil pointer dereference: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: [signal 0xc0000005 code=0x1 addr=0x18 pc=0xc23e52]: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: : driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: goroutine 49 [running]:: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: github.com/roblox/nomad-driver-iis/iis.(*taskHandle).handleError(0xc0002ff5e0, 0xe043b5, 0x52, 0xece940, 0xc0001fe880): driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: 	C:/Users/jjj/go/src/github.com/Roblox/nomad-driver-iis/iis/handle.go:170 +0x72: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: github.com/roblox/nomad-driver-iis/iis.(*taskHandle).run(0xc0002ff5e0, 0xc000321c80): driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: 	C:/Users/jjj/go/src/github.com/Roblox/nomad-driver-iis/iis/handle.go:111 +0xdfc: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: created by github.com/roblox/nomad-driver-iis/iis.(*Driver).StartTask: driver=win_iis
2021-02-15T16:57:03.505-0600 [DEBUG] client.driver_mgr.win_iis.exe: 	C:/Users/jjj/go/src/github.com/Roblox/nomad-driver-iis/iis/driver.go:313 +0x74e: driver=win_iis

Even if there is an error, nomad-driver-iis should exit gracefully, rather than blowing up and throwing a nil pointer exception 🙂 . Something to address in the driver itself.

[Vulfox]

@shishir-a412ed I'll give running the driver a try for nomad 1.0+ and update our tests accordingly. Everything in the repo is still operating on 0.11.x. It looks like that fix is applicable to all drivers, however there may be some code change that is needed on this driver's behalf to correctly gather task and group level allocated ports like what was done with the docker driver. Thanks for the tips.

#41 is hopefully the issue to address the lack of transparency when it comes to errors and/or gracefully handling the situation.

[futuralogic]

@Vulfox Thanks for following-up, much appreciated.

I'm using nomad v1.0.3

[Vulfox]

Still looking into this @futuralogic . Been addressing some other things with this driver as well. I can confirm, that with nomad 1.0+ the driver does not work (create an app pool nor site). This is definitely related to the bindings issue as well as the issue noted about a task reporting as "healthy" when indeed, it did not do its job.

[futuralogic]

@Vulfox Thank you! Much appreciated!

[Vulfox]

@futuralogic , I have fixed the binding issue with Nomad 1.0+, if you build a binary from the master branch, you should be able to move forward. There are a couple more issues related to the Nomad 1.0+ readiness atm, but you should see your website get created at the least.

Edit: Once I have stabilized 0.2.0, I'll be putting it up as a release on github.

[futuralogic]

It worked successfully with a basic binding. I will aim to do some further testing with site + app xml config files.

Is there anything specific around your comment "There are a couple more issues related to the Nomad 1.0+ readiness atm" that you could elaborate on? Or is it mostly behind-the-scenes type stuff?

We're not (that I know of) going to be attempting complicated IIS configurations, etc. Likely sticking to site/app xml launches and potentially some straight "path" bindings as needed.

Thank you so much for your work getting this updated for 1.0+ - it's greatly appreciated. We're excited to be able to move forward with nomad for our IIS apps.

[Vulfox]

@futuralogic There are a couple of outstanding bugs with the system and a few enhancements to ensure that we have a stable product here (such as an end to end test framework). To help visualize what needs to be done, I created a 0.2.0 Milestone found here: https://github.com/Roblox/nomad-driver-iis/milestone/1 .

When I meant Nomad 1.0+ readiness, I think I got mixed up and meant driver 0.2.0 release readiness. I may have conflated the two, I do apologize.

I hope the milestone makes it a bit more clear as what is remaining and potential bugs to look out for. The biggest one I can think of being a blocker for you, is the use of importing XMLs. This issue covers Site XML interaction #52 . I have the code done that makes for a better XML interaction done. I am just going through the testing process now.

I will be drafting up a PR that will probably have all of the remaining issues fixed and eventually siphoning it into smaller PRs for folks to ingest. The drafted PR should be tied to a branch, so if you need that XML interaction sooner than later, you will have a way to work with it while I finish up other 0.2.0 items that are somewhat tied to it.

[tfrijsewijk]

Hi @Vulfox! Great work is being done here!

I've built a binary from the 0.2.0 branch but I'm still running into the bindings issue mentioned here. I'm running Nomad 1.0.4 with Windows 2016. Any pointers here?

[Vulfox]

Hey @tfrijsewijk , are there any errors being thrown in the nomad logs when trying to set the bindings? What does the end result in IIS look like (empty, corrupted)? An example of what your Nomad config might help too, especially with the 0.2.0 not being terribly well documented and somewhat experimental.

[tfrijsewijk]

Hi @Vulfox! I am such a dummy -> I was doing something completely wrong.. I want my application to run on port 80, so in the binding stanza I added port = "80" (or port = 80).. But of course, that's now how it works 🤡

I wanted to provide you with a good example, but doing that I fixed my own problem.

Thanks for replying anyway!

How is the development coming along? I'm looking forward to 0.2.0. Is Roblox actually using this driver or an internal fork?

[Vulfox]

@tfrijsewijk Development is unsteady as you can see. I have higher Roblox priorities than this driver. It was initially made as a POC and it hasn't taken flight as much as I would like. What you see here is what we tried to utilize internally. There's a lot of weird things we do that needed addressing and we might be rounding a corner where this driver might be feasible again.

I've been meaning to loop back on this repo in my free time to deal with dated dependencies and release a bulk of 0.2.0, but Elden Ring is eating up all of my free time right now :)

[tfrijsewijk]

So if I understand correctly, the driver still has a future :)

I've heard that Elden Ring isn't a small indie game.. I hope you'll finish the game quickly so I can stalk you around here again ;-)