panic during shutdown flow

[Dieterbe]

good instance:

[Sarama] 2018/03/15 17:16:48 kafka: error while consuming m-12574-M-ops/27: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
[Sarama] 2018/03/15 17:16:48 consumer/m-12574-M-ops/27 shutting down because kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
2018/03/15 17:16:48 [kafkamdm.go:310 consumePartition()] [E] kafka-mdm: kafka consumer for m-12574-M-ops:27 has shutdown. stop consuming
2018/03/15 17:16:48 [I] An input plugin signalled a fatal error. Shutting down
2018/03/15 17:16:48 [I] CLU manager: HTTPNode mt-read03-12574-medium-ops-b-1218431600-1d3fz has left the cluster
2018/03/15 17:16:48 [I] CLU manager: HTTPNode mt-read01-12574-medium-ops-b-3464612116-st4js has left the cluster
2018/03/15 17:16:48 [I] API shutdown started.
2018/03/15 17:16:49 [I] API accept tcp [::]:6060: use of closed network connection
2018/03/15 17:16:49 [I] Shutting down kafka-mdm consumer
[Sarama] 2018/03/15 17:16:49 consumer/broker/786437 closed dead subscription to m-12574-M-ops/31
[Sarama] 2018/03/15 17:16:49 consumer/broker/786439 closed dead subscription to m-12574-M-ops/25
[Sarama] 2018/03/15 17:16:49 consumer/broker/786439 closed dead subscription to m-12574-M-ops/29
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:31 ended.
[Sarama] 2018/03/15 17:16:49 consumer/broker/786438 closed dead subscription to m-12574-M-ops/24
[Sarama] 2018/03/15 17:16:49 consumer/broker/786438 closed dead subscription to m-12574-M-ops/28
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:29 ended.
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:25 ended.
[Sarama] 2018/03/15 17:16:49 consumer/broker/786440 closed dead subscription to m-12574-M-ops/26
[Sarama] 2018/03/15 17:16:49 consumer/broker/786440 closed dead subscription to m-12574-M-ops/30
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:28 ended.
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:24 ended.
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:30 ended.
2018/03/15 17:16:49 [I] kafka-mdm consumer for m-12574-M-ops:26 ended.
2018/03/15 17:16:49 [I] Closing partitionsOffset DB.
[Sarama] 2018/03/15 17:16:49 Closing Client
[Sarama] 2018/03/15 17:16:49 Closed connection to broker kafka:9092
[Sarama] 2018/03/15 17:16:49 Closed connection to broker 10.12.0.7:9092
[Sarama] 2018/03/15 17:16:49 Closed connection to broker 10.12.0.6:9092
[Sarama] 2018/03/15 17:16:49 Closed connection to broker 10.12.0.5:9092
2018/03/15 17:16:49 [I] kafka-mdm consumer finished shutdown
2018/03/15 17:16:49 [I] closing store
[Sarama] 2018/03/15 17:16:49 Closed connection to broker 10.12.0.8:9092
2018/03/15 17:16:49 [I] cassandra-idx stopping
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] cassandra-idx writeQueue handler ended.
2018/03/15 17:16:49 [I] terminating.

bad instance:

[Sarama] 2018/03/15 17:16:48 kafka: error while consuming m-12574-M-ops/23: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
[Sarama] 2018/03/15 17:16:48 consumer/m-12574-M-ops/23 shutting down because kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
[Sarama] 2018/03/15 17:16:48 kafka: error while consuming m-12574-M-ops/19: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
[Sarama] 2018/03/15 17:16:48 consumer/m-12574-M-ops/19 shutting down because kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
2018/03/15 17:16:48 [kafkamdm.go:310 consumePartition()] [E] kafka-mdm: kafka consumer for m-12574-M-ops:19 has shutdown. stop consuming
2018/03/15 17:16:48 [I] An input plugin signalled a fatal error. Shutting down
2018/03/15 17:16:48 [I] CLU manager: HTTPNode mt-read02-12574-medium-ops-b-1100597842-p69pz has left the cluster
2018/03/15 17:16:48 [kafkamdm.go:310 consumePartition()] [E] kafka-mdm: kafka consumer for m-12574-M-ops:23 has shutdown. stop consuming
panic: close of closed channel

goroutine 1264 [running]:
github.com/grafana/metrictank/input/kafkamdm.(*KafkaMdm).consumePartition(0xc4208d65a0, 0xc420034f47, 0xd, 0x300000017, 0x3baa4f7e7)
	/go/src/github.com/grafana/metrictank/input/kafkamdm/kafkamdm.go:314 +0x9bc
created by github.com/grafana/metrictank/input/kafkamdm.(*KafkaMdm).Start
	/go/src/github.com/grafana/metrictank/input/kafkamdm/kafkamdm.go:225 +0x106

[Dieterbe]

it's caused by multiple goroutines running into a kafka error and each trying to close the same fatal channel.
may need to rethink this a bit.. e.g. send values on the channel instead of closing.

not urgent though