Unknown Error

[israrbacha]

VariableV2: CPU
Add: CPU XLA_CPU XLA_GPU
Mul: CPU XLA_CPU XLA_GPU
RandomStandardNormal: CPU XLA_CPU XLA_GPU
Const: CPU XLA_CPU XLA_GPU

Colocation members, user-requested devices, and framework assigned devices, if any:
G_synthesis/noise0/Initializer/random_normal/shape (Const)
G_synthesis/noise0/Initializer/random_normal/mean (Const)
G_synthesis/noise0/Initializer/random_normal/stddev (Const)
G_synthesis/noise0/Initializer/random_normal/RandomStandardNormal (RandomStandardNormal)
G_synthesis/noise0/Initializer/random_normal/mul (Mul)
G_synthesis/noise0/Initializer/random_normal (Add)
G_synthesis/noise0 (VariableV2) /device:GPU:0
G_synthesis/noise0/Assign (Assign) /device:GPU:0

 [[node G_synthesis/noise0/Initializer/random_normal/RandomStandardNormal (defined at ghome/syed/few-shotTF/training/networks_stylegan2.py:660) ]]Additional information about colocations:No node-device colocations were active during op 'G_synthesis/noise0/Initializer/random_normal/RandomStandardNormal' creation.

Device assignments active during op 'G_synthesis/noise0/Initializer/random_normal/RandomStandardNormal' creation:
with tf.device(None): </opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py:1696>
with tf.device(/gpu:0): </ghome/syed/few-shotTF/training/training_loop.py:167>

Original stack trace for 'G_synthesis/noise0/Initializer/random_normal/RandomStandardNormal':
File "ghome/syed/few-shotTF/run_training.py", line 254, in
main()
File "ghome/syed/few-shotTF/run_training.py", line 249, in main
run(**vars(args))
File "ghome/syed/few-shotTF/run_training.py", line 169, in run
dnnlib.submit_run(**kwargs)
File "ghome/syed/few-shotTF/dnnlib/submission/submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "ghome/syed/few-shotTF/dnnlib/submission/internal/local.py", line 22, in submit
return run_wrapper(submit_config)
File "ghome/syed/few-shotTF/dnnlib/submission/submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "ghome/syed/few-shotTF/training/training_loop.py", line 170, in training_loop
G = tflib.Network('G', num_channels=training_set.shape[0], resolution=training_set.shape[1], label_size=training_set.label_size, **G_args)
File "ghome/syed/few-shotTF/dnnlib/tflib/network.py", line 96, in init
self._init_graph()
File "ghome/syed/few-shotTF/dnnlib/tflib/network.py", line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File "ghome/syed/few-shotTF/training/networks_stylegan2.py", line 355, in G_main
components.synthesis = tflib.Network('G_synthesis', func_name=globals()[synthesis_func], **kwargs)
File "ghome/syed/few-shotTF/dnnlib/tflib/network.py", line 96, in init
self._init_graph()
File "ghome/syed/few-shotTF/dnnlib/tflib/network.py", line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File "ghome/syed/few-shotTF/training/networks_stylegan2.py", line 660, in G_synthesis_stylegan2
noise_inputs.append(tf.get_variable('noise%d' % layer_idx, shape=shape, initializer=tf.initializers.random_normal(), trainable=False))
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable
aggregation=aggregation)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable
aggregation=aggregation)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable
aggregation=aggregation)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter
aggregation=aggregation)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 929, in _get_single_variable
aggregation=aggregation)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 259, in call
return cls._variable_v1_call(*args, **kwargs)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
shape=shape)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 198, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2511, in default_variable_creator
shape=shape)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 263, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1568, in init
shape=shape)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1698, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 901, in
partition_info=partition_info)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py", line 323, in call
shape, self.mean, self.stddev, dtype, seed=self.seed)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 79, in random_normal
shape_tensor, dtype, seed=seed1, seed2=seed2)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/gen_random_ops.py", line 728, in random_standard_normal
seed2=seed2, name=name)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()