what are the outputs?

[IQ17]

Hi, thanks for the code!
I just started to learn this paper with your code.
As far as I understand, there are two neural networks, and I trained them jointly using the train.py, with 410 basketball images.

When I run render.py with the trained model, there are something I don't understand on the outputted image. What are they and how can I erase them?

Thanks!

[SSRSGJYD]

I have been faced with similar situations where there were artifacts. It may be helpful to use more training images and adjust hyperparameters. You can also try training seperately.

[IQ17]

After adjusting several things, the quality of the image improved considerably. see image, yet the artefact remained.

The artefact seems to be a result of something like a RGB grid mis-alignment.
I thought it was caused by F.interpolate and setted align_corners=True, but no help.

Or it may be caused by saturation?

[oKatanaaa]

After adjusting several things, the quality of the image improved considerably. see image, yet the artefact remained.

The artefact seems to be a result of something like a RGB grid mis-alignment.
I thought it was caused by F.interpolate and setted align_corners=True, but no help.

Or it may be caused by saturation?

Probably it is caused by the transposed convolution which causes a grid-like pattern on the output tensor.
Replacing all the transposed convolutions with conv+resize except for the last layer may alleviate this issue. It is preferable to use nearest neighbour interpolation, but others may work as well.

[wuxiaolianggit]

can you solve the problem?
When I run render.py with the trained model,there is a problems?
TypeError: Traceback (most recent call last):
File "/devdata/gutai_2t/liao/auto_clone_206/Anaconda3/envs/wuxl/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/devdata/gutai_2t/wuxl/NeuralTexture/dataset/eval_dataset.py", line 57, in
collect_fn = lambda x: EvalDataset._collect_fn(x, view_direction)
File "/devdata/gutai_2t/wuxl/NeuralTexture/dataset/eval_dataset.py", line 46, in _collect_fn
extrinsics = torch.stack(tuple(extrinsics), dim=0)
TypeError: expected Tensor as element 0 in argument 0, but got numpy.ndarray

@ @SSRSGJYD @IQ17

[SSRSGJYD]

You can first change variable extrinsics from numpy.ndarray to torch.Tensor.

[wuxiaolianggit]

extrinsics is a tuple, I modified extrinsics = torch.stack(tuple(extrinsics), dim=0) to extrinsics = torch.stack(tuple(torch.from_numpy（extrinsics）), dim=0)， but still not。 @SSRSGJYD

[SSRSGJYD]

You can just write:

extrinsics = torch.FloatTensor(extrinsics)

[wuxiaolianggit]

Thank you very much for your help, now it works @SSRSGJYD

[ChenFengYe]

@IQ17 Hi, I see your results with this neural renderer. I am quite curious about what this neural texture map looks like. I mean the texture map (in RGB space with parameterized UV space) not the rendered image. Could you please provide this neural texture data or figure to me? Quite thanks! I even can not find these results in the paper of deferred neural rendering. or could you provide some of this neural texture?@SSRSGJYD Thanks again!

[oKatanaaa]

@ChenFengYe Hi, here some of my results of training the neural renderer. These are slices of the neural texture converted to an RGB image. The first 3 'layers' (first image) of the neural texture were forced to learn RGB values and therefore they look similar to a common RGB texture.

[ChenFengYe]

@oKatanaaa Thanks for these inspiring results! Btw, how could you force the first 3 channels for RGB? It is quite similar to my current work.

[oKatanaaa]

@ChenFengYe Sorry for the late response. When the texture is being sampled, the first three channels are forced to learn RGB values via minimizing L2 loss between these sampled channels and the image. I will try to illustrate that with a pseudo code:
text - the neural texture;
img - image we are trying to render;
model - the neural renderer;
sampler - function that samples the texture according to UV map;
uv - the UV map;
optimizer - the object that performs the loss minimization.

Pseudo code:
x = sampler(uv, text) # has a shape of [image height, image width, channels]
rgb_layers = x[:, :, :3] # take first three channels
text_l2_loss = (rgb_layers - img)**2 # create a loss to force the texture to learn rgb values

rendered_image = model.predict(x) # note that the first three channels are also being trained with the network
render_l2_loss = (rendered_image - img)**2

total_l2_loss = text_l2_loss + render_l2_loss # construct the total loss
optimizer.minimize(total_l2_loss)

[ChenFengYe]

@oKatanaaa Thanks for your sharing. This code is very helpful!

[hzhao1997]

@ChenFengYe Hi, here some of my results of training the neural renderer. These are slices of the neural texture converted to an RGB image. The first 3 'layers' (first image) of the neural texture were forced to learn RGB values and therefore they look similar to a common RGB texture.

Hi, I try to visualize the neural texture, but it looks like this. I think I run into trouble but I can not deal with it. Could you share your visualization code? Thanks~

And this is the output from the trained model:

[szulm]

Hi, thanks for the code!
I just started to learn this paper with your code.
As far as I understand, there are two neural networks, and I trained them jointly using the train.py, with 410 basketball images.

When I run render.py with the trained model, there are something I don't understand on the outputted image. What are they and how can I erase them?

Thanks!

hi,I have the same question.What is your final solution?

[davidvfx07]

Hi, thanks for the code!
I just started to learn this paper with your code.
As far as I understand, there are two neural networks, and I trained them jointly using the train.py, with 410 basketball images.
When I run render.py with the trained model, there are something I don't understand on the outputted image. What are they and how can I erase them?
Thanks!

hi,I have the same question.What is your final solution?

Same here.

[QinjieXiao]

After adjusting several things, the quality of the image improved considerably. see image, yet the artefact remained.
The artefact seems to be a result of something like a RGB grid mis-alignment.
I thought it was caused by F.interpolate and setted align_corners=True, but no help.
Or it may be caused by saturation?

Probably it is caused by the transposed convolution which causes a grid-like pattern on the output tensor. Replacing all the transposed convolutions with conv+resize except for the last layer may alleviate this issue. It is preferable to use nearest neighbour interpolation, but others may work as well.

@oKatanaaa I modified the up layer as follows,

class up(nn.Module):
def init(self, in_ch, out_ch, output_pad=0, concat=True, final=False):
super(up, self).init()
self.concat = concat
self.final = final
if self.final:
self.conv = nn.Sequential(
nn.ConvTranspose2d(in_ch, out_ch, 4, stride=2, padding=1, output_padding=output_pad),
nn.InstanceNorm2d(out_ch),
nn.Tanh()
)
else:
self.conv = nn.Sequential(
nn.UpsamplingNearest2d(scale_factor=2),
nn.Conv2d(in_ch, out_ch, kernel_size=3, stride=1, padding=1),
nn.InstanceNorm2d(out_ch),
nn.LeakyReLU(0.2, inplace=True)
)
but the results still had grid-like artifacts, and light points artifacts, the predicted images are a little bit darker than the test images:

I'm wondering if you might have any suggestions for resolving this issue?