bilinear upsampling and resizing #9303

ahundt · 2018-02-04T00:23:24Z

This is an implementation of bilinear interpolation for the backend and layers. It introduces the parameter resize_images(interpolation='bilinear') and UpSampling2D(interpolation='bilinear').

All defaults remain interpolation='nearest' for backwards compatibility.

elephantgrass7 · 2018-02-06T03:06:24Z

keras/backend/cntk_backend.py

+            output = repeat_elements(output, width_factor, axis=2)
+            return output
+        else:
+            raise ValueError('CNTK Backend: Invalid data_format:', data_format)


maybe tell the user the valid data_formats

elephantgrass7 · 2018-02-06T03:07:13Z

keras/backend/tensorflow_backend.py

+            x = tf.image.resize_nearest_neighbor(x, new_shape)
+        elif interpolation == 'bilinear':
+            x = tf.image.resize_bilinear(x, new_shape)
+        else:


this value error is good but do in other places

elephantgrass7 · 2018-02-06T03:07:54Z

keras/backend/theano_backend.py

@@ -884,24 +884,42 @@ def repeat_elements(x, rep, axis):
    return y


-def resize_images(x, height_factor, width_factor, data_format):
+def resize_images(x, height_factor, width_factor, data_format, interpolation='nearest'):
    """Resize the images contained in a 4D tensor of shape
    - [batch, channels, height, width] (for 'channels_first' data_format)
    - [batch, height, width, channels] (for 'channels_last' data_format)
    by a factor of (height_factor, width_factor). Both factors should be
    positive integers.
    """
    if data_format == 'channels_first':


add comments to make easier to read

elephantgrass7 · 2018-02-06T03:09:10Z

keras/layers/convolutional.py

@@ -1565,6 +1565,7 @@ class UpSampling2D(Layer):
            It defaults to the `image_data_format` value found in your
            Keras config file at `~/.keras/keras.json`.
            If you never set it, then it will be "channels_last".
+        interpolation: A string, one of `nearest` or `bilinear`.


other doc string stuff has default marked

elephantgrass7 · 2018-02-06T03:09:57Z

keras/backend/cntk_backend.py

+            output = repeat_elements(output, width_factor, axis=2)
+            return output
+        else:
+            raise ValueError('CNTK Backend: Invalid data_format:', data_format)


in the tests k.backend() == cntk might want to lower case here

elephantgrass7 · 2018-02-06T03:11:18Z

tests/keras/layers/convolutional_test.py

+                    data_format=data_format)
+                layer.build(inputs.shape)
+                outputs = layer(K.variable(inputs))
+                np_output = K.eval(outputs)


this tests need to be more rigorous. only checking shape

ozabluda · 2018-02-13T01:53:59Z

This is an implementation of bilinear interpolation for the backend and layers. It introduces the parameter resize_images(interpolation='bilinear') and UpSampling2D(interpolation='bilinear').

All defaults remain interpolation='nearest' for backwards compatibility.

default for UpSampling2D has to remain nearest, but maybe after PR #8849, default for resize_images should be bilinear?

BTW, this seems to be how you do it in CNTK:
https://microsoft.github.io/CNTK-R/reference/transform_scale.html

ahundt · 2018-02-13T04:34:10Z

Can anyone suggest how to fix the theano errors? I'm not really a theano user so I was just implementing my best guess.

Dref360 · 2018-02-13T17:02:01Z

keras/backend/theano_backend.py

+        ratio = height_factor / width_factor
+        th_padding = _preprocess_padding('same')
+        output = theano.tensor.nnet.abstract_conv.bilinear_upsampling(
+            x, ratio=ratio, border_mode=th_padding)


border_mode is not required. The output is of value : (batch size, num_input_channels, input row size * row ratio,
input column size * column ratio)

So fi the ratio is good, everything should be.

bhack · 2018-04-24T10:27:22Z

Any update?

wtreible · 2018-04-26T16:45:20Z

I'm new to this, but it seems pretty important that the resizing function should support not only bilinear interpolation, but also bicubic, at least when using the tf backend. Many of my colleagues who have tried Keras are surprised that it only supports nn interpolation, especially since upsampling is used in almost every hourglass-style network.

Also, volume resizing w/ trilinear interp would help Keras keep up with Torch for 3D hourglass implementations.

bhack · 2018-04-26T16:53:12Z

@wtreible Exactly

ahundt · 2018-04-30T18:23:13Z

I think getting bilinear merged is step one. Other contributors can add new functionality later.

Maybe I can just leave bilinear unsupported and throwing an error on the one backend and just make the unit tests expect that. Then if someone wants it they can add it later.

fchollet · 2018-05-17T18:08:13Z

So, what should we do with this PR? What's the current status? @Dref360 what's your take on it?

Dref360 · 2018-05-17T18:18:20Z

I don't think we should merge it till the Theano version is fixed. The documentation of Theano doesn't match what's happening.

It's a great feature, I just don't think leaving Theano behind TF/CNTK is a good idea.

ahundt · 2018-05-17T18:21:40Z

It works correctly on the tf backend, I'm just not sure how to fix the theano error and I'm a bit tight on time to learn new backends so it has been in a holding pattern.

@Dref360 isn't theano no longer being developed? I feel like it can't possibly stay on par forever without ongoing work in the upstream library.

Dref360 · 2018-05-17T18:45:25Z

It's still being supported for at least 4 months. (https://groups.google.com/forum/#!topic/theano-users/7Poq8BZutbY)

My main concern is that it's the first major feature that Theano won't implement.

Currently, the only test Theano is not able to pass is: test_batchnorm_convnet_no_center_no_scale

Should we stop supporting Theano (ie. drop the CI) before October?
We could merge this PR and advise people to switch to another backend.

If we do so, this PR requires little changes (ie. throws if Theano) and the changes for TF are LGTM

fchollet · 2018-05-17T22:29:27Z

Should we stop supporting Theano (ie. drop the CI) before October?

It's okay if new features we add in Keras aren't supported in Theano. Existing workflows should keep working (and we should keep Theano on CI) for at least 1.5 years from now (Theano EoL + 1 year).

We will gradually drop features when Theano becomes a friction. Whenever it is "cheap" to make things work with Theano, we'll make it work. One recent example is that we dropped recurrent dropout support in Theano because the new implementation couldn't easily be made to work with Theano.

Basically: let's not make Theano a bottleneck of Keras development. But let's not break it when that can be avoided at a low cost.

This PR doesn't change the default behavior, so it would be okay to skip Theano support (won't break existing users).

joelthchao

I have checked why theano backend is failed. Please check my reviews and make it pass theano unit test.

joelthchao · 2018-08-12T09:51:19Z

keras/backend/theano_backend.py

+        output = repeat_elements(x, height_factor, axis=axis_1)
+        output = repeat_elements(output, width_factor, axis=axis_2)
+    elif interpolation == 'bilinear':
+        ratio = height_factor / width_factor


For theano, ratio needs to be integer.

ratio = height_factor // width_factor

how should this be done? Am I correct that the values might be tensors and not something that can be cast to int on every backend?

joelthchao · 2018-08-12T09:51:52Z

keras/backend/theano_backend.py

+            repeat_dim_1 = x._keras_shape[axis_1]
+            repeat_dim_2 = x._keras_shape[axis_2]
+            output._keras_shape[axis_1] = repeat_dim_1 * rep
+            output._keras_shape[axis_2] = repeat_dim_2 * rep


Unresolved referencerep

hmm... it has been a while since I looked at this... what was rep supposed to be?

gabrieldemarmiesse · 2018-08-25T14:34:22Z

I tried to take a stab at it. See #10994.

ahundt · 2018-08-26T03:05:53Z

oh cool just saw #10994, thanks @gabrieldemarmiesse!

gabrieldemarmiesse · 2018-08-30T08:00:35Z

@ahundt can you close the PR now that #10994 is merged?

And thanks again for your work! I didn't have to change it much.

bilinear upsampling initial implementation

0855745

ahundt changed the title ~~bilinear upsampling and resizing initial implementation~~ bilinear upsampling and resizing Feb 4, 2018

ahundt added 3 commits February 3, 2018 22:50

bilinear interpolation docstrings added

4b03e6a

thano_backend.py fix bilinear interpolation ratio

b7c10bd

theano_backend.py pep8 & padding fix

104f209

elephantgrass7 reviewed Feb 6, 2018

View reviewed changes

ahundt closed this Feb 6, 2018

ahundt reopened this Feb 6, 2018

fchollet added the Reviewers wanted label Feb 13, 2018

Dref360 reviewed Feb 13, 2018

View reviewed changes

theano_backend.py remove border_mode as per review

73f0311

bhack mentioned this pull request Jul 24, 2018

Add Support for BilinearUpSampling2D Layer for Keras model tensorflow/tfjs#437

Closed

joelthchao suggested changes Aug 12, 2018

View reviewed changes

gabrieldemarmiesse mentioned this pull request Aug 25, 2018

Added the mode "bilinear" in the upscaling2D layer. #10994

Merged

4 tasks

fchollet closed this Aug 30, 2018

ahundt deleted the bilinear_upsampling branch September 27, 2018 02:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bilinear upsampling and resizing #9303

bilinear upsampling and resizing #9303

ahundt commented Feb 4, 2018

elephantgrass7 Feb 6, 2018

elephantgrass7 Feb 6, 2018

elephantgrass7 Feb 6, 2018

elephantgrass7 Feb 6, 2018

elephantgrass7 Feb 6, 2018

elephantgrass7 Feb 6, 2018

ozabluda commented Feb 13, 2018

ahundt commented Feb 13, 2018

Dref360 Feb 13, 2018

bhack commented Apr 24, 2018

wtreible commented Apr 26, 2018

bhack commented Apr 26, 2018

ahundt commented Apr 30, 2018

fchollet commented May 17, 2018

Dref360 commented May 17, 2018

ahundt commented May 17, 2018

Dref360 commented May 17, 2018

fchollet commented May 17, 2018 •

edited

Loading

joelthchao left a comment

joelthchao Aug 12, 2018

ahundt Aug 26, 2018

joelthchao Aug 12, 2018

ahundt Aug 26, 2018

gabrieldemarmiesse commented Aug 25, 2018

ahundt commented Aug 26, 2018 •

edited

Loading

gabrieldemarmiesse commented Aug 30, 2018

bilinear upsampling and resizing #9303

bilinear upsampling and resizing #9303

Conversation

ahundt commented Feb 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ozabluda commented Feb 13, 2018

ahundt commented Feb 13, 2018

Choose a reason for hiding this comment

bhack commented Apr 24, 2018

wtreible commented Apr 26, 2018

bhack commented Apr 26, 2018

ahundt commented Apr 30, 2018

fchollet commented May 17, 2018

Dref360 commented May 17, 2018

ahundt commented May 17, 2018

Dref360 commented May 17, 2018

fchollet commented May 17, 2018 • edited Loading

joelthchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabrieldemarmiesse commented Aug 25, 2018

ahundt commented Aug 26, 2018 • edited Loading

gabrieldemarmiesse commented Aug 30, 2018

fchollet commented May 17, 2018 •

edited

Loading

ahundt commented Aug 26, 2018 •

edited

Loading