[mesh] Texturing: Multi-band blending #629

cvere · 2019-04-26T17:02:39Z

Description

Largely improve the texturing quality by using multi-band blending technique.

Split each input camera into frequency bands, by calculating its laplacian pyramid. For each output texture file, it computes the texture fusion per frequency bands with different number of contributions. The color averaging is now weighted by a score defined by the triangle reprojection area.
The textures per frequency bands are then combined to create the final output texture files.

Features list

Split an image in frequency bands and reconstitute the image from its frequency bands
Add triangles contributions to a specific frequency band
Improve padding performance (2 image visit instead of 2*paddingSize)

Implementation remarks

Add colorspace option to ImageCache
Improve colorspace option of writeImage (allow change of input colorspace)

Fix #644

Use an Image class instead of vector<Color> and width, height.

Does not change the results yet, but start to use the multiband.

Calculate the laplacian pyramid of an image with successives downscales (instead of successives blurs) -> memory saving

valid operators =, [] & resize without copying elements

calculate difference between an image of size s and an other of size s/2

For each camera, calculate the corresponding laplacian pyramid. For each texture triangle, fuse the contributions of frequencies bands from the best scoring cameras

save all frequency bands for all textures

Let openImageIO choose the size of the convolution kernel when downscaling by two & replace maxNbImagesForFusion by multiBandNbContrib

Create the texture padding with only two passes of the image instead of 2*padding

Add param to control the level of blur between each level of the laplacian pyramid (previously fixed to 2)

Add number of bands param, to control the number of level of the laplacian pyramid

Refine calculus of the number of atlases simultaneously computable

Texture padding computed in 2 passes instead of padding*2 passes

Triangles contribute to atlas textures according to their score (ie. the area of reprojection on the camera) -> less artefacts, especially on low frequencies band

Create new file to test multi-band blending on one image

Add colorspace attribute to ImagesCache and read images in that colorspace (LINEAR for depthMap, SRGB for texturing)

New colorspace type (AUTO_FROM_SRGB) -> calculate texture atlases in sRGB for multi-band blending accuracy

readImage kept to much space in memory

Bool to choose to use scores + scores in float type + simple rename

texParams.multibandNbContrib holds contributions to each frequency band, not the sum of contributions to all higher frequency bands as before

Delete MultiBandBlending class & calculate laplacian pyramid directly in the image class. Delete useless functions and add comments [mvsData] Image: some cleaning + split into .hpp / .cpp Create a .cpp file to write the implementation of elaborate functions

Simplify imageIO management in MVS by having only one imageIO file. Override imageIO methods to take directly mvsData::Image class in argument (instead of width, height and buffer)

Simplify image management by having only one image class. [mvsData] Image: refactoring & add simple methods [mesh] Texturing: use Image class for AccuImage structure (instead of vector<Color>) + use imageIO methods taking an Image in argument [mvsUtils] fileIO: adapt the method for loading images to use mvsData::Image

fabiencastan · 2019-06-06T16:50:11Z

src/aliceVision/mesh/Texturing.cpp

@@ -451,7 +451,7 @@ void Texturing::generateTexturesSubSet(const mvsUtils::MultiViewParams& mp,
                if(contrib + 1 == nbContribLevel)


Simpler and avoid to access outside of the array.

if(contrib + 1 == texParams.multiBandNbContrib[band]) { ++band; }

So we could remove the variable nbContribLevel.

fabiencastan · 2019-06-06T16:52:34Z

src/aliceVision/mesh/Texturing.cpp

@@ -451,7 +451,7 @@ void Texturing::generateTexturesSubSet(const mvsUtils::MultiViewParams& mp,
                if(contrib + 1 == nbContribLevel)
                {


Few lines above, we have:
int nbContribMax = std::min(texParams.multiBandNbContrib.back(), static_cast<int>(scorePerCamId.size()));

I think yes, but just to be sure: Does texParams.multiBandNbContrib still contains cumulative values in the latest version?

fabiencastan · 2019-06-06T17:03:40Z

src/aliceVision/mesh/Texturing.cpp

@@ -259,17 +259,22 @@ void Texturing::generateTextures(const mvsUtils::MultiViewParams &mp,
                                 const boost::filesystem::path &outPath, EImageFileType textureFileType)
 {
    ALICEVISION_LOG_INFO("Texturing: Use multiband blending with the following contributions per band:");
+


{ // Ensure that contribution levels do not contain 0 and are sorted (as they represent cumulative values). auto& m = texParams.multiBandNbContrib; m.erase(std::remove(std::begin(m), std::end(m), 0), std::end(m)); texParams.nbBand = m.size(); if(!std::is_sorted(std::begin(m), std::end(m))) { ALICEVISION_LOG_INFO("Sorting contributions per band (necessary)."); std::sort(std::begin(m), std::end(m)); } }

fabiencastan · 2019-06-06T17:07:20Z

src/samples/CMakeLists.txt

@@ -10,7 +10,6 @@ add_subdirectory(featuresRepeatability)
 # add_subdirectory(imageData)
 add_subdirectory(imageDescriberMatches)
 add_subdirectory(kvldFilter)
-add_subdirectory(multiBandBlending)


Looks like a mistake as the sample itself is not removed...

fabiencastan · 2019-06-06T17:15:10Z

src/aliceVision/CMakeLists.txt

@@ -38,7 +38,6 @@ endif()
 # MVS modules

 if(ALICEVISION_BUILD_MVS)
-  add_subdirectory(imageIO)


We tried to keep a split between IO and data structures as almost all modules have dependencies to data structures.
So if we don't split it, all modules will have dependencies to IO.

But maybe this commit is a needed intermediate solution before a new change? If not, we do not want to merge mvsData and imageIO.

We have a several files related to image IO:

image/io.hpp // SfM: functions to read/writes images

imageIO/image.hpp // MVS: functions to read/write images, relies on mvsData for data structure types (rgb, Color and now Image)

mvsData/imageIO.hpp (image.hpp on develop) // MVS: containing EImageFileType enum

The idea was to start simplifying things. We will probably want to merge image types and IO at some point, but it already seems weird to have, for the MVS part, this split between imageIO/image.hpp and mvsData/imageIO.hpp. And everything that uses mvsData also uses imageIO, so the merge does not seem that problematic IMHO. But if we want to keep this imageIO module as the one that will stay, maybe it could be the other way around: move what's inside mvsData/imageIO.hpp into the imageIO module.

fabiencastan · 2019-06-06T17:18:46Z

src/aliceVision/mesh/Texturing.cpp

-        //conversion ImagesCache::Img -> Image
-        mvsUtils::ImagesCache::ImgPtr imgPtr = imageCache.getImg_sync(camId);
-        Image camImg(imgPtr->data, imgPtr->getWidth(), imgPtr->getHeight());
+        const Image& camImg = *(imageCache.getImg_sync(camId));


getImg_sync returns an ImgPtr which is a std::shared_ptr<Img>.
If you remove the temporary variable and directly get the value from the shared_ptr, the shared_ptr ownership is removed.

fabiencastan · 2019-06-06T17:20:11Z

src/aliceVision/mesh/Texturing.hpp

@@ -152,24 +152,24 @@ struct Texturing
    // Create buffer for the set of output textures
    struct AccuImage
    {
-        std::vector<Color> img;
+        Image img;


fabiencastan · 2019-06-06T17:21:17Z

src/aliceVision/mvsUtils/ImagesCache.hpp

@@ -21,48 +22,11 @@ namespace mvsUtils {

 class ImagesCache
 {
-public:
-    class Img


fabiencastan · 2019-06-12T15:52:19Z

src/aliceVision/mvsUtils/MultiViewParams.cpp

+    const camera::IntrinsicBase* intrinsicPtr = _sfmData.getIntrinsicPtr(view.getIntrinsicId());
+
+    Vec2 pix_disto = intrinsicPtr->get_d_pixel({pixRC.x, pixRC.y});
+    return isPixelInImage(Pixel(pix_disto.x(), pix_disto.y()), camId, margin);


All the MVS structures have a global downscale (backed in all matrices, etc.) but it's not the case for the sfmData which contains the original SfM.
So we need to compensate for that:

const double s = getDownscaleFactor(camId); const Vec2 pix_disto = intrinsicPtr->get_d_pixel({pixRC.x * s, pixRC.y * s}) / s;

fabiencastan · 2019-06-12T16:19:43Z

src/aliceVision/mesh/Texturing.cpp

-        imageCache.refreshData(camId);        
-        const Image& camImg = *(imageCache.getImg_sync(camId));
+        imageCache.refreshData(camId);
+        mvsUtils::ImagesCache::ImgPtr imgPtr = imageCache.getImg_sync(camId);


mvsUtils::ImagesCache::ImgPtr imgPtr = imageCache.getImg_sync(camId); // keep ownership

It would be good to rename ImgPtr into ImgSharedPtr.

When texturing: check if mesh triangles reproject into both source and undisto images. [mvsUtils] MultiViewParams: new method to project a triangle into source image (= image with distortion) [mesh] Mesh: adapt triangle reprojection functions

Update nbContribMax (= max number of contribution for a camera): number of contributions used to be cumulative but not anymore (more practical for the user)

Incorrect use of shared_ptr: if you remove the temporary variable and directly get the value from the shared_ptr, the shared_ptr ownership is removed.

code cleaning + setting multiBandDownscale default value to 4 + renaming ImagesCache::ImgPtr to ImagesCache::ImgSharedPtr

only keep triangles with sufficient reprojection area

simogasp · 2019-06-13T16:23:05Z

src/aliceVision/mvsData/imageIO.hpp

+        else
+            to = to_;
+    }
+    OutputFileColorSpace()


simogasp · 2019-06-13T16:23:44Z

src/aliceVision/mvsData/imageIO.cpp

@@ -294,29 +379,34 @@ void writeImage(const std::string& path,
    fs::rename(tmpPath, path);
 }

-void writeImage(const std::string& path, int width, int height, const std::vector<unsigned char>& buffer, EImageQuality imageQuality, EImageColorSpace imageColorSpace, const oiio::ParamValueList& metadata)
+void writeImage(const std::string& path, int width, int height, const std::vector<unsigned char>& buffer, EImageQuality imageQuality, OutputFileColorSpace colorspace, const oiio::ParamValueList& metadata)


const OutputFileColorSpace& colorspace

simogasp · 2019-06-13T16:24:31Z

src/aliceVision/mvsData/imageIO.cpp

@@ -238,7 +323,7 @@ void writeImage(const std::string& path,
                int nchannels,
                const std::vector<T>& buffer,
                EImageQuality imageQuality,
-                EImageColorSpace imageColorSpace,
+                OutputFileColorSpace colorspace,


const OutputFileColorSpace& colorspace

simogasp · 2019-06-13T16:25:34Z

src/aliceVision/mvsData/imageIO.hpp

+
+struct OutputFileColorSpace
+{
+    EImageColorSpace from = EImageColorSpace::LINEAR;


EImageColorSpace from{EImageColorSpace::LINEAR};
EImageColorSpace to{EImageColorSpace::AUTO};

simogasp · 2019-06-13T16:26:26Z

src/aliceVision/mvsUtils/ImagesCache.cpp

+    mapIdCamId.resize( N_PRELOADED_IMAGES, -1 );
+    mapIdClock.resize( N_PRELOADED_IMAGES, clock() );
+}
+
 ImagesCache::~ImagesCache()


fabiencastan · 2019-06-20T11:38:24Z

LGTM!

cvere added 30 commits March 22, 2019 11:12

[mvsData] Create Image class

9152f4b

Use an Image class instead of vector<Color> and width, height.

[mesh] Texturing: add missing reference

5dfaf62

[mesh] texturing: create laplacian pyramid and recompose the image

84ee7ba

Does not change the results yet, but start to use the multiband.

[mvsUtils] ImageCache : function to setCacheSize

cf02002

[mesh] Texturing : create laplacian pyramid from downscaled images

885d38d

Calculate the laplacian pyramid of an image with successives downscales (instead of successives blurs) -> memory saving

[mvsData] Image : Color interpolation valid on borders

6ea3306

[mvsData] Image : operators

f4e9ebb

valid operators =, [] & resize without copying elements

[mvsData] Image : difference between different sizes images

b124297

calculate difference between an image of size s and an other of size s/2

[mesh] Texturing : new function to write a texture + fix padding

94ac053

[mesh] Texturing : memory management

d7b38d1

[mesh] Texturing : multiBandBlending

9160841

For each camera, calculate the corresponding laplacian pyramid. For each texture triangle, fuse the contributions of frequencies bands from the best scoring cameras

[mesh] Texturing : debug mode

553e178

save all frequency bands for all textures

[mesh] Texturing : delete sizeKernel & maxNbImagesForFusion

4e3383d

Let openImageIO choose the size of the convolution kernel when downscaling by two & replace maxNbImagesForFusion by multiBandNbContrib

[mesh] Texturing : optimize padding

95f373d

Create the texture padding with only two passes of the image instead of 2*padding

[mesh] Texturing: add downscale param for multiband blending

7dc8eb6

Add param to control the level of blur between each level of the laplacian pyramid (previously fixed to 2)

[mesh] Texturing: add nbBand param for multiband blending

f41101e

Add number of bands param, to control the number of level of the laplacian pyramid

[mesh] Texturing: keep margin in memory use

35d33a7

Refine calculus of the number of atlases simultaneously computable

[mesh] Texturing : optimize texture padding

fa0f07c

Texture padding computed in 2 passes instead of padding*2 passes

[mvsData] Image: handle invalid pixels in color interpolation

d2f29dd

Merge remote-tracking branch 'origin/develop' into dev_multiBandBlending

77b549d

[mesh] Texturing : error fix

8e6b830

[mesh] Texturing : final memory optimization

19dac6f

[mesh] Texturing: taking triangles scores into account

770499e

Triangles contribute to atlas textures according to their score (ie. the area of reprojection on the camera) -> less artefacts, especially on low frequencies band

[utils] New main to test multi-band blending

000aba7

Create new file to test multi-band blending on one image

[mvsUtils] load images from cache in a chosen colorspace

22096c7

Add colorspace attribute to ImagesCache and read images in that colorspace (LINEAR for depthMap, SRGB for texturing)

[mesh] texturing: write textures from sRGB colorspace

9a85cd0

New colorspace type (AUTO_FROM_SRGB) -> calculate texture atlases in sRGB for multi-band blending accuracy

[imageIO] image: fix error

c7929e8

readImage kept to much space in memory

[mvsData] Image: fix color interpolation and resize functions

e841b64

[mesh] texturing: taking triangles scores into account

70f7a75

Bool to choose to use scores + scores in float type + simple rename

[mesh] Texturing: fix number of contributions for multi-band blending

3202d80

texParams.multibandNbContrib holds contributions to each frequency band, not the sum of contributions to all higher frequency bands as before

cvere added 3 commits June 4, 2019 15:36

Merge imageIO module into mvsData/imageIO

c4d3678

Simplify imageIO management in MVS by having only one imageIO file. Override imageIO methods to take directly mvsData::Image class in argument (instead of width, height and buffer)

fabiencastan added this to the 2019.2 milestone Jun 5, 2019

fabiencastan reviewed Jun 6, 2019

View reviewed changes

src/aliceVision/mvsUtils/ImagesCache.hpp

@@ -21,48 +22,11 @@ namespace mvsUtils {

class ImagesCache

{

public:

class Img

Copy link

Member

fabiencastan Jun 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

fabiencastan mentioned this pull request Jun 8, 2019

BUG: incorrect index calculation for "bottom-right to up-left" scan in void Texturing::generateTexturesSubSet(...) #644

Closed

fabiencastan reviewed Jun 12, 2019

View reviewed changes

cvere added 5 commits June 13, 2019 15:31

[mesh] Texturing: fix nbContribMax calculus

1531052

Update nbContribMax (= max number of contribution for a camera): number of contributions used to be cumulative but not anymore (more practical for the user)

[mesh] Texturing: bug fix on a shared_ptr

850ca7a

Incorrect use of shared_ptr: if you remove the temporary variable and directly get the value from the shared_ptr, the shared_ptr ownership is removed.

[mesh] Texturing: code cleaning & setting param default value

118ed54

code cleaning + setting multiBandDownscale default value to 4 + renaming ImagesCache::ImgPtr to ImagesCache::ImgSharedPtr

[mesh] Texturing: set bestScoreThreshold default value to 0.1

7fdc82b

only keep triangles with sufficient reprojection area

cvere force-pushed the dev_multiBandBlending branch from 846158b to 7fdc82b Compare June 13, 2019 14:19

simogasp requested changes Jun 13, 2019

View reviewed changes

cvere added 3 commits June 14, 2019 14:37

code cleaning on struct OutputFileColorSpace

d794c42

Merge remote-tracking branch 'origin/develop' into dev_multiBandBlending

8bc2e79

Explicit constructor for OutputFileColorSpace struct & missing reference

9f189bc

fabiencastan approved these changes Jun 20, 2019

View reviewed changes

Merge branch 'develop' into dev_multiBandBlending

956df2a

fabiencastan merged commit c43f68c into develop Jun 20, 2019

fabiencastan deleted the dev_multiBandBlending branch June 20, 2019 19:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mesh] Texturing: Multi-band blending #629

[mesh] Texturing: Multi-band blending #629

cvere commented Apr 26, 2019 •

edited by fabiencastan

Loading

fabiencastan Jun 6, 2019

fabiencastan Jun 6, 2019

fabiencastan Jun 6, 2019

fabiencastan Jun 6, 2019

fabiencastan Jun 6, 2019

yann-lty Jun 7, 2019

fabiencastan Jun 6, 2019

fabiencastan Jun 6, 2019

fabiencastan Jun 6, 2019

fabiencastan Jun 12, 2019

fabiencastan Jun 12, 2019

fabiencastan Jun 12, 2019

simogasp Jun 13, 2019

simogasp Jun 13, 2019

simogasp Jun 13, 2019

simogasp Jun 13, 2019

simogasp Jun 13, 2019

fabiencastan commented Jun 20, 2019

		@@ -451,7 +451,7 @@ void Texturing::generateTexturesSubSet(const mvsUtils::MultiViewParams& mp,
		if(contrib + 1 == nbContribLevel)

[mesh] Texturing: Multi-band blending #629

[mesh] Texturing: Multi-band blending #629

Conversation

cvere commented Apr 26, 2019 • edited by fabiencastan Loading

Description

Features list

Implementation remarks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabiencastan commented Jun 20, 2019

cvere commented Apr 26, 2019 •

edited by fabiencastan

Loading