MPI fixes + smoother builtin #931

rhodrin · 2019-09-09T17:32:52Z

This PR:

Includes some bug fixes associated with MPI data slicing along with the relevant tests.
A gaussian smoother builtin for Devito functions that mimics the scipy.ndimage gaussian_filter.

Note: Some changes and tidying are still required.

devito/builtins.py

FabioLuporini · 2019-09-10T12:01:27Z

devito/builtins.py

+    """
+    Gaussian smooth function.
+    """
+    class DataDomain(dv.SubDomain):


Is it worth moving this inside Devito?
BTW, I wouldn't call it "DataDomain" -- all subdomains are data domains after all ?

Is it worth moving this inside Devito?

Not at the moment I don't think - if we're going to have some 'pre-set' subdomain methods it's probably better we introduce them after a little more thought. Or is that not what you meant?

BTW, I wouldn't call it "DataDomain" -- all subdomains are data domains after all ?

Yeah, sure. I think ObjectiveDomain is a better name.

mloubout · 2019-09-10T12:20:43Z

devito/builtins.py

+        def define(self, dimensions):
+            return {d: ('middle', self.lw, self.lw) for d in dimensions}
+
+    def fill_data(g, f, dim, lw, mode):


isn't this basically model/initialize_function? Would a be lot shorter/more readable to reuse it IMO

Will take a look, but I would have thought model/initialize_function can't do the 'reflect' conditions?

I didn't put it because didn't need it but it's just a small change in the rhs of the equation. Currently is Eq(u(d), u(npml) can just do Eq(u(d), u(nbpm+d) or something like that and you got the reflect

Oki, will check it out.

is it worth promoting initialize_function to a builtin ?

mloubout · 2019-09-10T12:21:12Z

devito/builtins.py

+
+        return
+
+    def subfloor(f, g):


Probably don't need this with the right sizes + Operator rather than python fill_data

Yeah, probably, will test.

mloubout · 2019-09-10T12:22:34Z

devito/builtins.py

+    weights = np.exp(-0.5/sigma**2*(np.linspace(-lw, lw, 2*lw+1))**2)
+    weights = weights/weights.sum()
+
+    for d in dims:


Can make it in one go with a 3D kernel from outer product of 1D kernel and avoid the loop as 3D smooth is
W[i,j,k] = w[i]*w[j]*w[k] and need one 3d conv rather than multiple 1D ones with padding.

Sure, but then the stencil is a square/cube (2d/3d). I think the symbolic coefficients stuff may need an update to get that to work, will double check. (It should of course be done that way, and will be at some point, but I don't think it'll work immediately which is why it's currently like this).

Yes but you can just do sum w[i]*f.subs(...) rather than symbolic coeefs and do just all dims at once

codecov-io · 2019-09-10T20:00:11Z

Codecov Report

Merging #931 into master will increase coverage by <.01%.
The diff coverage is 91.82%.

@@            Coverage Diff            @@
##           master    #931      +/-   ##
=========================================
+ Coverage   90.09%   90.1%   +<.01%     
=========================================
  Files         148     149       +1     
  Lines       19815   19991     +176     
  Branches     3080    3111      +31     
=========================================
+ Hits        17852   18012     +160     
- Misses       1596    1611      +15     
- Partials      367     368       +1

Impacted Files	Coverage Δ
devito/types/dense.py	`92.95% <ø> (-0.02%)`	⬇️
devito/types/grid.py	`93.12% <100%> (-0.04%)`	⬇️
devito/data/data.py	`96.93% <100%> (+0.01%)`	⬆️
tests/test_builtins.py	`100% <100%> (ø)`
devito/types/dimension.py	`93.64% <100%> (+0.01%)`	⬆️
devito/data/utils.py	`92.57% <100%> (+1.52%)`	⬆️
tests/test_data.py	`97.5% <100%> (+0.06%)`	⬆️
devito/builtins.py	`72.18% <80.68%> (+8.77%)`	⬆️
devito/mpi/distributed.py	`91.37% <0%> (+0.43%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3911117...10ff2d6. Read the comment docs.

FabioLuporini · 2019-09-11T09:01:05Z

devito/data/data.py

@@ -176,7 +176,9 @@ def __getitem__(self, glb_idx, comm_type):
        if comm_type is index_by_index:
            # Retrieve the pertinent local data prior to mpi send/receive operations
            data_idx = loc_data_idx(loc_idx)
+            self._index_stash = flip_idx(glb_idx, self._decomposition)


Probably time to introduce a context manager

devito/data/utils.py

FabioLuporini · 2019-09-11T09:02:54Z

devito/data/utils.py

+    return as_tuple(processed)
+
+
+def distributed_data_size(shape, coords, topology, nprocs):


just to be sure: does this name mean "compute_distributed_data_size" ?

It's to compute the 'cumulative distrusted data size' (could maybe call it cum_dist_dat_size? Anyway, alternate name suggestions are welcome!). Quick example:
Rank 0-3 dat shapes are (2, 2) and grid is a square.
This function will return:

[[(2, 2), (2, 4)], [(4, 2), (4, 4)]]

FabioLuporini · 2019-09-19T06:48:23Z

devito/builtins.py

+                                         thickness=lw)
+            dim_r = dv.SubDimension.right(name='abc_%s_r' % d.name, parent=d,
+                                          thickness=lw)
+            if mode == 'constant':


aside from the to_copy variables, the construction of the eqs is pretty much identical in the two cases (constant, reflect). So we can probably trim down the bodies of this if-else ?

Yes, can do.

FabioLuporini · 2019-09-19T06:50:31Z

devito/builtins.py

+    return f
+
+
+def initialize_function(function, data, nbpml, mode='constant'):


Maybe this should be squashed with the assign builtin so as to create something unique and sufficiently general for all our needs ? otherwise, there's an overlap between these builtins...although their names are totally different.

Yeah, should probably do something along those lines.

FabioLuporini · 2019-09-19T06:52:59Z

devito/data/utils.py

+
+    Parameters
+    ----------
+    idx: Tuple of slices/ints/tuples


numpydoc nitpicking:

idx : tuple of slices/ints/tuples ...

Yeah, still need to fix a few docstrings.

FabioLuporini · 2019-09-19T06:54:09Z

devito/data/utils.py

+    """
+    Compute the cumulative shape of the distributed data (cshape).
+
+    coords: Array containing the local shape of data to each rank.


use numpydoc?

Parameters ----------- ...

devito/data/utils.py

devito/types/grid.py

devito/types/dense.py

devito/builtins.py

FabioLuporini · 2019-10-16T06:43:20Z

just noticed that basically this is a single commit with loads of changes in it 😬

rhodrin · 2019-10-16T14:34:32Z

just noticed that basically this is a single commit with loads of changes in it

Oh. I've messed up a rebase somewhere. Will take a look.

FabioLuporini · 2019-10-16T16:14:22Z

devito/builtins.py



-def assign(f, v=0):
+def assign(f, RHS=0, options=None, name='asign', **kwargs):


typo (asign)

FabioLuporini · 2019-10-16T16:14:59Z

devito/builtins.py

    """
-    Assign a value to a Function.
+    Assign a list of RHS's to a list of Function's.


nitpicking: we don't use apostrophes for plurals such as Functions

perhaps it's now worth adding a couple examples?

Yes, there will be + quite a few tests.

FabioLuporini · 2019-10-16T16:15:12Z

devito/builtins.py

        The left-hand side of the assignment.
-    v : scalar, optional
+    RHS : expression or list of expression's, optional


expression -> expr-like

FabioLuporini · 2019-10-16T16:15:41Z

devito/builtins.py

    """
-    dv.Operator(dv.Eq(f, v), name='assign')()
+    if not isinstance(f, list):


does it work with

f = as_tuple(f)

?

or use the new as_list ?

FabioLuporini · 2019-10-16T16:18:08Z

devito/builtins.py

+        rhs.append(function.subs({d: subsr}))
+        options.extend([None, None])
+
+        if bool(additional_expressions):


pythonicity : no need for bool

FabioLuporini · 2019-10-16T16:18:17Z

devito/builtins.py

+    if not isinstance(RHS, list):
+        RHS = [RHS, ]*len(f)
+    eqs = []
+    if bool(options):


pythonicity : no need for bool

FabioLuporini · 2019-10-16T16:20:22Z

devito/builtins.py

+        expr['options'] = options
+        additional_expressions[d] = expr
+
+    if isinstance(f, dv.Function):


perhaps this if-else should be inside initialize-function ? ie, it's up to initialize-function to do the right thing based on whether the second argument is of type numpy.ndarray or Function?

FabioLuporini · 2019-10-16T16:21:26Z

devito/builtins.py

+    rhs = []
+    options = []
+
+    for d in function.dimensions:


@mloubout
Note that this won't work with TimeFunctions

perhaps we should use space_dimensions instead of dimensions ?

or am I missing something?

FabioLuporini

I've just added some more nitpicking, but I understand this is pretty much ready to go ?

@mloubout can you give a final look?

FabioLuporini · 2019-10-21T06:04:56Z

devito/builtins.py


    Parameters
    ----------
-    f : Function
+    f : Function or list of Function's


nitpicking: Functions

FabioLuporini · 2019-10-21T06:06:00Z

devito/builtins.py



-def assign(f, v=0):
+def assign(f, RHS=0, options=None, name='assign', **kwargs):


'rhs', not 'RHS': not sure it should be all capital letters

FabioLuporini · 2019-10-21T06:08:57Z

devito/builtins.py

+
+    additional_expressions = {}
+    for d in f_c.dimensions:
+        expr = {}


nitpicking: u probably don't need this one (see below)

FabioLuporini · 2019-10-21T06:09:32Z

devito/builtins.py

+        expr['lhs'] = lhs
+        expr['rhs'] = rhs
+        expr['options'] = options
+        additional_expressions[d] = expr


additional_expressions[d] = {'lhs': lhs, 'rhs': rhs, 'options': options}

FabioLuporini · 2019-10-21T06:11:20Z

devito/builtins.py

+
+        b = [min(l) for l in (w for w in (buff(i, j) for i, j in zip(local_size, halo)))]
+        if any(np.array(b) < 0):
+            raise ValueError("Function's halo is not sufficiently thick.")


Best if we say which Function

"Function `%s` ... " % function

devito/builtins.py

FabioLuporini · 2019-10-21T06:24:59Z

devito/builtins.py

+    weights = np.exp(-0.5/sigma**2*(np.linspace(-lw, lw, 2*lw+1))**2)
+    weights = weights/weights.sum()
+
+    additional_expressions = {}


nitpicking: for this sort of things, throughout devito we use the conventional name "mapper"

FabioLuporini · 2019-10-21T06:25:58Z

devito/builtins.py

+    if mode == 'reflect' and function.grid.distributor.is_parallel:
+        # Check that HALO size is appropriate
+        halo = function.halo
+        local_size = function.grid.distributor.shape


why not just function.shape ?

rhodrin · 2019-10-21T15:55:54Z

I've just added some more nitpicking, but I understand this is pretty much ready to go ?

Yes, should be.
Will address the comments above with a new commit shortly.

…with negative step. devito/data/data.py: Update slice step normalisation, devito/data/decomposition.py: Modify the manner in which negative slices are dealt with. test_data.py: Fix 'test_advanced_indexing' hopefully. Fix test. Some not final changes to ensure we now get tests to pass. tests/test_data.py: Move slice tests with negative step to their own 'tests' and skip them if yask enabled. Remove unused import. test_data.py: Seeing what works and what doesn't. data.py: Breaking for testing purposes. Example update. Edit notebook. data.py; decomposition.py: Some fixes. f.data[::-1, ::-1] ordering still broken. Start clean up. data.py, test_data.py: Enable slicing with step < 0 when MPI is enabled and add test for this functionality. data.py: Modifications to allow slicing with MPI. Code is in a mess but working for basic operations. Still need to fix for some advanced slicing. data.py: Start updating __getitem__. Still not working as intended. data.py: tweaking. Fix an error. Misc. fixes. Breaking it. data.py: More fixes. Start tidying. Fixes + tidying. Testing. This is painful. data.py: A mess, but progress. Many tests passing now, lots of tidying and simplifying to do. data.py, test_data.py: Some updates + adding more tests. flake8. decomposition.py, data.py: Add fixes for step sizes > 1. Currently incomplete and some cases will still fail. data.py, decomposition.py: Fixes for negative slicing with step < -1. decomposition.py: Fix formatting errror. test_data.py: Add tests with step sizes > 1. data.py, decomposition.py: __setitem__ fix for when (with mpi_slicing) the dim of val is 1. Data: Fix for cases with dimension > 2. Add more tests. Fix. test_data.py: Add tests for when dim(slice) < dim(Data). data.py: Some tidying. data.py: Start moving code to new methods. data.py: Start moving code to new methods. data: Further tidying + trimming. data: Minor tidying + add one new test. Bug testing. data: Start adding support for mpi slicing with step size > 2 + appropriate tests. One important test still broken owing to bug. flake8 data: Working on final test. data: Fix bug and add another test. Most tests in place now, lots of tidying + re-structuring still required. test_data: Update test_convert_index + add local to global tests. Add notebook before rebase to my other branch. decomposition.py: Fix bug for negative step, test_data.py: Add tests with negative step. flake8. devito/data/data.py: Update slice step normalisation, devito/data/decomposition.py: Modify the manner in which negative slices are dealt with. Tidying (no important changes). test_data.py: Fix 'test_advanced_indexing' hopefully. Attempt to fix test. Some not final changes to ensure we now get tests to pass. tests/test_data.py: Move slice tests with negative step to their own 'tests' and skip them if yask enabled. Remove unused import. test_data.py: Seeing what works and what doesn't. data.py: Breaking for testing purposes. Example update. Edit notebook. Testing. data.py; decomposition.py: Some fixes. f.data[::-1, ::-1] ordering still broken. Start clean up. test_data.py: Remove test that's no longer needed. . data.py: Modifications to allow slicing with MPI. Code is in a mess but working for basic operations. Still need to fix for some advanced slicing. data.py: Start updating __getitem__. Still not working as intended. data.py: tweaking. Fix one error. Misc. fixes. Breaking it. . data.py: More fixes. Start tidying. Fixes + tidying. Testing. This is painful. data.py: A mess, but progress. Many tests passing now, lots of tidying and simplifying to do. data.py, test_data.py: Some updates + adding more tests. flake8. decomposition.py, data.py: Add fixes for step sizes > 1. Currently incomplete and some cases will still fail. data.py, decomposition.py: Fixes for negative slicing with step < -1. data.py, decomposition.py: __setitem__ fix for when (with mpi_slicing) the dim of val is 1. Data: Fix for cases with dimension > 2. Add more tests. data.py: Some tidying. data.py: Start moving code to new methods. data.py: Start moving code to new methods. data: Further tidying + trimming. data: Minor tidying + add one new test. Bug testing. data: Working on final test. data: Fix bug and add another test. Most tests in place now, lots of tidying + re-structuring still required. data: Tidying. data: Further tidying. Note: Still some things to address (see Fabio's comments above) but approaching complete code. data.py: Change mpi_slicing to SLICING tag. data.py: Fix rebase errors. data, test_data: Further rebase fixes. data.py: Update docstrings. Add gaussian_smooth fn + example. Remove needless expression Fix. Add gaussian_smooth fn + example. Remove needless expression Fix. builtins.py: Begin adding boundary conditions to the Gaussian filter. Tinkering. builtins.py: Modify gaussian smoother. Now returns same results and numpy in examples tested. Add builtins test file. updates. Working on MPI. Bug fixing. Delete. Testing. Remove. MPI slicing fixes. Smoother now working. Update. Add fixme reminder. Fix tests for 'N' procs. Enable N-dims. gird, dimension: Memoize _arg_defaults for performance. gird.py: Chane _arg_defaults decorator from memoized_meth -> memoized_func. Bug fix. data: Add test and fixme. data.utils: Fix flip for slices with negative start/stops. builtins: Improve index creation. grid: Change _arg_defaults mpi test. grid: Remove commented out import. Tidying. Tidying. test_builtins: Skip mpi test if no mpi. No need for list data.utils: Add docstrings. builtins: Move init func to builtins. Start tidying Gaussian smooth. Making test function. builtins: 'New' Gaussian smoother fixes. builtins: Minor tidying of gaussian_smooth + initialize_function. Minor tidying. data.utils: Fix docstrings. types.dense: Add note _arg_defaults note. test_mpi: Add test. test_mpi.py: Fix rebase error.

…penmp are enabled.

…utils: Add as_list function.

…n_smooth' accordingly. Testing.

…builtins: Minor tidying. test_builtins: Start adding 'assign' tests. dle.transformer: Try another 'noop' override config. dle.transformer: Swtich the strings stupid!

…on. tests: Add further builtins tests.

FabioLuporini · 2019-10-22T06:12:53Z

devito/builtins.py

+    >>> import numpy as np
+    >>> from devito import Grid, SubDomain, Function, initialize_function
+
+    Create a subdomain representing the `inner` region:


this is actually grid.interior ?

FabioLuporini · 2019-10-22T06:15:26Z

devito/builtins.py

+def initialize_function(function, data, nbl, mapper=dict(), mode='constant',
+                        name='padfunc'):
+    """
+    Initialize a `Function` with the given ``data``. ``data``


we don't use backtips for types (so `Function` -> Function)

FabioLuporini · 2019-10-22T06:15:43Z

devito/builtins.py

+    ----------
+    function : Function
+        The initialised object.
+    data : ndarray of Function


FabioLuporini · 2019-10-22T06:18:11Z

devito/builtins.py

+    return f
+
+
+def initialize_function(function, data, nbl, mapper=dict(), mode='constant',


never use mutable objects as default values . Just google for it, you'll find loads of explanations, for example here

FabioLuporini · 2019-10-22T06:19:26Z

devito/builtins.py

+    nbl : int
+        Number of outer layers (such as PML layers for boundary damping).
+    mapper : dict, optional
+        Dictionary containing, for each dimension of function, a sub-dictionary


here you probably mean `function` as you're referring to a specific variable name

FabioLuporini · 2019-10-22T06:22:10Z

devito/builtins.py

+    rhs = []
+    options = []
+
+    for d in function.dimensions:


FabioLuporini · 2019-10-22T06:22:28Z

devito/builtins.py

    """
-    dv.Operator(dv.Eq(f, v), name='assign')()
+    if not isinstance(f, list):


or use the new as_list ?

FabioLuporini · 2019-10-22T06:23:02Z

devito/builtins.py

+            rhs_extra = exprs['rhs']
+            lhs.extend(as_list(lhs_extra))
+            rhs.extend(as_list(rhs_extra))
+            if 'options' in exprs.keys() and exprs['options']:


no need for .keys()

FabioLuporini · 2019-10-22T06:23:35Z

devito/builtins.py

+                options_extra = exprs['options']
+            else:
+                options_extra = len(as_list(lhs_extra))*[None, ]
+            if isinstance(options_extra, list):


you may be able to use as_list again, instead of this if-else ?

Not in this case since is could be a single dictionary which the current versions of as_tuple and as_list mess up. But I'm not sure if this behavior was intended or is a bug?

devito/tools/utils.py

FabioLuporini · 2019-10-23T07:55:42Z

devito/builtins.py

+            rhs_extra = exprs['rhs']
+            lhs.extend(as_list(lhs_extra))
+            rhs.extend(as_list(rhs_extra))
+            if 'options' in exprs and exprs['options']:


the pythonic way to write these 4 lines would be

options_extra = exprs.get('options', len(as_list(lhs_extra))*[None, ])

FabioLuporini · 2019-10-23T07:55:53Z

devito/builtins.py

+    >>> grid = Grid(shape=(6, 6))
+    >>> x, y = grid.dimensions
+
+    Create the function we wish to set along with the data to set it:


FabioLuporini · 2019-10-23T07:57:42Z

merged

rhodrin added the testing label Sep 9, 2019

FabioLuporini reviewed Sep 9, 2019

View reviewed changes

devito/builtins.py Outdated Show resolved Hide resolved

FabioLuporini reviewed Sep 10, 2019

View reviewed changes

rhodrin force-pushed the gaussian_smooth branch from 9a4ce68 to 84fcba9 Compare September 10, 2019 12:09

mloubout reviewed Sep 10, 2019

View reviewed changes

FabioLuporini reviewed Sep 11, 2019

View reviewed changes

devito/data/utils.py Show resolved Hide resolved

FabioLuporini reviewed Sep 11, 2019

View reviewed changes

rhodrin force-pushed the gaussian_smooth branch 2 times, most recently from 097ed68 to a41aa15 Compare September 18, 2019 21:15

FabioLuporini reviewed Sep 19, 2019

View reviewed changes

devito/data/utils.py Show resolved Hide resolved

FabioLuporini reviewed Sep 19, 2019

View reviewed changes

devito/types/grid.py Show resolved Hide resolved

rhodrin removed the testing label Sep 20, 2019

mloubout reviewed Sep 23, 2019

View reviewed changes

devito/types/dense.py Show resolved Hide resolved

rhodrin force-pushed the gaussian_smooth branch from dab510b to 6189693 Compare September 24, 2019 11:25

FabioLuporini reviewed Sep 30, 2019

View reviewed changes

devito/builtins.py Show resolved Hide resolved

rhodrin force-pushed the gaussian_smooth branch from 6189693 to 072a738 Compare October 14, 2019 16:34

FabioLuporini reviewed Oct 16, 2019

View reviewed changes

devito/builtins.py Outdated

def assign(f, v=0):

def assign(f, RHS=0, options=None, name='asign', **kwargs):

Copy link

Contributor

FabioLuporini Oct 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo (asign)

FabioLuporini reviewed Oct 16, 2019

View reviewed changes

FabioLuporini reviewed Oct 21, 2019

View reviewed changes

rhodrin added 8 commits October 21, 2019 19:21

builtins: Fix assign bug.

acd34e6

dle/transformer: Fix for when dle='noop' passed to operator but mpi/o…

0af2a89

…penmp are enabled.

builtins: Tidying + add initialize_function docstring example. tools.…

50494fc

…utils: Add as_list function.

builtins: Upgrade 'assign' and 'initialize_function'. Modify 'gaussia…

4fd9684

…n_smooth' accordingly. Testing.

builtins, dle/transformer: Start tidying.

f59e746

dle.transformer: Fix bug causing mode to be incorrectly overwritten. …

adaa06b

…builtins: Minor tidying. test_builtins: Start adding 'assign' tests. dle.transformer: Try another 'noop' override config. dle.transformer: Swtich the strings stupid!

builtins: Add 'inputs test' and add to docstring of initialize_functi…

a99d70b

…on. tests: Add further builtins tests.

rhodrin force-pushed the gaussian_smooth branch from c5ac469 to 50494fc Compare October 21, 2019 18:23

FabioLuporini reviewed Oct 22, 2019

View reviewed changes

rhodrin added 3 commits October 22, 2019 12:47

Tidying.

be06afc

Fix typos.

6564a92

builtins: Raise error if TimeFunction passed to initialize_function.

7efd7b3

FabioLuporini reviewed Oct 23, 2019

View reviewed changes

FabioLuporini merged commit c4ea844 into master Oct 23, 2019

FabioLuporini deleted the gaussian_smooth branch October 23, 2019 07:57

FabioLuporini mentioned this pull request Oct 23, 2019

renaming variable: from nbpml to nbl #975

Closed

This was referenced Oct 25, 2019

Nbpml to nbl4 #979

Closed

Sympy standard function with our types #984

Closed

		return as_tuple(processed)


		def distributed_data_size(shape, coords, topology, nprocs):

		return f


		def initialize_function(function, data, nbpml, mode='constant'):



		def assign(f, v=0):
		def assign(f, RHS=0, options=None, name='asign', **kwargs):

MPI fixes + smoother builtin #931

MPI fixes + smoother builtin #931

Conversation

rhodrin commented Sep 9, 2019

Choose a reason for hiding this comment

rhodrin Sep 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Sep 10, 2019

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini commented Oct 16, 2019

rhodrin commented Oct 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhodrin commented Oct 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini commented Oct 23, 2019

rhodrin Sep 10, 2019 •

edited

Loading

rhodrin commented Oct 16, 2019 •

edited

Loading