misc: Add MPI0 logging level #2130

georgebisbas · 2023-05-17T17:51:13Z

This logging mode helps to reduce the output of MPI runs to only emit performance numbers once per MPI rank.
It is mostly aimed to reduce the redundant verbose performance output.

e.g. instead of:

Allocating host memory for u(2, 52, 72, 72) [2 MB]
Allocating host memory for u(2, 52, 72, 72) [2 MB]
Operator `Kernel` generated in 1.72 s
  * lowering.IET: 0.71 s (41.5 %)
     * specializing.IET: 0.66 s (38.6 %)
        * make_mpi: 0.45 s (26.3 %)
  * lowering.Expressions: 0.56 s (32.8 %)
  * lowering.Clusters: 0.41 s (24.0 %)
Flops reduction after symbolic optimization: [209 --> 90]
Operator `Kernel` generated in 1.81 s
  * lowering.IET: 0.72 s (40.0 %)
     * specializing.IET: 0.66 s (36.6 %)
        * make_mpi: 0.45 s (25.0 %)
  * lowering.Expressions: 0.58 s (32.2 %)
  * lowering.Clusters: 0.48 s (26.7 %)
Flops reduction after symbolic optimization: [209 --> 90]
Operator `Kernel` fetched `/tmp/devito-jitcache-uid1000/562b2d8ee0a6d254108e1dfb5522a645f2d8d8e2.c` in 0.06 s from jit-cache
Operator `Kernel` fetched `/tmp/devito-jitcache-uid1000/562b2d8ee0a6d254108e1dfb5522a645f2d8d8e2.c` in 0.06 s from jit-cache
Operator `Kernel` ran in 0.01 s
Operator `Kernel` ran in 0.01 s
Global performance: [OI=2.81, 11.16 GFlops/s, 0.14 GPts/s]
Global performance: [OI=2.81, 11.16 GFlops/s, 0.14 GPts/s]
Local performance:
Local performance:
  * section0[rank0]<21,2,5,8,8,40> ran in 0.01 s [OI=2.81, 10.95 GFlops/s, 0.14 GPts/s]
  * section0[rank0]<21,2,5,8,8,40> ran in 0.01 s [OI=2.81, 10.95 GFlops/s, 0.14 GPts/s]
  * section0[rank1]<21,2,5,8,8,40> ran in 0.01 s [OI=2.81, 10.95 GFlops/s, 0.14 GPts/s]
Performance[mode=advanced] arguments: {'x0_blk0_size': 8, 'y0_blk0_size': 8}
  * section0[rank1]<21,2,5,8,8,40> ran in 0.01 s [OI=2.81, 10.95 GFlops/s, 0.14 GPts/s]
Performance[mode=advanced] arguments: {'x0_blk0_size': 8, 'y0_blk0_size': 8}
Norm u is 2325.5583
Norm u is 2325.5583

we get:

Allocating host memory for u(2, 52, 72, 72) [2 MB]
Allocating host memory for u(2, 52, 72, 72) [2 MB]
Operator `Kernel` generated in 1.81 s
  * lowering.IET: 0.73 s (40.6 %)
     * specializing.IET: 0.67 s (37.3 %)
        * make_mpi: 0.46 s (25.6 %)
  * lowering.Expressions: 0.54 s (30.0 %)
  * lowering.Clusters: 0.51 s (28.4 %)
     * specializing.Clusters: 0.39 s (21.7 %)
Flops reduction after symbolic optimization: [209 --> 90]
Operator `Kernel` generated in 1.80 s
  * lowering.IET: 0.75 s (41.8 %)
     * specializing.IET: 0.69 s (38.4 %)
        * make_mpi: 0.48 s (26.8 %)
  * lowering.Expressions: 0.56 s (31.2 %)
  * lowering.Clusters: 0.46 s (25.6 %)
Flops reduction after symbolic optimization: [209 --> 90]
Operator `Kernel` fetched `/tmp/devito-jitcache-uid1000/562b2d8ee0a6d254108e1dfb5522a645f2d8d8e2.c` in 0.06 s from jit-cache
Operator `Kernel` fetched `/tmp/devito-jitcache-uid1000/562b2d8ee0a6d254108e1dfb5522a645f2d8d8e2.c` in 0.06 s from jit-cache
Operator `Kernel` ran in 0.01 s
Global performance: [OI=2.81, 11.16 GFlops/s, 0.14 GPts/s]
Local performance:
  * section0[rank0]<21,2,5,8,8,40> ran in 0.01 s [OI=2.81, 11.65 GFlops/s, 0.15 GPts/s]
  * section0[rank1]<21,2,5,8,8,40> ran in 0.01 s [OI=2.81, 11.67 GFlops/s, 0.15 GPts/s]
Performance[mode=advanced] arguments: {'x0_blk0_size': 8, 'y0_blk0_size': 8}
Norm u is 2325.5583
Norm u is 2325.558

codecov · 2023-05-17T18:02:49Z

Codecov Report

Attention: Patch coverage is 93.75000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 79.46%. Comparing base (8729cd2) to head (e459d52).

Files	Patch %	Lines
devito/operator/profiling.py	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2130      +/-   ##
==========================================
+ Coverage   79.44%   79.46%   +0.01%     
==========================================
  Files         232      232              
  Lines       43607    43618      +11     
  Branches     8072     8073       +1     
==========================================
+ Hits        34645    34660      +15     
+ Misses       8207     8203       -4     
  Partials      755      755

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mloubout · 2023-05-17T18:36:54Z

devito/mpi/distributed.py

@@ -190,6 +191,10 @@ def __init__(self, shape, dimensions, input_comm=None, topology=None):
                global init_by_devito
                init_by_devito = True

+                # If MPI0 logging is selected, only emit from rank 0


I don't think this is a safe way to do it. You also end up here quite often when devito is called from an external program that initialized MPI already and that would completely turn off the logging on all ranks including 0.

If this is something we want it needs to be done outside MPI.Is_initialized() and on a per-rank base using get_Rank

I second Mathias. Here you're potentially altering the overarching application behavior.

Also you probably don't need to add a new log level... just restrict logging to rank 0 (if MPI active) inside op._emit_perf_profile or whatever is called. No?

I think this is addressed now

FabioLuporini · 2023-05-18T07:28:26Z

devito/mpi/distributed.py

@@ -190,6 +191,10 @@ def __init__(self, shape, dimensions, input_comm=None, topology=None):
                global init_by_devito
                init_by_devito = True

+                # If MPI0 logging is selected, only emit from rank 0


I second Mathias. Here you're potentially altering the overarching application behavior.

Also you probably don't need to add a new log level... just restrict logging to rank 0 (if MPI active) inside op._emit_perf_profile or whatever is called. No?

mloubout · 2023-05-26T17:15:43Z

devito/operator/operator.py

+
+        # In case 'MPI0' is selected for logging, restrict result printing to one rank
+        if configuration['mpi']:
+            set_log_level(configuration['log-level'], comm=args.comm)


I still don't really like changing configuration like that unless it's a temporary swithconfig

Based on this comment: #2130 (comment)

I thought that we would be happy to happen here?

no, this would change user-level setup after the first Operator run, so afterwards we would only see output from one rank

FabioLuporini · 2023-06-28T07:32:45Z

devito/operator/operator.py

+
+        # In case 'MPI0' is selected for logging, restrict result printing to one rank
+        if configuration['mpi']:
+            set_log_level(configuration['log-level'], comm=args.comm)


no, this would change user-level setup after the first Operator run, so afterwards we would only see output from one rank

FabioLuporini · 2023-06-28T07:36:25Z

devito/operator/operator.py

@@ -952,7 +957,12 @@ def _emit_apply_profiling(self, args):
                    if a in args:
                        perf_args[a] = args[a]
                        break
-        perf("Performance[mode=%s] arguments: %s" % (self._mode, perf_args))
+
+        if configuration['mpi']:


nitpicking: the following way more easily generalisable to any number of items

items = [] items.append('mode=%s' % self._mode) if configuration['mpi']: #Do we really want this `if` ? maybe we just spell it out regardless items.append('mpi=%s' % configuration['mpi']) perf("Performance[%s] arguments: %s" % (', '.join(items), perf_args))

georgebisbas · 2023-06-29T13:05:47Z

To add:
-Until Devito v3.5 included, domain decomposition occurs along the fastest axis. As of later versions, domain decomposition occurs along the slowest axis, for performance reasons. And yes, it is possible to control the domain decomposition in user code, but this is not neatly documented. Take a look at test_custom_topology in this file. In essence, Grid accepts the optional argument topology, which allows the user to pass a custom topology as an n-tuple, where n is the number of distributed dimensions. For example, for a two-dimensional grid, the topology (4, 1) will decompose the slowest axis into four partitions, one partition per MPI rank, while the fastest axis will be replicated over all MPI ranks.
+Until Devito v3.5 included, domain decomposition occurs along the fastest axis. As of later versions, domain decomposition occurs along the slowest axis, for performance reasons. And yes, it is possible to control the domain decomposition in user code, but this is not neatly documented. Take a look at
+class CustomTopology in distributed.py
+and test_custom_topology in this file. In essence, Grid accepts the optional argument topology, which allows the user to pass a custom topology as an n-tuple, where n is the number of distributed dimensions. For example, for a two-dimensional grid, the topology (4, 1) will decompose the slowest axis into four partitions, one partition per MPI rank, while the fastest axis will be replicated over all MPI ranks.

georgebisbas · 2023-08-02T06:19:45Z

Would this PR help: #2175
to avoid changing the user-level setup and only make a temp switchconfig as per the original plan?

FAQ.md

georgebisbas · 2023-11-03T17:17:16Z

Is this PR fine?

FAQ.md

FabioLuporini · 2023-11-09T08:06:33Z

devito/operator/operator.py

+        else:
+            perf("Performance[mode=%s] arguments: %s" % (self._mode, perf_args))
+
+        # Restore logging configuration to all ranks


why don't u use switchconfig?

Answered on slack: https://devitocodes.slack.com/archives/C7JMLMSG0/p1699874271414939?thread_ts=1699467879.080049&cid=C7JMLMSG0

georgebisbas · 2023-12-20T09:29:50Z

Converting to draft as apprantly switchcongfig does not work

FabioLuporini · 2024-02-15T08:34:06Z

devito/operator/operator.py

+        # In case MPI is used restrict result logging to one rank only
+        if configuration['mpi']:
+            # Only temporarily change configuration
+            with switchconfig(mpi=True):


what does mpi=True mean?

FabioLuporini · 2024-03-25T08:49:46Z

devito/operator/operator.py

+        if configuration['mpi']:
+            # Only temporarily change configuration
+            with switchconfig(mpi=True):
+                set_log_level('DEBUG', comm=args.comm)


Again: you're switching to DEBUG and then it remains DEBUG once control is handed back to the user

it shouldn't be DEBUG either, it should be whatever was set up to this point

and finally we shouldn't switch to mpi=True (why btw?) because it will trigger a compiler rebuild which we should (can) avoid: https://github.com/devitocodes/devito/blob/master/devito/__init__.py#L74

Oh right. Makes sense. Thanks

FabioLuporini

I suggest to add a logger context manager as per https://devitocodes.slack.com/archives/C7JMLMSG0/p1711357031925169?thread_ts=1711125928.651539&cid=C7JMLMSG0

FabioLuporini · 2024-04-10T15:54:15Z

devito/logger.py

-# Add extra logging levels (note: INFO has value=20, WARNING has value=30)
-DEBUG = logging.DEBUG
+# Add extra logging levels
+DEBUG = logging.DEBUG  # value=10


implementation detail

Not sure I understand what to change here

I think the numbers should be dropped

I am reverting as before, some detail is useful

FabioLuporini · 2024-04-10T15:55:13Z

devito/operator/operator.py

-        # Output summary of performance achieved
-        return self._emit_apply_profiling(args)
+        # In case MPI is used restrict result logging to one rank only
+        if configuration['mpi']:


this shouldn't be needed

This is needed only if the user accidentally asks for copies of an execution.
i.e. DEVITO_MPI=0 mpirun -n 2.
if not it will break with:

Attempting to use an MPI routine (internal_Comm_rank) before initializing or after finalizing MPICH

I can get away with a try-except in mpi_switch_log ?

added another condition in set_log_level

FabioLuporini · 2024-04-10T15:55:24Z

devito/operator/operator.py

-        return self._emit_apply_profiling(args)
+        # In case MPI is used restrict result logging to one rank only
+        if configuration['mpi']:
+            with mpi_switch_log(log_level='DEBUG', comm=args.comm):


why make the "switch log" mpi-specific?

I wanted to avoid confusion between having sth like set_log_level and switch_log
I am happy if you propose a name for that

switch_log_level is fine

FabioLuporini · 2024-04-11T12:13:40Z

devito/parameters.py

+
+    def __init__(self, **params):
+        self.params = {k.replace('_', '-'): v for k, v in params.items()}
+        self.previous = {}


this is never updated, how can this even work?

it was using the parent class enter

FabioLuporini · 2024-04-11T12:14:20Z

devito/parameters.py

+    """
+
+    def __init__(self, **params):
+        self.params = {k.replace('_', '-'): v for k, v in params.items()}


what's params for?

this function should expect, imho, two arguments: a level and a comm (the latter being optional)

Yes, maybe I can do it simpler

FabioLuporini · 2024-04-16T07:16:13Z

devito/logger.py

@@ -77,7 +77,7 @@ def set_log_level(level, comm=None):
    """
    from devito import configuration

-    if comm is not None:
+    if comm is not None and configuration['mpi']:


why is this addition needed?

To elegantly alleviate the case of DEVITO_MPI=0 mpirun -n 2 (running copies of the problem) as discussed

FabioLuporini · 2024-04-16T07:17:08Z

devito/operator/operator.py

+        # In case MPI is used restrict result logging to one rank only
+        with switch_log_level(comm=args.comm):
+            return self._emit_apply_profiling(args)
+
        return self._emit_apply_profiling(args)


emitting twice ?

ha! Actually not emmiting twice, but dead code never executed! dropping

FabioLuporini · 2024-04-16T07:17:23Z

devito/parameters.py

+class switch_log_level(object):
+    """
+    A context manager subclassing `switchconfig` to temporarily change
+    MPI logging.


old docstring

FabioLuporini · 2024-04-16T07:18:16Z

devito/parameters.py

+        self.comm = comm
+
+        # Limit logging to rank 0
+        set_log_level(self.level, comm)


theoretically shouldn't this be inside __enter__? though in practice it might not change much

FabioLuporini · 2024-04-16T07:18:44Z

devito/parameters.py

@@ -258,6 +258,29 @@ def wrapper(*args, **kwargs):
        return wrapper


+class switch_log_level(object):


doesn't this belong to logger.py?

moved, but inserted import configuration in in __init__ to avoid circular imports

devito/parameters.py

devito/logger.py

FabioLuporini

couple of nitpicks, but this is now GTG basically

FabioLuporini · 2024-04-18T07:25:13Z

devito/logger.py

+
+    def __exit__(self, *args):
+        set_log_level(self.level)
+        logger.addHandler(stream_handler)


shouldn't this line go inside set_log_level in an else branch representing the comm is None case? this way, a set_log_level in user code will behave as expected

note: the above is nitpicking

EdCaunt · 2024-04-18T09:10:15Z

FAQ.md


-Until Devito v3.5 included, domain decomposition occurs along the fastest axis. As of later versions, domain decomposition occurs along the slowest axis, for performance reasons.  And yes, it is possible to control the domain decomposition in user code, but this is not neatly documented. Take a look at `test_custom_topology` in [this file](https://github.com/devitocodes/devito/blob/master/tests/test_mpi.py). In essence, `Grid` accepts the optional argument `topology`, which allows the user to pass a custom topology as an n-tuple, where `n` is the number of distributed dimensions. For example, for a two-dimensional grid, the topology `(4, 1)` will decompose the slowest axis into four partitions, one partition per MPI rank, while the fastest axis will be replicated over all MPI ranks.
+Until Devito v3.5 included, domain decomposition occurs along the fastest axis. As of later versions, domain decomposition occurs along the slowest axis, for performance reasons.  And yes, it is possible to control the domain decomposition in user code, but this is not neatly documented. Take a look at `class CustomTopology` in [distributed.py](https://github.com/devitocodes/devito/blob/master/devito/mpi/distributed.py) and `test_custom_topology` in [this file](https://github.com/devitocodes/devito/blob/master/tests/test_mpi.py). In essence, `Grid` accepts the optional argument `topology`, which allows the user to pass a custom topology as an n-tuple, where `n` is the number of distributed dimensions. For example, for a two-dimensional grid, the topology `(4, 1)` will decompose the slowest axis into four partitions, one partition per MPI rank, while the fastest axis will be replicated over all MPI ranks.


"Up to and including Devito v3.5," would be a more natural wording

EdCaunt · 2024-04-18T09:11:57Z

devito/logger.py


    # Triggers a callback to `_set_log_level`
    configuration['log-level'] = level


+class switch_log_level(object):


object is no longer needed. Just class switch_log_level:

nitpicking: class names start with upercase and don't use underscore

For context managers lower case is probably acceptable

mloubout · 2024-04-18T11:55:29Z

devito/logger.py


    # Triggers a callback to `_set_log_level`
    configuration['log-level'] = level


+class switch_log_level(object):


nitpicking: class names start with upercase and don't use underscore

mloubout · 2024-04-18T11:56:28Z

devito/logger.py

+    A context manager to temporarily change MPI logging.
+    """
+
+    def __init__(self, comm):


comm is confusing we use it everywhere for the MPI comm, rename to level

it is actually an MPI comm

mloubout · 2024-04-18T11:57:16Z

devito/operator/operator.py

-        # Output summary of performance achieved
-        return self._emit_apply_profiling(args)
+        # In case MPI is used restrict result logging to one rank only
+        with switch_log_level(comm=args.comm):


remind me why switchconfig doesn't work here?

you have to pass the comm

georgebisbas added the documentation label May 17, 2023

georgebisbas requested review from FabioLuporini and mloubout May 17, 2023 17:51

mloubout reviewed May 17, 2023

View reviewed changes

FabioLuporini requested changes May 18, 2023

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch 2 times, most recently from ef111d5 to 76cca39 Compare May 18, 2023 15:22

georgebisbas requested a review from FabioLuporini May 18, 2023 15:23

georgebisbas self-assigned this May 18, 2023

georgebisbas requested a review from mloubout May 19, 2023 07:48

mloubout reviewed May 26, 2023

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch from 76cca39 to e7f0d97 Compare June 12, 2023 15:17

georgebisbas requested a review from mloubout June 19, 2023 16:02

georgebisbas force-pushed the mpi0_logging branch 2 times, most recently from 6a21332 to 3544c7a Compare June 27, 2023 10:15

FabioLuporini requested changes Jun 28, 2023

View reviewed changes

FabioLuporini reviewed Jun 28, 2023

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch from 3544c7a to c571185 Compare June 30, 2023 15:06

georgebisbas force-pushed the mpi0_logging branch from c571185 to a5ef5eb Compare August 1, 2023 09:00

georgebisbas force-pushed the mpi0_logging branch from a5ef5eb to 51fb9f6 Compare September 4, 2023 15:30

georgebisbas force-pushed the mpi0_logging branch from 51fb9f6 to 97b9804 Compare September 18, 2023 12:47

EdCaunt reviewed Oct 2, 2023

View reviewed changes

FAQ.md Show resolved Hide resolved

georgebisbas requested review from FabioLuporini and EdCaunt October 24, 2023 14:57

georgebisbas force-pushed the mpi0_logging branch from 423e563 to 4543339 Compare November 2, 2023 19:59

FabioLuporini requested changes Nov 9, 2023

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch 2 times, most recently from 00a4c34 to c67c473 Compare January 17, 2024 13:58

FabioLuporini reviewed Feb 15, 2024

View reviewed changes

FabioLuporini requested changes Mar 25, 2024

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch 2 times, most recently from 9dcab9f to 5ff0adf Compare April 10, 2024 14:01

georgebisbas requested a review from FabioLuporini April 10, 2024 14:14

georgebisbas marked this pull request as ready for review April 10, 2024 14:14

FabioLuporini reviewed Apr 10, 2024

View reviewed changes

FabioLuporini reviewed Apr 11, 2024

View reviewed changes

FabioLuporini reviewed Apr 16, 2024

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch from e48ff09 to 3529090 Compare April 16, 2024 10:45

georgebisbas requested a review from FabioLuporini April 16, 2024 10:46

georgebisbas added the MPI mpi-related label Apr 16, 2024

FabioLuporini reviewed Apr 18, 2024

View reviewed changes

devito/logger.py Outdated Show resolved Hide resolved

FabioLuporini approved these changes Apr 18, 2024

View reviewed changes

georgebisbas force-pushed the mpi0_logging branch from 3529090 to c2d85ed Compare April 18, 2024 08:02

misc: Restrict MPI perf logging to rank 0

e859b21

georgebisbas force-pushed the mpi0_logging branch from c2d85ed to 52cb671 Compare April 18, 2024 08:03

georgebisbas added 2 commits April 18, 2024 11:05

docs: Update for Custom topology and restore logging

6ddce4c

misc: Use mpi_switch_log for mpi rank 0 logging

e459d52

georgebisbas force-pushed the mpi0_logging branch from 52cb671 to e459d52 Compare April 18, 2024 08:05

EdCaunt reviewed Apr 18, 2024

View reviewed changes

mloubout reviewed Apr 18, 2024

View reviewed changes

mloubout approved these changes Apr 18, 2024

View reviewed changes

mloubout merged commit 3294158 into master Apr 18, 2024
31 checks passed

mloubout deleted the mpi0_logging branch April 18, 2024 12:28

		@@ -258,6 +258,29 @@ def wrapper(args, *kwargs):
		return wrapper


		class switch_log_level(object):


		Until Devito v3.5 included, domain decomposition occurs along the fastest axis. As of later versions, domain decomposition occurs along the slowest axis, for performance reasons. And yes, it is possible to control the domain decomposition in user code, but this is not neatly documented. Take a look at `test_custom_topology` in [this file](https://github.com/devitocodes/devito/blob/master/tests/test_mpi.py). In essence, `Grid` accepts the optional argument `topology`, which allows the user to pass a custom topology as an n-tuple, where `n` is the number of distributed dimensions. For example, for a two-dimensional grid, the topology `(4, 1)` will decompose the slowest axis into four partitions, one partition per MPI rank, while the fastest axis will be replicated over all MPI ranks.
		Until Devito v3.5 included, domain decomposition occurs along the fastest axis. As of later versions, domain decomposition occurs along the slowest axis, for performance reasons. And yes, it is possible to control the domain decomposition in user code, but this is not neatly documented. Take a look at `class CustomTopology` in [distributed.py](https://github.com/devitocodes/devito/blob/master/devito/mpi/distributed.py) and `test_custom_topology` in [this file](https://github.com/devitocodes/devito/blob/master/tests/test_mpi.py). In essence, `Grid` accepts the optional argument `topology`, which allows the user to pass a custom topology as an n-tuple, where `n` is the number of distributed dimensions. For example, for a two-dimensional grid, the topology `(4, 1)` will decompose the slowest axis into four partitions, one partition per MPI rank, while the fastest axis will be replicated over all MPI ranks.

misc: Add MPI0 logging level #2130

misc: Add MPI0 logging level #2130

Conversation

georgebisbas commented May 17, 2023 • edited Loading

codecov bot commented May 17, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgebisbas commented Jun 29, 2023

georgebisbas commented Aug 2, 2023

georgebisbas commented Nov 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgebisbas commented Dec 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgebisbas commented May 17, 2023 •

edited

Loading

codecov bot commented May 17, 2023 •

edited

Loading