Handle horovod errors #24
Conversation
horovod/mxnet/mpi_ops.cc
Outdated
| #include "tensor_util.h" | ||
|
|
||
| namespace horovod { | ||
| namespace mxnet { | ||
|
|
||
| static StatusManager status_manager; |
There was a problem hiding this comment.
Do we want to move op_count into status_manager and make it a tuple with the last_error like (op_count, last_error). Would it be helpful for debugging?
There was a problem hiding this comment.
This is a good one. Maybe we can save the name (which contains the op_count) to make it more informative.
|
@ctcyang As we discussed offline, MX_API_BEGIN/MX_API_END was used to catch C++ errors and make it possible to throw MXNetError at Python level. It is not profiler related as far as I know. |
|
TBH, it seems a bit hacky. This doesn't resolve the situation when the last Horovod operation fails. I have another idea. I see that this code can push operation in the engine, which may throw How would that error be handled? What if our Horovod error handling will enqueue another high-priority operation which will just throw |
|
New API is way nicer! :-) |
This PR populates error originated in Horovod back to MXNet engine through callback. The error will be thrown in MXNet when a sync action is performed, such as wait_to_read(). This approach has no performance impact. This is dependent on a PR in MXNet.
The approaches using synchronize and C++ future library shows 80~85% performance regression.