Skip to content

Fix RoiAlign CPU EP issues #7354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fdwr wants to merge 27 commits into
base: main
Choose a base branch
from
Open

Fix RoiAlign CPU EP issues #7354

Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
aa5461d
Fix RoiAlign CPU (1) pixel alignment is 0.5 off (2) 0 size regions ar…
fdwr Apr 16, 2021
b826842
Set max to 0, not 1
fdwr Apr 16, 2021
c49fc02
Dummy commit to retrigger CI
fdwr Apr 19, 2021
9c246e3
Remove -0.5 for now until ONNX update, at least enabling other two fi…
fdwr Jun 26, 2021
2a42fe3
Update RoiAlign test, previously compared against wrong values
fdwr Jul 1, 2021
a9a7e62
Fix build error about if being constant in Eigen and Protobuf
fdwr Jul 2, 2021
acb6e5c
Merge branch 'master' into user/dwayner/FixRoiAlign
fdwr Jul 16, 2021
3024d7e
Merge branch 'master' into user/dwayner/FixIfConstexprBuildError
fdwr Jul 16, 2021
1235ceb
Update with Changming's proposal
fdwr Jul 16, 2021
4fb7edb
Merge branch 'user/dwayner/FixIfConstexprBuildError' into user/dwayne…
fdwr Jul 16, 2021
68cb8b2
Merge branch 'master' into user/dwayner/FixRoiAlign
fdwr Jul 18, 2021
65e1302
Merge branch 'master' into user/dwayner/FixRoiAlign
fdwr Jul 5, 2022
031dce3
Add mask_rcnn_R_50_FPN_1x to GetSkippedModels
fdwr Jul 5, 2022
813ab94
Appease noisy linter
fdwr Jul 8, 2022
b9c89d9
Merge branch 'master' into user/dwayner/FixRoiAlign
fdwr Jul 9, 2022
4624031
Fix broken logic - it's the bucket size which should be clamped, not …
fdwr Jul 9, 2022
ba0fe83
Merge branch 'master' into user/dwayner/FixRoiAlign
fdwr Sep 23, 2022
9a9ba5c
Merge branch 'user/dwayner/FixRoiAlign' of https://github.com/microso…
fdwr Sep 24, 2022
8798fed
Merge branch 'main' into user/dwayner/FixRoiAlign
fdwr Sep 24, 2022
385c924
Appease linter
fdwr Sep 24, 2022
22eab88
Merge branch 'main' into user/dwayner/FixRoiAlign
fdwr Oct 27, 2022
5546fe9
Skip buggy model as it was generated with incorrect implementation
fdwr Oct 27, 2022
159452b
Model suppression comments and cast feedback
fdwr Oct 28, 2022
c4cadb6
Merge branch 'main' into user/dwayner/FixRoiAlign
fdwr Oct 31, 2022
bb11ce8
Merge branch 'main' into user/dwayner/FixRoiAlign
fdwr Nov 3, 2022
7c1638f
Disable 2 RoiAlign tests for CUDA EP until it is fixed too
fdwr Nov 3, 2022
5d721f6
Merge remote-tracking branch 'origin/main' into user/dwayner/FixRoiAlign
May 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions csharp/test/Microsoft.ML.OnnxRuntime.Tests.NetCoreApp/InferenceTest.netcore.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,8 @@ private static Dictionary<string, string> GetSkippedModels(DirectoryInfo modelsD
{ "cntk_simple_seg", "Bad onnx test output caused by wrong SAME_UPPER/SAME_LOWER for ConvTranspose" },
{ "coreml_Imputer-LogisticRegression_sklearn_load_breast_cancer", "Can't determine model file name" },
{ "mask_rcnn_keras", "Model should be edited to remove the extra outputs" },
{ "mask_rcnn", "Expected model values were recorded with buggy implementation of RoiAlign. See https://github.com/microsoft/onnxruntime/pull/7354/." },
{ "mask_rcnn_R_50_FPN_1x", "Expected model values were recorded with buggy implementation of RoiAlign. See https://github.com/microsoft/onnxruntime/pull/7354/." },
{ "test_strnormalizer_export_monday_casesensintive_lower", "ElementType not currently supported"},
{ "test_max_float64", "node test error"},
{ "test_min_uint8", "node test error"},
Expand Down
18 changes: 7 additions & 11 deletions onnxruntime/core/providers/cpu/object_detection/roialign.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,19 +181,17 @@ void RoiAlignForward(const TensorShape& output_shape, const T* bottom_data, floa

T roi_width = roi_end_w - roi_start_w;
T roi_height = roi_end_h - roi_start_h;
if (!half_pixel) {
// Force malformed ROIs to be 1x1
roi_width = std::max(roi_width, (T)1.);
roi_height = std::max(roi_height, (T)1.);
}
// Note that 0 size ROI's are legal, meaning the sample a single point in the input.
// Even inverted ROI's are acceptable, meaning mirrored images.

T bin_size_h = static_cast<T>(roi_height) / static_cast<T>(pooled_height);
T bin_size_w = static_cast<T>(roi_width) / static_cast<T>(pooled_width);

// We use roi_bin_grid to sample the grid and mimic integral
int64_t roi_bin_grid_h = (sampling_ratio > 0) ? sampling_ratio : static_cast<int64_t>(std::ceil(roi_height / pooled_height)); // e.g., = 2
int64_t roi_bin_grid_w =
(sampling_ratio > 0) ? sampling_ratio : static_cast<int64_t>(std::ceil(roi_width / pooled_width));
int64_t roi_bin_grid_w = (sampling_ratio > 0) ? sampling_ratio : static_cast<int64_t>(std::ceil(roi_width / pooled_width));
roi_bin_grid_h = std::max(roi_bin_grid_h, (int64_t)1);
Comment thread
fdwr marked this conversation as resolved.
Outdated
roi_bin_grid_w = std::max(roi_bin_grid_h, (int64_t)1);

// We do average (integral) pooling inside a bin
const int64_t count = std::max(roi_bin_grid_h * roi_bin_grid_w, static_cast<int64_t>(1)); // e.g. = 4
Expand Down Expand Up @@ -232,10 +230,8 @@ void RoiAlignForward(const TensorShape& output_shape, const T* bottom_data, floa
for (int64_t iy = 0; iy < roi_bin_grid_h; iy++) {
for (int64_t ix = 0; ix < roi_bin_grid_w; ix++) {
const auto& pc = pre_calc[pre_calc_index];
T val = std::max(
std::max(std::max(pc.w1 * offset_bottom_data[pc.pos1], pc.w2 * offset_bottom_data[pc.pos2]),
pc.w3 * offset_bottom_data[pc.pos3]),
pc.w4 * offset_bottom_data[pc.pos4]);
T val = pc.w1 * offset_bottom_data[pc.pos1] + pc.w2 * offset_bottom_data[pc.pos2] +
pc.w3 * offset_bottom_data[pc.pos3] + pc.w4 * offset_bottom_data[pc.pos4];
if (!max_flag) {
output_val = val;
max_flag = true;
Expand Down
119 changes: 74 additions & 45 deletions onnxruntime/test/providers/cpu/object_detection/roialign_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,44 @@ TEST(RoiAlignTest, AvgModePositive) {
constexpr int H = 5;
constexpr int W = 5;

std::vector<float> rois{0.,7.,5.,7.,5.,0.,-15.,-15.,-15.,-15.,0.,-10.,21.,-10.,21.,0.,13.,8.,13.,8.,0.,-14.,19.,-14.,19.};
test.AddInput<float>("X", {N, C, H, W}, {0.,1.,2.,3.,4.,5.,6.,7.,8.,9.,10.,11.,12.,13.,14.,15.,16.,17.,18.,19.,20.,21.,22.,23.,24.,
25.,26.,27.,28.,29.,30.,31.,32.,33.,34.,35.,36.,37.,38.,39.,40.,41.,42.,43.,44.,45.,46.,
47.,48.,49.,50.,51.,52.,53.,54.,55.,56.,57.,58.,59.,60.,61.,62.,63.,64.,65.,66.,67.,68.,
69.,70.,71.,72.,73.,74.});
test.AddInput<float>("rois", {5, 4}, {7.,5.,7.,5.,-15.,-15.,-15.,-15.,-10.,21.,-10.,21.,13.,8.,13.,8.,-14.,19.,-14.,19.});
// A series of empty rectangles yields point sampling.
std::vector<float> rois{0.,7.,5.,7.,5.,0.,-15.,-15.,-15.,-15.,0.,-10.,21.,-10.,21.,0.,13.,8.,13.,8.,0.,-14.,19.,-14.,19.}; // NOLINT
test.AddInput<float>(
"X",
{N, C, H, W},
{
0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., // NOLINT
25., 26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39., 40., 41., 42., 43., 44., 45., 46., 47., 48., 49., // NOLINT
50., 51., 52., 53., 54., 55., 56., 57., 58., 59., 60., 61., 62., 63., 64., 65., 66., 67., 68., 69., 70., 71., 72., 73., 74., // NOLINT
}
); // NOLINT
test.AddInput<float>("rois", {5, 4}, {7.,5.,7.,5., -15.,-15.,-15.,-15., -10.,21.,-10.,21., 13.,8.,13.,8., -14.,19.,-14.,19.}); // NOLINT
test.AddInput<int64_t>("batch_indices", {5}, {0, 0, 0, 0, 0});
test.AddOutput<float>("Y", {5,3,3,4}, {2.95833f,3.20833f,3.45833f,3.70833f,4.625f,4.875f,5.125f,5.375f,
6.29167f,6.54167f,6.79167f,7.04167f,27.9583f,28.2083f,28.4583f,
28.7083f,29.625f,29.875f,30.125f,30.375f,31.2917f,31.5417f,31.7917f,
32.0417f,52.9583f,53.2083f,53.4583f,53.7083f,54.625f,54.875f,55.125f,
55.375f,56.2917f,56.5417f,56.7917f,57.0417f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,
25.f,25.f,25.f,25.f,25.f,25.f,25.f,25.f,25.f,25.f,25.f,25.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,
50.f,50.f,50.f,7.39583f,7.39583f,7.42708f,7.64583f,9.0625f,9.0625f,9.09375f,
9.3125f,10.7292f,10.7292f,10.7604f,10.9792f,32.3958f,32.3958f,32.4271f,
32.6458f,34.0625f,34.0625f,34.0938f,34.3125f,35.7292f,35.7292f,35.7604f,
35.9792f,57.3958f,57.3958f,57.4271f,57.6458f,59.0625f,59.0625f,59.0938f,
59.3125f,60.7292f,60.7292f,60.7604f,60.9792f,4.27083f,4.52083f,4.77083f,
5.02083f,5.9375f,6.1875f,6.4375f,6.6875f,7.60417f,7.85417f,8.10417f,8.35417f,
29.2708f,29.5208f,29.7708f,30.0208f,30.9375f,31.1875f,31.4375f,31.6875f,32.6042f,
32.8542f,33.1042f,33.3542f,54.2708f,54.5208f,54.7708f,55.0208f,55.9375f,56.1875f,
56.4375f,56.6875f,57.6042f,57.8542f,58.1042f,58.3542f,6.77083f,6.77083f,6.77083f,
6.80208f,8.4375f,8.4375f,8.4375f,8.46875f,10.1042f,10.1042f,10.1042f,10.1354f,31.7708f,
31.7708f,31.7708f,31.8021f,33.4375f,33.4375f,33.4375f,33.4688f,35.1042f,35.1042f,35.1042f,
35.1354f,56.7708f,56.7708f,56.7708f,56.8021f,58.4375f,58.4375f,58.4375f,58.4688f,60.1042f,
60.1042f,60.1042f,60.1354f});
test.AddOutput<float>(
"Y",
{5, 3, 3, 4},
{
2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000,
27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000,
52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000,

0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000,
50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000,

6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625,
31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625,
56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625,

3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125,
28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125,
53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125,

5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375,
30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375,
55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375,
}
); // NOLINT

test.Run();
}
Expand Down Expand Up @@ -117,27 +129,44 @@ TEST(RoiAlignTest, MaxModePositive) {
constexpr int H = 5;
constexpr int W = 5;

std::vector<float> rois{0.,7.,5.,7.,5.,0.,-15.,-15.,-15.,-15.,0.,-10.,21.,-10.,21.,0.,13.,8.,13.,8.,0.,-14.,19.,-14.,19.};
test.AddInput<float>("X", {N, C, H, W}, {0.,1.,2.,3.,4.,5.,6.,7.,8.,9.,10.,11.,12.,13.,14.,15.,16.,17.,18.,19.,20.,21.,22.,23.,24.,
25.,26.,27.,28.,29.,30.,31.,32.,33.,34.,35.,36.,37.,38.,39.,40.,41.,42.,43.,44.,45.,46.,
47.,48.,49.,50.,51.,52.,53.,54.,55.,56.,57.,58.,59.,60.,61.,62.,63.,64.,65.,66.,67.,68.,
69.,70.,71.,72.,73.,74.});
test.AddInput<float>("rois", {5, 4}, {7.,5.,7.,5.,-15.,-15.,-15.,-15.,-10.,21.,-10.,21.,13.,8.,13.,8.,-14.,19.,-14.,19.});
// A series of empty rectangles yields point sampling.
std::vector<float> rois{0., 7., 5., 7., 5., 0., -15., -15., -15., -15., 0., -10., 21., -10., 21., 0., 13., 8., 13., 8., 0., -14., 19., -14., 19.}; // NOLINT
test.AddInput<float>(
"X",
{N, C, H, W},
{
0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.,11.,12.,13.,14., 15.,16.,17.,18.,19., 20.,21.,22.,23.,24., // NOLINT
25.,26.,27.,28.,29., 30.,31.,32.,33.,34., 35.,36.,37.,38.,39., 40.,41.,42.,43.,44., 45.,46.,47.,48.,49., // NOLINT
50.,51.,52.,53.,54., 55.,56.,57.,58.,59., 60.,61.,62.,63.,64., 65.,66.,67.,68.,69., 70.,71.,72.,73.,74., // NOLINT
}
);
test.AddInput<float>("rois", {5, 4}, {7., 5., 7., 5., -15., -15., -15., -15., -10., 21., -10., 21., 13., 8., 13., 8., -14., 19., -14., 19.}); // NOLINT
test.AddInput<int64_t>("batch_indices", {5}, {0, 0, 0, 0, 0});
test.AddOutput<float>("Y", {5,3,3,4}, {2.10938f,2.95313f,3.375f,2.53125f,3.35938f,4.70313f,5.375f,4.03125f,3.51563f,4.92188f,5.625f,
4.21875f,10.8984f,15.2578f,17.4375f,13.0781f,17.3568f,24.2995f,27.7708f,20.8281f,18.1641f,
25.4297f,29.0625f,21.7969f,19.6875f,27.5625f,31.5f,23.625f,31.3542f,43.8958f,50.1667f,37.625f,
32.8125f,45.9375f,52.5f,39.375f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,0.f,25.f,25.f,25.f,25.f,25.f,25.f,25.f,
25.f,25.f,25.f,25.f,25.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,50.f,5.625f,5.625f,5.625f,4.57031f,
8.95833f,8.95833f,8.95833f,7.27865f,9.375f,9.375f,9.375f,7.61719f,19.6875f,19.6875f,19.6875f,15.9961f,
31.3542f,31.3542f,31.3542f,25.4753f,32.8125f,32.8125f,32.8125f,26.6602f,33.75f,33.75f,33.75f,27.4219f,
53.75f,53.75f,53.75f,43.6719f,56.25f,56.25f,56.25f,45.7031f,4.5f,3.9375f,2.8125f,3.9375f,5.5f,4.8125f,
3.4375f,4.8125f,4.58333f,4.01042f,2.86458f,3.9375f,23.25f,20.3438f,14.5313f,18.f,28.4167f,24.86458f,
17.76042f,22.f,23.25f,20.3437f,14.5312f,18.f,42.f,36.75f,26.25f,32.0625f,51.3333f,44.9167f,32.08333f,39.1875f,
42.f,36.75f,26.25f,32.0625f,4.375f,4.375f,4.375f,4.375f,7.70833f,7.70833f,7.70833f,7.70833f,9.375f,
9.375f,9.375f,9.375f,21.875f,21.875f,21.875f,21.875f,26.9792f,26.9792f,26.9792f,26.9792f,32.8125f,
32.8125f,32.8125f,32.8125f,40.1042f,40.1042f,40.1042f,40.1042f,46.25f,46.25f,46.25f,46.25f,56.25f,56.25f,
56.25f,56.25f});
test.AddOutput<float>(
"Y",
{5, 3, 3, 4},
{
2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000, 2.0000,
27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000, 27.0000,
52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000, 52.0000,

0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000, 25.0000,
50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000, 50.0000,

6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625, 6.5625,
31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625, 31.5625,
56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625, 56.5625,

3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125, 3.3125,
28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125, 28.3125,
53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125, 53.3125,

5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375, 5.9375,
30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375, 30.9375,
55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375, 55.9375,
}
); // NOLINT

test.Run();
}
Expand Down