Programmable camera distortion #3039

saihv · 2020-09-25T02:30:47Z

Introduces programmable camera distortion through a distortion material and a material parameter collection hosting the Brown distortion coefficients (3 radial, 2 tangential). Coefficients can be changed dynamically through the API.

Currently only works for the scene (RGB) camera.

m1baldwin · 2020-10-02T15:58:44Z

@saihv Will there be a separate PR to also allow the ROS wrapper for airsim to retrieve the distortion parameters of the camera and publish them in the 'D' field of the sensor_msgs/Image message that it publishes per-camera?

ironclownfish

This looks good to me! There are a couple unnecessary line deletes/adds here and there, but in my opinion we don't need to worry so much about a small amount of such things.
Cool feature!

saihv · 2020-10-03T00:08:29Z

@m1baldwin I just added a getDistortionParams() API to retrieve the current distortion params of a camera. Perhaps you can reference this function from the ROS side as part of your PR to populate the distortion related fields?

PythonClient/airsim/client.py

rajat2004 · 2020-10-03T09:11:05Z

Unreal/Plugins/AirSim/Source/SimMode/SimModeBase.cpp

@@ -201,7 +200,7 @@ void ASimModeBase::initializeTimeOfDay()
        sky_sphere_ = sky_spheres[0];
        static const FName sun_prop_name(TEXT("Directional light actor"));
        auto* p = sky_sphere_class_->FindPropertyByName(sun_prop_name);
-        UObjectProperty* sun_prop = Cast<UObjectProperty>(p);
+        FObjectProperty* sun_prop = Cast<UObjectProperty>(p);


This change breaks UE 4.24, which is okay IMO, just need to update docs to 4.25
It still produces warnings on 4.25, so it should be fixed completely like
FObjectProperty* sun_prop = CastField<FObjectProperty>(p)

Good catch! I will revert this, because officially we still wish to support older UE versions.

Sure, that's also good, however I think the latest PRs like the segmentation one and even this has asset modifications which were done on 4.25 I assume, and wouldn't work in 4.24. Need to test all this though
While trying to maintain compatibility with 4.24, it might be good to update the docs to recommend 4.25, since some features could stop working on 4.24 and also raise support questions. Currently docs say 4.22 is supported, which I'm pretty sure is not the case

rajat2004 · 2020-10-03T09:30:09Z

Also, this would normally be caught by Travis, but needs some non-implemented methods in Unity/PawnSimApi so as to not break Unity compilation
https://travis-ci.com/github/rajat2004/AirSim/builds/187945054

m1baldwin · 2020-10-05T00:11:22Z

@m1baldwin I just added a getDistortionParams() API to retrieve the current distortion params of a camera. Perhaps you can reference this function from the ROS side as part of your PR to populate the distortion related fields?

@saihv sure, I think this is a better approach to use the new api rather than how I have it hard coded currently. I need to update the PR.

m1baldwin · 2020-10-05T00:12:31Z

Is it fine to use “plumb bob” model as the distortion model string here?

saihv · 2020-10-05T21:50:30Z

@m1baldwin Yes, plumb bob is appropriate.

m1baldwin · 2020-10-06T19:49:42Z

@saihv What is the difference between the simGetDistortion and getDistortion methods you have added? And, would the ros wrapper need to select between the two?

Update on this; it seems I am getting a seg fault when I try this in the ros wrapper. I am using simGetDistortion. When I try with the python API, I get "AttributeError: 'MultirotorClient' object has no attribute 'simGetDistortionParams'"

saihv · 2020-10-07T03:43:51Z

@m1baldwin Can you doublecheck if you're using the right Python client? (i.e., the one in this branch and not another pre-installed version)

simGetDistortionParams is the user-accessible API call, getDistortionParams is its internal corresponding method. The ROS wrapper would use the simGetDistortionParams one.

rajat2004 · 2020-10-07T06:00:49Z

@m1baldwin Which version of UE are you using? I think you'll have to use the version used by @saihv (probably 4.25.3) or later, since earlier versions will most likely fail to load the assets
As for the Python API, you'll have to use the files in this branch, You could create a file in the multirotor/, etc folders and use import setup_path. Another way which I use is to install airsim using pip install -e . from Pythonclient folder, that'll use the local package. Useful if changing branches, etc. since no need to use setup_path for quick tests

m1baldwin · 2020-10-07T12:22:16Z

@rajat2004 I had upgraded to UE4.25.3 for the purpose of testing this PR with the ros wrapper. Everything seems to run and build fine. For the python client, might be that I am not using the changes in the local, so I will check that. That will be the first thing I check. I merged this PR into my PR I had been testing to bring them to the same place, I had made minimal changes so what I am testing now seems to be mostly this PR.

Assuming this isn't an issue though for the cpp codes, I wonder why I am getting a seg fault?

Also, I have noticed that if I remove the simGetDistortion API from the wrapper, and then run the wrapper as is, seemingly randomly the simulation is breaking. In VScode, it is throwing a debug breakpoint at line 494 in the enableCapture function in PIPCamera (https://github.com/saihv/AirSim/blob/PR/distortion/Unreal/Plugins/AirSim/Source/PIPCamera.cpp#L494). This breakout only happens when I run something which is continuously calling the simGetImages API (ros_wrapper in this case)

rajat2004 · 2020-10-07T15:39:26Z

@m1baldwin Could you perhaps try out #3064? It might be causing the problem, it reverts the PR #2881, I'm not sure why this didn't appear when I tested with 10-15 min runs on my system, and with other testers as well. But seems to be same as here, so reverted the PR

m1baldwin · 2020-10-07T15:47:26Z

@rajat2004 sure, I could test with that next. Does it work with ue4.25.3?

rajat2004 · 2020-10-07T15:49:42Z

Yeah, it only has C++ code change, no asset modifications so works with any UE version

m1baldwin · 2020-10-07T15:54:16Z

@rajat2004 okay. It is weird, because I thought I had been using the latest master anyway before this, and I wasn't seeing this failure either. It was only when I tried this PR that the issue popped up. Maybe I was using a version of master from before Sep. 2? (edit, I was using the version from sep.2 with the image capture rate improvement)

m1baldwin · 2020-10-07T17:45:47Z

@rajat2004 Just to confirm, PR #3064 does fix the issue for me. The weird thing is I went back and double checked what I had been running previously. All last week, I was running the latest master which included your original PR to improve the image capture rate. For some reason after testing this PR, I started seeing the failure I described above.

I tried resetting everything to what I had last week, but now I see the failure still. Testing the PR you recommended works, but now of course my frame rate is low again. I am going to need to spend some more time testing as I will need the higher fps for my work.

tool3210

It looks good.

jonyMarino · 2020-12-15T22:14:08Z

Hi @saihv. We are getting this from Unity build:

C:\Users\USER\source\repos\AirSim\Unity\AirLibWrapper\AirsimWrapper\Source\PawnSimApi.cpp(231): error C4716: 'PawnSimApi::getDistortionParams': must return a value [C:\Users\USER\source\repos\AirSim\Unity\AirLi
bWrapper\AirsimWrapper\AirsimWrapper.vcxproj]

saihv · 2020-12-15T23:25:01Z

@jonyMarino Fixed

jonyMarino · 2020-12-17T15:13:42Z

Thank you, Sai! Great contribution! I had fun testing it.

saihv added 6 commits September 23, 2020 17:10

API for distortion parameters

4fa2107

Initial version of camera distortion material and API

08ee44c

Update PythonClient, remove distortion postprocess at EndPlay()

cf8d7fb

Remove irrelevant files

42f5aec

Refactor - move distortion params to PIPCamera

d736207

Remove unneeded includes

4aaa6e3

saihv requested review from ironclownfish and madratman September 25, 2020 02:30

Fix includes

eab194e

ironclownfish approved these changes Oct 2, 2020

View reviewed changes

Add missing material instancing, API to retrieve distortion params

db5e99e

rajat2004 reviewed Oct 3, 2020

View reviewed changes

PythonClient/airsim/client.py Outdated Show resolved Hide resolved

Fix docstring

ed31286

saihv removed the request for review from madratman October 3, 2020 07:25

rajat2004 reviewed Oct 3, 2020

View reviewed changes

m1baldwin mentioned this pull request Oct 6, 2020

Add distortion model and distortion params to ros wrapper #3023

Draft

Revert change for compatibility

e41becf

rajat2004 mentioned this pull request Oct 7, 2020

Disable cameras after fetching images, projection matrix #2881

Merged

tool3210 approved these changes Oct 30, 2020

View reviewed changes

saihv added 4 commits December 9, 2020 01:26

Add placeholders for Unity compilation

e5967a0

Revert irrelevant changes

57f458e

Fix typo

5b3e1cd

Fix returntype Unity

ded679f

jonyMarino closed this Dec 15, 2020

jonyMarino reopened this Dec 15, 2020

Fix unimpl Unity methods

2667c3a

jonyMarino merged commit 5196910 into microsoft:master Dec 17, 2020

ahmed-elsaharti mentioned this pull request Jan 7, 2021

Some problem about the Unitydemo/carDemo #3276

Closed

Chanchanhan mentioned this pull request Feb 2, 2021

server could not find function 'simGetDistortionParams' with argument count 2 #3362

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Programmable camera distortion #3039

Programmable camera distortion #3039

saihv commented Sep 25, 2020

m1baldwin commented Oct 2, 2020

ironclownfish left a comment

saihv commented Oct 3, 2020

rajat2004 Oct 3, 2020

saihv Oct 5, 2020

rajat2004 Oct 6, 2020

rajat2004 commented Oct 3, 2020

m1baldwin commented Oct 5, 2020

m1baldwin commented Oct 5, 2020

saihv commented Oct 5, 2020

m1baldwin commented Oct 6, 2020 •

edited

Loading

saihv commented Oct 7, 2020 •

edited

Loading

rajat2004 commented Oct 7, 2020

m1baldwin commented Oct 7, 2020

rajat2004 commented Oct 7, 2020

m1baldwin commented Oct 7, 2020

rajat2004 commented Oct 7, 2020

m1baldwin commented Oct 7, 2020 •

edited

Loading

m1baldwin commented Oct 7, 2020

tool3210 left a comment

jonyMarino commented Dec 15, 2020

saihv commented Dec 15, 2020

jonyMarino commented Dec 17, 2020

Programmable camera distortion #3039

Programmable camera distortion #3039

Conversation

saihv commented Sep 25, 2020

m1baldwin commented Oct 2, 2020

ironclownfish left a comment

Choose a reason for hiding this comment

saihv commented Oct 3, 2020

rajat2004 Oct 3, 2020

Choose a reason for hiding this comment

saihv Oct 5, 2020

Choose a reason for hiding this comment

rajat2004 Oct 6, 2020

Choose a reason for hiding this comment

rajat2004 commented Oct 3, 2020

m1baldwin commented Oct 5, 2020

m1baldwin commented Oct 5, 2020

saihv commented Oct 5, 2020

m1baldwin commented Oct 6, 2020 • edited Loading

saihv commented Oct 7, 2020 • edited Loading

rajat2004 commented Oct 7, 2020

m1baldwin commented Oct 7, 2020

rajat2004 commented Oct 7, 2020

m1baldwin commented Oct 7, 2020

rajat2004 commented Oct 7, 2020

m1baldwin commented Oct 7, 2020 • edited Loading

m1baldwin commented Oct 7, 2020

tool3210 left a comment

Choose a reason for hiding this comment

jonyMarino commented Dec 15, 2020

saihv commented Dec 15, 2020

jonyMarino commented Dec 17, 2020

m1baldwin commented Oct 6, 2020 •

edited

Loading

saihv commented Oct 7, 2020 •

edited

Loading

m1baldwin commented Oct 7, 2020 •

edited

Loading