diff --git a/build_linux/index.html b/build_linux/index.html index 69ee867dff..52646639a0 100644 --- a/build_linux/index.html +++ b/build_linux/index.html @@ -310,13 +310,14 @@

Linux - Build Unreal Engine# go to the folder where you clone GitHub projects +git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git +cd UnrealEngine +./Setup.sh +./GenerateProjectFiles.sh +make + +

macOS - Download Unreal Engine#

  1. Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required.
    2. @@ -329,14 +330,17 @@

    Build AirSim
  2. Clone AirSim and build it:
    3. -

    bash - # go to the folder where you clone GitHub projects - git clone https://github.com/Microsoft/AirSim.git - cd AirSim

    -

    By default AirSim recommends using clang 8 to build the binaries as those will be compatible with UE 4.24. The setup script will install the right version of cmake, llvm, and eigen.

    -

    bash - ./setup.sh - ./build.sh

    +
    # go to the folder where you clone GitHub projects
+git clone https://github.com/Microsoft/AirSim.git
+cd AirSim
+
    + +

    By default AirSim uses clang 8 to build for compatibility with UE 4.24. The setup script will install the right version of cmake, llvm, and eigen.

    +
    ./setup.sh
+./build.sh
+# use ./build.sh --debug to build in debug mode
+
    +

    Build Unreal Environment#

    Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in "Blocks Environment" which you can use, or you can create your own. Please see setting up Unreal Environment if you'd like to setup your own environment.

    How to Use AirSim#

    diff --git a/search/search_index.json b/search/search_index.json index cb288b0637..5567e10e51 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to AirSim # AirSim is a simulator for drones, cars and more, built on Unreal Engine (we now also have an experimental Unity release). It is open-source, cross platform, and supports hardware-in-loop with popular flight controllers such as PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. Similarly, we have an experimental release for a Unity plugin. Our goal is to develop AirSim as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles. For this purpose, AirSim also exposes APIs to retrieve data and control vehicles in a platform independent way. Check out the quick 1.5 minute demo Drones in AirSim Cars in AirSim What's New # Latest release v1.3.1 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added support for docker in ubuntu For complete list of changes, view our Changelog How to Get It # Windows # Download binaries Build it Linux # Download binaries Build it macOS # Build it How to Use It # Documentation # View our detailed documentation on all aspects of AirSim. Manual drive # If you have remote control (RC) as shown below, you can manually control the drone in the simulator. For cars, you can use arrow keys to drive manually. More details Programmatic control # AirSim exposes APIs so you can interact with the vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on. The APIs are exposed through the RPC, and are accessible via a variety of languages, including C++, Python, C# and Java. These APIs are also available as part of a separate, independent cross-platform library, so you can deploy them on a companion computer on your vehicle. This way you can write and test your code in the simulator, and later execute it on the real vehicles. Transfer learning and related research is one of our focus areas. Note that you can use SimMode setting to specify the default vehicle or the new ComputerVision mode so you don't get prompted each time you start AirSim. More details Gathering training data # There are two ways you can generate training data from AirSim for deep learning. The easiest way is to simply press the record button in the lower right corner. This will start writing pose and images for each frame. The data logging code is pretty simple and you can modify it to your heart's content. A better way to generate training data exactly the way you want is by accessing the APIs. This allows you to be in full control of how, what, where and when you want to log data. Computer Vision mode # Yet another way to use AirSim is the so-called \"Computer Vision\" mode. In this mode, you don't have vehicles or physics. You can use the keyboard to move around the scene, or use APIs to position available cameras in any arbitrary pose, and collect images such as depth, disparity, surface normals or object segmentation. More details Weather Effects # Press F10 to see various options available for weather effects. You can also control the weather using APIs . Press F1 to see other options available. Tutorials # Video - Setting up AirSim with Pixhawk Tutorial by Chris Lovett Video - Using AirSim with Pixhawk Tutorial by Chris Lovett Video - Using off-the-self environments with AirSim by Jim Piavis Reinforcement Learning with AirSim by Ashish Kapoor The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter Using TensorFlow for simple collision avoidance by Simon Levy and WLU team Participate # Paper # More technical details are available in AirSim paper (FSR 2017 Conference) . Please cite this as: @inproceedings{airsim2017fsr, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, title = {AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles}, year = {2017}, booktitle = {Field and Service Robotics}, eprint = {arXiv:1705.05065}, url = {https://arxiv.org/abs/1705.05065} } Contribute # Please take a look at open issues if you are looking for areas to contribute to. More on AirSim design More on code structure Contribution Guidelines Trello Board Who is Using AirSim? # We are maintaining a list of a few projects, people and groups that we are aware of. If you would like to be featured in this list please make a request here . Contact # Join the AirSim group on Facebook to stay up to date or ask any questions. FAQ # If you run into problems, check the FAQ and feel free to post issues in the AirSim repository. License # This project is released under the MIT License. Please review the License file for more details.","title":"Home"},{"location":"#welcome-to-airsim","text":"AirSim is a simulator for drones, cars and more, built on Unreal Engine (we now also have an experimental Unity release). It is open-source, cross platform, and supports hardware-in-loop with popular flight controllers such as PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. Similarly, we have an experimental release for a Unity plugin. Our goal is to develop AirSim as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles. For this purpose, AirSim also exposes APIs to retrieve data and control vehicles in a platform independent way. Check out the quick 1.5 minute demo Drones in AirSim Cars in AirSim","title":"Welcome to AirSim"},{"location":"#whats-new","text":"Latest release v1.3.1 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added support for docker in ubuntu For complete list of changes, view our Changelog","title":"What's New"},{"location":"#how-to-get-it","text":"","title":"How to Get It"},{"location":"#windows","text":"Download binaries Build it","title":"Windows"},{"location":"#linux","text":"Download binaries Build it","title":"Linux"},{"location":"#macos","text":"Build it","title":"macOS"},{"location":"#how-to-use-it","text":"","title":"How to Use It"},{"location":"#documentation","text":"View our detailed documentation on all aspects of AirSim.","title":"Documentation"},{"location":"#manual-drive","text":"If you have remote control (RC) as shown below, you can manually control the drone in the simulator. For cars, you can use arrow keys to drive manually. More details","title":"Manual drive"},{"location":"#programmatic-control","text":"AirSim exposes APIs so you can interact with the vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on. The APIs are exposed through the RPC, and are accessible via a variety of languages, including C++, Python, C# and Java. These APIs are also available as part of a separate, independent cross-platform library, so you can deploy them on a companion computer on your vehicle. This way you can write and test your code in the simulator, and later execute it on the real vehicles. Transfer learning and related research is one of our focus areas. Note that you can use SimMode setting to specify the default vehicle or the new ComputerVision mode so you don't get prompted each time you start AirSim. More details","title":"Programmatic control"},{"location":"#gathering-training-data","text":"There are two ways you can generate training data from AirSim for deep learning. The easiest way is to simply press the record button in the lower right corner. This will start writing pose and images for each frame. The data logging code is pretty simple and you can modify it to your heart's content. A better way to generate training data exactly the way you want is by accessing the APIs. This allows you to be in full control of how, what, where and when you want to log data.","title":"Gathering training data"},{"location":"#computer-vision-mode","text":"Yet another way to use AirSim is the so-called \"Computer Vision\" mode. In this mode, you don't have vehicles or physics. You can use the keyboard to move around the scene, or use APIs to position available cameras in any arbitrary pose, and collect images such as depth, disparity, surface normals or object segmentation. More details","title":"Computer Vision mode"},{"location":"#weather-effects","text":"Press F10 to see various options available for weather effects. You can also control the weather using APIs . Press F1 to see other options available.","title":"Weather Effects"},{"location":"#tutorials","text":"Video - Setting up AirSim with Pixhawk Tutorial by Chris Lovett Video - Using AirSim with Pixhawk Tutorial by Chris Lovett Video - Using off-the-self environments with AirSim by Jim Piavis Reinforcement Learning with AirSim by Ashish Kapoor The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter Using TensorFlow for simple collision avoidance by Simon Levy and WLU team","title":"Tutorials"},{"location":"#participate","text":"","title":"Participate"},{"location":"#paper","text":"More technical details are available in AirSim paper (FSR 2017 Conference) . Please cite this as: @inproceedings{airsim2017fsr, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, title = {AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles}, year = {2017}, booktitle = {Field and Service Robotics}, eprint = {arXiv:1705.05065}, url = {https://arxiv.org/abs/1705.05065} }","title":"Paper"},{"location":"#contribute","text":"Please take a look at open issues if you are looking for areas to contribute to. More on AirSim design More on code structure Contribution Guidelines Trello Board","title":"Contribute"},{"location":"#who-is-using-airsim","text":"We are maintaining a list of a few projects, people and groups that we are aware of. If you would like to be featured in this list please make a request here .","title":"Who is Using AirSim?"},{"location":"#contact","text":"Join the AirSim group on Facebook to stay up to date or ask any questions.","title":"Contact"},{"location":"#faq","text":"If you run into problems, check the FAQ and feel free to post issues in the AirSim repository.","title":"FAQ"},{"location":"#license","text":"This project is released under the MIT License. Please review the License file for more details.","title":"License"},{"location":"CHANGELOG/","text":"What's new # Below is summarized list of important changes. This does not include minor/less important changes or bug fixes or documentation update. This list updated every few months. For complete detailed changes, please review commit history . April 2020 # Fix issues with PX4 latest master branch Fix Lidar DrawDebugPoints causing crash Add docstrings for Python API Add missing noise, weather texture materials Update AirSim.uplugin version to 1.3.1 Camera Roll angle control using Q,E keys in CV mode, manual camera Remove broken GCC build New API - simSetTraceLine() ROS package compilation fixes and updates Latest release v1.3.1 for Windows and Linux APIs added and fixed - simSetCameraFov , rotateToYaw airsim Python package update to 1.2.8 NoDisplay ViewMode render state fix March 2020 # Latest release v1.3.0 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging Low-level Multirotor APIs Updated Eigen version to 3.3.7 Distance Sensor API fix Add simSwapTextures API Fix simContinueForTime , simPause APIs Lidar Sensor Trace Casting fix Fix rare reset() bug which causes Unreal crash Lidar sensor improvements, add simGetLidarSegmentation API Add RpcLibPort in settings Recording thread deadlock fix Prevent environment crash when Sun is not present Africa Tracking feautre, add simListSceneObjects() API, fix camera projection matrix ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added sensor APIs for Barometer, IMU, GPS, Magnetometer, Distance Sensor Added support for docker in ubuntu November, 2018 # Added Weather Effects and APIs Added Time of Day API An experimental integration of AirSim on Unity is now available. Learn more in Unity blog post . New environments : Forest, Plains (windmill farm), TalkingHeads (human head simulation), TrapCam (animal detection via camera) Highly efficient NoDisplay view mode to turn off main screen rendering so you can capture images at high rate Enable/disable sensors via settings Lidar Sensor Support for Flysky FS-SM100 RC USB adapter Case Study: Formula Student Technion Driverless Multi-Vehicle Capability Custom speed units ROS publisher simSetObjectPose API Character Control APIs (works on TalkingHeads binaries in release) Arducopter Solo Support Linux install without sudo access Kinect like ROS publisher June, 2018 # Development workflow doc Better Python 2 compatibility OSX setup fixes Almost complete rewrite of our APIs with new threading model, merging old APIs and creating few newer ones April, 2018 # Upgraded to Unreal Engine 4.18 and Visual Studio 2017 API framework refactoring to support world-level APIs Latest PX4 firmware now supported CarState with more information ThrustMaster wheel support pause and continueForTime APIs for drone as well as car Allow drone simulation run at higher clock rate without any degradation Forward-only mode fully functional for drone (do orbits while looking at center) Better PID tuning to reduce wobble for drones Ability to set arbitrary vehicle blueprint for drone as well as car gimbal stabilization via settings Ability to segment skinned and skeletal meshes by their name moveByAngleThrottle API Car physics tuning for better maneuverability Configure additional cameras via settings Time of day with geographically computed sun position Better car steering via keyboard Added MeshNamingMethod in segmentation setting gimbal API getCameraParameters API Ability turn off main rendering to save GPU resources Projection mode for capture settings getRCData, setRCData APIs Ability to turn off segmentation using negative IDs OSX build improvements Segmentation working for very large environments with initial IDs Better and extensible hash calculation for segmentation IDs Extensible PID controller for custom integration methods Sensor architecture now enables renderer specific features like ray casting Laser altimeter sensor Jan 2018 # Config system rewrite, enable flexible config we are targeting in future Multi-Vehicle support Phase 1, core infrastructure changes MacOS support Infrared view 5 types of noise and interference for cameras WYSIWIG capture settings for cameras, preview recording settings in main view Azure support Phase 1, enable configurability of instances for headless mode Full kinematics APIs, ability to get pose, linear and angular velocities + accelerations via APIs Record multiple images from multiple cameras New segmentation APIs, ability to set configure object IDs, search via regex New object pose APIs, ability to get pose of objects (like animals) in environment Camera infrastructure enhancements, ability to add new image types like IR with just few lines Clock speed APIs for drone as well as car, simulation can be run with speed factor of 0 < x < infinity Support for Logitech G920 wheel Physics tuning of the car, Car doesn\u2019t roll over, responds to steering with better curve, releasing gas paddle behavior more realistic Debugging APIs Stress tested to 24+ hours of continuous runs Support for Landscape and sky segmentation Manual navigation with accelerated controls in CV mode, user can explore environment much more easily Collison APIs Recording enhancements, log several new data points including ground truth, multiple images, controls state Planner and Perspective Depth views Disparity view New Image APIs supports float, png or numpy formats 6 config settings for image capture, ability to set auto-exposure, motion blur, gamma etc Full multi-camera support through out including sub-windows, recording, APIs etc Command line script to build all environments in one shot Remove submodules, use rpclib as download Nov 2017 # We now have the car model . No need to build the code. Just download binaries and you are good to go! The reinforcement learning example with AirSim New built-in flight controller called simple_flight that \"just works\" without any additional setup. It is also now default . AirSim now also generates depth as well as disparity images that is in camera plan. We also have official Linux build now! Sep 2017 # We have added car model ! Aug 2017 # simple_flight is now default flight controller for drones. If you want to use PX4, you will need to modify settings.json as per PX4 setup doc . Linux build is official and currently uses Unreal 4.17 due to various bug fixes required ImageType enum has breaking changes with several new additions and clarifying existing ones SubWindows are now configurable from settings.json PythonClient is now complete and has parity with C++ APIs. Some of these would have breaking changes. Feb 2017 # First release!","title":"Changelog"},{"location":"CHANGELOG/#whats-new","text":"Below is summarized list of important changes. This does not include minor/less important changes or bug fixes or documentation update. This list updated every few months. For complete detailed changes, please review commit history .","title":"What's new"},{"location":"CHANGELOG/#april-2020","text":"Fix issues with PX4 latest master branch Fix Lidar DrawDebugPoints causing crash Add docstrings for Python API Add missing noise, weather texture materials Update AirSim.uplugin version to 1.3.1 Camera Roll angle control using Q,E keys in CV mode, manual camera Remove broken GCC build New API - simSetTraceLine() ROS package compilation fixes and updates Latest release v1.3.1 for Windows and Linux APIs added and fixed - simSetCameraFov , rotateToYaw airsim Python package update to 1.2.8 NoDisplay ViewMode render state fix","title":"April 2020"},{"location":"CHANGELOG/#march-2020","text":"Latest release v1.3.0 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging Low-level Multirotor APIs Updated Eigen version to 3.3.7 Distance Sensor API fix Add simSwapTextures API Fix simContinueForTime , simPause APIs Lidar Sensor Trace Casting fix Fix rare reset() bug which causes Unreal crash Lidar sensor improvements, add simGetLidarSegmentation API Add RpcLibPort in settings Recording thread deadlock fix Prevent environment crash when Sun is not present Africa Tracking feautre, add simListSceneObjects() API, fix camera projection matrix ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added sensor APIs for Barometer, IMU, GPS, Magnetometer, Distance Sensor Added support for docker in ubuntu","title":"March 2020"},{"location":"CHANGELOG/#november-2018","text":"Added Weather Effects and APIs Added Time of Day API An experimental integration of AirSim on Unity is now available. Learn more in Unity blog post . New environments : Forest, Plains (windmill farm), TalkingHeads (human head simulation), TrapCam (animal detection via camera) Highly efficient NoDisplay view mode to turn off main screen rendering so you can capture images at high rate Enable/disable sensors via settings Lidar Sensor Support for Flysky FS-SM100 RC USB adapter Case Study: Formula Student Technion Driverless Multi-Vehicle Capability Custom speed units ROS publisher simSetObjectPose API Character Control APIs (works on TalkingHeads binaries in release) Arducopter Solo Support Linux install without sudo access Kinect like ROS publisher","title":"November, 2018"},{"location":"CHANGELOG/#june-2018","text":"Development workflow doc Better Python 2 compatibility OSX setup fixes Almost complete rewrite of our APIs with new threading model, merging old APIs and creating few newer ones","title":"June, 2018"},{"location":"CHANGELOG/#april-2018","text":"Upgraded to Unreal Engine 4.18 and Visual Studio 2017 API framework refactoring to support world-level APIs Latest PX4 firmware now supported CarState with more information ThrustMaster wheel support pause and continueForTime APIs for drone as well as car Allow drone simulation run at higher clock rate without any degradation Forward-only mode fully functional for drone (do orbits while looking at center) Better PID tuning to reduce wobble for drones Ability to set arbitrary vehicle blueprint for drone as well as car gimbal stabilization via settings Ability to segment skinned and skeletal meshes by their name moveByAngleThrottle API Car physics tuning for better maneuverability Configure additional cameras via settings Time of day with geographically computed sun position Better car steering via keyboard Added MeshNamingMethod in segmentation setting gimbal API getCameraParameters API Ability turn off main rendering to save GPU resources Projection mode for capture settings getRCData, setRCData APIs Ability to turn off segmentation using negative IDs OSX build improvements Segmentation working for very large environments with initial IDs Better and extensible hash calculation for segmentation IDs Extensible PID controller for custom integration methods Sensor architecture now enables renderer specific features like ray casting Laser altimeter sensor","title":"April, 2018"},{"location":"CHANGELOG/#jan-2018","text":"Config system rewrite, enable flexible config we are targeting in future Multi-Vehicle support Phase 1, core infrastructure changes MacOS support Infrared view 5 types of noise and interference for cameras WYSIWIG capture settings for cameras, preview recording settings in main view Azure support Phase 1, enable configurability of instances for headless mode Full kinematics APIs, ability to get pose, linear and angular velocities + accelerations via APIs Record multiple images from multiple cameras New segmentation APIs, ability to set configure object IDs, search via regex New object pose APIs, ability to get pose of objects (like animals) in environment Camera infrastructure enhancements, ability to add new image types like IR with just few lines Clock speed APIs for drone as well as car, simulation can be run with speed factor of 0 < x < infinity Support for Logitech G920 wheel Physics tuning of the car, Car doesn\u2019t roll over, responds to steering with better curve, releasing gas paddle behavior more realistic Debugging APIs Stress tested to 24+ hours of continuous runs Support for Landscape and sky segmentation Manual navigation with accelerated controls in CV mode, user can explore environment much more easily Collison APIs Recording enhancements, log several new data points including ground truth, multiple images, controls state Planner and Perspective Depth views Disparity view New Image APIs supports float, png or numpy formats 6 config settings for image capture, ability to set auto-exposure, motion blur, gamma etc Full multi-camera support through out including sub-windows, recording, APIs etc Command line script to build all environments in one shot Remove submodules, use rpclib as download","title":"Jan 2018"},{"location":"CHANGELOG/#nov-2017","text":"We now have the car model . No need to build the code. Just download binaries and you are good to go! The reinforcement learning example with AirSim New built-in flight controller called simple_flight that \"just works\" without any additional setup. It is also now default . AirSim now also generates depth as well as disparity images that is in camera plan. We also have official Linux build now!","title":"Nov 2017"},{"location":"CHANGELOG/#sep-2017","text":"We have added car model !","title":"Sep 2017"},{"location":"CHANGELOG/#aug-2017","text":"simple_flight is now default flight controller for drones. If you want to use PX4, you will need to modify settings.json as per PX4 setup doc . Linux build is official and currently uses Unreal 4.17 due to various bug fixes required ImageType enum has breaking changes with several new additions and clarifying existing ones SubWindows are now configurable from settings.json PythonClient is now complete and has parity with C++ APIs. Some of these would have breaking changes.","title":"Aug 2017"},{"location":"CHANGELOG/#feb-2017","text":"First release!","title":"Feb 2017"},{"location":"CONTRIBUTING/","text":"Contributing # Quick Start # Please read our short and sweet coding guidelines . For big changes such as adding new feature or refactoring, file an issue first . Use our recommended development workflow to make changes and test it. Use usual steps to make contributions just like other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first . Checklist # Use same style and formatting as rest of code even if it's not your preferred one. Change any documentation that goes with code changes. Do not include OS specific header files or new 3rd party dependencies. Keep your pull request small, ideally under 10 files. Make sure you don't include large binary files. When adding new includes, make dependency is absolutely necessary. Rebase your branch frequently with master (once every 2-3 days is ideal). Make sure your code would compile on Windows, Linux and OSX.","title":"Contribute"},{"location":"CONTRIBUTING/#contributing","text":"","title":"Contributing"},{"location":"CONTRIBUTING/#quick-start","text":"Please read our short and sweet coding guidelines . For big changes such as adding new feature or refactoring, file an issue first . Use our recommended development workflow to make changes and test it. Use usual steps to make contributions just like other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first .","title":"Quick Start"},{"location":"CONTRIBUTING/#checklist","text":"Use same style and formatting as rest of code even if it's not your preferred one. Change any documentation that goes with code changes. Do not include OS specific header files or new 3rd party dependencies. Keep your pull request small, ideally under 10 files. Make sure you don't include large binary files. When adding new includes, make dependency is absolutely necessary. Rebase your branch frequently with master (once every 2-3 days is ideal). Make sure your code would compile on Windows, Linux and OSX.","title":"Checklist"},{"location":"InfraredCamera/","text":"This is a tutorial for generating simulated thermal infrared (IR) images using AirSim and the AirSim Africa environment. Pre-compiled Africa Environment can be downloaded from the Releases tab of this Github repo: Windows Pre-compiled binary To generate your own data, you may use two python files: create_ir_segmentation_map.py and capture_ir_segmentation.py . create_ir_segmentation_map.py uses temperature, emissivity, and camera response information to estimate the thermal digital count that could be expected for the objects in the environment, and then reassigns the segmentation IDs in AirSim to match these digital counts. It should be run before starting to capture thermal IR data. Otherwise, digital counts in the IR images will be incorrect. The camera response, temperature, and emissivity data are all included for the Africa environment. capture_ir_segmentation.py is run after the segmentation IDs have been reassigned. It tracks objects of interest and records the infrared and scene images from the multirotor. It uses Computer Vision mode. Finally, the details about how temperatures were estimated for plants and animals in the Africa environment, etc. can be found in this paper: @inproceedings{bondi2018airsim, title={AirSim-W: A Simulation Environment for Wildlife Conservation with UAVs}, author={Bondi, Elizabeth and Dey, Debadeepta and Kapoor, Ashish and Piavis, Jim and Shah, Shital and Fang, Fei and Dilkina, Bistra and Hannaford, Robert and Iyer, Arvind and Joppa, Lucas and others}, booktitle={Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies}, pages={40}, year={2018}, organization={ACM} } nb","title":"Infrared Camera"},{"location":"SUPPORT/","text":"Support # We highly recommend to take a look at source code and contribute to the project. Due to large number of incoming feature request we may not be able to get to your request in your desired timeframe. So please contribute :). Join AirSim Facebook Group File GitHub Issue","title":"Support"},{"location":"SUPPORT/#support","text":"We highly recommend to take a look at source code and contribute to the project. Due to large number of incoming feature request we may not be able to get to your request in your desired timeframe. So please contribute :). Join AirSim Facebook Group File GitHub Issue","title":"Support"},{"location":"Unity/","text":"AirSim on Unity # AirSim on Unity allows you to run your simulators in the Unity Engine . This project comes with some sample Unity projects and a wrapper around the AirLib library to run as a native plugin in Unity. Included are two basic Unity Projects, one for a Car simulator and another for a Drone simulator. They are meant to be lightweight, and can be used to verify your setup is correct. Check out the Unity blogpost for overview on the release. Warning: Experimental Release # This project is still in early development, expect some rough edges. We are working to fully support the full AirLib API and feature set, but some things may be missing. Click here for the list of currently supported APIs. Windows # Building from source # Install Unity # Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub from here . Detailed instructions here . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs. Build Airsim # Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start x64 Native Tools Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line. Build Unity Project # Go inside the AirSim\\Unity directory: cd Unity . Build the unity project: build.cmd . Additionally, there is a free environment Windridge City which you can download from Unity Asset Store . And, of course, you can always create your own environment. Linux # Dependencies # sudo apt-get install libboost-all-dev Download and Install Unity for Linux # Warning: Unity Editor for Linux is still in Beta. Expect some rough edges. Install Unity # Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs. Build Airsim # git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh Generate AirsimWrapper Shared Library # cd AirSim/Unity ./build.sh This will generate the necessary shared library and copy it to the UnityDemo Plugins folder. Usage # Start Unity Hub, click on Projects on left pane, and then click on the Add button Select the folder AirSim\\Unity\\UnityDemo , and then hit the OK button. Click on the new project which showed up in the Unity Hub menu to open it in Unity. In the bottom pane, click on Projects -> Assets -> Scenes . Then, Double-click on SimModeSelector . This will load the SimModeSelector scene into the scene hierarchy pane. DO NOT add CarDemo or DroneDemo scene into the scene hierarchy pane. Hit the play button to start the simulation (and hit play again to stop the simulation. . Alternatively, you can change the SimMode in your Settings.json file. (You can read more about Settings.json here ) Controlling the car: Use WASD or the Arrow keys or the AirSim client. Controlling the drone: Keyboard control is not currently available for drone flight. Changing camera views: Keys 0 , 1 , 2 , 3 are used to toggle windows of different camera views. Recording simulation data: Press Record button(Red button) located at the right bottom corner of the screen, to toggle recording of the simulation data. The recorded data can be found at Documents\\AirSim\\(Date of recording) on Windows and ~/Documents/AirSim/(Date of recording) on Linux. Building Custom Environments For AirSim # To use environments other than UnityDemo , follow the instructions written here Cross-Compiling to Linux # Unity Editor supports compiling projects to Linux systems. After following the steps to build AirSim and Unity on Windows, do the following: Linux Pre-Requisites # Before being able to run Unity Binaries with the Airsim plugin, be sure have airsim and airsim unity built on your linux machine by following the Linux build steps above. Package UnityDemo Binary On Windows # Install Necessary Components # In order to package your project for linux, the Linux Build Support Unity add-on must be installed. * Open Unity Hub , and click the Add component button in the dropdown window under more options to the right of your Unity 2018.2.15f1 tab. * Make sure the Linux Build Support Platform is selected Once this component is successfully installed, you are ready to build Unity Projects for Linux! Build the Project # On your Windows machine, build the Unity Demo by navigating to the build settings option in the toolbar File -> Build Settings Make sure the following scenes are set to be built: SimModeSelector CarDemo DroneDemo Set the target operating system to linux, and choose the version appropriate for your system (x86 vs x86_64) Click Build Transport the built project as well as the generated folder \"{project_name}_Data\" to your linux machine Copy The AirsimWrapper Library to the Project Plugins folder # On your linux machine, navigate to your AirSim repository, and run the following commands in a terminal window: cp Unity/linux-build/libAirsimWrapper.so path/to/your/project/{project_name}_Data/Plugins/{os_version} This will generate the necessary shared library to allow Airsim to communicate with Unity and copy it to the plugins folder of your project binary. Run the Project Binary # Open a terminal and navigate to your project directory Set your project binary as an executable file: chmod +x \"{project_name}.{configuration}\" Run the binary file ./{project_name}.{configuration} Using Airsim API # For quickstart with the Python APIs for the car or the drone, simply run the hello_car.py or the hello_drone.py script accordingly. Details of the AirSim C++ and Python APIs are here . Acknowledgements # The drone object was provided by user 31415926 on sketchfab . It is licensed under the CC License .","title":"AirSim with Unity"},{"location":"Unity/#airsim-on-unity","text":"AirSim on Unity allows you to run your simulators in the Unity Engine . This project comes with some sample Unity projects and a wrapper around the AirLib library to run as a native plugin in Unity. Included are two basic Unity Projects, one for a Car simulator and another for a Drone simulator. They are meant to be lightweight, and can be used to verify your setup is correct. Check out the Unity blogpost for overview on the release.","title":"AirSim on Unity"},{"location":"Unity/#warning-experimental-release","text":"This project is still in early development, expect some rough edges. We are working to fully support the full AirLib API and feature set, but some things may be missing. Click here for the list of currently supported APIs.","title":"Warning: Experimental Release"},{"location":"Unity/#windows","text":"","title":"Windows"},{"location":"Unity/#building-from-source","text":"","title":"Building from source"},{"location":"Unity/#install-unity","text":"Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub from here . Detailed instructions here . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs.","title":"Install Unity"},{"location":"Unity/#build-airsim","text":"Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start x64 Native Tools Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line.","title":"Build Airsim"},{"location":"Unity/#build-unity-project","text":"Go inside the AirSim\\Unity directory: cd Unity . Build the unity project: build.cmd . Additionally, there is a free environment Windridge City which you can download from Unity Asset Store . And, of course, you can always create your own environment.","title":"Build Unity Project"},{"location":"Unity/#linux","text":"","title":"Linux"},{"location":"Unity/#dependencies","text":"sudo apt-get install libboost-all-dev","title":"Dependencies"},{"location":"Unity/#download-and-install-unity-for-linux","text":"Warning: Unity Editor for Linux is still in Beta. Expect some rough edges.","title":"Download and Install Unity for Linux"},{"location":"Unity/#install-unity_1","text":"Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs.","title":"Install Unity"},{"location":"Unity/#build-airsim_1","text":"git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh","title":"Build Airsim"},{"location":"Unity/#generate-airsimwrapper-shared-library","text":"cd AirSim/Unity ./build.sh This will generate the necessary shared library and copy it to the UnityDemo Plugins folder.","title":"Generate AirsimWrapper Shared Library"},{"location":"Unity/#usage","text":"Start Unity Hub, click on Projects on left pane, and then click on the Add button Select the folder AirSim\\Unity\\UnityDemo , and then hit the OK button. Click on the new project which showed up in the Unity Hub menu to open it in Unity. In the bottom pane, click on Projects -> Assets -> Scenes . Then, Double-click on SimModeSelector . This will load the SimModeSelector scene into the scene hierarchy pane. DO NOT add CarDemo or DroneDemo scene into the scene hierarchy pane. Hit the play button to start the simulation (and hit play again to stop the simulation. . Alternatively, you can change the SimMode in your Settings.json file. (You can read more about Settings.json here ) Controlling the car: Use WASD or the Arrow keys or the AirSim client. Controlling the drone: Keyboard control is not currently available for drone flight. Changing camera views: Keys 0 , 1 , 2 , 3 are used to toggle windows of different camera views. Recording simulation data: Press Record button(Red button) located at the right bottom corner of the screen, to toggle recording of the simulation data. The recorded data can be found at Documents\\AirSim\\(Date of recording) on Windows and ~/Documents/AirSim/(Date of recording) on Linux.","title":"Usage"},{"location":"Unity/#building-custom-environments-for-airsim","text":"To use environments other than UnityDemo , follow the instructions written here","title":"Building Custom Environments For AirSim"},{"location":"Unity/#cross-compiling-to-linux","text":"Unity Editor supports compiling projects to Linux systems. After following the steps to build AirSim and Unity on Windows, do the following:","title":"Cross-Compiling to Linux"},{"location":"Unity/#linux-pre-requisites","text":"Before being able to run Unity Binaries with the Airsim plugin, be sure have airsim and airsim unity built on your linux machine by following the Linux build steps above.","title":"Linux Pre-Requisites"},{"location":"Unity/#package-unitydemo-binary-on-windows","text":"","title":"Package UnityDemo Binary On Windows"},{"location":"Unity/#install-necessary-components","text":"In order to package your project for linux, the Linux Build Support Unity add-on must be installed. * Open Unity Hub , and click the Add component button in the dropdown window under more options to the right of your Unity 2018.2.15f1 tab. * Make sure the Linux Build Support Platform is selected Once this component is successfully installed, you are ready to build Unity Projects for Linux!","title":"Install Necessary Components"},{"location":"Unity/#build-the-project","text":"On your Windows machine, build the Unity Demo by navigating to the build settings option in the toolbar File -> Build Settings Make sure the following scenes are set to be built: SimModeSelector CarDemo DroneDemo Set the target operating system to linux, and choose the version appropriate for your system (x86 vs x86_64) Click Build Transport the built project as well as the generated folder \"{project_name}_Data\" to your linux machine","title":"Build the Project"},{"location":"Unity/#copy-the-airsimwrapper-library-to-the-project-plugins-folder","text":"On your linux machine, navigate to your AirSim repository, and run the following commands in a terminal window: cp Unity/linux-build/libAirsimWrapper.so path/to/your/project/{project_name}_Data/Plugins/{os_version} This will generate the necessary shared library to allow Airsim to communicate with Unity and copy it to the plugins folder of your project binary.","title":"Copy The AirsimWrapper Library to the Project Plugins folder"},{"location":"Unity/#run-the-project-binary","text":"Open a terminal and navigate to your project directory Set your project binary as an executable file: chmod +x \"{project_name}.{configuration}\" Run the binary file ./{project_name}.{configuration}","title":"Run the Project Binary"},{"location":"Unity/#using-airsim-api","text":"For quickstart with the Python APIs for the car or the drone, simply run the hello_car.py or the hello_drone.py script accordingly. Details of the AirSim C++ and Python APIs are here .","title":"Using Airsim API"},{"location":"Unity/#acknowledgements","text":"The drone object was provided by user 31415926 on sketchfab . It is licensed under the CC License .","title":"Acknowledgements"},{"location":"airsim_ros_pkgs/","text":"airsim_ros_pkgs # A ROS wrapper over the AirSim C++ client library. Setup # Install gcc >= 8.0.0 sudo apt-get install gcc-8 g++-8 Verify version by gcc --version Ubuntu 16.04 Install ROS kinetic Install tf2 sensor and mavros packages: sudo apt-get install ros-kinetic-tf2-sensor-msgs ros-kinetic-mavros* Ubuntu 18.04 Install ROS melodic Install tf2 sensor and mavros packages: sudo apt-get install ros-melodic-tf2-sensor-msgs ros-melodic-mavros* Install catkin_tools sudo apt-get install python-catkin-tools or pip install catkin_tools Build # Build AirSim git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh; Build ROS package cd ros; catkin build; # or catkin_make If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 Running # source devel/setup.bash; roslaunch airsim_ros_pkgs airsim_node.launch; roslaunch airsim_ros_pkgs rviz.launch; Using AirSim ROS wrapper # The ROS wrapper is composed of two ROS nodes - the first is a wrapper over AirSim's multirotor C++ client library, and the second is a simple PD position controller. Let's look at the ROS API for both nodes: AirSim ROS Wrapper Node # Publishers: # /airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw GPS coordinates corresponding to global NED frame. This is set in the airsim's settings.json file under the OriginGeopoint key. /airsim_node/VEHICLE_NAME/global_gps sensor_msgs/NavSatFix This the current GPS coordinates of the drone in airsim. /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Odometry in NED frame (default name: odom_local_ned, launch name and frame type are configurable) wrt take-off point. /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE/camera_info sensor_msgs/CameraInfo /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE sensor_msgs/Image RGB or float image depending on image type requested in settings.json. /tf tf2_msgs/TFMessage /airsim_node/VEHICLE_NAME/altimeter/SENSOR_NAME [airsim_ros_pkgs::Altimeter] This the current altimeter reading for altitude, pressure, and QNH (https://en.wikipedia.org/wiki/QNH) /airsim_node/VEHICLE_NAME/imu/SENSOR_NAME [sensor_msgs::Imu] (http://docs.ros.org/api/sensor_msgs/html/msg/Imu.html) IMU sensor data /airsim_node/VEHICLE_NAME/magnetometer/SENSOR_NAME [sensor_msgs::MagneticField] (http://docs.ros.org/api/sensor_msgs/html/msg/MagneticField.html) Meausrement of magnetic field vector/compass /airsim_node/VEHICLE_NAME/distance/SENSOR_NAME [sensor_msgs::Range] (http://docs.ros.org/api/sensor_msgs/html/msg/Range.html) Meausrement of distance from an active ranger, such as infrared or IR /airsim_node/VEHICLE_NAME/lidar/SENSOR_NAME [sensor_msgs::PointCloud2] (http://docs.ros.org/api/sensor_msgs/html/msg/PointCloud2.html) LIDAR pointcloud Subscribers: # /airsim_node/vel_cmd_body_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /airsim_node/vel_cmd_world_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /gimbal_angle_euler_cmd airsim_ros_pkgs/GimbalAngleEulerCmd Gimbal set point in euler angles. /gimbal_angle_quat_cmd airsim_ros_pkgs/GimbalAngleQuatCmd Gimbal set point in quaternion. /airsim_node/VEHICLE_NAME/car_cmd [airsim_ros_pkgs/CarControls] Throttle, brake, steering and gear selections for control. Both automatic and manual transmission control possible, see the car_joy.py script for use. Services: # /airsim_node/VEHICLE_NAME/land airsim_ros_pkgs/Takeoff /airsim_node/takeoff airsim_ros_pkgs/Takeoff /airsim_node/reset airsim_ros_pkgs/Reset Resets all drones Parameters: # /airsim_node/world_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: world_ned Set to \"world_enu\" to switch to ENU frames automatically /airsim_node/odom_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: odom_local_ned If you set world_frame_id to \"world_enu\", the default odom name will instead default to \"odom_local_enu\" /airsim_node/coordinate_system_enu [boolean] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false If you set world_frame_id to \"world_enu\", this setting will instead default to true /airsim_node/update_airsim_control_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for updating drone odom and state from airsim, and sending in control commands. The current RPClib interface to unreal engine maxes out at 50 Hz. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/update_airsim_img_response_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for receiving images from all cameras in airsim. The speed will depend on number of images requested and their resolution. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/publish_clock [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false Will publish the ros /clock topic if set to true. Simple PID Position Controller Node # Parameters: # PD controller parameters: /pd_position_node/kd_x [double], /pd_position_node/kp_y [double], /pd_position_node/kp_z [double], /pd_position_node/kp_yaw [double] Proportional gains /pd_position_node/kd_x [double], /pd_position_node/kd_y [double], /pd_position_node/kd_z [double], /pd_position_node/kd_yaw [double] Derivative gains /pd_position_node/reached_thresh_xyz [double] Threshold euler distance (meters) from current position to setpoint position /pd_position_node/reached_yaw_degrees [double] Threshold yaw distance (degrees) from current position to setpoint position /pd_position_node/update_control_every_n_sec [double] Default: 0.01 seconds Services: # /airsim_node/VEHICLE_NAME/gps_goal [Request: srv/SetGPSPosition ] Target gps position + yaw. In absolute altitude. /airsim_node/VEHICLE_NAME/local_position_goal [Request: srv/SetLocalPosition Target local position + yaw in global NED frame. Subscribers: # /airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw Listens to home geo coordinates published by airsim_node . /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Listens to odometry published by airsim_node Publishers: # /vel_cmd_world_frame airsim_ros_pkgs/VelCmd Sends velocity command to airsim_node Global params # Dynamic constraints. These can be changed in dynamic_constraints.launch : /max_vel_horz_abs [double] Maximum horizontal velocity of the drone (meters/second) /max_vel_vert_abs [double] Maximum vertical velocity of the drone (meters/second) /max_yaw_rate_degree [double] Maximum yaw rate (degrees/second) Misc # Windows Subsytem for Linux on Windows 10 # WSL setup: Get Windows Subsystem for Linux Get Ubuntu 16.04 or Ubuntu 18.04 Go to Ubuntu 16 / 18 instructions! Setup for X apps (like RViz, rqt_image_view, terminator) in Windows + WSL Install Xming X Server . Find and run XLaunch from the Windows start menu. Select Multiple Windows in first popup, Start no client in second popup, only Clipboard in third popup. Do not select Native Opengl . Open Ubuntu 16.04 / 18.04 session by typing Ubuntu 16.04 / Ubuntu 18.04 in Windows start menu. Recommended: Install terminator : $ sudo apt-get install terminator. You can open terminator in a new window by entering $ DISPLAY=:0 terminator -u .","title":"ROS: AirSim ROS Wrapper"},{"location":"airsim_ros_pkgs/#airsim_ros_pkgs","text":"A ROS wrapper over the AirSim C++ client library.","title":"airsim_ros_pkgs"},{"location":"airsim_ros_pkgs/#setup","text":"Install gcc >= 8.0.0 sudo apt-get install gcc-8 g++-8 Verify version by gcc --version Ubuntu 16.04 Install ROS kinetic Install tf2 sensor and mavros packages: sudo apt-get install ros-kinetic-tf2-sensor-msgs ros-kinetic-mavros* Ubuntu 18.04 Install ROS melodic Install tf2 sensor and mavros packages: sudo apt-get install ros-melodic-tf2-sensor-msgs ros-melodic-mavros* Install catkin_tools sudo apt-get install python-catkin-tools or pip install catkin_tools","title":"Setup"},{"location":"airsim_ros_pkgs/#build","text":"Build AirSim git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh; Build ROS package cd ros; catkin build; # or catkin_make If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8","title":"Build"},{"location":"airsim_ros_pkgs/#running","text":"source devel/setup.bash; roslaunch airsim_ros_pkgs airsim_node.launch; roslaunch airsim_ros_pkgs rviz.launch;","title":"Running"},{"location":"airsim_ros_pkgs/#using-airsim-ros-wrapper","text":"The ROS wrapper is composed of two ROS nodes - the first is a wrapper over AirSim's multirotor C++ client library, and the second is a simple PD position controller. Let's look at the ROS API for both nodes:","title":"Using AirSim ROS wrapper"},{"location":"airsim_ros_pkgs/#airsim-ros-wrapper-node","text":"","title":"AirSim ROS Wrapper Node"},{"location":"airsim_ros_pkgs/#publishers","text":"/airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw GPS coordinates corresponding to global NED frame. This is set in the airsim's settings.json file under the OriginGeopoint key. /airsim_node/VEHICLE_NAME/global_gps sensor_msgs/NavSatFix This the current GPS coordinates of the drone in airsim. /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Odometry in NED frame (default name: odom_local_ned, launch name and frame type are configurable) wrt take-off point. /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE/camera_info sensor_msgs/CameraInfo /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE sensor_msgs/Image RGB or float image depending on image type requested in settings.json. /tf tf2_msgs/TFMessage /airsim_node/VEHICLE_NAME/altimeter/SENSOR_NAME [airsim_ros_pkgs::Altimeter] This the current altimeter reading for altitude, pressure, and QNH (https://en.wikipedia.org/wiki/QNH) /airsim_node/VEHICLE_NAME/imu/SENSOR_NAME [sensor_msgs::Imu] (http://docs.ros.org/api/sensor_msgs/html/msg/Imu.html) IMU sensor data /airsim_node/VEHICLE_NAME/magnetometer/SENSOR_NAME [sensor_msgs::MagneticField] (http://docs.ros.org/api/sensor_msgs/html/msg/MagneticField.html) Meausrement of magnetic field vector/compass /airsim_node/VEHICLE_NAME/distance/SENSOR_NAME [sensor_msgs::Range] (http://docs.ros.org/api/sensor_msgs/html/msg/Range.html) Meausrement of distance from an active ranger, such as infrared or IR /airsim_node/VEHICLE_NAME/lidar/SENSOR_NAME [sensor_msgs::PointCloud2] (http://docs.ros.org/api/sensor_msgs/html/msg/PointCloud2.html) LIDAR pointcloud","title":"Publishers:"},{"location":"airsim_ros_pkgs/#subscribers","text":"/airsim_node/vel_cmd_body_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /airsim_node/vel_cmd_world_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /gimbal_angle_euler_cmd airsim_ros_pkgs/GimbalAngleEulerCmd Gimbal set point in euler angles. /gimbal_angle_quat_cmd airsim_ros_pkgs/GimbalAngleQuatCmd Gimbal set point in quaternion. /airsim_node/VEHICLE_NAME/car_cmd [airsim_ros_pkgs/CarControls] Throttle, brake, steering and gear selections for control. Both automatic and manual transmission control possible, see the car_joy.py script for use.","title":"Subscribers:"},{"location":"airsim_ros_pkgs/#services","text":"/airsim_node/VEHICLE_NAME/land airsim_ros_pkgs/Takeoff /airsim_node/takeoff airsim_ros_pkgs/Takeoff /airsim_node/reset airsim_ros_pkgs/Reset Resets all drones","title":"Services:"},{"location":"airsim_ros_pkgs/#parameters","text":"/airsim_node/world_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: world_ned Set to \"world_enu\" to switch to ENU frames automatically /airsim_node/odom_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: odom_local_ned If you set world_frame_id to \"world_enu\", the default odom name will instead default to \"odom_local_enu\" /airsim_node/coordinate_system_enu [boolean] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false If you set world_frame_id to \"world_enu\", this setting will instead default to true /airsim_node/update_airsim_control_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for updating drone odom and state from airsim, and sending in control commands. The current RPClib interface to unreal engine maxes out at 50 Hz. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/update_airsim_img_response_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for receiving images from all cameras in airsim. The speed will depend on number of images requested and their resolution. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/publish_clock [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false Will publish the ros /clock topic if set to true.","title":"Parameters:"},{"location":"airsim_ros_pkgs/#simple-pid-position-controller-node","text":"","title":"Simple PID Position Controller Node"},{"location":"airsim_ros_pkgs/#parameters_1","text":"PD controller parameters: /pd_position_node/kd_x [double], /pd_position_node/kp_y [double], /pd_position_node/kp_z [double], /pd_position_node/kp_yaw [double] Proportional gains /pd_position_node/kd_x [double], /pd_position_node/kd_y [double], /pd_position_node/kd_z [double], /pd_position_node/kd_yaw [double] Derivative gains /pd_position_node/reached_thresh_xyz [double] Threshold euler distance (meters) from current position to setpoint position /pd_position_node/reached_yaw_degrees [double] Threshold yaw distance (degrees) from current position to setpoint position /pd_position_node/update_control_every_n_sec [double] Default: 0.01 seconds","title":"Parameters:"},{"location":"airsim_ros_pkgs/#services_1","text":"/airsim_node/VEHICLE_NAME/gps_goal [Request: srv/SetGPSPosition ] Target gps position + yaw. In absolute altitude. /airsim_node/VEHICLE_NAME/local_position_goal [Request: srv/SetLocalPosition Target local position + yaw in global NED frame.","title":"Services:"},{"location":"airsim_ros_pkgs/#subscribers_1","text":"/airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw Listens to home geo coordinates published by airsim_node . /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Listens to odometry published by airsim_node","title":"Subscribers:"},{"location":"airsim_ros_pkgs/#publishers_1","text":"/vel_cmd_world_frame airsim_ros_pkgs/VelCmd Sends velocity command to airsim_node","title":"Publishers:"},{"location":"airsim_ros_pkgs/#global-params","text":"Dynamic constraints. These can be changed in dynamic_constraints.launch : /max_vel_horz_abs [double] Maximum horizontal velocity of the drone (meters/second) /max_vel_vert_abs [double] Maximum vertical velocity of the drone (meters/second) /max_yaw_rate_degree [double] Maximum yaw rate (degrees/second)","title":"Global params"},{"location":"airsim_ros_pkgs/#misc","text":"","title":"Misc"},{"location":"airsim_ros_pkgs/#windows-subsytem-for-linux-on-windows-10","text":"WSL setup: Get Windows Subsystem for Linux Get Ubuntu 16.04 or Ubuntu 18.04 Go to Ubuntu 16 / 18 instructions! Setup for X apps (like RViz, rqt_image_view, terminator) in Windows + WSL Install Xming X Server . Find and run XLaunch from the Windows start menu. Select Multiple Windows in first popup, Start no client in second popup, only Clipboard in third popup. Do not select Native Opengl . Open Ubuntu 16.04 / 18.04 session by typing Ubuntu 16.04 / Ubuntu 18.04 in Windows start menu. Recommended: Install terminator : $ sudo apt-get install terminator. You can open terminator in a new window by entering $ DISPLAY=:0 terminator -u .","title":"Windows Subsytem for Linux on Windows 10"},{"location":"airsim_tutorial_pkgs/","text":"AirSim ROS Tutorials # This is a set of sample AirSim settings.json s, roslaunch and rviz files to give a starting point for using AirSim with ROS. See airsim_ros_pkgs for the ROS API. Setup # Make sure that the airsim_ros_pkgs Setup has been completed and the prerequisites installed. $ cd PATH_TO/AirSim/ros $ catkin build airsim_tutorial_pkgs If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build airsim_tutorial_pkgs -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 Note: For running examples, and also whenever a new terminal is opened, sourcing the setup.bash file is necessary. If you're using the ROS wrapper frequently, it might be helpful to add the source PATH_TO/AirSim/ros/devel/setup.bash to your ~/.profile or ~/.bashrc to avoid the need to run the command every time a new terminal is opened Examples # Single drone with monocular and depth cameras, and lidar # Settings.json - front_stereo_and_center_mono.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/front_stereo_and_center_mono.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; # in a new pane / terminal $ source PATH_TO/AirSim/ros/devel/setup.bash $ roslaunch airsim_tutorial_pkgs front_stereo_and_center_mono.launch `` The above would start rviz with tf's, registered RGBD cloud using [depth_image_proc](https://wiki.ros.org/depth_image_proc) using the [ depth_to_pointcloud` launch file](https://github.com/microsoft/AirSim/master/ros/src/airsim_tutorial_pkgs/launch/front_stereo_and_center_mono/depth_to_pointcloud.launch), and the lidar point cloud. Two drones, with cameras, lidar, IMU each # Settings.json - two_drones_camera_lidar_imu.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/two_drones_camera_lidar_imu.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable. Twenty-five drones in a square pattern # Settings.json - twenty_five_drones.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/twenty_five_drones.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable.","title":"ROS: AirSim Tutorial Packages"},{"location":"airsim_tutorial_pkgs/#airsim-ros-tutorials","text":"This is a set of sample AirSim settings.json s, roslaunch and rviz files to give a starting point for using AirSim with ROS. See airsim_ros_pkgs for the ROS API.","title":"AirSim ROS Tutorials"},{"location":"airsim_tutorial_pkgs/#setup","text":"Make sure that the airsim_ros_pkgs Setup has been completed and the prerequisites installed. $ cd PATH_TO/AirSim/ros $ catkin build airsim_tutorial_pkgs If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build airsim_tutorial_pkgs -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 Note: For running examples, and also whenever a new terminal is opened, sourcing the setup.bash file is necessary. If you're using the ROS wrapper frequently, it might be helpful to add the source PATH_TO/AirSim/ros/devel/setup.bash to your ~/.profile or ~/.bashrc to avoid the need to run the command every time a new terminal is opened","title":"Setup"},{"location":"airsim_tutorial_pkgs/#examples","text":"","title":"Examples"},{"location":"airsim_tutorial_pkgs/#single-drone-with-monocular-and-depth-cameras-and-lidar","text":"Settings.json - front_stereo_and_center_mono.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/front_stereo_and_center_mono.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; # in a new pane / terminal $ source PATH_TO/AirSim/ros/devel/setup.bash $ roslaunch airsim_tutorial_pkgs front_stereo_and_center_mono.launch `` The above would start rviz with tf's, registered RGBD cloud using [depth_image_proc](https://wiki.ros.org/depth_image_proc) using the [ depth_to_pointcloud` launch file](https://github.com/microsoft/AirSim/master/ros/src/airsim_tutorial_pkgs/launch/front_stereo_and_center_mono/depth_to_pointcloud.launch), and the lidar point cloud.","title":"Single drone with monocular and depth cameras, and lidar"},{"location":"airsim_tutorial_pkgs/#two-drones-with-cameras-lidar-imu-each","text":"Settings.json - two_drones_camera_lidar_imu.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/two_drones_camera_lidar_imu.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable.","title":"Two drones, with cameras, lidar, IMU each"},{"location":"airsim_tutorial_pkgs/#twenty-five-drones-in-a-square-pattern","text":"Settings.json - twenty_five_drones.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/twenty_five_drones.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable.","title":"Twenty-five drones in a square pattern"},{"location":"apis/","text":"AirSim APIs # Introduction # AirSim exposes APIs so you can interact with vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on. Python Quickstart # If you want to use Python to call AirSim APIs, we recommend using Anaconda with Python 3.5 or later versions however some code may also work with Python 2.7 ( help us improve compatibility!). First install this package: pip install msgpack-rpc-python You can either get AirSim binaries from releases or compile from the source ( Windows , Linux ). Once you can run AirSim, choose Car as vehicle and then navigate to PythonClient\\car\\ folder and run: python hello_car.py If you are using Visual Studio 2019 then just open AirSim.sln, set PythonClient as startup project and choose car\\hello_car.py as your startup script. Installing AirSim Package # You can also install airsim package simply by, pip install airsim You can find source code and samples for this package in PythonClient folder in your repo. Notes 1. You may notice a file setup_path.py in our example folders. This file has simple code to detect if airsim package is available in parent folder and in that case we use that instead of pip installed package so you always use latest code. 2. AirSim is still under heavy development which means you might frequently need to update the package to use new APIs. C++ Users # If you want to use C++ APIs and examples, please see C++ APIs Guide . Hello Car # Here's how to use AirSim APIs using Python to control simulated car (see also C++ example ): # ready to run example: PythonClient/car/hello_car.py import airsim import time # connect to the AirSim simulator client = airsim.CarClient() client.confirmConnection() client.enableApiControl(True) car_controls = airsim.CarControls() while True: # get state of the car car_state = client.getCarState() print(\"Speed %d, Gear %d\" % (car_state.speed, car_state.gear)) # set the controls for car car_controls.throttle = 1 car_controls.steering = 1 client.setCarControls(car_controls) # let car drive a bit time.sleep(1) # get camera images from the car responses = client.simGetImages([ airsim.ImageRequest(0, airsim.ImageType.DepthVis), airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm('py1.pfm', airsim.get_pfm_array(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file('py1.png', response.image_data_uint8) Hello Drone # Here's how to use AirSim APIs using Python to control simulated quadrotor (see also C++ example ): # ready to run example: PythonClient/multirotor/hello_drone.py import airsim # connect to the AirSim simulator client = airsim.MultirotorClient() client.confirmConnection() client.enableApiControl(True) client.armDisarm(True) # Async methods returns Future. Call join() to wait for task to complete. client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() # take images responses = client.simGetImages([ airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis), airsim.ImageRequest(\"1\", airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with the images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm(os.path.normpath('/temp/py1.pfm'), airsim.getPfmArray(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file(os.path.normpath('/temp/py1.png'), response.image_data_uint8) Common APIs # reset : This resets the vehicle to its original starting state. Note that you must call enableApiControl and armDisarm again after the call to reset . confirmConnection : Checks state of connection every 1 sec and reports it in Console so user can see the progress for connection. enableApiControl : For safety reasons, by default API control for autonomous vehicle is not enabled and human operator has full control (usually via RC or joystick in simulator). The client must make this call to request control via API. It is likely that human operator of vehicle might have disallowed API control which would mean that enableApiControl has no effect. This can be checked by isApiControlEnabled . isApiControlEnabled : Returns true if API control is established. If false (which is default) then API calls would be ignored. After a successful call to enableApiControl , the isApiControlEnabled should return true. ping : If connection is established then this call will return true otherwise it will be blocked until timeout. simPrintLogMessage : Prints the specified message in the simulator's window. If message_param is also supplied then its printed next to the message and in that case if this API is called with same message value but different message_param again then previous line is overwritten with new line (instead of API creating new line on display). For example, simPrintLogMessage(\"Iteration: \", to_string(i)) keeps updating same line on display when API is called with different values of i. The valid values of severity parameter is 0 to 3 inclusive that corresponds to different colors. simGetObjectPose , simSetObjectPose : Gets and sets the pose of specified object in Unreal environment. Here the object means \"actor\" in Unreal terminology. They are searched by tag as well as name. Please note that the names shown in UE Editor are auto-generated in each run and are not permanent. So if you want to refer to actor by name, you must change its auto-generated name in UE Editor. Alternatively you can add a tag to actor which can be done by clicking on that actor in Unreal Editor and then going to Tags property , click \"+\" sign and add some string value. If multiple actors have same tag then the first match is returned. If no matches are found then NaN pose is returned. The returned pose is in NED coordinates in SI units with its origin at Player Start. For simSetObjectPose , the specified actor must have Mobility set to Movable or otherwise you will get undefined behavior. The simSetObjectPose has parameter teleport which means object is moved through other objects in its way and it returns true if move was successful Image / Computer Vision APIs # AirSim offers comprehensive images APIs to retrieve synchronized images from multiple cameras along with ground truth including depth, disparity, surface normals and vision. You can set the resolution, FOV, motion blur etc parameters in settings.json . There is also API for detecting collision state. See also complete code that generates specified number of stereo images and ground truth depth with normalization to camera plan, computation of disparity image and saving it to pfm format . More on image APIs and Computer Vision mode . For vision problems that can benefit from domain randomization, there is also an object retexturing API , which can be used in supported scenes. Pause and Continue APIs # AirSim allows to pause and continue the simulation through pause(is_paused) API. To pause the simulation call pause(True) and to continue the simulation call pause(False) . You may have scenario, especially while using reinforcement learning, to run the simulation for specified amount of time and then automatically pause. While simulation is paused, you may then do some expensive computation, send a new command and then again run the simulation for specified amount of time. This can be achieved by API continueForTime(seconds) . This API runs the simulation for the specified number of seconds and then pauses the simulation. For example usage, please see pause_continue_car.py and pause_continue_drone.py . Collision API # The collision information can be obtained using simGetCollisionInfo API. This call returns a struct that has information not only whether collision occurred but also collision position, surface normal, penetration depth and so on. Time of Day API # AirSim assumes there exist sky sphere of class EngineSky/BP_Sky_Sphere in your environment with ADirectionalLight actor . By default, the position of the sun in the scene doesn't move with time. You can use settings to set up latitude, longitude, date and time which AirSim uses to compute the position of sun in the scene. You can also use following API call to set the sun position according to given date time: simSetTimeOfDay(self, is_enabled, start_datetime = \"\", is_start_datetime_dst = False, celestial_clock_speed = 1, update_interval_secs = 60, move_sun = True) The is_enabled parameter must be True to enable time of day effect. If it is False then sun position is reset to its original in the environment. Other parameters are same as in settings . Weather APIs # By default all weather effects are disabled. To enable weather effect, first call: simEnableWeather(True) Various weather effects can be enabled by using simSetWeatherParameter method which takes WeatherParameter , for example, client.simSetWeatherParameter(airsim.WeatherParameter.Rain, 0.25); The second parameter value is from 0 to 1. The first parameter provides following options: class WeatherParameter: Rain = 0 Roadwetness = 1 Snow = 2 RoadSnow = 3 MapleLeaf = 4 RoadLeaf = 5 Dust = 6 Fog = 7 Please note that Roadwetness , RoadSnow and RoadLeaf effects requires adding materials to your scene. Please see example code for more details. Recording APIs # Recording APIs can be used to start recording data through APIs. Data to be recorded can be specified using settings . To start recording, use - client.startRecording() Similarly, to stop recording, use client.stopRecording() . To check whether Recording is running, call client.isRecording() , returns a bool . This API works alongwith toggling Recording using R button, therefore if it's enabled using R key, isRecording() will return True , and recording can be stopped via API using stopRecording() . Similarly, recording started using API will be stopped if R key is pressed in Viewport. LogMessage will also appear in the top-left of the viewport if recording is started or stopped using API. Note that this will only save the data as specfied in the settings. For full freedom in storing data such as certain sensor information, or in a different format or layout, use the other APIs to fetch the data and save as desired. Lidar APIs # AirSim offers API to retrieve point cloud data from Lidar sensors on vehicles. You can set the number of channels, points per second, horizontal and vertical FOV, etc parameters in settings.json . More on lidar APIs and settings and sensor settings Multiple Vehicles # AirSim supports multiple vehicles and control them through APIs. Please Multiple Vehicles doc. Coordinate System # All AirSim API uses NED coordinate system, i.e., +X is North, +Y is East and +Z is Down. All units are in SI system. Please note that this is different from coordinate system used internally by Unreal Engine. In Unreal Engine, +Z is up instead of down and length unit is in centimeters instead of meters. AirSim APIs takes care of the appropriate conversions. The starting point of the vehicle is always coordinates (0, 0, 0) in NED system. Thus when converting from Unreal coordinates to NED, we first subtract the starting offset and then scale by 100 for cm to m conversion. The vehicle is spawned in Unreal environment where the Player Start component is placed. There is a setting called OriginGeopoint in settings.json which assigns geographic longitude, longitude and altitude to the Player Start component. Vehicle Specific APIs # APIs for Car # Car has followings APIs available: setCarControls : This allows you to set throttle, steering, handbrake and auto or manual gear. getCarState : This retrieves the state information including speed, current gear and 6 kinematics quantities: position, orientation, linear and angular velocity, linear and angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame. Image APIs . APIs for Multirotor # Multirotor can be controlled by specifying angles, velocity vector, destination position or some combination of these. There are corresponding move* APIs for this purpose. When doing position control, we need to use some path following algorithm. By default AirSim uses carrot following algorithm. This is often referred to as \"high level control\" because you just need to specify high level goal and the firmware takes care of the rest. Currently lowest level control available in AirSim is moveByAngleThrottleAsync API. getMultirotorState # This API returns the state of the vehicle in one call. The state includes, collision, estimated kinematics (i.e. kinematics computed by fusing sensors), and timestamp (nano seconds since epoch). The kinematics here means 6 quantities: position, orientation, linear and angular velocity, linear and angular acceleration. Please note that simple_slight currently doesn't support state estimator which means estimated and ground truth kinematics values would be same for simple_flight. Estimated kinematics are however available for PX4 except for angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame. Async methods, duration and max_wait_seconds # Many API methods has parameters named duration or max_wait_seconds and they have Async as suffix, for example, takeoffAsync . These methods will return immediately after starting the task in AirSim so that your client code can do something else while that task is being executed. If you want to wait for this task to complete then you can call waitOnLastTask like this: //C++ client.takeoffAsync()->waitOnLastTask(); # Python client.takeoffAsync().join() If you start another command then it automatically cancels the previous task and starts new command. This allows to use pattern where your coded continuously does the sensing, computes a new trajectory to follow and issues that path to vehicle in AirSim. Each newly issued trajectory cancels the previous trajectory allowing your code to continuously do the update as new sensor data arrives. All Async method returns concurrent.futures.Future in Python ( std::future in C++). Please note that these future classes currently do not allow to check status or cancel the task; they only allow to wait for task to complete. AirSim does provide API cancelLastTask , however. drivetrain # There are two modes you can fly vehicle: drivetrain parameter is set to airsim.DrivetrainType.ForwardOnly or airsim.DrivetrainType.MaxDegreeOfFreedom . When you specify ForwardOnly, you are saying that vehicle's front should always point in the direction of travel. So if you want drone to take left turn then it would first rotate so front points to left. This mode is useful when you have only front camera and you are operating vehicle using FPV view. This is more or less like travelling in car where you always have front view. The MaxDegreeOfFreedom means you don't care where the front points to. So when you take left turn, you just start going left like crab. Quadrotors can go in any direction regardless of where front points to. The MaxDegreeOfFreedom enables this mode. yaw_mode # yaw_mode is a struct YawMode with two fields, yaw_or_rate and is_rate . If is_rate field is True then yaw_or_rate field is interpreted as angular velocity in degrees/sec which means you want vehicle to rotate continuously around its axis at that angular velocity while moving. If is_rate is False then yaw_or_rate is interpreted as angle in degrees which means you want vehicle to rotate to specific angle (i.e. yaw) and keep that angle while moving. You can probably see that when yaw_mode.is_rate == true , the drivetrain parameter shouldn't be set to ForwardOnly because you are contradicting by saying that keep front pointing ahead but also rotate continuously. However if you have yaw_mode.is_rate = false in ForwardOnly mode then you can do some funky stuff. For example, you can have drone do circles and have yaw_or_rate set to 90 so camera is always pointed to center (\"super cool selfie mode\"). In MaxDegreeofFreedom also you can get some funky stuff by setting yaw_mode.is_rate = true and say yaw_mode.yaw_or_rate = 20 . This will cause drone to go in its path while rotating which may allow to do 360 scanning. In most cases, you just don't want yaw to change which you can do by setting yaw rate of 0. The shorthand for this is airsim.YawMode.Zero() (or in C++: YawMode::Zero() ). lookahead and adaptive_lookahead # When you ask vehicle to follow a path, AirSim uses \"carrot following\" algorithm. This algorithm operates by looking ahead on path and adjusting its velocity vector. The parameters for this algorithm is specified by lookahead and adaptive_lookahead . For most of the time you want algorithm to auto-decide the values by simply setting lookahead = -1 and adaptive_lookahead = 0 . Using APIs on Real Vehicles # We want to be able to run same code that runs in simulation as on real vehicle. This allows you to test your code in simulator and deploy to real vehicle. Generally speaking, APIs therefore shouldn't allow you to do something that cannot be done on real vehicle (for example, getting the ground truth). But, of course, simulator has much more information and it would be useful in applications that may not care about running things on real vehicle. For this reason, we clearly delineate between sim-only APIs by attaching sim prefix, for example, simGetGroundTruthKinematics . This way you can avoid using these simulation-only APIs if you care about running your code on real vehicles. The AirLib is self-contained library that you can put on an offboard computing module such as the Gigabyte barebone Mini PC. This module then can talk to the flight controllers such as PX4 using exact same code and flight controller protocol. The code you write for testing in the simulator remains unchanged. See AirLib on custom drones . Adding New APIs to AirSim # Adding new APIs requires modifying the source code. Much of the changes are mechanical and required for various levels of abstractions that AirSim supports. This commit demonstrates how to add a simple API simPrintLogMessage that prints message in simulator window. Some Internals # The APIs use msgpack-rpc protocol over TCP/IP through rpclib developed by Tam\u00c3\u00a1s Szelei which allows you to use variety of programming languages including C++, C#, Python, Java etc. When AirSim starts, it opens port 41451 (this can be changed via settings ) and listens for incoming request. The Python or C++ client code connects to this port and sends RPC calls using msgpack serialization format . References and Examples # C++ API Examples Car Examples Multirotor Examples Computer Vision Examples Move on Path demo showing video of fast multirotor flight through Modular Neighborhood environment Building a Hexacopter Building Point Clouds FAQ # Unreal is slowed down dramatically when I run API # If you see Unreal getting slowed down dramatically when Unreal Engine window loses focus then go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. Do I need anything else on Windows? # You should install VS2019 with VC++, Windows SDK 10.0 and Python. To use Python APIs you will need Python 3.5 or later (install it using Anaconda). Which version of Python should I use? # We recommend Anaconda to get Python tools and libraries. Our code is tested with Python 3.5.3 :: Anaconda 4.4.0. This is important because older version have been known to have problems . I get error on import cv2 # You can install OpenCV using: conda install opencv pip install opencv-python TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float' # This error happens if you install Jupyter, which somehow breaks the msgpackrpc library. Create a new python environment which the minimal required packages.","title":"Core APIs"},{"location":"apis/#airsim-apis","text":"","title":"AirSim APIs"},{"location":"apis/#introduction","text":"AirSim exposes APIs so you can interact with vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on.","title":"Introduction"},{"location":"apis/#python-quickstart","text":"If you want to use Python to call AirSim APIs, we recommend using Anaconda with Python 3.5 or later versions however some code may also work with Python 2.7 ( help us improve compatibility!). First install this package: pip install msgpack-rpc-python You can either get AirSim binaries from releases or compile from the source ( Windows , Linux ). Once you can run AirSim, choose Car as vehicle and then navigate to PythonClient\\car\\ folder and run: python hello_car.py If you are using Visual Studio 2019 then just open AirSim.sln, set PythonClient as startup project and choose car\\hello_car.py as your startup script.","title":"Python Quickstart"},{"location":"apis/#installing-airsim-package","text":"You can also install airsim package simply by, pip install airsim You can find source code and samples for this package in PythonClient folder in your repo. Notes 1. You may notice a file setup_path.py in our example folders. This file has simple code to detect if airsim package is available in parent folder and in that case we use that instead of pip installed package so you always use latest code. 2. AirSim is still under heavy development which means you might frequently need to update the package to use new APIs.","title":"Installing AirSim Package"},{"location":"apis/#c-users","text":"If you want to use C++ APIs and examples, please see C++ APIs Guide .","title":"C++ Users"},{"location":"apis/#hello-car","text":"Here's how to use AirSim APIs using Python to control simulated car (see also C++ example ): # ready to run example: PythonClient/car/hello_car.py import airsim import time # connect to the AirSim simulator client = airsim.CarClient() client.confirmConnection() client.enableApiControl(True) car_controls = airsim.CarControls() while True: # get state of the car car_state = client.getCarState() print(\"Speed %d, Gear %d\" % (car_state.speed, car_state.gear)) # set the controls for car car_controls.throttle = 1 car_controls.steering = 1 client.setCarControls(car_controls) # let car drive a bit time.sleep(1) # get camera images from the car responses = client.simGetImages([ airsim.ImageRequest(0, airsim.ImageType.DepthVis), airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm('py1.pfm', airsim.get_pfm_array(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file('py1.png', response.image_data_uint8)","title":"Hello Car"},{"location":"apis/#hello-drone","text":"Here's how to use AirSim APIs using Python to control simulated quadrotor (see also C++ example ): # ready to run example: PythonClient/multirotor/hello_drone.py import airsim # connect to the AirSim simulator client = airsim.MultirotorClient() client.confirmConnection() client.enableApiControl(True) client.armDisarm(True) # Async methods returns Future. Call join() to wait for task to complete. client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() # take images responses = client.simGetImages([ airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis), airsim.ImageRequest(\"1\", airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with the images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm(os.path.normpath('/temp/py1.pfm'), airsim.getPfmArray(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file(os.path.normpath('/temp/py1.png'), response.image_data_uint8)","title":"Hello Drone"},{"location":"apis/#common-apis","text":"reset : This resets the vehicle to its original starting state. Note that you must call enableApiControl and armDisarm again after the call to reset . confirmConnection : Checks state of connection every 1 sec and reports it in Console so user can see the progress for connection. enableApiControl : For safety reasons, by default API control for autonomous vehicle is not enabled and human operator has full control (usually via RC or joystick in simulator). The client must make this call to request control via API. It is likely that human operator of vehicle might have disallowed API control which would mean that enableApiControl has no effect. This can be checked by isApiControlEnabled . isApiControlEnabled : Returns true if API control is established. If false (which is default) then API calls would be ignored. After a successful call to enableApiControl , the isApiControlEnabled should return true. ping : If connection is established then this call will return true otherwise it will be blocked until timeout. simPrintLogMessage : Prints the specified message in the simulator's window. If message_param is also supplied then its printed next to the message and in that case if this API is called with same message value but different message_param again then previous line is overwritten with new line (instead of API creating new line on display). For example, simPrintLogMessage(\"Iteration: \", to_string(i)) keeps updating same line on display when API is called with different values of i. The valid values of severity parameter is 0 to 3 inclusive that corresponds to different colors. simGetObjectPose , simSetObjectPose : Gets and sets the pose of specified object in Unreal environment. Here the object means \"actor\" in Unreal terminology. They are searched by tag as well as name. Please note that the names shown in UE Editor are auto-generated in each run and are not permanent. So if you want to refer to actor by name, you must change its auto-generated name in UE Editor. Alternatively you can add a tag to actor which can be done by clicking on that actor in Unreal Editor and then going to Tags property , click \"+\" sign and add some string value. If multiple actors have same tag then the first match is returned. If no matches are found then NaN pose is returned. The returned pose is in NED coordinates in SI units with its origin at Player Start. For simSetObjectPose , the specified actor must have Mobility set to Movable or otherwise you will get undefined behavior. The simSetObjectPose has parameter teleport which means object is moved through other objects in its way and it returns true if move was successful","title":"Common APIs"},{"location":"apis/#image-computer-vision-apis","text":"AirSim offers comprehensive images APIs to retrieve synchronized images from multiple cameras along with ground truth including depth, disparity, surface normals and vision. You can set the resolution, FOV, motion blur etc parameters in settings.json . There is also API for detecting collision state. See also complete code that generates specified number of stereo images and ground truth depth with normalization to camera plan, computation of disparity image and saving it to pfm format . More on image APIs and Computer Vision mode . For vision problems that can benefit from domain randomization, there is also an object retexturing API , which can be used in supported scenes.","title":"Image / Computer Vision APIs"},{"location":"apis/#pause-and-continue-apis","text":"AirSim allows to pause and continue the simulation through pause(is_paused) API. To pause the simulation call pause(True) and to continue the simulation call pause(False) . You may have scenario, especially while using reinforcement learning, to run the simulation for specified amount of time and then automatically pause. While simulation is paused, you may then do some expensive computation, send a new command and then again run the simulation for specified amount of time. This can be achieved by API continueForTime(seconds) . This API runs the simulation for the specified number of seconds and then pauses the simulation. For example usage, please see pause_continue_car.py and pause_continue_drone.py .","title":"Pause and Continue APIs"},{"location":"apis/#collision-api","text":"The collision information can be obtained using simGetCollisionInfo API. This call returns a struct that has information not only whether collision occurred but also collision position, surface normal, penetration depth and so on.","title":"Collision API"},{"location":"apis/#time-of-day-api","text":"AirSim assumes there exist sky sphere of class EngineSky/BP_Sky_Sphere in your environment with ADirectionalLight actor . By default, the position of the sun in the scene doesn't move with time. You can use settings to set up latitude, longitude, date and time which AirSim uses to compute the position of sun in the scene. You can also use following API call to set the sun position according to given date time: simSetTimeOfDay(self, is_enabled, start_datetime = \"\", is_start_datetime_dst = False, celestial_clock_speed = 1, update_interval_secs = 60, move_sun = True) The is_enabled parameter must be True to enable time of day effect. If it is False then sun position is reset to its original in the environment. Other parameters are same as in settings .","title":"Time of Day API"},{"location":"apis/#weather-apis","text":"By default all weather effects are disabled. To enable weather effect, first call: simEnableWeather(True) Various weather effects can be enabled by using simSetWeatherParameter method which takes WeatherParameter , for example, client.simSetWeatherParameter(airsim.WeatherParameter.Rain, 0.25); The second parameter value is from 0 to 1. The first parameter provides following options: class WeatherParameter: Rain = 0 Roadwetness = 1 Snow = 2 RoadSnow = 3 MapleLeaf = 4 RoadLeaf = 5 Dust = 6 Fog = 7 Please note that Roadwetness , RoadSnow and RoadLeaf effects requires adding materials to your scene. Please see example code for more details.","title":"Weather APIs"},{"location":"apis/#recording-apis","text":"Recording APIs can be used to start recording data through APIs. Data to be recorded can be specified using settings . To start recording, use - client.startRecording() Similarly, to stop recording, use client.stopRecording() . To check whether Recording is running, call client.isRecording() , returns a bool . This API works alongwith toggling Recording using R button, therefore if it's enabled using R key, isRecording() will return True , and recording can be stopped via API using stopRecording() . Similarly, recording started using API will be stopped if R key is pressed in Viewport. LogMessage will also appear in the top-left of the viewport if recording is started or stopped using API. Note that this will only save the data as specfied in the settings. For full freedom in storing data such as certain sensor information, or in a different format or layout, use the other APIs to fetch the data and save as desired.","title":"Recording APIs"},{"location":"apis/#lidar-apis","text":"AirSim offers API to retrieve point cloud data from Lidar sensors on vehicles. You can set the number of channels, points per second, horizontal and vertical FOV, etc parameters in settings.json . More on lidar APIs and settings and sensor settings","title":"Lidar APIs"},{"location":"apis/#multiple-vehicles","text":"AirSim supports multiple vehicles and control them through APIs. Please Multiple Vehicles doc.","title":"Multiple Vehicles"},{"location":"apis/#coordinate-system","text":"All AirSim API uses NED coordinate system, i.e., +X is North, +Y is East and +Z is Down. All units are in SI system. Please note that this is different from coordinate system used internally by Unreal Engine. In Unreal Engine, +Z is up instead of down and length unit is in centimeters instead of meters. AirSim APIs takes care of the appropriate conversions. The starting point of the vehicle is always coordinates (0, 0, 0) in NED system. Thus when converting from Unreal coordinates to NED, we first subtract the starting offset and then scale by 100 for cm to m conversion. The vehicle is spawned in Unreal environment where the Player Start component is placed. There is a setting called OriginGeopoint in settings.json which assigns geographic longitude, longitude and altitude to the Player Start component.","title":"Coordinate System"},{"location":"apis/#vehicle-specific-apis","text":"","title":"Vehicle Specific APIs"},{"location":"apis/#apis-for-car","text":"Car has followings APIs available: setCarControls : This allows you to set throttle, steering, handbrake and auto or manual gear. getCarState : This retrieves the state information including speed, current gear and 6 kinematics quantities: position, orientation, linear and angular velocity, linear and angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame. Image APIs .","title":"APIs for Car"},{"location":"apis/#apis-for-multirotor","text":"Multirotor can be controlled by specifying angles, velocity vector, destination position or some combination of these. There are corresponding move* APIs for this purpose. When doing position control, we need to use some path following algorithm. By default AirSim uses carrot following algorithm. This is often referred to as \"high level control\" because you just need to specify high level goal and the firmware takes care of the rest. Currently lowest level control available in AirSim is moveByAngleThrottleAsync API.","title":"APIs for Multirotor"},{"location":"apis/#getmultirotorstate","text":"This API returns the state of the vehicle in one call. The state includes, collision, estimated kinematics (i.e. kinematics computed by fusing sensors), and timestamp (nano seconds since epoch). The kinematics here means 6 quantities: position, orientation, linear and angular velocity, linear and angular acceleration. Please note that simple_slight currently doesn't support state estimator which means estimated and ground truth kinematics values would be same for simple_flight. Estimated kinematics are however available for PX4 except for angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame.","title":"getMultirotorState"},{"location":"apis/#async-methods-duration-and-max_wait_seconds","text":"Many API methods has parameters named duration or max_wait_seconds and they have Async as suffix, for example, takeoffAsync . These methods will return immediately after starting the task in AirSim so that your client code can do something else while that task is being executed. If you want to wait for this task to complete then you can call waitOnLastTask like this: //C++ client.takeoffAsync()->waitOnLastTask(); # Python client.takeoffAsync().join() If you start another command then it automatically cancels the previous task and starts new command. This allows to use pattern where your coded continuously does the sensing, computes a new trajectory to follow and issues that path to vehicle in AirSim. Each newly issued trajectory cancels the previous trajectory allowing your code to continuously do the update as new sensor data arrives. All Async method returns concurrent.futures.Future in Python ( std::future in C++). Please note that these future classes currently do not allow to check status or cancel the task; they only allow to wait for task to complete. AirSim does provide API cancelLastTask , however.","title":"Async methods, duration and max_wait_seconds"},{"location":"apis/#drivetrain","text":"There are two modes you can fly vehicle: drivetrain parameter is set to airsim.DrivetrainType.ForwardOnly or airsim.DrivetrainType.MaxDegreeOfFreedom . When you specify ForwardOnly, you are saying that vehicle's front should always point in the direction of travel. So if you want drone to take left turn then it would first rotate so front points to left. This mode is useful when you have only front camera and you are operating vehicle using FPV view. This is more or less like travelling in car where you always have front view. The MaxDegreeOfFreedom means you don't care where the front points to. So when you take left turn, you just start going left like crab. Quadrotors can go in any direction regardless of where front points to. The MaxDegreeOfFreedom enables this mode.","title":"drivetrain"},{"location":"apis/#yaw_mode","text":"yaw_mode is a struct YawMode with two fields, yaw_or_rate and is_rate . If is_rate field is True then yaw_or_rate field is interpreted as angular velocity in degrees/sec which means you want vehicle to rotate continuously around its axis at that angular velocity while moving. If is_rate is False then yaw_or_rate is interpreted as angle in degrees which means you want vehicle to rotate to specific angle (i.e. yaw) and keep that angle while moving. You can probably see that when yaw_mode.is_rate == true , the drivetrain parameter shouldn't be set to ForwardOnly because you are contradicting by saying that keep front pointing ahead but also rotate continuously. However if you have yaw_mode.is_rate = false in ForwardOnly mode then you can do some funky stuff. For example, you can have drone do circles and have yaw_or_rate set to 90 so camera is always pointed to center (\"super cool selfie mode\"). In MaxDegreeofFreedom also you can get some funky stuff by setting yaw_mode.is_rate = true and say yaw_mode.yaw_or_rate = 20 . This will cause drone to go in its path while rotating which may allow to do 360 scanning. In most cases, you just don't want yaw to change which you can do by setting yaw rate of 0. The shorthand for this is airsim.YawMode.Zero() (or in C++: YawMode::Zero() ).","title":"yaw_mode"},{"location":"apis/#lookahead-and-adaptive_lookahead","text":"When you ask vehicle to follow a path, AirSim uses \"carrot following\" algorithm. This algorithm operates by looking ahead on path and adjusting its velocity vector. The parameters for this algorithm is specified by lookahead and adaptive_lookahead . For most of the time you want algorithm to auto-decide the values by simply setting lookahead = -1 and adaptive_lookahead = 0 .","title":"lookahead and adaptive_lookahead"},{"location":"apis/#using-apis-on-real-vehicles","text":"We want to be able to run same code that runs in simulation as on real vehicle. This allows you to test your code in simulator and deploy to real vehicle. Generally speaking, APIs therefore shouldn't allow you to do something that cannot be done on real vehicle (for example, getting the ground truth). But, of course, simulator has much more information and it would be useful in applications that may not care about running things on real vehicle. For this reason, we clearly delineate between sim-only APIs by attaching sim prefix, for example, simGetGroundTruthKinematics . This way you can avoid using these simulation-only APIs if you care about running your code on real vehicles. The AirLib is self-contained library that you can put on an offboard computing module such as the Gigabyte barebone Mini PC. This module then can talk to the flight controllers such as PX4 using exact same code and flight controller protocol. The code you write for testing in the simulator remains unchanged. See AirLib on custom drones .","title":"Using APIs on Real Vehicles"},{"location":"apis/#adding-new-apis-to-airsim","text":"Adding new APIs requires modifying the source code. Much of the changes are mechanical and required for various levels of abstractions that AirSim supports. This commit demonstrates how to add a simple API simPrintLogMessage that prints message in simulator window.","title":"Adding New APIs to AirSim"},{"location":"apis/#some-internals","text":"The APIs use msgpack-rpc protocol over TCP/IP through rpclib developed by Tam\u00c3\u00a1s Szelei which allows you to use variety of programming languages including C++, C#, Python, Java etc. When AirSim starts, it opens port 41451 (this can be changed via settings ) and listens for incoming request. The Python or C++ client code connects to this port and sends RPC calls using msgpack serialization format .","title":"Some Internals"},{"location":"apis/#references-and-examples","text":"C++ API Examples Car Examples Multirotor Examples Computer Vision Examples Move on Path demo showing video of fast multirotor flight through Modular Neighborhood environment Building a Hexacopter Building Point Clouds","title":"References and Examples"},{"location":"apis/#faq","text":"","title":"FAQ"},{"location":"apis/#unreal-is-slowed-down-dramatically-when-i-run-api","text":"If you see Unreal getting slowed down dramatically when Unreal Engine window loses focus then go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked.","title":"Unreal is slowed down dramatically when I run API"},{"location":"apis/#do-i-need-anything-else-on-windows","text":"You should install VS2019 with VC++, Windows SDK 10.0 and Python. To use Python APIs you will need Python 3.5 or later (install it using Anaconda).","title":"Do I need anything else on Windows?"},{"location":"apis/#which-version-of-python-should-i-use","text":"We recommend Anaconda to get Python tools and libraries. Our code is tested with Python 3.5.3 :: Anaconda 4.4.0. This is important because older version have been known to have problems .","title":"Which version of Python should I use?"},{"location":"apis/#i-get-error-on-import-cv2","text":"You can install OpenCV using: conda install opencv pip install opencv-python","title":"I get error on import cv2"},{"location":"apis/#typeerror-unsupported-operand-types-for-asyncioloop-and-float","text":"This error happens if you install Jupyter, which somehow breaks the msgpackrpc library. Create a new python environment which the minimal required packages.","title":"TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float'"},{"location":"apis_cpp/","text":"Using C++ APIs for AirSim # Please read general API doc first if you haven't already. This document describes C++ examples and other C++ specific details. Quick Start # Fastest way to get started is to open AirSim.sln in Visual Studio 2019. You will see Hello Car and Hello Drone examples in the solution. These examples will show you the include paths and lib paths you will need to setup in your VC++ projects. If you are using Linux then you will specify these paths either in your cmake file or on compiler command line. Include and Lib Folders # Include folders: $(ProjectDir)..\\AirLib\\deps\\rpclib\\include;include;$(ProjectDir)..\\AirLib\\deps\\eigen3;$(ProjectDir)..\\AirLib\\include Dependencies: rpc.lib Lib folders: $(ProjectDir)\\..\\AirLib\\deps\\MavLinkCom\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\deps\\rpclib\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\lib\\$(Platform)\\$(Configuration) Hello Car # Here's how to use AirSim APIs using C++ to control simulated car (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloCar/main.cpp #include #include \"vehicles/car/api/CarRpcLibClient.hpp\" int main() { msr::airlib::CarRpcLibClient client; client.enableApiControl(true); //this disables manual control CarControllerBase::CarControls controls; std::cout << \"Press enter to drive forward\" << std::endl; std::cin.get(); controls.throttle = 1; client.setCarControls(controls); std::cout << \"Press Enter to activate handbrake\" << std::endl; std::cin.get(); controls.handbrake = true; client.setCarControls(controls); std::cout << \"Press Enter to take turn and drive backward\" << std::endl; std::cin.get(); controls.handbrake = false; controls.throttle = -1; controls.steering = 1; client.setCarControls(controls); std::cout << \"Press Enter to stop\" << std::endl; std::cin.get(); client.setCarControls(CarControllerBase::CarControls()); return 0; } Hello Drone # Here's how to use AirSim APIs using C++ to control simulated quadrotor (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloDrone/main.cpp #include #include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int main() { using namespace std; msr::airlib::MultirotorRpcLibClient client; cout << \"Press Enter to enable API control\" << endl; cin.get(); client.enableApiControl(true); cout << \"Press Enter to arm the drone\" << endl; cin.get(); client.armDisarm(true); cout << \"Press Enter to takeoff\" << endl; cin.get(); client.takeoffAsync(5)->waitOnLastTask(); cout << \"Press Enter to move 5 meters in x direction with 1 m/s velocity\" << endl; cin.get(); auto position = client.getMultirotorState().getPosition(); // from current location client.moveToPositionAsync(position.x() + 5, position.y(), position.z(), 1)->waitOnLastTask(); cout << \"Press Enter to land\" << endl; cin.get(); client.landAsync()->waitOnLastTask(); return 0; } See Also # Examples of how to use internal infrastructure in AirSim in your other projects DroneShell app shows how to make simple interface using C++ APIs to control drones Python APIs","title":"C++ APIs"},{"location":"apis_cpp/#using-c-apis-for-airsim","text":"Please read general API doc first if you haven't already. This document describes C++ examples and other C++ specific details.","title":"Using C++ APIs for AirSim"},{"location":"apis_cpp/#quick-start","text":"Fastest way to get started is to open AirSim.sln in Visual Studio 2019. You will see Hello Car and Hello Drone examples in the solution. These examples will show you the include paths and lib paths you will need to setup in your VC++ projects. If you are using Linux then you will specify these paths either in your cmake file or on compiler command line.","title":"Quick Start"},{"location":"apis_cpp/#include-and-lib-folders","text":"Include folders: $(ProjectDir)..\\AirLib\\deps\\rpclib\\include;include;$(ProjectDir)..\\AirLib\\deps\\eigen3;$(ProjectDir)..\\AirLib\\include Dependencies: rpc.lib Lib folders: $(ProjectDir)\\..\\AirLib\\deps\\MavLinkCom\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\deps\\rpclib\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\lib\\$(Platform)\\$(Configuration)","title":"Include and Lib Folders"},{"location":"apis_cpp/#hello-car","text":"Here's how to use AirSim APIs using C++ to control simulated car (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloCar/main.cpp #include #include \"vehicles/car/api/CarRpcLibClient.hpp\" int main() { msr::airlib::CarRpcLibClient client; client.enableApiControl(true); //this disables manual control CarControllerBase::CarControls controls; std::cout << \"Press enter to drive forward\" << std::endl; std::cin.get(); controls.throttle = 1; client.setCarControls(controls); std::cout << \"Press Enter to activate handbrake\" << std::endl; std::cin.get(); controls.handbrake = true; client.setCarControls(controls); std::cout << \"Press Enter to take turn and drive backward\" << std::endl; std::cin.get(); controls.handbrake = false; controls.throttle = -1; controls.steering = 1; client.setCarControls(controls); std::cout << \"Press Enter to stop\" << std::endl; std::cin.get(); client.setCarControls(CarControllerBase::CarControls()); return 0; }","title":"Hello Car"},{"location":"apis_cpp/#hello-drone","text":"Here's how to use AirSim APIs using C++ to control simulated quadrotor (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloDrone/main.cpp #include #include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int main() { using namespace std; msr::airlib::MultirotorRpcLibClient client; cout << \"Press Enter to enable API control\" << endl; cin.get(); client.enableApiControl(true); cout << \"Press Enter to arm the drone\" << endl; cin.get(); client.armDisarm(true); cout << \"Press Enter to takeoff\" << endl; cin.get(); client.takeoffAsync(5)->waitOnLastTask(); cout << \"Press Enter to move 5 meters in x direction with 1 m/s velocity\" << endl; cin.get(); auto position = client.getMultirotorState().getPosition(); // from current location client.moveToPositionAsync(position.x() + 5, position.y(), position.z(), 1)->waitOnLastTask(); cout << \"Press Enter to land\" << endl; cin.get(); client.landAsync()->waitOnLastTask(); return 0; }","title":"Hello Drone"},{"location":"apis_cpp/#see-also","text":"Examples of how to use internal infrastructure in AirSim in your other projects DroneShell app shows how to make simple interface using C++ APIs to control drones Python APIs","title":"See Also"},{"location":"azure/","text":"AirSim Development Environment on Azure # This document explains how to automate the creation of a development environment on Azure and code and debug a Python application connected to AirSim using Visual Studio Code Automatically Deploy Your Azure VM # Use this template to create, deploy and configure an Azure VM to work with AirSim Note: the VM deployment and configuration process may take 20+ minutes to complete Regarding the deployment of the Azure VM # When using an Azure Trial account, the default vCPU quota may not be enough to allocate the required VM to run AirSim. If that's the case, you will see an error when trying to create the VM and will have to make a support request to increase the default quota. To avoid charges for the Virtual Machine usage while not in use, remember to deallocate its resources from the Azure Portal or use the following command from the Azure CLI: bash az vm deallocate --resource-group MyResourceGroup --name MyVMName Code and debug from Visual Studio Code and Remote SSH # Install Visual Studio Code Install the Remote - SSH extension Press F1 and run the Remote - SSH: Connect to host... command Add the recently create VM details. For instance, AzureUser@11.22.33.44 Run the Remote - SSH: Connect to host... command again, and now select the newly added connection. Once connected, click on the Clone Repository button in Visual Studio Code, and either clone this repository in the remote VM and open just the azure folder , or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. It is important to open that directory so Visual Studio Code can use the specific .vscode directory for the scenario and not the general AirSim .vscode directory. It contains the recommended extensions to install, the task to start AirSim remotely and the launch configuration for the Python application. Install all the recommended extensions Press F1 and select the Tasks: Run Task option. Then, select the Start AirSim task from Visual Studio Code to execute the start-airsim.ps1 script from Visual Studio Code. Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges Code and debug from a local Visual Studio Code and connect to AirSim via forwarded ports # Note: this scenario, will be using two Visual Studio Code instances. The first one will be used as a bridge to forward ports via SSH to the Azure VM and execute remote processes, and the second one will be used for local Python development. To be able to reach the VM from the local Python code, it is required to keep the Remote - SSH instance of Visual Studio Code opened, while working with the local Python environment on the second instance Open the first Visual Studio Code instance Follow the steps in the previous section to connect via Remote - SSH In the Remote Explorer , add the port 41451 as a forwarded port to localhost Either run the Start AirSim task on the Visual Studio Code with the remote session as explained in the previous scenario or manually start the AirSim binary in the VM Open a second Visual Studio Code instance, without disconnecting or closing the first one Either clone this repository locally and open just the azure folder in Visual Studio Code, or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. Run pip install -r requirements.txt inside the app directory Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges Running with Docker # Once both the AirSim environment and the Python application are ready, you can package everything as a Docker image. The sample project inside the azure directory is already prepared to run a prebuilt AirSim binary and Python code using Docker. This would be a perfect scenario when you want to run the simulation at scale. For instance, you could have several different configurations for the same simulation and execute them in a parallel, unattended way using a Docker image on Azure Container Services Since AirSim requires access to the host GPU, it is required to use a Docker runtime that supports it. For more information about running AirSim in Docker, click here . When using Azure Container Services to run this image, the only extra-requirement is to add GPU support to the Container Group where it will be deployed. It can use either public docker images from DockerHub or images deployed to a private Azure Container Registry Building the docker image # docker build -t / -f ./docker/Dockerfile .` Using a different AirSim binary # To use a different AirSim binary, first check the official documentation on How to Build AirSim on Windows and How to Build AirSim on Linux if you also want to run it with Docker Once you have a zip file with the new AirSim environment (or prefer to use one from the Official Releases ), you need to modify some of the scripts in the azure directory of the repository to point to the new environment: - In azure/azure-env-creation/configure-vm.ps1 , modify all the parameters starting with $airSimBinary with the new values - In azure/start-airsim.ps1 , modify $airSimExecutable and $airSimProcessName with the new values If you are using the docker image, you also need a linux binary zip file and modify the following Docker-related files: - In azure/docker/Dockerfile , modify the AIRSIM_BINARY_ZIP_URL and AIRSIM_BINARY_ZIP_FILENAME ENV declarations with the new values - In azure/docker/docker-entrypoint.sh , modify AIRSIM_EXECUTABLE with the new value Maintaining this development environment # Several components of this development environment (ARM templates, initialization scripts and VSCode tasks) directly depend on the current directory structures file names and repository locations. When planning to modify/fork any of those, make sure to check every script and template to make any required adjustment.","title":"AirSim on Azure"},{"location":"azure/#airsim-development-environment-on-azure","text":"This document explains how to automate the creation of a development environment on Azure and code and debug a Python application connected to AirSim using Visual Studio Code","title":"AirSim Development Environment on Azure"},{"location":"azure/#automatically-deploy-your-azure-vm","text":"Use this template to create, deploy and configure an Azure VM to work with AirSim Note: the VM deployment and configuration process may take 20+ minutes to complete","title":"Automatically Deploy Your Azure VM"},{"location":"azure/#regarding-the-deployment-of-the-azure-vm","text":"When using an Azure Trial account, the default vCPU quota may not be enough to allocate the required VM to run AirSim. If that's the case, you will see an error when trying to create the VM and will have to make a support request to increase the default quota. To avoid charges for the Virtual Machine usage while not in use, remember to deallocate its resources from the Azure Portal or use the following command from the Azure CLI: bash az vm deallocate --resource-group MyResourceGroup --name MyVMName","title":"Regarding the deployment of the Azure VM"},{"location":"azure/#code-and-debug-from-visual-studio-code-and-remote-ssh","text":"Install Visual Studio Code Install the Remote - SSH extension Press F1 and run the Remote - SSH: Connect to host... command Add the recently create VM details. For instance, AzureUser@11.22.33.44 Run the Remote - SSH: Connect to host... command again, and now select the newly added connection. Once connected, click on the Clone Repository button in Visual Studio Code, and either clone this repository in the remote VM and open just the azure folder , or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. It is important to open that directory so Visual Studio Code can use the specific .vscode directory for the scenario and not the general AirSim .vscode directory. It contains the recommended extensions to install, the task to start AirSim remotely and the launch configuration for the Python application. Install all the recommended extensions Press F1 and select the Tasks: Run Task option. Then, select the Start AirSim task from Visual Studio Code to execute the start-airsim.ps1 script from Visual Studio Code. Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges","title":"Code and debug from Visual Studio Code and Remote SSH"},{"location":"azure/#code-and-debug-from-a-local-visual-studio-code-and-connect-to-airsim-via-forwarded-ports","text":"Note: this scenario, will be using two Visual Studio Code instances. The first one will be used as a bridge to forward ports via SSH to the Azure VM and execute remote processes, and the second one will be used for local Python development. To be able to reach the VM from the local Python code, it is required to keep the Remote - SSH instance of Visual Studio Code opened, while working with the local Python environment on the second instance Open the first Visual Studio Code instance Follow the steps in the previous section to connect via Remote - SSH In the Remote Explorer , add the port 41451 as a forwarded port to localhost Either run the Start AirSim task on the Visual Studio Code with the remote session as explained in the previous scenario or manually start the AirSim binary in the VM Open a second Visual Studio Code instance, without disconnecting or closing the first one Either clone this repository locally and open just the azure folder in Visual Studio Code, or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. Run pip install -r requirements.txt inside the app directory Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges","title":"Code and debug from a local Visual Studio Code and connect to AirSim via forwarded ports"},{"location":"azure/#running-with-docker","text":"Once both the AirSim environment and the Python application are ready, you can package everything as a Docker image. The sample project inside the azure directory is already prepared to run a prebuilt AirSim binary and Python code using Docker. This would be a perfect scenario when you want to run the simulation at scale. For instance, you could have several different configurations for the same simulation and execute them in a parallel, unattended way using a Docker image on Azure Container Services Since AirSim requires access to the host GPU, it is required to use a Docker runtime that supports it. For more information about running AirSim in Docker, click here . When using Azure Container Services to run this image, the only extra-requirement is to add GPU support to the Container Group where it will be deployed. It can use either public docker images from DockerHub or images deployed to a private Azure Container Registry","title":"Running with Docker"},{"location":"azure/#building-the-docker-image","text":"docker build -t / -f ./docker/Dockerfile .`","title":"Building the docker image"},{"location":"azure/#using-a-different-airsim-binary","text":"To use a different AirSim binary, first check the official documentation on How to Build AirSim on Windows and How to Build AirSim on Linux if you also want to run it with Docker Once you have a zip file with the new AirSim environment (or prefer to use one from the Official Releases ), you need to modify some of the scripts in the azure directory of the repository to point to the new environment: - In azure/azure-env-creation/configure-vm.ps1 , modify all the parameters starting with $airSimBinary with the new values - In azure/start-airsim.ps1 , modify $airSimExecutable and $airSimProcessName with the new values If you are using the docker image, you also need a linux binary zip file and modify the following Docker-related files: - In azure/docker/Dockerfile , modify the AIRSIM_BINARY_ZIP_URL and AIRSIM_BINARY_ZIP_FILENAME ENV declarations with the new values - In azure/docker/docker-entrypoint.sh , modify AIRSIM_EXECUTABLE with the new value","title":"Using a different AirSim binary"},{"location":"azure/#maintaining-this-development-environment","text":"Several components of this development environment (ARM templates, initialization scripts and VSCode tasks) directly depend on the current directory structures file names and repository locations. When planning to modify/fork any of those, make sure to check every script and template to make any required adjustment.","title":"Maintaining this development environment"},{"location":"build_linux/","text":"Build AirSim on Linux & MacOS # The current recommended and tested environment is Ubuntu 18.04 LTS . Theoretically, you can build on other distros as well, but we haven't tested it. Only macOS Catalina (10.15) is supported. We've two options - you can either build inside docker containers or your host machine. Docker # Please see instructions here Host machine # Pre-build Setup # Linux - Build Unreal Engine # Make sure you are registered with Epic Games . This is required to get source code access for Unreal Engine. Clone Unreal in your favorite folder and build it (this may take a while!). Note : We only support Unreal >= 4.22 at present. We recommend using 4.24. bash # go to the folder where you clone GitHub projects git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git cd UnrealEngine ./Setup.sh ./GenerateProjectFiles.sh make macOS - Download Unreal Engine # Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects. Build AirSim # Clone AirSim and build it: bash # go to the folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git cd AirSim By default AirSim recommends using clang 8 to build the binaries as those will be compatible with UE 4.24. The setup script will install the right version of cmake, llvm, and eigen. bash ./setup.sh ./build.sh Build Unreal Environment # Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment if you'd like to setup your own environment. How to Use AirSim # Linux # Once AirSim is setup: Go to UnrealEngine installation folder and start Unreal by running ./Engine/Binaries/Linux/UE4Editor . When Unreal Engine prompts for opening or creating project, select Browse and choose AirSim/Unreal/Environments/Blocks (or your custom Unreal project). Alternatively, the project file can be passed as a commandline argument. For Blocks: ./Engine/Binaries/Linux/UE4Editor /Unreal/Environments/Blocks/Blocks.uproject If you get prompts to convert project, look for More Options or Convert-In-Place option. If you get prompted to build, choose Yes. If you get prompted to disable AirSim plugin, choose No. After Unreal Editor loads, press Play button. Mac # Browse to AirSim/Unreal/Environments/Blocks . Run ./GenerateProjectFiles.sh from the terminal, where UE_PATH is the path to the Unreal installation folder. (By default, this is /Users/Shared/Epic\\ Games/UE_4.24/ ) The script creates an XCode workspace by the name Blocks.xcworkspace. Open the XCode workspace, and press the Build and run button in the top left. After Unreal Editor loads, press Play button. See Using APIs and settings.json for various options available for AirSim usage. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. [Optional] Setup Remote Control (Multirotor Only) # A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard. FAQs # I'm getting error \" could not be compiled. Try rebuilding from source manually\". This could either happen because of compile error or the fact that your gch files are outdated. Look in to your console window. Do you see something like below? fatal error: file '/usr/include/linux/version.h''/usr/include/linux/version.h' has been modified since the precompiled header If this is the case then look for *.gch file(s) that follows after that message, delete them and try again. Here's relevant thread on Unreal Engine forums. If you see other compile errors in console then open up those source files and see if it is due to changes you made. If not, then report it as issue on GitHub. Unreal crashed! How do I know what went wrong? Go to the MyUnrealProject/Saved/Crashes folder and search for the file MyProject.log within its subdirectories. At the end of this file you will see the stack trace and messages. You can also take a look at the Diagnostics.txt file. How do I use an IDE on Linux? You can use Qt Creator or CodeLite. Instructions for Qt Creator are available here . Can I cross compile for Linux from a Windows machine? Yes, you can, but we haven't tested it. You can find the instructions here . What compiler and stdlib does AirSim use? We use the same compiler that Unreal Engine uses, Clang 8 , and stdlib, libc++ . AirSim's setup.sh will automatically download them. What version of CMake does the AirSim build use? 3.10.0 or higher. This is not the default in Ubuntu 16.04 so setup.sh installs it for you. You can check your CMake version using cmake --version . If you have an older version, follow these instructions or see the CMake website . Can I compile AirSim in BashOnWindows? Yes, however, you can't run Unreal from BashOnWindows. So this is kind of useful to check a Linux compile, but not for an end-to-end run. See the BashOnWindows install guide . Make sure to have the latest version (Windows 10 Creators Edition) as previous versions had various issues. Also, don't invoke bash from Visual Studio Command Prompt , otherwise CMake might find VC++ and try and use that! Where can I find more info on running Unreal on Linux? Start here: Unreal on Linux Building Unreal on Linux Unreal Linux Support Unreal Cross Compilation","title":"Build on Linux"},{"location":"build_linux/#build-airsim-on-linux-macos","text":"The current recommended and tested environment is Ubuntu 18.04 LTS . Theoretically, you can build on other distros as well, but we haven't tested it. Only macOS Catalina (10.15) is supported. We've two options - you can either build inside docker containers or your host machine.","title":"Build AirSim on Linux & MacOS"},{"location":"build_linux/#docker","text":"Please see instructions here","title":"Docker"},{"location":"build_linux/#host-machine","text":"","title":"Host machine"},{"location":"build_linux/#pre-build-setup","text":"","title":"Pre-build Setup"},{"location":"build_linux/#linux-build-unreal-engine","text":"Make sure you are registered with Epic Games . This is required to get source code access for Unreal Engine. Clone Unreal in your favorite folder and build it (this may take a while!). Note : We only support Unreal >= 4.22 at present. We recommend using 4.24. bash # go to the folder where you clone GitHub projects git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git cd UnrealEngine ./Setup.sh ./GenerateProjectFiles.sh make","title":"Linux - Build Unreal Engine"},{"location":"build_linux/#macos-download-unreal-engine","text":"Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects.","title":"macOS - Download Unreal Engine"},{"location":"build_linux/#build-airsim","text":"Clone AirSim and build it: bash # go to the folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git cd AirSim By default AirSim recommends using clang 8 to build the binaries as those will be compatible with UE 4.24. The setup script will install the right version of cmake, llvm, and eigen. bash ./setup.sh ./build.sh","title":"Build AirSim"},{"location":"build_linux/#build-unreal-environment","text":"Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment if you'd like to setup your own environment.","title":"Build Unreal Environment"},{"location":"build_linux/#how-to-use-airsim","text":"","title":"How to Use AirSim"},{"location":"build_linux/#linux","text":"Once AirSim is setup: Go to UnrealEngine installation folder and start Unreal by running ./Engine/Binaries/Linux/UE4Editor . When Unreal Engine prompts for opening or creating project, select Browse and choose AirSim/Unreal/Environments/Blocks (or your custom Unreal project). Alternatively, the project file can be passed as a commandline argument. For Blocks: ./Engine/Binaries/Linux/UE4Editor /Unreal/Environments/Blocks/Blocks.uproject If you get prompts to convert project, look for More Options or Convert-In-Place option. If you get prompted to build, choose Yes. If you get prompted to disable AirSim plugin, choose No. After Unreal Editor loads, press Play button.","title":"Linux"},{"location":"build_linux/#mac","text":"Browse to AirSim/Unreal/Environments/Blocks . Run ./GenerateProjectFiles.sh from the terminal, where UE_PATH is the path to the Unreal installation folder. (By default, this is /Users/Shared/Epic\\ Games/UE_4.24/ ) The script creates an XCode workspace by the name Blocks.xcworkspace. Open the XCode workspace, and press the Build and run button in the top left. After Unreal Editor loads, press Play button. See Using APIs and settings.json for various options available for AirSim usage. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked.","title":"Mac"},{"location":"build_linux/#optional-setup-remote-control-multirotor-only","text":"A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard.","title":"[Optional] Setup Remote Control (Multirotor Only)"},{"location":"build_linux/#faqs","text":"I'm getting error \" could not be compiled. Try rebuilding from source manually\". This could either happen because of compile error or the fact that your gch files are outdated. Look in to your console window. Do you see something like below? fatal error: file '/usr/include/linux/version.h''/usr/include/linux/version.h' has been modified since the precompiled header If this is the case then look for *.gch file(s) that follows after that message, delete them and try again. Here's relevant thread on Unreal Engine forums. If you see other compile errors in console then open up those source files and see if it is due to changes you made. If not, then report it as issue on GitHub. Unreal crashed! How do I know what went wrong? Go to the MyUnrealProject/Saved/Crashes folder and search for the file MyProject.log within its subdirectories. At the end of this file you will see the stack trace and messages. You can also take a look at the Diagnostics.txt file. How do I use an IDE on Linux? You can use Qt Creator or CodeLite. Instructions for Qt Creator are available here . Can I cross compile for Linux from a Windows machine? Yes, you can, but we haven't tested it. You can find the instructions here . What compiler and stdlib does AirSim use? We use the same compiler that Unreal Engine uses, Clang 8 , and stdlib, libc++ . AirSim's setup.sh will automatically download them. What version of CMake does the AirSim build use? 3.10.0 or higher. This is not the default in Ubuntu 16.04 so setup.sh installs it for you. You can check your CMake version using cmake --version . If you have an older version, follow these instructions or see the CMake website . Can I compile AirSim in BashOnWindows? Yes, however, you can't run Unreal from BashOnWindows. So this is kind of useful to check a Linux compile, but not for an end-to-end run. See the BashOnWindows install guide . Make sure to have the latest version (Windows 10 Creators Edition) as previous versions had various issues. Also, don't invoke bash from Visual Studio Command Prompt , otherwise CMake might find VC++ and try and use that! Where can I find more info on running Unreal on Linux? Start here: Unreal on Linux Building Unreal on Linux Unreal Linux Support Unreal Cross Compilation","title":"FAQs"},{"location":"build_windows/","text":"Build AirSim on Windows # Install Unreal Engine # Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects. Build AirSim # Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start Developer Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line. This will create ready to use plugin bits in the Unreal\\Plugins folder that can be dropped into any Unreal project. Build Unreal Project # Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment . Setup Remote Control (Multirotor only) # A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard. How to Use AirSim # Once AirSim is set up by following above steps, you can, Double click on .sln file to load the Blocks project in Unreal\\Environments\\Blocks (or .sln file in your own custom Unreal project). If you don't see .sln file then you probably haven't completed steps in Build Unreal Project section above. Select your Unreal project as Start Up project (for example, Blocks project) and make sure Build config is set to \"Develop Editor\" and x64. After Unreal Editor loads, press Play button. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. See Using APIs and settings.json for various options available. AirSim on Unity (Experimental) # Unity is another great game engine platform and we have an experimental integration of AirSim with Unity . Please note that this is work in progress and all features may not work yet. FAQ # How to force Unreal to use Visual Studio 2019? # If the default update_from_git.bat file results in VS 2017 project, then you may need to run the C:\\Program Files\\Epic Games\\UE_4.24\\Engine\\Binaries\\DotNET\\UnrealBuildTool.exe tool manually, with the command line options -projectfiles -project= -game -rocket -progress -2019 . If you are upgrading from 4.18 to 4.24 you may also need to add BuildSettingsVersion.V2 to your *.Target.cs and *Editor.Target.cs build files, like this: public AirSimNHTestTarget(TargetInfo Target) : base(Target) { Type = TargetType.Game; DefaultBuildSettings = BuildSettingsVersion.V2; ExtraModuleNames.AddRange(new string[] { \"AirSimNHTest\" }); } You may also need to edit this file: \"%APPDATA%\\Unreal Engine\\UnrealBuildTool\\BuildConfiguration.xml And add this Compiler version setting: VisualStudio2019 I get error C100 : An internal error has occurred in the compiler when running build.cmd # We have noticed this happening with VS version 15.9.0 and have checked-in a workaround in AirSim code. If you have this VS version, please make sure to pull the latest AirSim code. I get error \"'corecrt.h': No such file or directory\" or \"Windows SDK version 8.1 not found\" # Very likely you don't have Windows SDK installed with Visual Studio. How do I use PX4 firmware with AirSim? # By default, AirSim uses its own built-in firmware called simple_flight . There is no additional setup if you just want to go with it. If you want to switch to using PX4 instead then please see this guide . I made changes in Visual Studio but there is no effect # Sometimes the Unreal + VS build system doesn't recompile if you make changes to only header files. To ensure a recompile, make some Unreal based cpp file \"dirty\" like AirSimGameMode.cpp. Unreal still uses VS2015 or I'm getting some link error # Running several versions of VS can lead to issues when compiling UE projects. One problem that may arise is that UE will try to compile with an older version of VS which may or may not work. There are two settings in Unreal, one for for the engine and one for the project, to adjust the version of VS to be used. 1. Edit -> Editor preferences -> General -> Source code 2. Edit -> Project Settings -> Platforms -> Windows -> Toolchain ->CompilerVersion In some cases, these settings will still not lead to the desired result and errors such as the following might be produced: LINK : fatal error LNK1181: cannot open input file 'ws2_32.lib' To resolve such issues the following procedure can be applied: 1. Uninstall all old versions of VS using the VisualStudioUninstaller 2. Repair/Install VS 2019 3. Restart machine and install Epic launcher and desired version of the engine","title":"Build on Windows"},{"location":"build_windows/#build-airsim-on-windows","text":"","title":"Build AirSim on Windows"},{"location":"build_windows/#install-unreal-engine","text":"Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects.","title":"Install Unreal Engine"},{"location":"build_windows/#build-airsim","text":"Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start Developer Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line. This will create ready to use plugin bits in the Unreal\\Plugins folder that can be dropped into any Unreal project.","title":"Build AirSim"},{"location":"build_windows/#build-unreal-project","text":"Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment .","title":"Build Unreal Project"},{"location":"build_windows/#setup-remote-control-multirotor-only","text":"A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard.","title":"Setup Remote Control (Multirotor only)"},{"location":"build_windows/#how-to-use-airsim","text":"Once AirSim is set up by following above steps, you can, Double click on .sln file to load the Blocks project in Unreal\\Environments\\Blocks (or .sln file in your own custom Unreal project). If you don't see .sln file then you probably haven't completed steps in Build Unreal Project section above. Select your Unreal project as Start Up project (for example, Blocks project) and make sure Build config is set to \"Develop Editor\" and x64. After Unreal Editor loads, press Play button. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. See Using APIs and settings.json for various options available.","title":"How to Use AirSim"},{"location":"build_windows/#airsim-on-unity-experimental","text":"Unity is another great game engine platform and we have an experimental integration of AirSim with Unity . Please note that this is work in progress and all features may not work yet.","title":"AirSim on Unity (Experimental)"},{"location":"build_windows/#faq","text":"","title":"FAQ"},{"location":"build_windows/#how-to-force-unreal-to-use-visual-studio-2019","text":"If the default update_from_git.bat file results in VS 2017 project, then you may need to run the C:\\Program Files\\Epic Games\\UE_4.24\\Engine\\Binaries\\DotNET\\UnrealBuildTool.exe tool manually, with the command line options -projectfiles -project= -game -rocket -progress -2019 . If you are upgrading from 4.18 to 4.24 you may also need to add BuildSettingsVersion.V2 to your *.Target.cs and *Editor.Target.cs build files, like this: public AirSimNHTestTarget(TargetInfo Target) : base(Target) { Type = TargetType.Game; DefaultBuildSettings = BuildSettingsVersion.V2; ExtraModuleNames.AddRange(new string[] { \"AirSimNHTest\" }); } You may also need to edit this file: \"%APPDATA%\\Unreal Engine\\UnrealBuildTool\\BuildConfiguration.xml And add this Compiler version setting: VisualStudio2019 ","title":"How to force Unreal to use Visual Studio 2019?"},{"location":"build_windows/#i-get-error-c100-an-internal-error-has-occurred-in-the-compiler-when-running-buildcmd","text":"We have noticed this happening with VS version 15.9.0 and have checked-in a workaround in AirSim code. If you have this VS version, please make sure to pull the latest AirSim code.","title":"I get error C100 : An internal error has occurred in the compiler when running build.cmd"},{"location":"build_windows/#i-get-error-corecrth-no-such-file-or-directory-or-windows-sdk-version-81-not-found","text":"Very likely you don't have Windows SDK installed with Visual Studio.","title":"I get error \"'corecrt.h': No such file or directory\" or \"Windows SDK version 8.1 not found\""},{"location":"build_windows/#how-do-i-use-px4-firmware-with-airsim","text":"By default, AirSim uses its own built-in firmware called simple_flight . There is no additional setup if you just want to go with it. If you want to switch to using PX4 instead then please see this guide .","title":"How do I use PX4 firmware with AirSim?"},{"location":"build_windows/#i-made-changes-in-visual-studio-but-there-is-no-effect","text":"Sometimes the Unreal + VS build system doesn't recompile if you make changes to only header files. To ensure a recompile, make some Unreal based cpp file \"dirty\" like AirSimGameMode.cpp.","title":"I made changes in Visual Studio but there is no effect"},{"location":"build_windows/#unreal-still-uses-vs2015-or-im-getting-some-link-error","text":"Running several versions of VS can lead to issues when compiling UE projects. One problem that may arise is that UE will try to compile with an older version of VS which may or may not work. There are two settings in Unreal, one for for the engine and one for the project, to adjust the version of VS to be used. 1. Edit -> Editor preferences -> General -> Source code 2. Edit -> Project Settings -> Platforms -> Windows -> Toolchain ->CompilerVersion In some cases, these settings will still not lead to the desired result and errors such as the following might be produced: LINK : fatal error LNK1181: cannot open input file 'ws2_32.lib' To resolve such issues the following procedure can be applied: 1. Uninstall all old versions of VS using the VisualStudioUninstaller 2. Repair/Install VS 2019 3. Restart machine and install Epic launcher and desired version of the engine","title":"Unreal still uses VS2015 or I'm getting some link error"},{"location":"camera_views/","text":"Camera Views # The camera views that are shown on screen are the camera views you can fetch via the simGetImages API . From left to right is the depth view, segmentation view and the FPV view. See Image APIs for description of various available views. Turning ON/OFF Views # Press F1 key to see keyboard shortcuts for turning on/off any or all views. You can also select various view modes there, such as \"Fly with Me\" mode, FPV mode and \"Ground View\" mode. Configuring Sub-Windows # Now you can select what is shown by each of above sub windows. For instance, you can chose to show surface normals in first window (instead of depth) and disparity in second window (instead of segmentation). Below is the settings value you can use in settings.json : { \"SubWindows\": [ {\"Index\": 1, \"ImageType\": 5}, {\"Index\": 2, \"ImageType\": 3} ] } Performance Impact # Note : This section is outdated and has not been updated for new performance enhancement changes. Now rendering these views does impact the FPS performance of the game, since this is additional work for the GPU. The following shows the impact on FPS when you open these views. This is measured on Intel core i7 computer with 32 gb RAM and a GeForce GTX 1080 graphics card running the Modular Neighborhood map, using cooked debug bits, no debugger or GameEditor open. The normal state with no subviews open is measuring around 16 ms per frame, which means it is keeping a nice steady 60 FPS (which is the target FPS). As it climbs up to 35ms the FPS drops to around 28 frames per second, spiking to 40ms means a few drops to 25 fps. The simulator can still function and fly correctly when all this is going on even in the worse case because the physics is decoupled from the rendering. However if the delay gets too high such that the communication with PX4 hardware is interrupted due to overly busy CPU then the flight can stall due to timeout in the offboard control messages. On the computer where this was measured the drone could fly the path.py program without any problems with all views open, and with 3 python scripts running to capture each view type. But there was one stall during this flight, but it recovered gracefully and completed the path. So it was right on the limit. The following shows the impact on CPU, perhaps a bit surprisingly, the CPU impact is also non trivial.","title":"Camera Views"},{"location":"camera_views/#camera-views","text":"The camera views that are shown on screen are the camera views you can fetch via the simGetImages API . From left to right is the depth view, segmentation view and the FPV view. See Image APIs for description of various available views.","title":"Camera Views"},{"location":"camera_views/#turning-onoff-views","text":"Press F1 key to see keyboard shortcuts for turning on/off any or all views. You can also select various view modes there, such as \"Fly with Me\" mode, FPV mode and \"Ground View\" mode.","title":"Turning ON/OFF Views"},{"location":"camera_views/#configuring-sub-windows","text":"Now you can select what is shown by each of above sub windows. For instance, you can chose to show surface normals in first window (instead of depth) and disparity in second window (instead of segmentation). Below is the settings value you can use in settings.json : { \"SubWindows\": [ {\"Index\": 1, \"ImageType\": 5}, {\"Index\": 2, \"ImageType\": 3} ] }","title":"Configuring Sub-Windows"},{"location":"camera_views/#performance-impact","text":"Note : This section is outdated and has not been updated for new performance enhancement changes. Now rendering these views does impact the FPS performance of the game, since this is additional work for the GPU. The following shows the impact on FPS when you open these views. This is measured on Intel core i7 computer with 32 gb RAM and a GeForce GTX 1080 graphics card running the Modular Neighborhood map, using cooked debug bits, no debugger or GameEditor open. The normal state with no subviews open is measuring around 16 ms per frame, which means it is keeping a nice steady 60 FPS (which is the target FPS). As it climbs up to 35ms the FPS drops to around 28 frames per second, spiking to 40ms means a few drops to 25 fps. The simulator can still function and fly correctly when all this is going on even in the worse case because the physics is decoupled from the rendering. However if the delay gets too high such that the communication with PX4 hardware is interrupted due to overly busy CPU then the flight can stall due to timeout in the offboard control messages. On the computer where this was measured the drone could fly the path.py program without any problems with all views open, and with 3 python scripts running to capture each view type. But there was one stall during this flight, but it recovered gracefully and completed the path. So it was right on the limit. The following shows the impact on CPU, perhaps a bit surprisingly, the CPU impact is also non trivial.","title":"Performance Impact"},{"location":"cmake_linux/","text":"Installing cmake on Linux # If you don't have cmake version 3.10 (for example, 3.2.2 is the default on Ubuntu 14) you can run the following: mkdir ~/cmake-3.10.2 cd ~/cmake-3.10.2 wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.sh Now you have to run this command by itself (it is interactive) sh cmake-3.10.2-Linux-x86_64.sh --prefix ~/cmake-3.10.2 Answer 'n' to the question about creating another cmake-3.10.2-Linux-x86_64 folder and then sudo update-alternatives --install /usr/bin/cmake cmake ~/cmake-3.10.2/bin/cmake 60 Now type cmake --version to make sure your cmake version is 3.10.2.","title":"Installing cmake on Linux"},{"location":"cmake_linux/#installing-cmake-on-linux","text":"If you don't have cmake version 3.10 (for example, 3.2.2 is the default on Ubuntu 14) you can run the following: mkdir ~/cmake-3.10.2 cd ~/cmake-3.10.2 wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.sh Now you have to run this command by itself (it is interactive) sh cmake-3.10.2-Linux-x86_64.sh --prefix ~/cmake-3.10.2 Answer 'n' to the question about creating another cmake-3.10.2-Linux-x86_64 folder and then sudo update-alternatives --install /usr/bin/cmake cmake ~/cmake-3.10.2/bin/cmake 60 Now type cmake --version to make sure your cmake version is 3.10.2.","title":"Installing cmake on Linux"},{"location":"code_structure/","text":"AirLib # Majority of the code is located in AirLib. This is a self-contained library that you should be able to compile with any C++11 compiler. AirLib consists of the following components: 1. Physics engine: This is header-only physics engine. It is designed to be fast and extensible to implement different vehicles. 2. Sensor models: This is header-only models for Barometer, IMU, GPS and Magnetometer 3. Vehicle models: This is header-only models for vehicle configurations and models. Currently we have implemented model for a MultiRotor and a configuration for PX4 QuadRotor in the X config. 4. Control library: This part of AirLib provides abstract base class for our APIs and concrete implementation for specific vehicle platforms such as MavLink. It also has classes for the RPC client and server. Unreal/Plugins/AirSim # This is the only portion of project which is dependent on Unreal engine. We have kept it isolated so we can implement simulator for other platforms as well (for example, Unity). The Unreal code takes advantage of its UObject based classes including Blueprints. 1. SimMode_ classes : We wish to support various simulator modes such as pure Computer Vision mode where there is no drone. The SimMode classes help implement many different modes. 2. VehiclePawnBase : This is the base class for all vehicle pawn visualizations. 3. VehicleBase : This class provides abstract interface to implement a combination of rendering component (i.e. Unreal pawn), physics component (i.e. MultiRotor) and controller (i.e. MavLinkHelper). MavLinkCom # This is the library developed by our own team member Chris Lovett that provides C++ classes to talk to the MavLink devices. This library is stand alone and can be used in any project. See MavLinkCom for more info. Sample Programs # We have created a few sample programs to demonstrate how to use the API. See HelloDrone and DroneShell. DroneShell demonstrates how to connect to the simulator using UDP. The simulator is running a server (similar to DroneServer). Contributing # See Contribution Guidelines Unreal Framework # The following picture illustrates how AirSim is loaded and invoked by the Unreal Game Engine:","title":"Code Structure"},{"location":"code_structure/#airlib","text":"Majority of the code is located in AirLib. This is a self-contained library that you should be able to compile with any C++11 compiler. AirLib consists of the following components: 1. Physics engine: This is header-only physics engine. It is designed to be fast and extensible to implement different vehicles. 2. Sensor models: This is header-only models for Barometer, IMU, GPS and Magnetometer 3. Vehicle models: This is header-only models for vehicle configurations and models. Currently we have implemented model for a MultiRotor and a configuration for PX4 QuadRotor in the X config. 4. Control library: This part of AirLib provides abstract base class for our APIs and concrete implementation for specific vehicle platforms such as MavLink. It also has classes for the RPC client and server.","title":"AirLib"},{"location":"code_structure/#unrealpluginsairsim","text":"This is the only portion of project which is dependent on Unreal engine. We have kept it isolated so we can implement simulator for other platforms as well (for example, Unity). The Unreal code takes advantage of its UObject based classes including Blueprints. 1. SimMode_ classes : We wish to support various simulator modes such as pure Computer Vision mode where there is no drone. The SimMode classes help implement many different modes. 2. VehiclePawnBase : This is the base class for all vehicle pawn visualizations. 3. VehicleBase : This class provides abstract interface to implement a combination of rendering component (i.e. Unreal pawn), physics component (i.e. MultiRotor) and controller (i.e. MavLinkHelper).","title":"Unreal/Plugins/AirSim"},{"location":"code_structure/#mavlinkcom","text":"This is the library developed by our own team member Chris Lovett that provides C++ classes to talk to the MavLink devices. This library is stand alone and can be used in any project. See MavLinkCom for more info.","title":"MavLinkCom"},{"location":"code_structure/#sample-programs","text":"We have created a few sample programs to demonstrate how to use the API. See HelloDrone and DroneShell. DroneShell demonstrates how to connect to the simulator using UDP. The simulator is running a server (similar to DroneServer).","title":"Sample Programs"},{"location":"code_structure/#contributing","text":"See Contribution Guidelines","title":"Contributing"},{"location":"code_structure/#unreal-framework","text":"The following picture illustrates how AirSim is loaded and invoked by the Unreal Game Engine:","title":"Unreal Framework"},{"location":"coding_guidelines/","text":"Modern C++ Coding Guidelines # We are using Modern C++11. Smart pointers, Lambdas, and C++11 multithreading primitives are your friend. Quick Note # The great thing about \"standards\" is that there are many to chose from: ISO , Sutter & Stroustrup , ROS , LINUX , Google's , Microsoft's , CERN's , GCC's , ARM's , LLVM's and probably thousands of others. Unfortunately most of these can't even agree on something as basic as how to name a class or a constant. This is probably due to the fact that these standards often carry lots of legacy issues due to supporting existing code bases. The intention behind this document is to create guidance that remains as close to ISO, Sutter & Stroustrup and ROS while resolving as many conflicts, disadvantages and inconsistencies as possible among them. Naming Conventions # Avoid using any sort of Hungarian notation on names and \"_ptr\" on pointers. Code Element Style Comment Namespace under_scored Differentiate from class names Class name CamelCase To differentiate from STL types which ISO recommends (do not use \"C\" or \"T\" prefixes) Function name camelCase Lower case start is almost universal except for .Net world Parameters/Locals under_scored Vast majority of standards recommends this because _ is more readable to C++ crowd (although not much to Java/.Net crowd) Member variables under_scored_with_ The prefix _ is heavily discouraged as ISO has rules around reserving _identifiers, so we recommend suffix instead Enums and its members CamelCase Most except very old standards agree with this one Globals g_under_scored You shouldn't have these in first place! Constants UPPER_CASE Very contentious and we just have to pick one here, unless if is a private constant in class or method, then use naming for Members or Locals File names Match case of class name in file Lot of pro and cons either way but this removes inconsistency in auto generated code (important for ROS) Header Files # Use a namespace qualified #ifdef to protect against multiple inclusion: #ifndef msr_airsim_MyHeader_hpp #define msr_airsim_MyHeader_hpp //--your code #endif The reason we don't use #pragma once is because it's not supported if same header file exists at multiple places (which might be possible under ROS build system!). Bracketing # Inside function or method body place curly bracket on same line. Outside that the Namespace, Class and methods levels use separate line. This is called K&R style and its variants are widely used in C++ vs other styles which are more popular in other languages. Notice that curlies are not required if you have single statement, but complex statements are easier to keep correct with the braces. int main(int argc, char* argv[]) { while (x == y) { f0(); if (cont()) { f1(); } else { f2(); f3(); } if (x > 100) break; } } Const and References # Religiously review all non-scalar parameters you declare to be candidate for const and references. If you are coming from languages such as C#/Java/Python, the most often mistake you would make is to pass parameters by value instead of const T&; Especially most of the strings, vectors and maps you want to pass as const T&; (if they are readonly) or T& (if they are writable). Also add const suffix to methods as much as possible. Overriding # When overriding virtual method, use override suffix. Pointers # This is really about memory management. A simulator has much performance critical code, so we try and avoid overloading the memory manager with lots of calls to new/delete. We also want to avoid too much copying of things on the stack, so we pass things by reference when ever possible. But when the object really needs to live longer than the call stack you often need to allocate that object on the heap, and so you have a pointer. Now, if management of the lifetime of that object is going to be tricky we recommend using C++ 11 smart pointers . But smart pointers do have a cost, so don\u2019t use them blindly everywhere. For private code where performance is paramount, raw pointers can be used. Raw pointers are also often needed when interfacing with legacy systems that only accept pointer types, for example, sockets API. But we try to wrap those legacy interfaces as much as possible and avoid that style of programming from leaking into the larger code base. Religiously check if you can use const everywhere, for example, const float * const xP . Avoid using prefix or suffix to indicate pointer types in variable names, i.e. use my_obj instead of myobj_ptr except in cases where it might make sense to differentiate variables better, for example, int mynum = 5; int* mynum_ptr = mynum; This is Too Short, ye? # Yes, and it's on purpose because no one likes to read 200 page coding guidelines. The goal here is to cover only most significant things which are already not covered by strict mode compilation in GCC and Level 4 warnings-as-errors in VC++. If you had like to know about how to write better code in C++, please see GotW and Effective Modern C++ book.","title":"Coding Guidelines"},{"location":"coding_guidelines/#modern-c-coding-guidelines","text":"We are using Modern C++11. Smart pointers, Lambdas, and C++11 multithreading primitives are your friend.","title":"Modern C++ Coding Guidelines"},{"location":"coding_guidelines/#quick-note","text":"The great thing about \"standards\" is that there are many to chose from: ISO , Sutter & Stroustrup , ROS , LINUX , Google's , Microsoft's , CERN's , GCC's , ARM's , LLVM's and probably thousands of others. Unfortunately most of these can't even agree on something as basic as how to name a class or a constant. This is probably due to the fact that these standards often carry lots of legacy issues due to supporting existing code bases. The intention behind this document is to create guidance that remains as close to ISO, Sutter & Stroustrup and ROS while resolving as many conflicts, disadvantages and inconsistencies as possible among them.","title":"Quick Note"},{"location":"coding_guidelines/#naming-conventions","text":"Avoid using any sort of Hungarian notation on names and \"_ptr\" on pointers. Code Element Style Comment Namespace under_scored Differentiate from class names Class name CamelCase To differentiate from STL types which ISO recommends (do not use \"C\" or \"T\" prefixes) Function name camelCase Lower case start is almost universal except for .Net world Parameters/Locals under_scored Vast majority of standards recommends this because _ is more readable to C++ crowd (although not much to Java/.Net crowd) Member variables under_scored_with_ The prefix _ is heavily discouraged as ISO has rules around reserving _identifiers, so we recommend suffix instead Enums and its members CamelCase Most except very old standards agree with this one Globals g_under_scored You shouldn't have these in first place! Constants UPPER_CASE Very contentious and we just have to pick one here, unless if is a private constant in class or method, then use naming for Members or Locals File names Match case of class name in file Lot of pro and cons either way but this removes inconsistency in auto generated code (important for ROS)","title":"Naming Conventions"},{"location":"coding_guidelines/#header-files","text":"Use a namespace qualified #ifdef to protect against multiple inclusion: #ifndef msr_airsim_MyHeader_hpp #define msr_airsim_MyHeader_hpp //--your code #endif The reason we don't use #pragma once is because it's not supported if same header file exists at multiple places (which might be possible under ROS build system!).","title":"Header Files"},{"location":"coding_guidelines/#bracketing","text":"Inside function or method body place curly bracket on same line. Outside that the Namespace, Class and methods levels use separate line. This is called K&R style and its variants are widely used in C++ vs other styles which are more popular in other languages. Notice that curlies are not required if you have single statement, but complex statements are easier to keep correct with the braces. int main(int argc, char* argv[]) { while (x == y) { f0(); if (cont()) { f1(); } else { f2(); f3(); } if (x > 100) break; } }","title":"Bracketing"},{"location":"coding_guidelines/#const-and-references","text":"Religiously review all non-scalar parameters you declare to be candidate for const and references. If you are coming from languages such as C#/Java/Python, the most often mistake you would make is to pass parameters by value instead of const T&; Especially most of the strings, vectors and maps you want to pass as const T&; (if they are readonly) or T& (if they are writable). Also add const suffix to methods as much as possible.","title":"Const and References"},{"location":"coding_guidelines/#overriding","text":"When overriding virtual method, use override suffix.","title":"Overriding"},{"location":"coding_guidelines/#pointers","text":"This is really about memory management. A simulator has much performance critical code, so we try and avoid overloading the memory manager with lots of calls to new/delete. We also want to avoid too much copying of things on the stack, so we pass things by reference when ever possible. But when the object really needs to live longer than the call stack you often need to allocate that object on the heap, and so you have a pointer. Now, if management of the lifetime of that object is going to be tricky we recommend using C++ 11 smart pointers . But smart pointers do have a cost, so don\u2019t use them blindly everywhere. For private code where performance is paramount, raw pointers can be used. Raw pointers are also often needed when interfacing with legacy systems that only accept pointer types, for example, sockets API. But we try to wrap those legacy interfaces as much as possible and avoid that style of programming from leaking into the larger code base. Religiously check if you can use const everywhere, for example, const float * const xP . Avoid using prefix or suffix to indicate pointer types in variable names, i.e. use my_obj instead of myobj_ptr except in cases where it might make sense to differentiate variables better, for example, int mynum = 5; int* mynum_ptr = mynum;","title":"Pointers"},{"location":"coding_guidelines/#this-is-too-short-ye","text":"Yes, and it's on purpose because no one likes to read 200 page coding guidelines. The goal here is to cover only most significant things which are already not covered by strict mode compilation in GCC and Level 4 warnings-as-errors in VC++. If you had like to know about how to write better code in C++, please see GotW and Effective Modern C++ book.","title":"This is Too Short, ye?"},{"location":"create_issue/","text":"How to Create Issue or Ask Question Effectively # AirSim is open source project and contributors like you keeps it going. It is important to respect contributors time and effort when you are asking a question or filing an issue. Your chances of receiving helpful response would increase if you follow below guidelines: DOs # Search issues to see if someone already has asked it. Chose title that is short and summarizes well. Copy and paste full error message. Precisely describe steps you used that produced the error message or symptom. Describe what vehicle, mode, OS, AirSim version and other settings you are using. Copy and paste minimal version of code that reproduces the problem. Tell us what the goal you want to achieve or expected output. Tell us what you did so far to debug this issue. DONT'S # Do not use \"Please help\" etc in the title. See above. Do not copy and paste screen shot of error message. Copy and paste text. Do not use \"it doesn't work\". Precisely state what is the error message or symptom. Do not ask to write code for you. Contribute !","title":"Create Issue"},{"location":"create_issue/#how-to-create-issue-or-ask-question-effectively","text":"AirSim is open source project and contributors like you keeps it going. It is important to respect contributors time and effort when you are asking a question or filing an issue. Your chances of receiving helpful response would increase if you follow below guidelines:","title":"How to Create Issue or Ask Question Effectively"},{"location":"create_issue/#dos","text":"Search issues to see if someone already has asked it. Chose title that is short and summarizes well. Copy and paste full error message. Precisely describe steps you used that produced the error message or symptom. Describe what vehicle, mode, OS, AirSim version and other settings you are using. Copy and paste minimal version of code that reproduces the problem. Tell us what the goal you want to achieve or expected output. Tell us what you did so far to debug this issue.","title":"DOs"},{"location":"create_issue/#donts","text":"Do not use \"Please help\" etc in the title. See above. Do not copy and paste screen shot of error message. Copy and paste text. Do not use \"it doesn't work\". Precisely state what is the error message or symptom. Do not ask to write code for you. Contribute !","title":"DONT'S"},{"location":"custom_drone/","text":"AirLib on a Real Drone # The AirLib library can be compiled and deployed on the companion computer on a real drone. For our testing, we mounted a Gigabyte Brix BXi7-5500 ultra compact PC on the drone connected to the Pixhawk flight controller over USB. The Gigabyte PC is running Ubuntu, so we are able to SSH into it over Wi-Fi: Once connected you can run MavLinkTest with this command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. And this will produce a log file of the flight which can then be used for playback in the simulator . You can also add -proxy:192.168.1.100:14550 to connect MavLinkTest to a remote computer where you can run QGroundControl or our PX4 Log Viewer which is another handy way to see what is going on with your drone. MavLinkTest then has some simple commands for testing your drone, here's a simple example of some commands: arm takeoff 5 orbit 10 2 This will arm the drone, takeoff of 5 meters, then do an orbit pattern radius 10 meters, at 2 m/s. Type '?' to find all available commands. Note: Some commands (for example, orbit ) are named differently and have different syntax in MavLinkTest and DroneShell (for example, circlebypath -radius 10 -velocity 21 ). When you land the drone you can stop MavLinkTest and copy the *.mavlink log file that was generated. DroneServer and DroneShell # Once you are happy that the MavLinkTest is working, you can also run DroneServer and DroneShell as follows. First, run MavLinkTest with a local proxy to send everything to DroneServer: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. -proxy:127.0.0.1:14560 Change ~/Documents/AirSim/settings.json to say \"serial\":false, because we want DroneServer to look for this UDP connection. DroneServer 0 Lastly, you can now connect DroneShell to this instance of DroneServer and use the DroneShell commands to fly your drone: DroneShell ==||=> Welcome to DroneShell 1.0. Type ? for help. Microsoft Research (c) 2016. Waiting for drone to report a valid GPS location... ==||=> requestcontrol ==||=> arm ==||=> takeoff ==||=> circlebypath -radius 10 -velocity 2 PX4 Specific Tools # You can run the MavlinkCom library and MavLinkTest app to test the connection between your companion computer and flight controller. How Does This Work? # AirSim uses MavLinkCom component developed by @lovettchris. The MavLinkCom has a proxy architecture where you can open a connection to PX4 either using serial or UDP and then other components share this connection. When PX4 sends MavLink message, all components receive that message. If any component sends a message then it's received by PX4 only. This allows you to connect any number of components to PX4 This code opens a connection for LogViewer and QGC. You can add something more if you like. If you want to use QGC + AirSim together than you will need QGC to let own the serial port. QGC opens up TCP connection that acts as a proxy so any other component can connect to QGC and send MavLinkMessage to QGC and then QGC forwards that message to PX4. So you tell AirSim to connect to QGC and let QGC own serial port. For companion board, the way we did it earlier was to have Gigabyte Brix on the drone. This x86 full-fledged computer that will connect to PX4 through USB. We had Ubuntu on Brix and ran DroneServer . The DroneServer created an API endpoint that we can talk to via C++ client code (or Python code) and it translated API calls to MavLink messages. That way you can write your code against the same API, test it in the simulator and then run the same code on an actual vehicle. So the companion computer has DroneServer running along with client code.","title":"AirSim on Real Drones"},{"location":"custom_drone/#airlib-on-a-real-drone","text":"The AirLib library can be compiled and deployed on the companion computer on a real drone. For our testing, we mounted a Gigabyte Brix BXi7-5500 ultra compact PC on the drone connected to the Pixhawk flight controller over USB. The Gigabyte PC is running Ubuntu, so we are able to SSH into it over Wi-Fi: Once connected you can run MavLinkTest with this command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. And this will produce a log file of the flight which can then be used for playback in the simulator . You can also add -proxy:192.168.1.100:14550 to connect MavLinkTest to a remote computer where you can run QGroundControl or our PX4 Log Viewer which is another handy way to see what is going on with your drone. MavLinkTest then has some simple commands for testing your drone, here's a simple example of some commands: arm takeoff 5 orbit 10 2 This will arm the drone, takeoff of 5 meters, then do an orbit pattern radius 10 meters, at 2 m/s. Type '?' to find all available commands. Note: Some commands (for example, orbit ) are named differently and have different syntax in MavLinkTest and DroneShell (for example, circlebypath -radius 10 -velocity 21 ). When you land the drone you can stop MavLinkTest and copy the *.mavlink log file that was generated.","title":"AirLib on a Real Drone"},{"location":"custom_drone/#droneserver-and-droneshell","text":"Once you are happy that the MavLinkTest is working, you can also run DroneServer and DroneShell as follows. First, run MavLinkTest with a local proxy to send everything to DroneServer: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. -proxy:127.0.0.1:14560 Change ~/Documents/AirSim/settings.json to say \"serial\":false, because we want DroneServer to look for this UDP connection. DroneServer 0 Lastly, you can now connect DroneShell to this instance of DroneServer and use the DroneShell commands to fly your drone: DroneShell ==||=> Welcome to DroneShell 1.0. Type ? for help. Microsoft Research (c) 2016. Waiting for drone to report a valid GPS location... ==||=> requestcontrol ==||=> arm ==||=> takeoff ==||=> circlebypath -radius 10 -velocity 2","title":"DroneServer and DroneShell"},{"location":"custom_drone/#px4-specific-tools","text":"You can run the MavlinkCom library and MavLinkTest app to test the connection between your companion computer and flight controller.","title":"PX4 Specific Tools"},{"location":"custom_drone/#how-does-this-work","text":"AirSim uses MavLinkCom component developed by @lovettchris. The MavLinkCom has a proxy architecture where you can open a connection to PX4 either using serial or UDP and then other components share this connection. When PX4 sends MavLink message, all components receive that message. If any component sends a message then it's received by PX4 only. This allows you to connect any number of components to PX4 This code opens a connection for LogViewer and QGC. You can add something more if you like. If you want to use QGC + AirSim together than you will need QGC to let own the serial port. QGC opens up TCP connection that acts as a proxy so any other component can connect to QGC and send MavLinkMessage to QGC and then QGC forwards that message to PX4. So you tell AirSim to connect to QGC and let QGC own serial port. For companion board, the way we did it earlier was to have Gigabyte Brix on the drone. This x86 full-fledged computer that will connect to PX4 through USB. We had Ubuntu on Brix and ran DroneServer . The DroneServer created an API endpoint that we can talk to via C++ client code (or Python code) and it translated API calls to MavLink messages. That way you can write your code against the same API, test it in the simulator and then run the same code on an actual vehicle. So the companion computer has DroneServer running along with client code.","title":"How Does This Work?"},{"location":"custom_unity_environments/","text":"Adding AirSim to Custom Unity Projects # Before completing these steps, make sure you have properly set up AirSim for Unity 1. Open the Containing folder of your custom unity project 2. Copy and paste the following items from Unity demo into the main project folder of your custom environment: Assets ProjectSettings Open your custom environment in Unity Drag your desired scene into the Scene Hierarchy panel Drag CarDemo into the Scene Hierarchy panel Copy the following items from CarDemo into your custom scene: Main Camera Directional Light AirSimHUD Car After removing CarDemo from the Hierarchy panel, save your modified scene as CarDemo . Repeat Steps 6 and 7 with DroneDemo . This time, save your custom scene as DroneDemo . Your custom environment is now ready to interface with AirSim!","title":"Custom Unity Environment"},{"location":"custom_unity_environments/#adding-airsim-to-custom-unity-projects","text":"Before completing these steps, make sure you have properly set up AirSim for Unity 1. Open the Containing folder of your custom unity project 2. Copy and paste the following items from Unity demo into the main project folder of your custom environment: Assets ProjectSettings Open your custom environment in Unity Drag your desired scene into the Scene Hierarchy panel Drag CarDemo into the Scene Hierarchy panel Copy the following items from CarDemo into your custom scene: Main Camera Directional Light AirSimHUD Car After removing CarDemo from the Hierarchy panel, save your modified scene as CarDemo . Repeat Steps 6 and 7 with DroneDemo . This time, save your custom scene as DroneDemo . Your custom environment is now ready to interface with AirSim!","title":"Adding AirSim to Custom Unity Projects"},{"location":"design/","text":"Paper # You can read more about our architecture and design in our paper (work in progress) . You may cite this as, @techreport{MSR-TR-2017-9, title = {{A}erial {I}nformatics and {R}obotics Platform}, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, year = {2017}, institution = {Microsoft Research}, number = {{M}{S}{R}-{T}{R}-2017-9}} } Architecture # Below is high level overview of how different components interact with each other.","title":"Architecture"},{"location":"design/#paper","text":"You can read more about our architecture and design in our paper (work in progress) . You may cite this as, @techreport{MSR-TR-2017-9, title = {{A}erial {I}nformatics and {R}obotics Platform}, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, year = {2017}, institution = {Microsoft Research}, number = {{M}{S}{R}-{T}{R}-2017-9}} }","title":"Paper"},{"location":"design/#architecture","text":"Below is high level overview of how different components interact with each other.","title":"Architecture"},{"location":"dev_workflow/","text":"Development Workflow # Below is the guide on how to perform different development activities while working with AirSim. If you are new to Unreal Engine based projects and want to contribute to AirSim or make your own forks for your custom requirements, this might save you some time. Development Environment # OS # We highly recommend Windows 10 and Visual Studio 2019 as your development environment. The support for other OSes and IDE is unfortunately not as mature on the Unreal Engine side and you may risk severe loss of productivity trying to do workarounds and jumping through the hoops. Hardware # We recommend GPUs such as NVidia 1080 or NVidia Titan series with powerful desktop such as one with 64GB RAM, 6+ cores, SSDs and 2-3 displays (ideally 4K). We have found HP Z840 work quite well for our needs. The development experience on high-end laptops is generally sub-par compared to powerful desktops however they might be useful in a pinch. You generally want laptops with discrete NVidia GPU (at least M2000 or better) with 64GB RAM, SSDs and hopefully 4K display. We have found models such as Lenovo P50 work well for our needs. Laptops with only integrated graphics might not work well. Updating and Changing AirSim Code # Overview # AirSim is designed as plugin. This means it can't run by itself, you need to put it in an Unreal project (we call it \"environment\"). So building and testing AirSim has two steps: (1) build the plugin (2) deploy plugin in Unreal project and run the project. The first step is accomplished by build.cmd available in AirSim root. This command will update everything you need for the plugin in the Unreal\\Plugins folder. So to deploy the plugin, you just need to copy Unreal\\Plugins folder in to your Unreal project folder. Next you should remove all intermediate files in your Unreal project and then regenerate .sln file for your Unreal project. To do this, we have two handy .bat files in Unreal\\Environments\\Blocks folder: clean.bat and GenerateProjectFiles.bat . So just run these bat files in sequence from root of your Unreal project. Now you are ready to open new .sln in Visual Studio and press F5 to run it. Steps # Below are the steps we use to make changes in AirSim and test them out. The best way to do development in AirSim code is to use Blocks project . This is the light weight project so compile time is relatively faster. Generally the workflow is, REM //Use x64 Native Tools Command Prompt for VS 2019 REM //Navigate to AirSim repo folder git pull build.cmd cd Unreal\\Environments\\Blocks update_from_git.bat start Blocks.sln Above commands first builds the AirSim plugin and then deploys it to Blocks project using handy update_from_git.bat . Now you can work inside Visual Studio solution, make changes to the code and just run F5 to build, run and test your changes. The debugging, break points etc should work as usual. After you are done with you code changes, you might want to push your changes back to AirSim repo or your own fork or you may deploy the new plugin to your custom Unreal project. To do this, go back to command prompt and first update the AirSim repo folder: REM //Use x64 Native Tools Command Prompt for VS 2019 REM //run this from Unreal\\Environments\\Blocks update_to_git.bat build.cmd Above command will transfer your code changes from Unreal project folder back to Unreal\\Plugins folder. Now your changes are ready to be pushed to AirSim repo or your own fork. You can also copy Unreal\\Plugins to your custom Unreal engine project and see if everything works in your custom project as well. Take Away # Once you understand how Unreal Build system and plugin model works as well as why we are doing above steps, you should feel quite comfortable in following this workflow. Don't be afraid of opening up .bat files to peek inside and see what its doing. They are quite minimal and straightforward (except, of course, build.cmd - don't look in to that one). FAQ # I made changes in code in Blocks project but its not working. # When you press F5 or F6 in Visual Studio to start build, the Unreal Build system kicks in and it tries to find out if any files are dirty and what it needs to build. Unfortunately, it often fails to recognize dirty files that is not the code that uses Unreal headers and object hierarchy. So, the trick is to just make some file dirty that Unreal Build system always recognizes. My favorite one is AirSimGameMode.cpp. Just insert a line, delete it and save the file. I made changes in the code outside of Visual Studio but its not working. # Don't do that! Unreal Build system assumes that you are using Visual Studio and it does bunch of things to integrate with Visual Studio. If you do insist on using other editors then look up how to do command line builds in Unreal projects OR see docs on your editor on how it can integrate with Unreal build system OR run clean.bat + GenerateProjectFiles.bat to make sure VS solution is in sync. I'm trying to add new file in the Unreal Project and its not working. # It won't! While you are indeed using Visual Studio solution, remember that this solution was actually generated by Unreal Build system. If you want to add new files in your project, first shut down Visual Studio, add an empty file at desired location and then run GenerateProjectFiles.bat which will scan all files in your project and then re-create the .sln file. Now open this new .sln file and you are in business. I copied Unreal\\Plugins folder but nothing happens in Unreal Project. # First make sure your project's .uproject file is referencing the plugin. Then make sure you have run clean.bat and then GenerateProjectFiles.bat as described in Overview above. I have multiple Unreal projects with AirSim plugin. How do I update them easily? # You are in luck! We have build_all_ue_projects.bat which exactly does that. Don't treat it as black box (at least not yet), open it up and see what it does. It has 4 variables that are being set from command line args. If these args is not supplied they are set to default values in next set of statements. You might want to change default values for the paths. This batch file builds AirSim plugin, deploys it to all listed projects (see CALL statements later in the batch file), runs packaging for those projects and puts final binaries in specified folder - all in one step! This is what we use to create our own binary releases. How do I contribute back to AirSim? # Before making any changes make sure you have created your feature branch. After you test your code changes in Blocks environment, follow the usual steps to make contributions just like any other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first .","title":"Development Workflow"},{"location":"dev_workflow/#development-workflow","text":"Below is the guide on how to perform different development activities while working with AirSim. If you are new to Unreal Engine based projects and want to contribute to AirSim or make your own forks for your custom requirements, this might save you some time.","title":"Development Workflow"},{"location":"dev_workflow/#development-environment","text":"","title":"Development Environment"},{"location":"dev_workflow/#os","text":"We highly recommend Windows 10 and Visual Studio 2019 as your development environment. The support for other OSes and IDE is unfortunately not as mature on the Unreal Engine side and you may risk severe loss of productivity trying to do workarounds and jumping through the hoops.","title":"OS"},{"location":"dev_workflow/#hardware","text":"We recommend GPUs such as NVidia 1080 or NVidia Titan series with powerful desktop such as one with 64GB RAM, 6+ cores, SSDs and 2-3 displays (ideally 4K). We have found HP Z840 work quite well for our needs. The development experience on high-end laptops is generally sub-par compared to powerful desktops however they might be useful in a pinch. You generally want laptops with discrete NVidia GPU (at least M2000 or better) with 64GB RAM, SSDs and hopefully 4K display. We have found models such as Lenovo P50 work well for our needs. Laptops with only integrated graphics might not work well.","title":"Hardware"},{"location":"dev_workflow/#updating-and-changing-airsim-code","text":"","title":"Updating and Changing AirSim Code"},{"location":"dev_workflow/#overview","text":"AirSim is designed as plugin. This means it can't run by itself, you need to put it in an Unreal project (we call it \"environment\"). So building and testing AirSim has two steps: (1) build the plugin (2) deploy plugin in Unreal project and run the project. The first step is accomplished by build.cmd available in AirSim root. This command will update everything you need for the plugin in the Unreal\\Plugins folder. So to deploy the plugin, you just need to copy Unreal\\Plugins folder in to your Unreal project folder. Next you should remove all intermediate files in your Unreal project and then regenerate .sln file for your Unreal project. To do this, we have two handy .bat files in Unreal\\Environments\\Blocks folder: clean.bat and GenerateProjectFiles.bat . So just run these bat files in sequence from root of your Unreal project. Now you are ready to open new .sln in Visual Studio and press F5 to run it.","title":"Overview"},{"location":"dev_workflow/#steps","text":"Below are the steps we use to make changes in AirSim and test them out. The best way to do development in AirSim code is to use Blocks project . This is the light weight project so compile time is relatively faster. Generally the workflow is, REM //Use x64 Native Tools Command Prompt for VS 2019 REM //Navigate to AirSim repo folder git pull build.cmd cd Unreal\\Environments\\Blocks update_from_git.bat start Blocks.sln Above commands first builds the AirSim plugin and then deploys it to Blocks project using handy update_from_git.bat . Now you can work inside Visual Studio solution, make changes to the code and just run F5 to build, run and test your changes. The debugging, break points etc should work as usual. After you are done with you code changes, you might want to push your changes back to AirSim repo or your own fork or you may deploy the new plugin to your custom Unreal project. To do this, go back to command prompt and first update the AirSim repo folder: REM //Use x64 Native Tools Command Prompt for VS 2019 REM //run this from Unreal\\Environments\\Blocks update_to_git.bat build.cmd Above command will transfer your code changes from Unreal project folder back to Unreal\\Plugins folder. Now your changes are ready to be pushed to AirSim repo or your own fork. You can also copy Unreal\\Plugins to your custom Unreal engine project and see if everything works in your custom project as well.","title":"Steps"},{"location":"dev_workflow/#take-away","text":"Once you understand how Unreal Build system and plugin model works as well as why we are doing above steps, you should feel quite comfortable in following this workflow. Don't be afraid of opening up .bat files to peek inside and see what its doing. They are quite minimal and straightforward (except, of course, build.cmd - don't look in to that one).","title":"Take Away"},{"location":"dev_workflow/#faq","text":"","title":"FAQ"},{"location":"dev_workflow/#i-made-changes-in-code-in-blocks-project-but-its-not-working","text":"When you press F5 or F6 in Visual Studio to start build, the Unreal Build system kicks in and it tries to find out if any files are dirty and what it needs to build. Unfortunately, it often fails to recognize dirty files that is not the code that uses Unreal headers and object hierarchy. So, the trick is to just make some file dirty that Unreal Build system always recognizes. My favorite one is AirSimGameMode.cpp. Just insert a line, delete it and save the file.","title":"I made changes in code in Blocks project but its not working."},{"location":"dev_workflow/#i-made-changes-in-the-code-outside-of-visual-studio-but-its-not-working","text":"Don't do that! Unreal Build system assumes that you are using Visual Studio and it does bunch of things to integrate with Visual Studio. If you do insist on using other editors then look up how to do command line builds in Unreal projects OR see docs on your editor on how it can integrate with Unreal build system OR run clean.bat + GenerateProjectFiles.bat to make sure VS solution is in sync.","title":"I made changes in the code outside of Visual Studio but its not working."},{"location":"dev_workflow/#im-trying-to-add-new-file-in-the-unreal-project-and-its-not-working","text":"It won't! While you are indeed using Visual Studio solution, remember that this solution was actually generated by Unreal Build system. If you want to add new files in your project, first shut down Visual Studio, add an empty file at desired location and then run GenerateProjectFiles.bat which will scan all files in your project and then re-create the .sln file. Now open this new .sln file and you are in business.","title":"I'm trying to add new file in the Unreal Project and its not working."},{"location":"dev_workflow/#i-copied-unrealplugins-folder-but-nothing-happens-in-unreal-project","text":"First make sure your project's .uproject file is referencing the plugin. Then make sure you have run clean.bat and then GenerateProjectFiles.bat as described in Overview above.","title":"I copied Unreal\\Plugins folder but nothing happens in Unreal Project."},{"location":"dev_workflow/#i-have-multiple-unreal-projects-with-airsim-plugin-how-do-i-update-them-easily","text":"You are in luck! We have build_all_ue_projects.bat which exactly does that. Don't treat it as black box (at least not yet), open it up and see what it does. It has 4 variables that are being set from command line args. If these args is not supplied they are set to default values in next set of statements. You might want to change default values for the paths. This batch file builds AirSim plugin, deploys it to all listed projects (see CALL statements later in the batch file), runs packaging for those projects and puts final binaries in specified folder - all in one step! This is what we use to create our own binary releases.","title":"I have multiple Unreal projects with AirSim plugin. How do I update them easily?"},{"location":"dev_workflow/#how-do-i-contribute-back-to-airsim","text":"Before making any changes make sure you have created your feature branch. After you test your code changes in Blocks environment, follow the usual steps to make contributions just like any other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first .","title":"How do I contribute back to AirSim?"},{"location":"docker_ubuntu/","text":"AirSim on Docker in Linux # We've two options for docker. You can either build an image for running airsim linux binaries , or for compiling Unreal Engine + AirSim from source Binaries # Requirements: # Install nvidia-docker2 Build the docker image # Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on Ubuntu 18.04 with CUDA 10.0. You can specify any NVIDIA cudagl at your own risk. --target_image is the desired name of your docker image. Defaults to airsim_binary with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --base_image=nvidia/cudagl:10.0-devel-ubuntu18.04 \\ --target_image=airsim_binary:10.0-devel-ubuntu18.04 Verify you have an image by: $ docker images | grep airsim Running an unreal binary inside a docker container # Get an unreal binary or package your own project in Ubuntu. Let's take the Blocks binary as an example. You can download it by running $ cd Airsim/docker; $ ./download_blocks_env_binary.sh Running an unreal binary inside a docker container The syntax is: $ ./run_airsim_image_binary.sh DOCKER_IMAGE_NAME UNREAL_BINARY_SHELL_SCRIPT UNREAL_BINARY_ARGUMENTS -- headless For blocks, you can do a $ ./run_airsim_image_binary.sh airsim_binary:10.0-devel-ubuntu18.04 Blocks/Blocks.sh -windowed -ResX=1080 -ResY=720 DOCKER_IMAGE_NAME : Same as target_image parameter in previous step. By default, enter airsim_binary:10.0-devel-ubuntu18.04 UNREAL_BINARY_SHELL_SCRIPT : for Blocks enviroment, it will be Blocks/Blocks.sh UNREAL_BINARY_ARGUMENTS : For airsim, most relevant would be -windowed , -ResX , -ResY . Click on link to see all options. Running in Headless mode: Suffix -- headless at the end: $ ./run_airsim_image_binary.sh Blocks/Blocks.sh -- headless Specifying a settings.json Source # Requirements: # Install nvidia-docker2 Install ue4-docker Build Unreal Engine inside docker: # To get access to Unreal Engine's source code, register on Epic Games' website and link it to your github account, as explained in the Required Steps section here . Note that you don't need to do Step 2: Downloading UE4 on Linux ! Build unreal engine 4.19.2 docker image. We're going to use CUDA 10.0 in our example. $ ue4-docker build 4.19.2 --cuda=10.0 --no-full [optional] $ ue4-docker clean to free up some space. Details here ue4-docker supports all CUDA version listed on NVIDIA's cudagl dockerhub here . Please see this page for advanced configurations using ue4-docker Disk space: The unreal images and containers can take up a lot of space, especially if you try more than one version. Here's a list of useful links to monitor space used by docker and clean up intermediate builds: Large container images primer $ docker system df $ docker container prune $ docker image prune $ docker system prune Building AirSim inside UE4 docker container: # Build AirSim docker image (which lays over the unreal image we just built) Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on adamrehn/ue4-engine:4.19.2-cudagl10.0 . See ue4-docker for other versions. --target_image is the desired name of your docker image. Defaults to airsim_source with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --source \\ ----base_image adamrehn/ue4-engine:4.19.2-cudagl10.0 \\ --target_image=airsim_source:4.19.2-cudagl10.0 Running AirSim container # Run the airsim source image we built by: ./run_airsim_image_source.sh airsim_source:4.19.2-cudagl10.0 Syntax is ./run_airsim_image_source.sh DOCKER_IMAGE_NAME -- headless -- headless : suffix this to run in optional headless mode. Inside the container, you can see UnrealEngine and AirSim under /home/ue4 . Start unreal engine inside the container: ue4@HOSTMACHINE:~$ /home/ue4/UnrealEngine/Engine/Binaries/Linux/UE4Editor Specifying an airsim settings.json Continue with AirSim's Linux docs . [Misc] Packaging Unreal Environments in airsim_source containers # Let's take the Blocks environment as an example. In the following script, specify the full path to your unreal uproject file by project and the directory where you want the binaries to be placed by archivedirectory $ /home/ue4/UnrealEngine/Engine/Build/BatchFiles/RunUAT.sh BuildCookRun -platform=Linux -clientconfig=Shipping -serverconfig=Shipping -noP4 -cook -allmaps -build -stage -prereqs -pak -archive \\ -archivedirectory=/home/ue4/Binaries/Blocks/ \\ -project=/home/ue4/AirSim/Unreal/Environments/Blocks/Blocks.uproject This would create a Blocks binary in /home/ue4/Binaries/Blocks/ . You can test it by running /home/ue4/Binaries/Blocks/LinuxNoEditor/Blocks.sh -windowed Specifying an airsim settings.json # #### airsim_binary docker image: - We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . - Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in * run_airsim_image_binary.sh nvidia-docker run -it \\ -v $PATH_TO_YOUR/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -v $UNREAL_BINARY_PATH:$UNREAL_BINARY_PATH \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME \\ /bin/bash -c \"$UNREAL_BINARY_COMMAND\" #### airsim_source docker image: We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in run_airsim_image_source.sh : nvidia-docker run -it \\ -v $(pwd)/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME","title":"Docker on Linux"},{"location":"docker_ubuntu/#airsim-on-docker-in-linux","text":"We've two options for docker. You can either build an image for running airsim linux binaries , or for compiling Unreal Engine + AirSim from source","title":"AirSim on Docker in Linux"},{"location":"docker_ubuntu/#binaries","text":"","title":"Binaries"},{"location":"docker_ubuntu/#requirements","text":"Install nvidia-docker2","title":"Requirements:"},{"location":"docker_ubuntu/#build-the-docker-image","text":"Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on Ubuntu 18.04 with CUDA 10.0. You can specify any NVIDIA cudagl at your own risk. --target_image is the desired name of your docker image. Defaults to airsim_binary with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --base_image=nvidia/cudagl:10.0-devel-ubuntu18.04 \\ --target_image=airsim_binary:10.0-devel-ubuntu18.04 Verify you have an image by: $ docker images | grep airsim","title":"Build the docker image"},{"location":"docker_ubuntu/#running-an-unreal-binary-inside-a-docker-container","text":"Get an unreal binary or package your own project in Ubuntu. Let's take the Blocks binary as an example. You can download it by running $ cd Airsim/docker; $ ./download_blocks_env_binary.sh Running an unreal binary inside a docker container The syntax is: $ ./run_airsim_image_binary.sh DOCKER_IMAGE_NAME UNREAL_BINARY_SHELL_SCRIPT UNREAL_BINARY_ARGUMENTS -- headless For blocks, you can do a $ ./run_airsim_image_binary.sh airsim_binary:10.0-devel-ubuntu18.04 Blocks/Blocks.sh -windowed -ResX=1080 -ResY=720 DOCKER_IMAGE_NAME : Same as target_image parameter in previous step. By default, enter airsim_binary:10.0-devel-ubuntu18.04 UNREAL_BINARY_SHELL_SCRIPT : for Blocks enviroment, it will be Blocks/Blocks.sh UNREAL_BINARY_ARGUMENTS : For airsim, most relevant would be -windowed , -ResX , -ResY . Click on link to see all options. Running in Headless mode: Suffix -- headless at the end: $ ./run_airsim_image_binary.sh Blocks/Blocks.sh -- headless Specifying a settings.json","title":"Running an unreal binary inside a docker container"},{"location":"docker_ubuntu/#source","text":"","title":"Source"},{"location":"docker_ubuntu/#requirements_1","text":"Install nvidia-docker2 Install ue4-docker","title":"Requirements:"},{"location":"docker_ubuntu/#build-unreal-engine-inside-docker","text":"To get access to Unreal Engine's source code, register on Epic Games' website and link it to your github account, as explained in the Required Steps section here . Note that you don't need to do Step 2: Downloading UE4 on Linux ! Build unreal engine 4.19.2 docker image. We're going to use CUDA 10.0 in our example. $ ue4-docker build 4.19.2 --cuda=10.0 --no-full [optional] $ ue4-docker clean to free up some space. Details here ue4-docker supports all CUDA version listed on NVIDIA's cudagl dockerhub here . Please see this page for advanced configurations using ue4-docker Disk space: The unreal images and containers can take up a lot of space, especially if you try more than one version. Here's a list of useful links to monitor space used by docker and clean up intermediate builds: Large container images primer $ docker system df $ docker container prune $ docker image prune $ docker system prune","title":"Build Unreal Engine inside docker:"},{"location":"docker_ubuntu/#building-airsim-inside-ue4-docker-container","text":"Build AirSim docker image (which lays over the unreal image we just built) Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on adamrehn/ue4-engine:4.19.2-cudagl10.0 . See ue4-docker for other versions. --target_image is the desired name of your docker image. Defaults to airsim_source with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --source \\ ----base_image adamrehn/ue4-engine:4.19.2-cudagl10.0 \\ --target_image=airsim_source:4.19.2-cudagl10.0","title":"Building AirSim inside UE4 docker container:"},{"location":"docker_ubuntu/#running-airsim-container","text":"Run the airsim source image we built by: ./run_airsim_image_source.sh airsim_source:4.19.2-cudagl10.0 Syntax is ./run_airsim_image_source.sh DOCKER_IMAGE_NAME -- headless -- headless : suffix this to run in optional headless mode. Inside the container, you can see UnrealEngine and AirSim under /home/ue4 . Start unreal engine inside the container: ue4@HOSTMACHINE:~$ /home/ue4/UnrealEngine/Engine/Binaries/Linux/UE4Editor Specifying an airsim settings.json Continue with AirSim's Linux docs .","title":"Running AirSim container"},{"location":"docker_ubuntu/#misc-packaging-unreal-environments-in-airsim_source-containers","text":"Let's take the Blocks environment as an example. In the following script, specify the full path to your unreal uproject file by project and the directory where you want the binaries to be placed by archivedirectory $ /home/ue4/UnrealEngine/Engine/Build/BatchFiles/RunUAT.sh BuildCookRun -platform=Linux -clientconfig=Shipping -serverconfig=Shipping -noP4 -cook -allmaps -build -stage -prereqs -pak -archive \\ -archivedirectory=/home/ue4/Binaries/Blocks/ \\ -project=/home/ue4/AirSim/Unreal/Environments/Blocks/Blocks.uproject This would create a Blocks binary in /home/ue4/Binaries/Blocks/ . You can test it by running /home/ue4/Binaries/Blocks/LinuxNoEditor/Blocks.sh -windowed","title":"[Misc] Packaging Unreal Environments in airsim_source containers"},{"location":"docker_ubuntu/#specifying-an-airsim-settingsjson","text":"#### airsim_binary docker image: - We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . - Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in * run_airsim_image_binary.sh nvidia-docker run -it \\ -v $PATH_TO_YOUR/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -v $UNREAL_BINARY_PATH:$UNREAL_BINARY_PATH \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME \\ /bin/bash -c \"$UNREAL_BINARY_COMMAND\" #### airsim_source docker image: We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in run_airsim_image_source.sh : nvidia-docker run -it \\ -v $(pwd)/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME","title":"Specifying an airsim settings.json"},{"location":"drone_survey/","text":"Implementing a Drone Survey script # Moved here from https://github.com/microsoft/AirSim/wiki/Implementing-a-Drone-Survey-script Ever wanted to capture a bunch of top-down pictures of a certain location? Well, the Python API makes this really simple. See the code available here . Let's assume we want the following variables: Variable Description boxsize The overall size of the square box to survey stripewidth How far apart to drive the swim lanes, this can depend on the type of camera lens, for example. altitude The height to fly the survey. speed The speed of the survey can depend on how fast your camera can snap shots. So with these we can compute a square path box using this code: path = [] distance = 0 while x < self.boxsize: distance += self.boxsize path.append(Vector3r(x, self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, self.boxsize, z)) distance += self.boxsize path.append(Vector3r(x, -self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, -self.boxsize, z)) distance += self.boxsize Assuming we start in the corner of the box, increment x by the stripe width, then fly the full y-dimension of -boxsize to +boxsize , so in this case, boxsize is half the size of the actual box we will be covering. Once we have this list of Vector3r objects, we can fly this path very simply with the following call: result = self.client.moveOnPath(path, self.velocity, trip_time, DrivetrainType.ForwardOnly, YawMode(False,0), lookahead, 1) We can compute an appropriate trip_time timeout by dividing the distance of the path and the speed we are flying. The lookahead needed here for smooth path interpolation can be computed from the velocity using self.velocity + (self.velocity/2) . The more lookahead, the smoother the turns. This is why you see in the screenshot that the ends of each swimland are smooth turns rather than a square box pattern. This can result in a smoother video from your camera also. That's it, pretty simple, eh? Now of course you can add a lot more intelligence to this, make it avoid known obstacles on your map, make it climb up and down a hillside so you can survey a slope, etc. Lots of fun to be had.","title":"Surveying Using Drone"},{"location":"drone_survey/#implementing-a-drone-survey-script","text":"Moved here from https://github.com/microsoft/AirSim/wiki/Implementing-a-Drone-Survey-script Ever wanted to capture a bunch of top-down pictures of a certain location? Well, the Python API makes this really simple. See the code available here . Let's assume we want the following variables: Variable Description boxsize The overall size of the square box to survey stripewidth How far apart to drive the swim lanes, this can depend on the type of camera lens, for example. altitude The height to fly the survey. speed The speed of the survey can depend on how fast your camera can snap shots. So with these we can compute a square path box using this code: path = [] distance = 0 while x < self.boxsize: distance += self.boxsize path.append(Vector3r(x, self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, self.boxsize, z)) distance += self.boxsize path.append(Vector3r(x, -self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, -self.boxsize, z)) distance += self.boxsize Assuming we start in the corner of the box, increment x by the stripe width, then fly the full y-dimension of -boxsize to +boxsize , so in this case, boxsize is half the size of the actual box we will be covering. Once we have this list of Vector3r objects, we can fly this path very simply with the following call: result = self.client.moveOnPath(path, self.velocity, trip_time, DrivetrainType.ForwardOnly, YawMode(False,0), lookahead, 1) We can compute an appropriate trip_time timeout by dividing the distance of the path and the speed we are flying. The lookahead needed here for smooth path interpolation can be computed from the velocity using self.velocity + (self.velocity/2) . The more lookahead, the smoother the turns. This is why you see in the screenshot that the ends of each swimland are smooth turns rather than a square box pattern. This can result in a smoother video from your camera also. That's it, pretty simple, eh? Now of course you can add a lot more intelligence to this, make it avoid known obstacles on your map, make it climb up and down a hillside so you can survey a slope, etc. Lots of fun to be had.","title":"Implementing a Drone Survey script"},{"location":"faq/","text":"FAQ # General # Unreal editor is slow when it is not the active window # Go to Edit/Editor Preferences, select \"All Settings\" and type \"CPU\" in the search box. It should find the setting titled \"Use Less CPU when in Background\", and you want to uncheck this checkbox. My mouse disappears in Unreal # Yes, Unreal steals the mouse, and we don't draw one. So to get your mouse back just use Alt+TAB to switch to a different window. To avoid this entirely, go to Project settings in Unreal Editor, go to Input tab and disable all settings for mouse capture. Where is the setting file and how do I modify it? # AirSim will create empty settings file at ~/Documents/AirSim/settings.json . You can view the available settings options . How do I arm my drone? # If you're using simple_flight, your vehicle is already armed and ready to fly. For PX4 you can arm by holding both sticks on remote control down and to the center. When making API call I get error # If you are getting this error, TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float' its probably due to upgraded version of tornado package with version > 5.0 in Python that conflicts with msgpack-rpc-python which requires tornado package < 5.0. To fix this you can update the package like this: pip install --upgrade msgpack-rpc-python But this might break something (for example, PyTorch 0.4+) because it will uninstall newer tornado and re-install older one. To avoid this you should create new conda environment . I'm getting Eigen not found error when compiling Unreal project. # This is most likely because AirSim wasn't built and Plugin folder was copied in Unreal project folder. To fix this make sure you build AirSim first (run build.cmd in Windows). Something went wrong. How do I debug? # First turn on C++ exceptions from the Exceptions window: and copy the stack trace of all exceptions you see there during execution that look relevant (for example, there might be an initial exception from VSPerf140 that you can ignore) then paste these call stacks into a new AirSim GitHub issue, thanks. What do the colors mean in the Segmentation View ? # See Camera Views for information on the camera views and how to change them. Unreal 4.xx doesn't look as good as 4.yy # Unreal 4.15 added the ability for Foliage LOD dithering to be disabled on a case-by-case basis by unchecking the Dithered LOD Transition checkbox in the foliage materials. Note that all materials used on all LODs need to have the checkbox checked in order for dithered LOD transitions to work. When checked the transition of generated foliage will be a lot smoother and will look better than 4.14. Can I use an XBox controller to fly? # See XBox controller for details. Can I build a hexacopter with AirSim? # See how to build a hexacopter . How do I use AirSim with multiple vehicles? # Here is multi-vehicle setup guide . What computer do you need? # It depends on how big your Unreal Environment is. The Blocks environment that comes with AirSim is very basic and works on typical laptops. The Modular Neighborhood Pack that we use ourselves for research requires GPUs with at least 4GB of RAM. The Open World environment needs GPU with 8GB RAM. Our typical development machines have 32GB of RAM and NVIDIA TitanX and a fast hard drive . How do I report issues? # It's a good idea to include your configuration like below. If you can also include logs, that could also expedite the investigation. Operating System: Windows 10 64bit CPU: Intel Core i7 GPU: Nvidia GTX 1080 RAM: 32 GB Flight Controller: Pixhawk v2 Remote Control: Futaba If you have modified the default ~/Document/AirSim/settings.json , please include your settings also. If you are using PX4 then try to capture log from MavLink or PX4 . File an issue through GitHub Issues . Others # Linux Build FAQ Windows Build FAQ PX4 Setup FAQ Remote Control FAQ Unreal Blocks Environment FAQ Unreal Custom Environment FAQ","title":"FAQ"},{"location":"faq/#faq","text":"","title":"FAQ"},{"location":"faq/#general","text":"","title":"General"},{"location":"faq/#unreal-editor-is-slow-when-it-is-not-the-active-window","text":"Go to Edit/Editor Preferences, select \"All Settings\" and type \"CPU\" in the search box. It should find the setting titled \"Use Less CPU when in Background\", and you want to uncheck this checkbox.","title":"Unreal editor is slow when it is not the active window"},{"location":"faq/#my-mouse-disappears-in-unreal","text":"Yes, Unreal steals the mouse, and we don't draw one. So to get your mouse back just use Alt+TAB to switch to a different window. To avoid this entirely, go to Project settings in Unreal Editor, go to Input tab and disable all settings for mouse capture.","title":"My mouse disappears in Unreal"},{"location":"faq/#where-is-the-setting-file-and-how-do-i-modify-it","text":"AirSim will create empty settings file at ~/Documents/AirSim/settings.json . You can view the available settings options .","title":"Where is the setting file and how do I modify it?"},{"location":"faq/#how-do-i-arm-my-drone","text":"If you're using simple_flight, your vehicle is already armed and ready to fly. For PX4 you can arm by holding both sticks on remote control down and to the center.","title":"How do I arm my drone?"},{"location":"faq/#when-making-api-call-i-get-error","text":"If you are getting this error, TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float' its probably due to upgraded version of tornado package with version > 5.0 in Python that conflicts with msgpack-rpc-python which requires tornado package < 5.0. To fix this you can update the package like this: pip install --upgrade msgpack-rpc-python But this might break something (for example, PyTorch 0.4+) because it will uninstall newer tornado and re-install older one. To avoid this you should create new conda environment .","title":"When making API call I get error"},{"location":"faq/#im-getting-eigen-not-found-error-when-compiling-unreal-project","text":"This is most likely because AirSim wasn't built and Plugin folder was copied in Unreal project folder. To fix this make sure you build AirSim first (run build.cmd in Windows).","title":"I'm getting Eigen not found error when compiling Unreal project."},{"location":"faq/#something-went-wrong-how-do-i-debug","text":"First turn on C++ exceptions from the Exceptions window: and copy the stack trace of all exceptions you see there during execution that look relevant (for example, there might be an initial exception from VSPerf140 that you can ignore) then paste these call stacks into a new AirSim GitHub issue, thanks.","title":"Something went wrong. How do I debug?"},{"location":"faq/#what-do-the-colors-mean-in-the-segmentation-view","text":"See Camera Views for information on the camera views and how to change them.","title":"What do the colors mean in the Segmentation View ?"},{"location":"faq/#unreal-4xx-doesnt-look-as-good-as-4yy","text":"Unreal 4.15 added the ability for Foliage LOD dithering to be disabled on a case-by-case basis by unchecking the Dithered LOD Transition checkbox in the foliage materials. Note that all materials used on all LODs need to have the checkbox checked in order for dithered LOD transitions to work. When checked the transition of generated foliage will be a lot smoother and will look better than 4.14.","title":"Unreal 4.xx doesn't look as good as 4.yy"},{"location":"faq/#can-i-use-an-xbox-controller-to-fly","text":"See XBox controller for details.","title":"Can I use an XBox controller to fly?"},{"location":"faq/#can-i-build-a-hexacopter-with-airsim","text":"See how to build a hexacopter .","title":"Can I build a hexacopter with AirSim?"},{"location":"faq/#how-do-i-use-airsim-with-multiple-vehicles","text":"Here is multi-vehicle setup guide .","title":"How do I use AirSim with multiple vehicles?"},{"location":"faq/#what-computer-do-you-need","text":"It depends on how big your Unreal Environment is. The Blocks environment that comes with AirSim is very basic and works on typical laptops. The Modular Neighborhood Pack that we use ourselves for research requires GPUs with at least 4GB of RAM. The Open World environment needs GPU with 8GB RAM. Our typical development machines have 32GB of RAM and NVIDIA TitanX and a fast hard drive .","title":"What computer do you need?"},{"location":"faq/#how-do-i-report-issues","text":"It's a good idea to include your configuration like below. If you can also include logs, that could also expedite the investigation. Operating System: Windows 10 64bit CPU: Intel Core i7 GPU: Nvidia GTX 1080 RAM: 32 GB Flight Controller: Pixhawk v2 Remote Control: Futaba If you have modified the default ~/Document/AirSim/settings.json , please include your settings also. If you are using PX4 then try to capture log from MavLink or PX4 . File an issue through GitHub Issues .","title":"How do I report issues?"},{"location":"faq/#others","text":"Linux Build FAQ Windows Build FAQ PX4 Setup FAQ Remote Control FAQ Unreal Blocks Environment FAQ Unreal Custom Environment FAQ","title":"Others"},{"location":"flight_controller/","text":"Flight Controller # What is Flight Controller? # \"Wait!\" you ask, \"Why do you need flight controller for a simulator?\". The primary job of flight controller is to take in desired state as input, estimate actual state using sensors data and then drive the actuators in such a way so that actual state comes as close to the desired state. For quadrotors, desired state can be specified as roll, pitch and yaw, for example. It then estimates actual roll, pitch and yaw using gyroscope and accelerometer. Then it generates appropriate motor signals so actual state becomes desired state. You can find more in-depth in our paper . How Simulator uses Flight Controller? # Simulator consumes the motor signals generated by flight controller to figure out force and thrust generated by each actuator (i.e. propellers in case of quadrotor). This is then used by the physics engine to compute the kinetic properties of the vehicle. This in turn generates simulated sensor data and feed it back to the flight controller. You can find more in-depth in our paper . What is Hardware- and Software-in-Loop? # Hardware-in-Loop (HITL or HIL) means flight controller runs in actual hardware such as Naze32 or Pixhawk chip. You then connect this hardware to PC using USB port. Simulator talks to the device to retrieve actuator signals and send it simulated sensor data. This is obviously as close as you can get to real thing. However, it typically requires more steps to set up and usually hard to debug. One big issue is that simulator clock and device clock runs on their own speed and accuracy. Also, USB connection (which is usually only USB 2.0) may not be enough for real-time communication. In \"software-in-loop\" simulation (SITL or SIL) mode the firmware runs in your computer as opposed to separate board. This is generally fine except that now you are not touching any code paths that are specific to your device. Also, none of your code now runs with real-time clock usually provided by specialized hardware board. For well-designed flight controllers with software clock, these are usually not concerning issues. What Flight Controllers are Supported? # AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In the future, we also plan to support ROSFlight and Hackflight . Using AirSim Without Flight Controller # Yes, now it's possible to use AirSim without flight controller. Please see the instructions here for how to use so-called \"Computer Vision\" mode. If you don't need vehicle dynamics, we highly recommend using this mode.","title":"Flight Controller"},{"location":"flight_controller/#flight-controller","text":"","title":"Flight Controller"},{"location":"flight_controller/#what-is-flight-controller","text":"\"Wait!\" you ask, \"Why do you need flight controller for a simulator?\". The primary job of flight controller is to take in desired state as input, estimate actual state using sensors data and then drive the actuators in such a way so that actual state comes as close to the desired state. For quadrotors, desired state can be specified as roll, pitch and yaw, for example. It then estimates actual roll, pitch and yaw using gyroscope and accelerometer. Then it generates appropriate motor signals so actual state becomes desired state. You can find more in-depth in our paper .","title":"What is Flight Controller?"},{"location":"flight_controller/#how-simulator-uses-flight-controller","text":"Simulator consumes the motor signals generated by flight controller to figure out force and thrust generated by each actuator (i.e. propellers in case of quadrotor). This is then used by the physics engine to compute the kinetic properties of the vehicle. This in turn generates simulated sensor data and feed it back to the flight controller. You can find more in-depth in our paper .","title":"How Simulator uses Flight Controller?"},{"location":"flight_controller/#what-is-hardware-and-software-in-loop","text":"Hardware-in-Loop (HITL or HIL) means flight controller runs in actual hardware such as Naze32 or Pixhawk chip. You then connect this hardware to PC using USB port. Simulator talks to the device to retrieve actuator signals and send it simulated sensor data. This is obviously as close as you can get to real thing. However, it typically requires more steps to set up and usually hard to debug. One big issue is that simulator clock and device clock runs on their own speed and accuracy. Also, USB connection (which is usually only USB 2.0) may not be enough for real-time communication. In \"software-in-loop\" simulation (SITL or SIL) mode the firmware runs in your computer as opposed to separate board. This is generally fine except that now you are not touching any code paths that are specific to your device. Also, none of your code now runs with real-time clock usually provided by specialized hardware board. For well-designed flight controllers with software clock, these are usually not concerning issues.","title":"What is Hardware- and Software-in-Loop?"},{"location":"flight_controller/#what-flight-controllers-are-supported","text":"AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In the future, we also plan to support ROSFlight and Hackflight .","title":"What Flight Controllers are Supported?"},{"location":"flight_controller/#using-airsim-without-flight-controller","text":"Yes, now it's possible to use AirSim without flight controller. Please see the instructions here for how to use so-called \"Computer Vision\" mode. If you don't need vehicle dynamics, we highly recommend using this mode.","title":"Using AirSim Without Flight Controller"},{"location":"hard_drive/","text":"Busy Hard Drive # It is not required, but we recommend running your Unreal Environment on a Solid State Drive (SSD). Between debugging, logging, and Unreal asset loading the hard drive can become your bottle neck. It is normal that your hard drive will be slammed while Unreal is loading the environment, but if your hard drive performance looks like this while the Unreal game is running then you will probably not get a good flying experience. In fact, if the hard drive is this busy, chances are the drone will not fly properly at all. For some unknown reason this I/O bottle neck also interferes with the drone control loop and if that loop doesn't run at a high rate (300-500 Hz) then the drone will not fly. Not surprising, the control loop inside the PX4 firmware that runs on a Pixhawk flight controller runs at 1000 Hz. Reducing I/O # If you can't whip off to Fry's Electronics and pick up an overpriced super fast SSD this weekend, then the following steps can be taken to reduce the hard drive I/O: First run the Unreal Environment using Cooked content outside of the UE Editor or any debugging environment, and package the content to your fastest SSD drive. You can do that using this menu option: If you must use the UE editor (because you are actively modifying game assets), then at least don't run that in a debugger. If you are using Visual Studio use start without debugging. If you must debug the app, and you are using Visual Studio debugger, stop then Visual Studio from logging Intellitrace information. Go to Tools/Options/Debugging/Intellitrace, and turn off the main checkbox. Turn off any Unreal Analytics that your environment may have enabled, especially any file logging. I/O from Page Faults # If your system is running out of RAM it may start paging memory to disk. If your operating system has enabled paging to disk, make sure it is paging to your fastest SSD. Or if you have enough RAM disable paging all together. In fact, if you disable paging and the game stops working you will know for sure you are running out of RAM. Obviously, shutting down any other unnecessary apps should also free up memory so you don't run out. Ideal Runtime performance # This is what my slow hard drive looks like when flying from UE editor. You can see it's very busy, but the drone still flies ok: This is what my fast SSD looks like when the drone is flying in an Unreal Cooked app (no UE editor, no debugger). Not surprisingly it is flying perfectly in this case:","title":"Tips for Busy HDD"},{"location":"hard_drive/#busy-hard-drive","text":"It is not required, but we recommend running your Unreal Environment on a Solid State Drive (SSD). Between debugging, logging, and Unreal asset loading the hard drive can become your bottle neck. It is normal that your hard drive will be slammed while Unreal is loading the environment, but if your hard drive performance looks like this while the Unreal game is running then you will probably not get a good flying experience. In fact, if the hard drive is this busy, chances are the drone will not fly properly at all. For some unknown reason this I/O bottle neck also interferes with the drone control loop and if that loop doesn't run at a high rate (300-500 Hz) then the drone will not fly. Not surprising, the control loop inside the PX4 firmware that runs on a Pixhawk flight controller runs at 1000 Hz.","title":"Busy Hard Drive"},{"location":"hard_drive/#reducing-io","text":"If you can't whip off to Fry's Electronics and pick up an overpriced super fast SSD this weekend, then the following steps can be taken to reduce the hard drive I/O: First run the Unreal Environment using Cooked content outside of the UE Editor or any debugging environment, and package the content to your fastest SSD drive. You can do that using this menu option: If you must use the UE editor (because you are actively modifying game assets), then at least don't run that in a debugger. If you are using Visual Studio use start without debugging. If you must debug the app, and you are using Visual Studio debugger, stop then Visual Studio from logging Intellitrace information. Go to Tools/Options/Debugging/Intellitrace, and turn off the main checkbox. Turn off any Unreal Analytics that your environment may have enabled, especially any file logging.","title":"Reducing I/O"},{"location":"hard_drive/#io-from-page-faults","text":"If your system is running out of RAM it may start paging memory to disk. If your operating system has enabled paging to disk, make sure it is paging to your fastest SSD. Or if you have enough RAM disable paging all together. In fact, if you disable paging and the game stops working you will know for sure you are running out of RAM. Obviously, shutting down any other unnecessary apps should also free up memory so you don't run out.","title":"I/O from Page Faults"},{"location":"hard_drive/#ideal-runtime-performance","text":"This is what my slow hard drive looks like when flying from UE editor. You can see it's very busy, but the drone still flies ok: This is what my fast SSD looks like when the drone is flying in an Unreal Cooked app (no UE editor, no debugger). Not surprisingly it is flying perfectly in this case:","title":"Ideal Runtime performance"},{"location":"hello_drone/","text":"Hello Drone # How does Hello Drone work? # Hello Drone uses the RPC client to connect to the RPC server that is automatically started by the AirSim. The RPC server routes all the commands to a class that implements MultirotorApiBase . In essence, MultirotorApiBase defines our abstract interface for getting data from the quadrotor and sending back commands. We currently have concrete implementation for MultirotorApiBase for MavLink based vehicles. The implementation for DJI drone platforms, specifically Matrice, is in works.","title":"Hello Drone"},{"location":"hello_drone/#hello-drone","text":"","title":"Hello Drone"},{"location":"hello_drone/#how-does-hello-drone-work","text":"Hello Drone uses the RPC client to connect to the RPC server that is automatically started by the AirSim. The RPC server routes all the commands to a class that implements MultirotorApiBase . In essence, MultirotorApiBase defines our abstract interface for getting data from the quadrotor and sending back commands. We currently have concrete implementation for MultirotorApiBase for MavLink based vehicles. The implementation for DJI drone platforms, specifically Matrice, is in works.","title":"How does Hello Drone work?"},{"location":"image_apis/","text":"Image APIs # Please read general API doc first if you are not familiar with AirSim APIs. Getting a Single Image # Here's a sample code to get a single image from camera named \"0\". The returned value is bytes of png format image. To get uncompressed and other format as well as available cameras please see next sections. Python # import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() png_image = client.simGetImage(\"0\", airsim.ImageType.Scene) # do something with image C++ # #include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int getOneImage() { using namespace msr::airlib; // for car use CarRpcLibClient MultirotorRpcLibClient client; std::vector png_image = client.simGetImage(\"0\", VehicleCameraBase::ImageType::Scene); // do something with images } Getting Images with More Flexibility # The simGetImages API which is slightly more complex to use than simGetImage API, for example, you can get left camera view, right camera view and depth image from left camera in a single API call. The simGetImages API also allows you to get uncompressed images as well as floating point single channel images (instead of 3 channel (RGB), each 8 bit). Python # import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() responses = client.simGetImages([ # png format airsim.ImageRequest(0, airsim.ImageType.Scene), # uncompressed RGB array bytes airsim.ImageRequest(1, airsim.ImageType.Scene, False, False), # floating point uncompressed image airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) # do something with response which contains image data, pose, timestamp etc Using AirSim Images with NumPy # If you plan to use numpy for image manipulation, you should get uncompressed RGB image and then convert to numpy like this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.Scene, False, False)]) response = responses[0] # get numpy array img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) # reshape array to 4 channel image array H X W X 4 img_rgb = img1d.reshape(response.height, response.width, 3) # original image is fliped vertically img_rgb = np.flipud(img_rgb) # write to png airsim.write_png(os.path.normpath(filename + '.png'), img_rgb) Quick Tips # The API simGetImage returns binary string literal which means you can simply dump it in binary file to create a .png file. However if you want to process it in any other way than you can handy function airsim.string_to_uint8_array . This converts binary string literal to NumPy uint8 array. The API simGetImages can accept request for multiple image types from any cameras in single call. You can specify if image is png compressed, RGB uncompressed or float array. For png compressed images, you get binary string literal . For float array you get Python list of float64. You can convert this float array to NumPy 2D array using airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height) You can also save float array to .pfm file (Portable Float Map format) using airsim.write_pfm() function. C++ # int getStereoAndDepthImages() { using namespace msr::airlib; typedef VehicleCameraBase::ImageRequest ImageRequest; typedef VehicleCameraBase::ImageResponse ImageResponse; typedef VehicleCameraBase::ImageType ImageType; // for car use // CarRpcLibClient client; MultirotorRpcLibClient client; // get right, left and depth images. First two as png, second as float16. std::vector request = { //png format ImageRequest(\"0\", ImageType::Scene), //uncompressed RGB array bytes ImageRequest(\"1\", ImageType::Scene, false, false), //floating point uncompressed image ImageRequest(\"1\", ImageType::DepthPlanner, true) }; const std::vector& response = client.simGetImages(request); // do something with response which contains image data, pose, timestamp etc } Ready to Run Complete Examples # Python # For a more complete ready to run sample code please see sample code in AirSimClient project for multirotors or HelloCar sample . This code also demonstrates simple activities such as saving images in files or using numpy to manipulate images. C++ # For a more complete ready to run sample code please see sample code in HelloDrone project for multirotors or HelloCar project . See also other example code that generates specified number of stereo images along with ground truth depth and disparity and saving it to pfm format . Available Cameras # Car # The cameras on car can be accessed by following names in API calls: front_center , front_right , front_left , fpv and back_center . Here FPV camera is driver's head position in the car. Multirotor # The cameras in CV mode can be accessed by following names in API calls: front_center , front_right , front_left , bottom_center and back_center . Computer Vision Mode # Camera names are same as in multirotor. Backward compatibility for camera names # Before AirSim v1.2, cameras were accessed using ID numbers instead of names. For backward compatibility you can still use following ID numbers for above camera names in same order as above: \"0\" , \"1\" , \"2\" , \"3\" , \"4\" . In addition, camera name \"\" is also available to access the default camera which is generally the camera \"0\" . \"Computer Vision\" Mode # You can use AirSim in so-called \"Computer Vision\" mode. In this mode, physics engine is disabled and there is no vehicle, just cameras. You can move around using keyboard (use F1 to see help on keys). You can press Record button to continuously generate images. Or you can call APIs to move cameras around and take images. To active this mode, edit settings.json that you can find in your Documents\\AirSim folder (or ~/Documents/AirSim on Linux) and make sure following values exist at root level: { \"SettingsVersion\": 1.2, \"SimMode\": \"ComputerVision\" } Here's the Python code example to move camera around and capture images. This mode was inspired from UnrealCV project . Setting Pose in Computer Vision Mode # To move around the environment using APIs you can use simSetVehiclePose API. This API takes position and orientation and sets that on the invisible vehicle where the front-center camera is located. All rest of the cameras move along keeping the relative position. If you don't want to change position (or orientation) then just set components of position (or orientation) to floating point nan values. The simGetVehiclePose allows to retrieve the current pose. You can also use simGetGroundTruthKinematics to get the quantities kinematics quantities for the movement. Many other non-vehicle specific APIs are also available such as segmentation APIs, collision APIs and camera APIs. Camera APIs # The simGetCameraInfo returns the pose (in world frame, NED coordinates, SI units) and FOV (in degrees) for the specified camera. Please see example usage . The simSetCameraPose sets the pose for the specified camera while taking an input pose as a combination of relative position and a quaternion in NED frame. The handy airsim.to_quaternion() function allows to convert pitch, roll, yaw to quaternion. For example, to set camera-0 to 15-degree pitch while maintaining the same position, you can use: camera_pose = airsim.Pose(airsim.Vector3r(0, 0, 0), airsim.to_quaternion(0.261799, 0, 0)) #RPY in radians client.simSetCameraPose(0, camera_pose); Gimbal # You can set stabilization for pitch, roll or yaw for any camera using settings . Please see example usage . Changing Resolution and Camera Parameters # To change resolution, FOV etc, you can use settings.json . For example, below addition in settings.json sets parameters for scene capture and uses \"Computer Vision\" mode described above. If you omit any setting then below default values will be used. For more information see settings doc . If you are using stereo camera, currently the distance between left and right is fixed at 25 cm. { \"SettingsVersion\": 1.2, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"MotionBlurAmount\": 0 } ] }, \"SimMode\": \"ComputerVision\" } What Does Pixel Values Mean in Different Image Types? # Available ImageType Values # Scene = 0, DepthPlanner = 1, DepthPerspective = 2, DepthVis = 3, DisparityNormalized = 4, Segmentation = 5, SurfaceNormals = 6, Infrared = 7 DepthPlanner and DepthPerspective # You normally want to retrieve the depth image as float (i.e. set pixels_as_float = true ) and specify ImageType = DepthPlanner or ImageType = DepthPerspective in ImageRequest . For ImageType = DepthPlanner , you get depth in camera plan, i.e., all points that are in plan parallel to camera have same depth. For ImageType = DepthPerspective , you get depth from camera using a projection ray that hits that pixel. Depending on your use case, planner depth or perspective depth may be the ground truth image that you want. For example, you may be able to feed perspective depth to ROS package such as depth_image_proc to generate a point cloud. Or planner depth may be more compatible with estimated depth image generated by stereo algorithms such as SGM. DepthVis # When you specify ImageType = DepthVis in ImageRequest , you get an image that helps depth visualization. In this case, each pixel value is interpolated from black to white depending on depth in camera plane in meters. The pixels with pure white means depth of 100m or more while pure black means depth of 0 meters. DisparityNormalized # You normally want to retrieve disparity image as float (i.e. set pixels_as_float = true and specify ImageType = DisparityNormalized in ImageRequest ) in which case each pixel is (Xl - Xr)/Xmax , which is thereby normalized to values between 0 to 1. Segmentation # When you specify ImageType = Segmentation in ImageRequest , you get an image that gives you ground truth segmentation of the scene. At the startup, AirSim assigns value 0 to 255 to each mesh available in environment. This value is then mapped to a specific color in the pallet . The RGB values for each object ID can be found in this file . You can assign a specific value (limited to the range 0-255) to a specific mesh using APIs. For example, below Python code sets the object ID for the mesh called \"Ground\" to 20 in Blocks environment and hence changes its color in Segmentation view: success = client.simSetSegmentationObjectID(\"Ground\", 20); The return value is a boolean type that lets you know if the mesh was found. Notice that typical Unreal environments, like Blocks, usually have many other meshes that comprises of same object, for example, \"Ground_2\", \"Ground_3\" and so on. As it is tedious to set object ID for all of these meshes, AirSim also supports regular expressions. For example, the code below sets all meshes which have names starting with \"ground\" (ignoring case) to 21 with just one line: success = client.simSetSegmentationObjectID(\"ground[\\w]*\", 21, True); The return value is true if at least one mesh was found using regular expression matching. It is recommended that you request uncompressed image using this API to ensure you get precise RGB values for segmentation image: responses = client.simGetImages([ImageRequest(0, AirSimImageType.Segmentation, False, False)]) img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) #get numpy array img_rgb = img1d.reshape(response.height, response.width, 3) #reshape array to 3 channel image array H X W X 3 img_rgb = np.flipud(img_rgb) #original image is fliped vertically #find unique colors print(np.unique(img_rgb[:,:,0], return_counts=True)) #red print(np.unique(img_rgb[:,:,1], return_counts=True)) #green print(np.unique(img_rgb[:,:,2], return_counts=True)) #blue A complete ready-to-run example can be found in segmentation.py . Unsetting object ID # An object's ID can be set to -1 to make it not show up on the segmentation image. How to Find Mesh Names? # To get desired ground truth segmentation you will need to know the names of the meshes in your Unreal environment. To do this, you will need to open up Unreal Environment in Unreal Editor and then inspect the names of the meshes you are interested in using the World Outliner. For example, below we see the mesh names for the ground in Blocks environment in right panel in the editor: If you don't know how to open Unreal Environment in Unreal Editor then try following the guide for building from source . Once you decide on the meshes you are interested, note down their names and use above API to set their object IDs. There are few settings available to change object ID generation behavior. Changing Colors for Object IDs # At present the color for each object ID is fixed as in this pallet . We will be adding ability to change colors for object IDs to desired values shortly. In the meantime you can open the segmentation image in your favorite image editor and get the RGB values you are interested in. Startup Object IDs # At the start, AirSim assigns object ID to each object found in environment of type UStaticMeshComponent or ALandscapeProxy . It then either uses mesh name or owner name (depending on settings), lower cases it, removes any chars below ASCII 97 to remove numbers and some punctuations, sums int value of all chars and modulo 255 to generate the object ID. In other words, all object with same alphabet chars would get same object ID. This heuristic is simple and effective for many Unreal environments but may not be what you want. In that case, please use above APIs to change object IDs to your desired values. There are few settings available to change this behavior. Getting Object ID for Mesh # The simGetSegmentationObjectID API allows you get object ID for given mesh name. Infrared # Currently this is just a map from object ID to grey scale 0-255. So any mesh with object ID 42 shows up with color (42, 42, 42). Please see segmentation section for more details on how to set object IDs. Typically noise setting can be applied for this image type to get slightly more realistic effect. We are still working on adding other infrared artifacts and any contributions are welcome. Example Code # A complete example of setting vehicle positions at random locations and orientations and then taking images can be found in GenerateImageGenerator.hpp . This example generates specified number of stereo images and ground truth disparity image and saving it to pfm format .","title":"Image APIs"},{"location":"image_apis/#image-apis","text":"Please read general API doc first if you are not familiar with AirSim APIs.","title":"Image APIs"},{"location":"image_apis/#getting-a-single-image","text":"Here's a sample code to get a single image from camera named \"0\". The returned value is bytes of png format image. To get uncompressed and other format as well as available cameras please see next sections.","title":"Getting a Single Image"},{"location":"image_apis/#python","text":"import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() png_image = client.simGetImage(\"0\", airsim.ImageType.Scene) # do something with image","title":"Python"},{"location":"image_apis/#c","text":"#include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int getOneImage() { using namespace msr::airlib; // for car use CarRpcLibClient MultirotorRpcLibClient client; std::vector png_image = client.simGetImage(\"0\", VehicleCameraBase::ImageType::Scene); // do something with images }","title":"C++"},{"location":"image_apis/#getting-images-with-more-flexibility","text":"The simGetImages API which is slightly more complex to use than simGetImage API, for example, you can get left camera view, right camera view and depth image from left camera in a single API call. The simGetImages API also allows you to get uncompressed images as well as floating point single channel images (instead of 3 channel (RGB), each 8 bit).","title":"Getting Images with More Flexibility"},{"location":"image_apis/#python_1","text":"import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() responses = client.simGetImages([ # png format airsim.ImageRequest(0, airsim.ImageType.Scene), # uncompressed RGB array bytes airsim.ImageRequest(1, airsim.ImageType.Scene, False, False), # floating point uncompressed image airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) # do something with response which contains image data, pose, timestamp etc","title":"Python"},{"location":"image_apis/#using-airsim-images-with-numpy","text":"If you plan to use numpy for image manipulation, you should get uncompressed RGB image and then convert to numpy like this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.Scene, False, False)]) response = responses[0] # get numpy array img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) # reshape array to 4 channel image array H X W X 4 img_rgb = img1d.reshape(response.height, response.width, 3) # original image is fliped vertically img_rgb = np.flipud(img_rgb) # write to png airsim.write_png(os.path.normpath(filename + '.png'), img_rgb)","title":"Using AirSim Images with NumPy"},{"location":"image_apis/#quick-tips","text":"The API simGetImage returns binary string literal which means you can simply dump it in binary file to create a .png file. However if you want to process it in any other way than you can handy function airsim.string_to_uint8_array . This converts binary string literal to NumPy uint8 array. The API simGetImages can accept request for multiple image types from any cameras in single call. You can specify if image is png compressed, RGB uncompressed or float array. For png compressed images, you get binary string literal . For float array you get Python list of float64. You can convert this float array to NumPy 2D array using airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height) You can also save float array to .pfm file (Portable Float Map format) using airsim.write_pfm() function.","title":"Quick Tips"},{"location":"image_apis/#c_1","text":"int getStereoAndDepthImages() { using namespace msr::airlib; typedef VehicleCameraBase::ImageRequest ImageRequest; typedef VehicleCameraBase::ImageResponse ImageResponse; typedef VehicleCameraBase::ImageType ImageType; // for car use // CarRpcLibClient client; MultirotorRpcLibClient client; // get right, left and depth images. First two as png, second as float16. std::vector request = { //png format ImageRequest(\"0\", ImageType::Scene), //uncompressed RGB array bytes ImageRequest(\"1\", ImageType::Scene, false, false), //floating point uncompressed image ImageRequest(\"1\", ImageType::DepthPlanner, true) }; const std::vector& response = client.simGetImages(request); // do something with response which contains image data, pose, timestamp etc }","title":"C++"},{"location":"image_apis/#ready-to-run-complete-examples","text":"","title":"Ready to Run Complete Examples"},{"location":"image_apis/#python_2","text":"For a more complete ready to run sample code please see sample code in AirSimClient project for multirotors or HelloCar sample . This code also demonstrates simple activities such as saving images in files or using numpy to manipulate images.","title":"Python"},{"location":"image_apis/#c_2","text":"For a more complete ready to run sample code please see sample code in HelloDrone project for multirotors or HelloCar project . See also other example code that generates specified number of stereo images along with ground truth depth and disparity and saving it to pfm format .","title":"C++"},{"location":"image_apis/#available-cameras","text":"","title":"Available Cameras"},{"location":"image_apis/#car","text":"The cameras on car can be accessed by following names in API calls: front_center , front_right , front_left , fpv and back_center . Here FPV camera is driver's head position in the car.","title":"Car"},{"location":"image_apis/#multirotor","text":"The cameras in CV mode can be accessed by following names in API calls: front_center , front_right , front_left , bottom_center and back_center .","title":"Multirotor"},{"location":"image_apis/#computer-vision-mode","text":"Camera names are same as in multirotor.","title":"Computer Vision Mode"},{"location":"image_apis/#backward-compatibility-for-camera-names","text":"Before AirSim v1.2, cameras were accessed using ID numbers instead of names. For backward compatibility you can still use following ID numbers for above camera names in same order as above: \"0\" , \"1\" , \"2\" , \"3\" , \"4\" . In addition, camera name \"\" is also available to access the default camera which is generally the camera \"0\" .","title":"Backward compatibility for camera names"},{"location":"image_apis/#computer-vision-mode_1","text":"You can use AirSim in so-called \"Computer Vision\" mode. In this mode, physics engine is disabled and there is no vehicle, just cameras. You can move around using keyboard (use F1 to see help on keys). You can press Record button to continuously generate images. Or you can call APIs to move cameras around and take images. To active this mode, edit settings.json that you can find in your Documents\\AirSim folder (or ~/Documents/AirSim on Linux) and make sure following values exist at root level: { \"SettingsVersion\": 1.2, \"SimMode\": \"ComputerVision\" } Here's the Python code example to move camera around and capture images. This mode was inspired from UnrealCV project .","title":"\"Computer Vision\" Mode"},{"location":"image_apis/#setting-pose-in-computer-vision-mode","text":"To move around the environment using APIs you can use simSetVehiclePose API. This API takes position and orientation and sets that on the invisible vehicle where the front-center camera is located. All rest of the cameras move along keeping the relative position. If you don't want to change position (or orientation) then just set components of position (or orientation) to floating point nan values. The simGetVehiclePose allows to retrieve the current pose. You can also use simGetGroundTruthKinematics to get the quantities kinematics quantities for the movement. Many other non-vehicle specific APIs are also available such as segmentation APIs, collision APIs and camera APIs.","title":"Setting Pose in Computer Vision Mode"},{"location":"image_apis/#camera-apis","text":"The simGetCameraInfo returns the pose (in world frame, NED coordinates, SI units) and FOV (in degrees) for the specified camera. Please see example usage . The simSetCameraPose sets the pose for the specified camera while taking an input pose as a combination of relative position and a quaternion in NED frame. The handy airsim.to_quaternion() function allows to convert pitch, roll, yaw to quaternion. For example, to set camera-0 to 15-degree pitch while maintaining the same position, you can use: camera_pose = airsim.Pose(airsim.Vector3r(0, 0, 0), airsim.to_quaternion(0.261799, 0, 0)) #RPY in radians client.simSetCameraPose(0, camera_pose);","title":"Camera APIs"},{"location":"image_apis/#gimbal","text":"You can set stabilization for pitch, roll or yaw for any camera using settings . Please see example usage .","title":"Gimbal"},{"location":"image_apis/#changing-resolution-and-camera-parameters","text":"To change resolution, FOV etc, you can use settings.json . For example, below addition in settings.json sets parameters for scene capture and uses \"Computer Vision\" mode described above. If you omit any setting then below default values will be used. For more information see settings doc . If you are using stereo camera, currently the distance between left and right is fixed at 25 cm. { \"SettingsVersion\": 1.2, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"MotionBlurAmount\": 0 } ] }, \"SimMode\": \"ComputerVision\" }","title":"Changing Resolution and Camera Parameters"},{"location":"image_apis/#what-does-pixel-values-mean-in-different-image-types","text":"","title":"What Does Pixel Values Mean in Different Image Types?"},{"location":"image_apis/#available-imagetype-values","text":"Scene = 0, DepthPlanner = 1, DepthPerspective = 2, DepthVis = 3, DisparityNormalized = 4, Segmentation = 5, SurfaceNormals = 6, Infrared = 7","title":"Available ImageType Values"},{"location":"image_apis/#depthplanner-and-depthperspective","text":"You normally want to retrieve the depth image as float (i.e. set pixels_as_float = true ) and specify ImageType = DepthPlanner or ImageType = DepthPerspective in ImageRequest . For ImageType = DepthPlanner , you get depth in camera plan, i.e., all points that are in plan parallel to camera have same depth. For ImageType = DepthPerspective , you get depth from camera using a projection ray that hits that pixel. Depending on your use case, planner depth or perspective depth may be the ground truth image that you want. For example, you may be able to feed perspective depth to ROS package such as depth_image_proc to generate a point cloud. Or planner depth may be more compatible with estimated depth image generated by stereo algorithms such as SGM.","title":"DepthPlanner and DepthPerspective"},{"location":"image_apis/#depthvis","text":"When you specify ImageType = DepthVis in ImageRequest , you get an image that helps depth visualization. In this case, each pixel value is interpolated from black to white depending on depth in camera plane in meters. The pixels with pure white means depth of 100m or more while pure black means depth of 0 meters.","title":"DepthVis"},{"location":"image_apis/#disparitynormalized","text":"You normally want to retrieve disparity image as float (i.e. set pixels_as_float = true and specify ImageType = DisparityNormalized in ImageRequest ) in which case each pixel is (Xl - Xr)/Xmax , which is thereby normalized to values between 0 to 1.","title":"DisparityNormalized"},{"location":"image_apis/#segmentation","text":"When you specify ImageType = Segmentation in ImageRequest , you get an image that gives you ground truth segmentation of the scene. At the startup, AirSim assigns value 0 to 255 to each mesh available in environment. This value is then mapped to a specific color in the pallet . The RGB values for each object ID can be found in this file . You can assign a specific value (limited to the range 0-255) to a specific mesh using APIs. For example, below Python code sets the object ID for the mesh called \"Ground\" to 20 in Blocks environment and hence changes its color in Segmentation view: success = client.simSetSegmentationObjectID(\"Ground\", 20); The return value is a boolean type that lets you know if the mesh was found. Notice that typical Unreal environments, like Blocks, usually have many other meshes that comprises of same object, for example, \"Ground_2\", \"Ground_3\" and so on. As it is tedious to set object ID for all of these meshes, AirSim also supports regular expressions. For example, the code below sets all meshes which have names starting with \"ground\" (ignoring case) to 21 with just one line: success = client.simSetSegmentationObjectID(\"ground[\\w]*\", 21, True); The return value is true if at least one mesh was found using regular expression matching. It is recommended that you request uncompressed image using this API to ensure you get precise RGB values for segmentation image: responses = client.simGetImages([ImageRequest(0, AirSimImageType.Segmentation, False, False)]) img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) #get numpy array img_rgb = img1d.reshape(response.height, response.width, 3) #reshape array to 3 channel image array H X W X 3 img_rgb = np.flipud(img_rgb) #original image is fliped vertically #find unique colors print(np.unique(img_rgb[:,:,0], return_counts=True)) #red print(np.unique(img_rgb[:,:,1], return_counts=True)) #green print(np.unique(img_rgb[:,:,2], return_counts=True)) #blue A complete ready-to-run example can be found in segmentation.py .","title":"Segmentation"},{"location":"image_apis/#unsetting-object-id","text":"An object's ID can be set to -1 to make it not show up on the segmentation image.","title":"Unsetting object ID"},{"location":"image_apis/#how-to-find-mesh-names","text":"To get desired ground truth segmentation you will need to know the names of the meshes in your Unreal environment. To do this, you will need to open up Unreal Environment in Unreal Editor and then inspect the names of the meshes you are interested in using the World Outliner. For example, below we see the mesh names for the ground in Blocks environment in right panel in the editor: If you don't know how to open Unreal Environment in Unreal Editor then try following the guide for building from source . Once you decide on the meshes you are interested, note down their names and use above API to set their object IDs. There are few settings available to change object ID generation behavior.","title":"How to Find Mesh Names?"},{"location":"image_apis/#changing-colors-for-object-ids","text":"At present the color for each object ID is fixed as in this pallet . We will be adding ability to change colors for object IDs to desired values shortly. In the meantime you can open the segmentation image in your favorite image editor and get the RGB values you are interested in.","title":"Changing Colors for Object IDs"},{"location":"image_apis/#startup-object-ids","text":"At the start, AirSim assigns object ID to each object found in environment of type UStaticMeshComponent or ALandscapeProxy . It then either uses mesh name or owner name (depending on settings), lower cases it, removes any chars below ASCII 97 to remove numbers and some punctuations, sums int value of all chars and modulo 255 to generate the object ID. In other words, all object with same alphabet chars would get same object ID. This heuristic is simple and effective for many Unreal environments but may not be what you want. In that case, please use above APIs to change object IDs to your desired values. There are few settings available to change this behavior.","title":"Startup Object IDs"},{"location":"image_apis/#getting-object-id-for-mesh","text":"The simGetSegmentationObjectID API allows you get object ID for given mesh name.","title":"Getting Object ID for Mesh"},{"location":"image_apis/#infrared","text":"Currently this is just a map from object ID to grey scale 0-255. So any mesh with object ID 42 shows up with color (42, 42, 42). Please see segmentation section for more details on how to set object IDs. Typically noise setting can be applied for this image type to get slightly more realistic effect. We are still working on adding other infrared artifacts and any contributions are welcome.","title":"Infrared"},{"location":"image_apis/#example-code","text":"A complete example of setting vehicle positions at random locations and orientations and then taking images can be found in GenerateImageGenerator.hpp . This example generates specified number of stereo images and ground truth disparity image and saving it to pfm format .","title":"Example Code"},{"location":"lidar/","text":"How to Use Lidar in AirSim # AirSim supports Lidar for multirotors and cars. The enablement of lidar and the other lidar settings can be configured via AirSimSettings json. Please see general sensors for information on configruation of general/shared sensor settings. Enabling lidar on a vehicle # By default, lidars are not enabled. To enable lidar, set the SensorType and Enabled attributes in settings json. \"Lidar1\": { \"SensorType\": 6, \"Enabled\" : true, Multiple lidars can be enabled on a vehicle. Lidar configuration # The following parameters can be configured right now via settings json. Parameter Description NumberOfChannels Number of channels/lasers of the lidar Range Range, in meters PointsPerSecond Number of points captured per second RotationsPerSecond Rotations per second HorizontalFOVStart Horizontal FOV start for the lidar, in degrees HorizontalFOVEnd Horizontal FOV end for the lidar, in degrees VerticalFOVUpper Vertical FOV upper limit for the lidar, in degrees VerticalFOVLower Vertical FOV lower limit for the lidar, in degrees X Y Z Position of the lidar relative to the vehicle (in NED, in meters) Roll Pitch Yaw Orientation of the lidar relative to the vehicle (in degrees, yaw-pitch-roll order to front vector +X) DataFrame Frame for the points in output (\"VehicleInertialFrame\" or \"SensorLocalFrame\") e.g., { \"SeeDocsAt\": \"https://github.com/Microsoft/AirSim/blob/master/docs/settings_json.md\", \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"simpleflight\", \"AutoCreate\": true, \"Sensors\": { \"LidarSensor1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 100000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"HorizontalFOVStart\": -20, \"HorizontalFOVEnd\": 20, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" }, \"LidarSensor2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" } } } } } Server side visualization for debugging # Be default, the lidar points are not drawn on the viewport. To enable the drawing of hit laser points on the viewport, please enable setting 'DrawDebugPoints' via settings json. e.g., \"Lidar1\": { ... \"DrawDebugPoints\": true }, Client API # Use getLidarData() API to retrieve the Lidar data. * The API returns a Point-Cloud as a flat array of floats along with the timestamp of the capture and lidar pose. * Point-Cloud: * The floats represent [x,y,z] coordinate for each point hit within the range in the last scan. * The frame for the points in the output is configurable using \"DataFrame\" attribute \"\" or \"VehicleInertialFrame\" -- default; returned points are in vehicle inertial frame (in NED, in meters) \"SensorLocalFrame\" -- returned points are in lidar local frame (in NED, in meters) * Lidar Pose: * Lidar pose in the vehicle inertial frame (in NED, in meters) * Can be used to transform points to other frames. Python Examples # drone_lidar.py car_lidar.py Coming soon # Visualization of lidar data on client side.","title":"LIDAR"},{"location":"lidar/#how-to-use-lidar-in-airsim","text":"AirSim supports Lidar for multirotors and cars. The enablement of lidar and the other lidar settings can be configured via AirSimSettings json. Please see general sensors for information on configruation of general/shared sensor settings.","title":"How to Use Lidar in AirSim"},{"location":"lidar/#enabling-lidar-on-a-vehicle","text":"By default, lidars are not enabled. To enable lidar, set the SensorType and Enabled attributes in settings json. \"Lidar1\": { \"SensorType\": 6, \"Enabled\" : true, Multiple lidars can be enabled on a vehicle.","title":"Enabling lidar on a vehicle"},{"location":"lidar/#lidar-configuration","text":"The following parameters can be configured right now via settings json. Parameter Description NumberOfChannels Number of channels/lasers of the lidar Range Range, in meters PointsPerSecond Number of points captured per second RotationsPerSecond Rotations per second HorizontalFOVStart Horizontal FOV start for the lidar, in degrees HorizontalFOVEnd Horizontal FOV end for the lidar, in degrees VerticalFOVUpper Vertical FOV upper limit for the lidar, in degrees VerticalFOVLower Vertical FOV lower limit for the lidar, in degrees X Y Z Position of the lidar relative to the vehicle (in NED, in meters) Roll Pitch Yaw Orientation of the lidar relative to the vehicle (in degrees, yaw-pitch-roll order to front vector +X) DataFrame Frame for the points in output (\"VehicleInertialFrame\" or \"SensorLocalFrame\") e.g., { \"SeeDocsAt\": \"https://github.com/Microsoft/AirSim/blob/master/docs/settings_json.md\", \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"simpleflight\", \"AutoCreate\": true, \"Sensors\": { \"LidarSensor1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 100000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"HorizontalFOVStart\": -20, \"HorizontalFOVEnd\": 20, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" }, \"LidarSensor2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" } } } } }","title":"Lidar configuration"},{"location":"lidar/#server-side-visualization-for-debugging","text":"Be default, the lidar points are not drawn on the viewport. To enable the drawing of hit laser points on the viewport, please enable setting 'DrawDebugPoints' via settings json. e.g., \"Lidar1\": { ... \"DrawDebugPoints\": true },","title":"Server side visualization for debugging"},{"location":"lidar/#client-api","text":"Use getLidarData() API to retrieve the Lidar data. * The API returns a Point-Cloud as a flat array of floats along with the timestamp of the capture and lidar pose. * Point-Cloud: * The floats represent [x,y,z] coordinate for each point hit within the range in the last scan. * The frame for the points in the output is configurable using \"DataFrame\" attribute \"\" or \"VehicleInertialFrame\" -- default; returned points are in vehicle inertial frame (in NED, in meters) \"SensorLocalFrame\" -- returned points are in lidar local frame (in NED, in meters) * Lidar Pose: * Lidar pose in the vehicle inertial frame (in NED, in meters) * Can be used to transform points to other frames.","title":"Client API"},{"location":"lidar/#python-examples","text":"drone_lidar.py car_lidar.py","title":"Python Examples"},{"location":"lidar/#coming-soon","text":"Visualization of lidar data on client side.","title":"Coming soon"},{"location":"log_viewer/","text":"Log Viewer # The LogViewer is a Windows WPF app that presents the MavLink streams that it is getting from the Unreal Simulator. You can use this to monitor what is happening on the drone while it is flying. For example, the picture below shows a real time graph of the x, y an z gyro sensor information being generated by the simulator. Usage # To use this LogViewer, connect the simulator before you run the simulation. Simply press the blue connector button on the top right corner of the window, select the Socket tab , enter the port number 14388, and your localhost network. Then press the record button (triangle on the right hand side of the toolbar). Now start the simulator, pick some mavlink items to graph, you should see something like this: The drone view here is the actual estimated position coming from the PX4, so that is a great way to check whether the PX4 is in sync with the simulator. Sometimes you can see some drift here as the attitude estimation catches up with reality, this is more visible after a bad crash. Installation # If you can't build the LogViewer.sln, there is also a click once installer . Configuration # The magic port number 14388 can be configured in the simulator by editing the settings.json file .","title":"MavLink LogViewer"},{"location":"log_viewer/#log-viewer","text":"The LogViewer is a Windows WPF app that presents the MavLink streams that it is getting from the Unreal Simulator. You can use this to monitor what is happening on the drone while it is flying. For example, the picture below shows a real time graph of the x, y an z gyro sensor information being generated by the simulator.","title":"Log Viewer"},{"location":"log_viewer/#usage","text":"To use this LogViewer, connect the simulator before you run the simulation. Simply press the blue connector button on the top right corner of the window, select the Socket tab , enter the port number 14388, and your localhost network. Then press the record button (triangle on the right hand side of the toolbar). Now start the simulator, pick some mavlink items to graph, you should see something like this: The drone view here is the actual estimated position coming from the PX4, so that is a great way to check whether the PX4 is in sync with the simulator. Sometimes you can see some drift here as the attitude estimation catches up with reality, this is more visible after a bad crash.","title":"Usage"},{"location":"log_viewer/#installation","text":"If you can't build the LogViewer.sln, there is also a click once installer .","title":"Installation"},{"location":"log_viewer/#configuration","text":"The magic port number 14388 can be configured in the simulator by editing the settings.json file .","title":"Configuration"},{"location":"mavlinkcom/","text":"Welcome to MavLinkCom # MavLinkCom is a cross-platform C++ library that helps connect to and communicate with MavLink based vehicles. Specifically this library is designed to work well with PX4 based drones. Design # You can view and edit the Design.dgml diagram in Visual Studio. The following are the most important classes in this library. MavLinkNode # This is the base class for all MavLinkNodes (subclasses include MavLinkVehicle, MavLinkVideoClient and MavLinkVideoServer). The node connects to your mavlink enabled vehicle via a MavLinkConnection and provides methods for sending MavLinkMessages and MavLinkCommands and for subscribing to receive messages. This base class also stores the local system id and component id your app wants to use to identify itself to your remove vehicle. You can also call startHeartbeat to send regular heartbeat messages to keep the connection alive. MavLinkMessage # This is the encoded MavLinkMessage. For those who have used the mavlink.h C API, this is similar to mavlink_message_t. You do not create these manually, they are encoded from a strongly typed MavLinkMessageBase subclass. Strongly typed message and command classes # The MavLinkComGenerator parses the mavlink common.xml message definitions and generates all the MavLink* MavLinkMessageBase subclasses as well as a bunch of handy mavlink enums and a bunch of strongly typed MavLinkCommand subclasses. MavLinkMessageBase # This is the base class for a set of strongly typed message classes that are code generated by the MavLinkComGenerator project. This replaces the C messages defined in the mavlink C API and provides a slightly more object oriented way to send and receive messages via sendMessage on MavLinkNode. These classes have encode/decode methods that convert to and from the MavLinkMessage class. MavLinkCommand # This is the base class for a set of strongly typed command classes that are code generated by the MavLinkComGenerator project. This replaces the C definitions defined in the mavlink C API and provides a more object oriented way to send commands via the sendCommand method on MavLinkNode. The MavLinkNode takes care of turning these into the underlying mavlink COMMAND_LONG message. MavLinkConnection # This class provides static helper methods for creating connections to remote MavLink nodes, over serial ports, as well as UDP, or TCP sockets. This class provides a way to subscribe to receive messages from that node in a pub/sub way so you can have multiple subscribers on the same connection. MavLinkVehicle uses this to track various messages that define the overall vehicle state. MavLinkVehicle # MavLinkVehicle is a MavLinkNode that tracks various messages that define the overall vehicle state and provides a VehicleState struct containing a snapshot of that state, including home position, current orientation, local position, global position, and so on. This class also provides a bunch of helper methods that wrap commonly used commands providing simple method calls to do things like arm, disarm, takeoff, land, go to a local coordinate, and fly under offbaord control either by position or velocity control. MavLinkTcpServer # This helper class provides a way to setup a \"server\" that accepts MavLinkConnections from remote nodes. You can use this class to get a connection that you can then give to MavLinkVideoServer to serve images over MavLink. MavLinkFtpClient # This helper class takes a given MavLinkConnection and provides FTP client support for the MAVLINK_MSG_ID_FILE_TRANSFER_PROTOCOL for vehicles that support the FTP capability. This class provides simple methods to list directory contents, and the get and put files. MavLinkVideoClient # This helper class takes a given MavLinkConnection and provides helper methods for requesting video from remote node and packaging up the MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA messages into simple to use MavLinkVideoFrames. MavLinkVideoServer # This helper class takes a given MavLinkConnection and provides the server side of the MavLinkVideoClient protocol, including helper methods for notifying when there is a video request to process (hasVideoRequest) and a method to send video frames (sendFrame) which will generate the right MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA sequence. Examples # The following code from the UnitTest project shows how to connect to a Pixhawk flight controller over USB serial port, then wait for the first heartbeat message to be received: auto connection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); MavLinkHeartbeat heartbeat; if (!waitForHeartbeat(10000, heartbeat)) { throw std::runtime_error(\"Received no heartbeat from PX4 after 10 seconds\"); } The following code connects to serial port, and then forwards all messages to and from QGroundControl to that drone using another connection that is joined to the drone stream. auto droneConnection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); auto proxyConnection = MavLinkConnection::connectRemoteUdp(\"qgc\", \"127.0.0.1\", \"127.0.0.1\", 14550); droneConnection->join(proxyConnection); The following code then takes that connection and turns on heartBeats and starts tracking vehicle information using local system id 166 and component id 1. auto vehicle = std::make_shared(166, 1); vehicle->connect(connection); vehicle->startHeartbeat(); std::this_thread::sleep_for(std::chrono::seconds(5)); VehicleState state = vehicle->getVehicleState(); printf(\"Home position is %s, %f,%f,%f\\n\", state.home.is_set ? \"set\" : \"not set\", state.home.global_pos.lat, state.home.global_pos.lon, state.home.global_pos.alt); The following code uses the vehicle object to arm the drone and take off and wait for the takeoff altitude to be reached: bool rc = false; if (!vehicle->armDisarm(true).wait(3000, &rc) || !rc) { printf(\"arm command failed\\n\"); return; } if (!vehicle->takeoff(targetAlt).wait(3000, &rc) || !rc) { printf(\"takeoff command failed\\n\"); return; } int version = vehicle->getVehicleStateVersion(); while (true) { int newVersion = vehicle->getVehicleStateVersion(); if (version != newVersion) { VehicleState state = vehicle->getVehicleState(); float alt = state.local_est.pos.z; if (alt >= targetAlt - delta && alt <= targetAlt + delta) { reached = true; printf(\"Target altitude reached\\n\"); break; } } else { std::this_thread::sleep_for(std::chrono::milliseconds(10)); } } The following code uses offboard control to make the drone fly in a circle with camera pointed at the center. Here we use the subscribe method to check each new local position message to indicate so we can compute the new velocity vector as soon as that new position is received. We request a high rate for those messages using setMessageInterval to ensure smooth circular orbit. vehicle->setMessageInterval((int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED, 30); vehicle->requestControl(); int subscription = vehicle->getConnection()->subscribe( [&](std::shared_ptr connection, const MavLinkMessage& m) { if (m.msgid == (int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED) { float x = localPos.x; float y = localPos.y; float dx = x - cx; float dy = y - cy; float angle = atan2(dy, dx); if (angle < 0) angle += M_PI * 2; float tangent = angle + M_PI_2; double newvx = orbitSpeed * cos(tangent); double newvy = orbitSpeed * sin(tangent); float heading = angle + M_PI; vehicle->moveByLocalVelocityWithAltHold(newvx, newvy, altitude, true, heading); } }); The following code stops flying the drone in offboard mode and tells the drone to loiter at its current location. This version of the code shows how to use the AsyncResult without blocking on a wait call. vehicle->releaseControl(); if (vehicle->loiter().then([=](bool rc) { printf(\"loiter command %s\\n\", rc ? \"succeeded\" : \"failed\"); } The following code gets all configurable parameters from the drone and prints them: auto list = vehicle->getParamList(); auto end = list.end(); int count = 0; for (auto iter = list.begin(); iter < end; iter++) { count++; MavLinkParameter p = *iter; if (p.type == MAV_PARAM_TYPE_REAL32 || p.type == MAV_PARAM_TYPE_REAL64) { printf(\"%s=%f\\n\", p.name.c_str(), p.value); } else { printf(\"%s=%d\\n\", p.name.c_str(), static_cast(p.value)); } } The following code sets a parameter on the Pixhawk to disable the USB safety check (this is handy if you are controlling the Pixhawk over USB using another onboard computer that is part of the drone itself). You should NOT do this if you are connecting your PC or laptop to the drone over USB. MavLinkParameter p; p.name = \"CBRK_USB_CHK\"; p.value = 197848; if (!vehicle->setParameter(p).wait(3000,&rc) || !rc) { printf(\"Setting the CBRK_USB_CHK failed\"); } MavLinkVehicle actually has a helper method for this called allowFlightControlOverUsb, so now you know how it is implemented :-) Advanced Connections # You can wire up different configurations of mavlink pipelines using the MavLinkConnection class \"join\" method as shown below. Example 1, we connect to PX4 over serial, and proxy those messages through to QGroundControl and the LogViewer who are listening on remote ports. Example 2: simulation can talk to jMavSim and jMavSim connects to PX4. jMavSim can also manage multiple connections, so it can talk to unreal simulator. Another MavLinkConnection can be joined to proxy connections that jMavSim doesn't support, like the LogViewer or a remote camera node. Example 3: we use MavLinkConnection to connect to PX4 over serial, then join additional connections for all our remote nodes including jMavSim. Example 4: We can also do distributed systems to control the drone remotely:","title":"MavLinkCom"},{"location":"mavlinkcom/#welcome-to-mavlinkcom","text":"MavLinkCom is a cross-platform C++ library that helps connect to and communicate with MavLink based vehicles. Specifically this library is designed to work well with PX4 based drones.","title":"Welcome to MavLinkCom"},{"location":"mavlinkcom/#design","text":"You can view and edit the Design.dgml diagram in Visual Studio. The following are the most important classes in this library.","title":"Design"},{"location":"mavlinkcom/#mavlinknode","text":"This is the base class for all MavLinkNodes (subclasses include MavLinkVehicle, MavLinkVideoClient and MavLinkVideoServer). The node connects to your mavlink enabled vehicle via a MavLinkConnection and provides methods for sending MavLinkMessages and MavLinkCommands and for subscribing to receive messages. This base class also stores the local system id and component id your app wants to use to identify itself to your remove vehicle. You can also call startHeartbeat to send regular heartbeat messages to keep the connection alive.","title":"MavLinkNode"},{"location":"mavlinkcom/#mavlinkmessage","text":"This is the encoded MavLinkMessage. For those who have used the mavlink.h C API, this is similar to mavlink_message_t. You do not create these manually, they are encoded from a strongly typed MavLinkMessageBase subclass.","title":"MavLinkMessage"},{"location":"mavlinkcom/#strongly-typed-message-and-command-classes","text":"The MavLinkComGenerator parses the mavlink common.xml message definitions and generates all the MavLink* MavLinkMessageBase subclasses as well as a bunch of handy mavlink enums and a bunch of strongly typed MavLinkCommand subclasses.","title":"Strongly typed message and command classes"},{"location":"mavlinkcom/#mavlinkmessagebase","text":"This is the base class for a set of strongly typed message classes that are code generated by the MavLinkComGenerator project. This replaces the C messages defined in the mavlink C API and provides a slightly more object oriented way to send and receive messages via sendMessage on MavLinkNode. These classes have encode/decode methods that convert to and from the MavLinkMessage class.","title":"MavLinkMessageBase"},{"location":"mavlinkcom/#mavlinkcommand","text":"This is the base class for a set of strongly typed command classes that are code generated by the MavLinkComGenerator project. This replaces the C definitions defined in the mavlink C API and provides a more object oriented way to send commands via the sendCommand method on MavLinkNode. The MavLinkNode takes care of turning these into the underlying mavlink COMMAND_LONG message.","title":"MavLinkCommand"},{"location":"mavlinkcom/#mavlinkconnection","text":"This class provides static helper methods for creating connections to remote MavLink nodes, over serial ports, as well as UDP, or TCP sockets. This class provides a way to subscribe to receive messages from that node in a pub/sub way so you can have multiple subscribers on the same connection. MavLinkVehicle uses this to track various messages that define the overall vehicle state.","title":"MavLinkConnection"},{"location":"mavlinkcom/#mavlinkvehicle","text":"MavLinkVehicle is a MavLinkNode that tracks various messages that define the overall vehicle state and provides a VehicleState struct containing a snapshot of that state, including home position, current orientation, local position, global position, and so on. This class also provides a bunch of helper methods that wrap commonly used commands providing simple method calls to do things like arm, disarm, takeoff, land, go to a local coordinate, and fly under offbaord control either by position or velocity control.","title":"MavLinkVehicle"},{"location":"mavlinkcom/#mavlinktcpserver","text":"This helper class provides a way to setup a \"server\" that accepts MavLinkConnections from remote nodes. You can use this class to get a connection that you can then give to MavLinkVideoServer to serve images over MavLink.","title":"MavLinkTcpServer"},{"location":"mavlinkcom/#mavlinkftpclient","text":"This helper class takes a given MavLinkConnection and provides FTP client support for the MAVLINK_MSG_ID_FILE_TRANSFER_PROTOCOL for vehicles that support the FTP capability. This class provides simple methods to list directory contents, and the get and put files.","title":"MavLinkFtpClient"},{"location":"mavlinkcom/#mavlinkvideoclient","text":"This helper class takes a given MavLinkConnection and provides helper methods for requesting video from remote node and packaging up the MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA messages into simple to use MavLinkVideoFrames.","title":"MavLinkVideoClient"},{"location":"mavlinkcom/#mavlinkvideoserver","text":"This helper class takes a given MavLinkConnection and provides the server side of the MavLinkVideoClient protocol, including helper methods for notifying when there is a video request to process (hasVideoRequest) and a method to send video frames (sendFrame) which will generate the right MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA sequence.","title":"MavLinkVideoServer"},{"location":"mavlinkcom/#examples","text":"The following code from the UnitTest project shows how to connect to a Pixhawk flight controller over USB serial port, then wait for the first heartbeat message to be received: auto connection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); MavLinkHeartbeat heartbeat; if (!waitForHeartbeat(10000, heartbeat)) { throw std::runtime_error(\"Received no heartbeat from PX4 after 10 seconds\"); } The following code connects to serial port, and then forwards all messages to and from QGroundControl to that drone using another connection that is joined to the drone stream. auto droneConnection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); auto proxyConnection = MavLinkConnection::connectRemoteUdp(\"qgc\", \"127.0.0.1\", \"127.0.0.1\", 14550); droneConnection->join(proxyConnection); The following code then takes that connection and turns on heartBeats and starts tracking vehicle information using local system id 166 and component id 1. auto vehicle = std::make_shared(166, 1); vehicle->connect(connection); vehicle->startHeartbeat(); std::this_thread::sleep_for(std::chrono::seconds(5)); VehicleState state = vehicle->getVehicleState(); printf(\"Home position is %s, %f,%f,%f\\n\", state.home.is_set ? \"set\" : \"not set\", state.home.global_pos.lat, state.home.global_pos.lon, state.home.global_pos.alt); The following code uses the vehicle object to arm the drone and take off and wait for the takeoff altitude to be reached: bool rc = false; if (!vehicle->armDisarm(true).wait(3000, &rc) || !rc) { printf(\"arm command failed\\n\"); return; } if (!vehicle->takeoff(targetAlt).wait(3000, &rc) || !rc) { printf(\"takeoff command failed\\n\"); return; } int version = vehicle->getVehicleStateVersion(); while (true) { int newVersion = vehicle->getVehicleStateVersion(); if (version != newVersion) { VehicleState state = vehicle->getVehicleState(); float alt = state.local_est.pos.z; if (alt >= targetAlt - delta && alt <= targetAlt + delta) { reached = true; printf(\"Target altitude reached\\n\"); break; } } else { std::this_thread::sleep_for(std::chrono::milliseconds(10)); } } The following code uses offboard control to make the drone fly in a circle with camera pointed at the center. Here we use the subscribe method to check each new local position message to indicate so we can compute the new velocity vector as soon as that new position is received. We request a high rate for those messages using setMessageInterval to ensure smooth circular orbit. vehicle->setMessageInterval((int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED, 30); vehicle->requestControl(); int subscription = vehicle->getConnection()->subscribe( [&](std::shared_ptr connection, const MavLinkMessage& m) { if (m.msgid == (int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED) { float x = localPos.x; float y = localPos.y; float dx = x - cx; float dy = y - cy; float angle = atan2(dy, dx); if (angle < 0) angle += M_PI * 2; float tangent = angle + M_PI_2; double newvx = orbitSpeed * cos(tangent); double newvy = orbitSpeed * sin(tangent); float heading = angle + M_PI; vehicle->moveByLocalVelocityWithAltHold(newvx, newvy, altitude, true, heading); } }); The following code stops flying the drone in offboard mode and tells the drone to loiter at its current location. This version of the code shows how to use the AsyncResult without blocking on a wait call. vehicle->releaseControl(); if (vehicle->loiter().then([=](bool rc) { printf(\"loiter command %s\\n\", rc ? \"succeeded\" : \"failed\"); } The following code gets all configurable parameters from the drone and prints them: auto list = vehicle->getParamList(); auto end = list.end(); int count = 0; for (auto iter = list.begin(); iter < end; iter++) { count++; MavLinkParameter p = *iter; if (p.type == MAV_PARAM_TYPE_REAL32 || p.type == MAV_PARAM_TYPE_REAL64) { printf(\"%s=%f\\n\", p.name.c_str(), p.value); } else { printf(\"%s=%d\\n\", p.name.c_str(), static_cast(p.value)); } } The following code sets a parameter on the Pixhawk to disable the USB safety check (this is handy if you are controlling the Pixhawk over USB using another onboard computer that is part of the drone itself). You should NOT do this if you are connecting your PC or laptop to the drone over USB. MavLinkParameter p; p.name = \"CBRK_USB_CHK\"; p.value = 197848; if (!vehicle->setParameter(p).wait(3000,&rc) || !rc) { printf(\"Setting the CBRK_USB_CHK failed\"); } MavLinkVehicle actually has a helper method for this called allowFlightControlOverUsb, so now you know how it is implemented :-)","title":"Examples"},{"location":"mavlinkcom/#advanced-connections","text":"You can wire up different configurations of mavlink pipelines using the MavLinkConnection class \"join\" method as shown below. Example 1, we connect to PX4 over serial, and proxy those messages through to QGroundControl and the LogViewer who are listening on remote ports. Example 2: simulation can talk to jMavSim and jMavSim connects to PX4. jMavSim can also manage multiple connections, so it can talk to unreal simulator. Another MavLinkConnection can be joined to proxy connections that jMavSim doesn't support, like the LogViewer or a remote camera node. Example 3: we use MavLinkConnection to connect to PX4 over serial, then join additional connections for all our remote nodes including jMavSim. Example 4: We can also do distributed systems to control the drone remotely:","title":"Advanced Connections"},{"location":"mavlinkcom_mocap/","text":"Welcome to MavLinkMoCap # This folder contains the MavLinkMoCap library which connects to a OptiTrack camera system for accurate indoor location. Dependencies: # OptiTrack Motive . MavLinkCom . Setup RigidBody # First you need to define a RigidBody named 'Quadrocopter' using Motive. See Rigid_Body_Tracking . MavLinkTest # Use MavLinkTest to talk to your PX4 drone, with \"-server:addr:port\", for example, when connected to drone wifi use: MavLinkMoCap -server:10.42.0.228:14590 \"-project:D:\\OptiTrack\\Motive Project 2016-12-19 04.09.42 PM.ttp\" This publishes the ATT_POS_MOCAP messages and you can proxy those through to the PX4 by running MavLinkTest on the dronebrain using: MavLinkTest -serial:/dev/ttyACM0,115200 -proxy:10.42.0.228:14590 Now the drone will get the ATT_POS_MOCAP and you should see the light turn green meaning it is now has a home position and is ready to fly.","title":"MavLink MoCap"},{"location":"mavlinkcom_mocap/#welcome-to-mavlinkmocap","text":"This folder contains the MavLinkMoCap library which connects to a OptiTrack camera system for accurate indoor location.","title":"Welcome to MavLinkMoCap"},{"location":"mavlinkcom_mocap/#dependencies","text":"OptiTrack Motive . MavLinkCom .","title":"Dependencies:"},{"location":"mavlinkcom_mocap/#setup-rigidbody","text":"First you need to define a RigidBody named 'Quadrocopter' using Motive. See Rigid_Body_Tracking .","title":"Setup RigidBody"},{"location":"mavlinkcom_mocap/#mavlinktest","text":"Use MavLinkTest to talk to your PX4 drone, with \"-server:addr:port\", for example, when connected to drone wifi use: MavLinkMoCap -server:10.42.0.228:14590 \"-project:D:\\OptiTrack\\Motive Project 2016-12-19 04.09.42 PM.ttp\" This publishes the ATT_POS_MOCAP messages and you can proxy those through to the PX4 by running MavLinkTest on the dronebrain using: MavLinkTest -serial:/dev/ttyACM0,115200 -proxy:10.42.0.228:14590 Now the drone will get the ATT_POS_MOCAP and you should see the light turn green meaning it is now has a home position and is ready to fly.","title":"MavLinkTest"},{"location":"meshes/","text":"How to Access Meshes in AIRSIM # AirSim supports the ability to access the static meshes that make up the scene Mesh structure # Each mesh is represented with the below struct. struct MeshPositionVertexBuffersResponse { Vector3r position; Quaternionr orientation; std::vector vertices; std::vector indices; std::string name; }; The position and orientation are in the Unreal coordinate system. The mesh itself is a triangular mesh represented by the vertices and the indices. The triangular mesh type is typically called a Face-Vertex Mesh. This means every triplet of indices hold the indexes of the vertices that make up the triangle/face. The x,y,z coordinates of the vertices are all stored in a single vector. This means the vertices vertices is Nx3 where N is number of vertices. The position of the vertices are the global positions in the Unreal coordinate system. This means they have already been Transformed by the position and orientation. How to use # The API to get the meshes in the scene is quite simple. However, one should note that the function call is very expensive and should very rarely be called. In general this is ok because this function only accesses the static meshes which for most applications are not changing during the duration of your program. Note that you will have to use a 3rdparty library or your own custom code to actually interact with the recieved meshes. Below I utilize the python bindings of libigl to visualize the recieved meshes. import airsim AIRSIM_HOST_IP='127.0.0.1' client = airsim.VehicleClient(ip=AIRSIM_HOST_IP) client.confirmConnection() # List of returned meshes are received via this function meshes=client.simGetMeshPositionVertexBuffers() index=0 for m in meshes: # Finds one of the cube meshes in the Blocks environment if 'cube' in m.name: # Code from here on relies on libigl. Libigl uses pybind11 to wrap C++ code. So here the built pyigl.so # library is in the same directory as this example code. # This is here as code for your own mesh library should require something similar from pyigl import * from iglhelpers import * # Convert the lists to numpy arrays vertex_list=np.array(m.vertices,dtype=np.float32) indices=np.array(m.indices,dtype=np.uint32) num_vertices=int(len(vertex_list)/3) num_indices=len(indices) # Libigl requires the shape to be Nx3 where N is number of vertices or indices # It also requires the actual type to be double(float64) for vertices and int64 for the triangles/indices vertices_reshaped=vertex_list.reshape((num_vertices,3)) indices_reshaped=indices.reshape((int(num_indices/3),3)) vertices_reshaped=vertices_reshaped.astype(np.float64) indices_reshaped=indices_reshaped.astype(np.int64) #Libigl function to convert to internal Eigen format v_eig=p2e(vertices_reshaped) i_eig=p2e(indices_reshaped) # View the mesh viewer = igl.glfw.Viewer() viewer.data().set_mesh(v_eig,i_eig) viewer.launch() break","title":"Mesh Vertex Buffers"},{"location":"meshes/#how-to-access-meshes-in-airsim","text":"AirSim supports the ability to access the static meshes that make up the scene","title":"How to Access Meshes in AIRSIM"},{"location":"meshes/#mesh-structure","text":"Each mesh is represented with the below struct. struct MeshPositionVertexBuffersResponse { Vector3r position; Quaternionr orientation; std::vector vertices; std::vector indices; std::string name; }; The position and orientation are in the Unreal coordinate system. The mesh itself is a triangular mesh represented by the vertices and the indices. The triangular mesh type is typically called a Face-Vertex Mesh. This means every triplet of indices hold the indexes of the vertices that make up the triangle/face. The x,y,z coordinates of the vertices are all stored in a single vector. This means the vertices vertices is Nx3 where N is number of vertices. The position of the vertices are the global positions in the Unreal coordinate system. This means they have already been Transformed by the position and orientation.","title":"Mesh structure"},{"location":"meshes/#how-to-use","text":"The API to get the meshes in the scene is quite simple. However, one should note that the function call is very expensive and should very rarely be called. In general this is ok because this function only accesses the static meshes which for most applications are not changing during the duration of your program. Note that you will have to use a 3rdparty library or your own custom code to actually interact with the recieved meshes. Below I utilize the python bindings of libigl to visualize the recieved meshes. import airsim AIRSIM_HOST_IP='127.0.0.1' client = airsim.VehicleClient(ip=AIRSIM_HOST_IP) client.confirmConnection() # List of returned meshes are received via this function meshes=client.simGetMeshPositionVertexBuffers() index=0 for m in meshes: # Finds one of the cube meshes in the Blocks environment if 'cube' in m.name: # Code from here on relies on libigl. Libigl uses pybind11 to wrap C++ code. So here the built pyigl.so # library is in the same directory as this example code. # This is here as code for your own mesh library should require something similar from pyigl import * from iglhelpers import * # Convert the lists to numpy arrays vertex_list=np.array(m.vertices,dtype=np.float32) indices=np.array(m.indices,dtype=np.uint32) num_vertices=int(len(vertex_list)/3) num_indices=len(indices) # Libigl requires the shape to be Nx3 where N is number of vertices or indices # It also requires the actual type to be double(float64) for vertices and int64 for the triangles/indices vertices_reshaped=vertex_list.reshape((num_vertices,3)) indices_reshaped=indices.reshape((int(num_indices/3),3)) vertices_reshaped=vertices_reshaped.astype(np.float64) indices_reshaped=indices_reshaped.astype(np.int64) #Libigl function to convert to internal Eigen format v_eig=p2e(vertices_reshaped) i_eig=p2e(indices_reshaped) # View the mesh viewer = igl.glfw.Viewer() viewer.data().set_mesh(v_eig,i_eig) viewer.launch() break","title":"How to use"},{"location":"multi_vehicle/","text":"Multiple Vehicles in AirSim # Since release 1.2, AirSim is fully enabled for multiple vehicles. This capability allows you to create multiple vehicles easily and use APIs to control them. Creating Multiple Vehicles # It's as easy as specifying them in settings.json . The Vehicles element allows you to specify list of vehicles you want to create along with their initial positions and orientations. The positions are specified in NED coordinates in SI units with origin set at Player Start component in Unreal environment. The orientation is specified as Yaw, Pitch and Roll in degrees. Creating Multiple Cars # { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\", \"Vehicles\": { \"Car1\": { \"VehicleType\": \"PhysXCar\", \"X\": 4, \"Y\": 0, \"Z\": -2 }, \"Car2\": { \"VehicleType\": \"PhysXCar\", \"X\": -4, \"Y\": 0, \"Z\": -2, \"Yaw\": 90 } } } Creating Multiple Drones # { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"X\": 4, \"Y\": 0, \"Z\": -2, \"Yaw\": -180 }, \"Drone2\": { \"VehicleType\": \"SimpleFlight\", \"X\": 8, \"Y\": 0, \"Z\": -2 } } } Using APIs for Multiple Vehicles # The new APIs since AirSim 1.2 allows you to specify vehicle_name . This name corresponds to keys in json settings (for example, Car1 or Drone2 above). Example code for cars Example code for multirotors Demo #","title":"Multiple Vehicles"},{"location":"multi_vehicle/#multiple-vehicles-in-airsim","text":"Since release 1.2, AirSim is fully enabled for multiple vehicles. This capability allows you to create multiple vehicles easily and use APIs to control them.","title":"Multiple Vehicles in AirSim"},{"location":"multi_vehicle/#creating-multiple-vehicles","text":"It's as easy as specifying them in settings.json . The Vehicles element allows you to specify list of vehicles you want to create along with their initial positions and orientations. The positions are specified in NED coordinates in SI units with origin set at Player Start component in Unreal environment. The orientation is specified as Yaw, Pitch and Roll in degrees.","title":"Creating Multiple Vehicles"},{"location":"multi_vehicle/#creating-multiple-cars","text":"{ \"SettingsVersion\": 1.2, \"SimMode\": \"Car\", \"Vehicles\": { \"Car1\": { \"VehicleType\": \"PhysXCar\", \"X\": 4, \"Y\": 0, \"Z\": -2 }, \"Car2\": { \"VehicleType\": \"PhysXCar\", \"X\": -4, \"Y\": 0, \"Z\": -2, \"Yaw\": 90 } } }","title":"Creating Multiple Cars"},{"location":"multi_vehicle/#creating-multiple-drones","text":"{ \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"X\": 4, \"Y\": 0, \"Z\": -2, \"Yaw\": -180 }, \"Drone2\": { \"VehicleType\": \"SimpleFlight\", \"X\": 8, \"Y\": 0, \"Z\": -2 } } }","title":"Creating Multiple Drones"},{"location":"multi_vehicle/#using-apis-for-multiple-vehicles","text":"The new APIs since AirSim 1.2 allows you to specify vehicle_name . This name corresponds to keys in json settings (for example, Car1 or Drone2 above). Example code for cars Example code for multirotors","title":"Using APIs for Multiple Vehicles"},{"location":"multi_vehicle/#demo","text":"","title":"Demo"},{"location":"orbit/","text":"An Orbit Trajectory # Moved here from https://github.com/microsoft/AirSim/wiki/An-Orbit-Trajectory Have you ever wanted to fly a nice smooth circular orbit? This can be handy for capturing 3D objects from all sides especially if you get multiple orbits at different altitudes. So the PythonClient/multirotor folder contains a script named Orbit that will do precisely that. See demo video The demo video was created by running this command line: python orbit.py --radius 10 --altitude 5 --speed 1 --center \"0,1\" --iterations 1 This flies a 10 meter radius orbit around the center location at (startpos + radius * [0,1]), in other words, the center is located radius meters away in the direction of the provided center vector. It also keeps the front-facing camera on the drone always pointing at the center of the circle. If you watch the flight using LogViewer you will see a nice circular pattern get traced out on the GPS map: The core of the algorithm is not that complicated. At each point on the circle, we look ahead by a small delta in degrees, called the lookahead_angle , where that angle is computed based on our desired velocity. We then find that lookahead point on the circle using sin/cosine and make that our \"target point\". Calculating the velocity then is easy, just subtract our current position from that point and feed that into the AirSim method moveByVelocityZ .","title":"Orbit Trajectory"},{"location":"orbit/#an-orbit-trajectory","text":"Moved here from https://github.com/microsoft/AirSim/wiki/An-Orbit-Trajectory Have you ever wanted to fly a nice smooth circular orbit? This can be handy for capturing 3D objects from all sides especially if you get multiple orbits at different altitudes. So the PythonClient/multirotor folder contains a script named Orbit that will do precisely that. See demo video The demo video was created by running this command line: python orbit.py --radius 10 --altitude 5 --speed 1 --center \"0,1\" --iterations 1 This flies a 10 meter radius orbit around the center location at (startpos + radius * [0,1]), in other words, the center is located radius meters away in the direction of the provided center vector. It also keeps the front-facing camera on the drone always pointing at the center of the circle. If you watch the flight using LogViewer you will see a nice circular pattern get traced out on the GPS map: The core of the algorithm is not that complicated. At each point on the circle, we look ahead by a small delta in degrees, called the lookahead_angle , where that angle is computed based on our desired velocity. We then find that lookahead point on the circle using sin/cosine and make that our \"target point\". Calculating the velocity then is easy, just subtract our current position from that point and feed that into the AirSim method moveByVelocityZ .","title":"An Orbit Trajectory"},{"location":"pfm/","text":"pfm Format # Pfm (or Portable FloatMap) image format stores image as floating point pixels and hence are not restricted to usual 0-255 pixel value range. This is useful for HDR images or images that describes something other than colors like depth. One of the good viewer to view this file format is PfmPad . We don't recommend Maverick photo viewer because it doesn't seem to show depth images properly. AirSim has code to write pfm file for C++ and read as well as write for Python .","title":"pfm format"},{"location":"pfm/#pfm-format","text":"Pfm (or Portable FloatMap) image format stores image as floating point pixels and hence are not restricted to usual 0-255 pixel value range. This is useful for HDR images or images that describes something other than colors like depth. One of the good viewer to view this file format is PfmPad . We don't recommend Maverick photo viewer because it doesn't seem to show depth images properly. AirSim has code to write pfm file for C++ and read as well as write for Python .","title":"pfm Format"},{"location":"playback/","text":"Playback # AirSim supports playing back the high level commands in a *.mavlink log file that were recorded using the MavLinkTest app for the purpose of comparing real and simulated flight. The recording.mavlink is an example of a log file captured using a real drone using the following command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. Then the log file contains the commands performed, which included several \"orbit\" commands, the resulting GPS map of the flight looks like this: Side-by-side comparison # Now we can copy the *.mavlink log file recorded by MavLinkTest to the PC running the Unreal simulator with AirSim plugin. When the Simulator is running and the drone is parked in a place in a map that has room to do the same maneuvers we can run this MavLinkTest command line: MavLinkTest -server:127.0.0.1:14550 This should connect to the simulator. Now you can enter this command: PlayLog recording.mavlink The same commands you performed on the real drone will now play again in the simulator. You can then press 't' to see the trace, and it will show you the trace of the real drone and the simulated drone. Every time you press 't' again you can reset the lines so they are sync'd to the current position, this way I was able to capture a side-by-side trace of the \"orbit\" command performed in this recording, which generates the picture below. The pink line is the simulated flight and the red line is the real flight: Note: I'm using the ';' key in the simulator to take control of camera position using keyboard to get this shot. Parameters # It may help to set the simulator up with some of the same flight parameters that your real drone is using, for example, in my case I was using a lower than normal cruise speed, slow takeoff speed, and it helps to tell the simulator to wait a long time before disarming (COM_DISARM_LAND) and to turn off the safety switches NAV_RCL_ACT and NAV_DLL_ACT ( don't do that on a real drone). param MPC_XY_CRUISE 2 param MPC_XY_VEL_MAX 2 param MPC_TKO_SPEED 1 param COM_DISARM_LAND 60 param NAV_RCL_ACT 0 param NAV_DLL_ACT 0","title":"Playing Logs"},{"location":"playback/#playback","text":"AirSim supports playing back the high level commands in a *.mavlink log file that were recorded using the MavLinkTest app for the purpose of comparing real and simulated flight. The recording.mavlink is an example of a log file captured using a real drone using the following command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. Then the log file contains the commands performed, which included several \"orbit\" commands, the resulting GPS map of the flight looks like this:","title":"Playback"},{"location":"playback/#side-by-side-comparison","text":"Now we can copy the *.mavlink log file recorded by MavLinkTest to the PC running the Unreal simulator with AirSim plugin. When the Simulator is running and the drone is parked in a place in a map that has room to do the same maneuvers we can run this MavLinkTest command line: MavLinkTest -server:127.0.0.1:14550 This should connect to the simulator. Now you can enter this command: PlayLog recording.mavlink The same commands you performed on the real drone will now play again in the simulator. You can then press 't' to see the trace, and it will show you the trace of the real drone and the simulated drone. Every time you press 't' again you can reset the lines so they are sync'd to the current position, this way I was able to capture a side-by-side trace of the \"orbit\" command performed in this recording, which generates the picture below. The pink line is the simulated flight and the red line is the real flight: Note: I'm using the ';' key in the simulator to take control of camera position using keyboard to get this shot.","title":"Side-by-side comparison"},{"location":"playback/#parameters","text":"It may help to set the simulator up with some of the same flight parameters that your real drone is using, for example, in my case I was using a lower than normal cruise speed, slow takeoff speed, and it helps to tell the simulator to wait a long time before disarming (COM_DISARM_LAND) and to turn off the safety switches NAV_RCL_ACT and NAV_DLL_ACT ( don't do that on a real drone). param MPC_XY_CRUISE 2 param MPC_XY_VEL_MAX 2 param MPC_TKO_SPEED 1 param COM_DISARM_LAND 60 param NAV_RCL_ACT 0 param NAV_DLL_ACT 0","title":"Parameters"},{"location":"point_clouds/","text":"Point Clouds # Moved here from https://github.com/microsoft/AirSim/wiki/Point-Clouds A Python script point_cloud.py shows how to convert the depth image returned from AirSim into a point cloud. The following depth image was captured using the Modular Neighborhood environment: And with the appropriate projection matrix, the OpenCV reprojectImageTo3D function can turn this into a point cloud. The following is the result, which is also available here: https://skfb.ly/68r7y . SketchFab can upload the resulting file cloud.asc and render it for you. PS: you may notice the scene is reflected on the Y axis, so I may have a sign wrong in the projection matrix. An exercise for the reader :-)","title":"Building Point Clouds"},{"location":"point_clouds/#point-clouds","text":"Moved here from https://github.com/microsoft/AirSim/wiki/Point-Clouds A Python script point_cloud.py shows how to convert the depth image returned from AirSim into a point cloud. The following depth image was captured using the Modular Neighborhood environment: And with the appropriate projection matrix, the OpenCV reprojectImageTo3D function can turn this into a point cloud. The following is the result, which is also available here: https://skfb.ly/68r7y . SketchFab can upload the resulting file cloud.asc and render it for you. PS: you may notice the scene is reflected on the Y axis, so I may have a sign wrong in the projection matrix. An exercise for the reader :-)","title":"Point Clouds"},{"location":"px4_build/","text":"Building PX4 # Source code # Getting the PX4 source code is easy: git clone https://github.com/PX4/Firmware.git cd Firmware Oh, and if you don't have git yet just run this: sudo apt-get install git We are currently testing using the 1.6.0rc1 version, but the latest master branch should be ok too. Now to build it you will need the right tools. PX4 Build tools # The full instructions are available on the dev.px4.io website, but we've copied the relevant subset of those instructions here for your convenience. (Note that BashOnWindows ) can be used to build the PX4 firmware, just follow the BashOnWindows instructions at the bottom of this page). Build SITL version # Now you can make the SITL version that runs in posix, from the Firmware folder you created above: make posix_sitl_ekf2 Note: this build system is quite special, it knows how to update git submodules (and there's a lot of them), then it runs cmake (if necessary), then it runs the build itself. So in a way the root Makefile is a meta-meta makefile :-) It shouldn't take long, about 2 minutes. If all succeeds, the last line will link the px4 app, which you can then run using the following: make posix_sitl_ekf2 none_iris And you should see output that looks like this: creating new parameters file creating new dataman file ______ __ __ ___ | ___ \\ \\ \\ / / / | | |_/ / \\ V / / /| | | __/ / \\ / /_| | | | / /^\\ \\ \\___ | \\_| \\/ \\/ |_/ px4 starting. 18446744073709551615 WARNING: setRealtimeSched failed (not run as root?) ERROR [param] importing from 'rootfs/eeprom/parameters' failed (-1) Command 'param' failed, returned 1 SYS_AUTOSTART: curr: 0 -> new: 4010 SYS_MC_EST_GROUP: curr: 2 -> new: 1 INFO [dataman] Unkown restart, data manager file 'rootfs/fs/microsd/dataman' size is 11797680 bytes BAT_N_CELLS: curr: 0 -> new: 3 CAL_GYRO0_ID: curr: 0 -> new: 2293768 CAL_ACC0_ID: curr: 0 -> new: 1376264 CAL_ACC1_ID: curr: 0 -> new: 1310728 CAL_MAG0_ID: curr: 0 -> new: 196616 so this is good, first run sets up the px4 parameters for SITL mode. Second run has less output. This app is also an interactive console where you can type commands. Type 'help' to see what they are and just type ctrl-C to kill it. You can do that and restart it any time, that's a great way to reset any wonky state if you need to (it's equivalent to a Pixhawk hardware reboot). ARM embedded tools # If you plan to build the PX4 firmware for real Pixhawk hardware then you will need the gcc cross-compiler for ARM Cortex-M4 chipset. You can get this compiler by PX4 DevGuide, specifically this is in their ubuntu_sim_nuttx.sh setup script. After following those setup instructions you can verify the install by entering this command arm-none-eabi-gcc --version . You should see the following output: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204] Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Build PX4 for ARM hardware # Now you can build the PX4 firmware for running on real pixhawk hardware: make px4fmu-v2_default This build will take a little longer because it is building a lot more including the NuttX real time OS, all the drivers for the sensors in the Pixhawk flight controller, and more. It is also running the compiler in super size-squeezing mode so it can fit all that in a 1 megabyte ROM !! One nice tid bit is you can plug in your pixhawk USB, and type make px4fmu-v2_default upload to flash the hardware with these brand new bits, so you don't need to use QGroundControl for that. Some Useful Parameters # PX4 has many customizable parameters (over 700 of them, in fact) and to get best results with AirSim we have found the following parameters are handy: // be sure to enable the new position estimator module: param set SYS_MC_EST_GROUP 2 // increase default limits on cruise speed so you can move around a large map more quickly. param MPC_XY_CRUISE 10 param MPC_XY_VEL_MAX 10 param MPC_Z_VEL_MAX_DN 2 // increase timeout for auto-disarm on landing so that any long running app doesn't have to worry about it param COM_DISARM_LAND 60 // make it possible to fly without radio control attached (do NOT do this one on a real drone) param NAV_RCL_ACT 0 // enable new syslogger to get more information from PX4 logs param set SYS_LOGGER 1 Using BashOnWindows # See Bash on Windows Toolchain .","title":"Building PX4"},{"location":"px4_build/#building-px4","text":"","title":"Building PX4"},{"location":"px4_build/#source-code","text":"Getting the PX4 source code is easy: git clone https://github.com/PX4/Firmware.git cd Firmware Oh, and if you don't have git yet just run this: sudo apt-get install git We are currently testing using the 1.6.0rc1 version, but the latest master branch should be ok too. Now to build it you will need the right tools.","title":"Source code"},{"location":"px4_build/#px4-build-tools","text":"The full instructions are available on the dev.px4.io website, but we've copied the relevant subset of those instructions here for your convenience. (Note that BashOnWindows ) can be used to build the PX4 firmware, just follow the BashOnWindows instructions at the bottom of this page).","title":"PX4 Build tools"},{"location":"px4_build/#build-sitl-version","text":"Now you can make the SITL version that runs in posix, from the Firmware folder you created above: make posix_sitl_ekf2 Note: this build system is quite special, it knows how to update git submodules (and there's a lot of them), then it runs cmake (if necessary), then it runs the build itself. So in a way the root Makefile is a meta-meta makefile :-) It shouldn't take long, about 2 minutes. If all succeeds, the last line will link the px4 app, which you can then run using the following: make posix_sitl_ekf2 none_iris And you should see output that looks like this: creating new parameters file creating new dataman file ______ __ __ ___ | ___ \\ \\ \\ / / / | | |_/ / \\ V / / /| | | __/ / \\ / /_| | | | / /^\\ \\ \\___ | \\_| \\/ \\/ |_/ px4 starting. 18446744073709551615 WARNING: setRealtimeSched failed (not run as root?) ERROR [param] importing from 'rootfs/eeprom/parameters' failed (-1) Command 'param' failed, returned 1 SYS_AUTOSTART: curr: 0 -> new: 4010 SYS_MC_EST_GROUP: curr: 2 -> new: 1 INFO [dataman] Unkown restart, data manager file 'rootfs/fs/microsd/dataman' size is 11797680 bytes BAT_N_CELLS: curr: 0 -> new: 3 CAL_GYRO0_ID: curr: 0 -> new: 2293768 CAL_ACC0_ID: curr: 0 -> new: 1376264 CAL_ACC1_ID: curr: 0 -> new: 1310728 CAL_MAG0_ID: curr: 0 -> new: 196616 so this is good, first run sets up the px4 parameters for SITL mode. Second run has less output. This app is also an interactive console where you can type commands. Type 'help' to see what they are and just type ctrl-C to kill it. You can do that and restart it any time, that's a great way to reset any wonky state if you need to (it's equivalent to a Pixhawk hardware reboot).","title":"Build SITL version"},{"location":"px4_build/#arm-embedded-tools","text":"If you plan to build the PX4 firmware for real Pixhawk hardware then you will need the gcc cross-compiler for ARM Cortex-M4 chipset. You can get this compiler by PX4 DevGuide, specifically this is in their ubuntu_sim_nuttx.sh setup script. After following those setup instructions you can verify the install by entering this command arm-none-eabi-gcc --version . You should see the following output: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204] Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.","title":"ARM embedded tools"},{"location":"px4_build/#build-px4-for-arm-hardware","text":"Now you can build the PX4 firmware for running on real pixhawk hardware: make px4fmu-v2_default This build will take a little longer because it is building a lot more including the NuttX real time OS, all the drivers for the sensors in the Pixhawk flight controller, and more. It is also running the compiler in super size-squeezing mode so it can fit all that in a 1 megabyte ROM !! One nice tid bit is you can plug in your pixhawk USB, and type make px4fmu-v2_default upload to flash the hardware with these brand new bits, so you don't need to use QGroundControl for that.","title":"Build PX4 for ARM hardware"},{"location":"px4_build/#some-useful-parameters","text":"PX4 has many customizable parameters (over 700 of them, in fact) and to get best results with AirSim we have found the following parameters are handy: // be sure to enable the new position estimator module: param set SYS_MC_EST_GROUP 2 // increase default limits on cruise speed so you can move around a large map more quickly. param MPC_XY_CRUISE 10 param MPC_XY_VEL_MAX 10 param MPC_Z_VEL_MAX_DN 2 // increase timeout for auto-disarm on landing so that any long running app doesn't have to worry about it param COM_DISARM_LAND 60 // make it possible to fly without radio control attached (do NOT do this one on a real drone) param NAV_RCL_ACT 0 // enable new syslogger to get more information from PX4 logs param set SYS_LOGGER 1","title":"Some Useful Parameters"},{"location":"px4_build/#using-bashonwindows","text":"See Bash on Windows Toolchain .","title":"Using BashOnWindows"},{"location":"px4_logging/","text":"PX4/MavLink Logging # Thanks to Chris Lovett for developing various tools for PX4/MavLink logging mentioned on this page! Logging MavLink Messages # The following command will connect MavLinkTest app to the Simulator and enable logging of all mavlink commands to and from the PX4. MavLinkTest -server:127.0.0.1:14550 -logdir:d:\\temp Sometimes you will also want to log the \"output\" from the Simulator that is being sent to the PX4. Specifically this will capture the \"hilgps\" and \"hilsensor\" messages that are generated by the Simulator. To do that run this as well: MavLinkTest -server:127.0.0.1:14389 -logdir:d:\\temp\\output You will then see log files organized by date in d:\\temp\\logs, specifically input.mavlink and output.mavlink files. MavLink LogViewer # For MavLink enabled drones, you can also use our Log Viewer to visualize the streams of data. PX4 Log in SITL Mode # In SITL mode, please a log file is produced when drone is armed. The SITL terminal will contain the path to the log file, it should look something like this INFO [logger] Opened log file: rootfs/fs/microsd/log/2017-03-27/20_02_49.ulg PX4 Log in SITL Mode # If you are using Pixhawk hardware in HIL mode, then set parameter SYS_LOGGER=1 using QGroundControl. PX4 will write log file on device which you can download at later date using QGroundControl.","title":"PX4/MavLink Logging"},{"location":"px4_logging/#px4mavlink-logging","text":"Thanks to Chris Lovett for developing various tools for PX4/MavLink logging mentioned on this page!","title":"PX4/MavLink Logging"},{"location":"px4_logging/#logging-mavlink-messages","text":"The following command will connect MavLinkTest app to the Simulator and enable logging of all mavlink commands to and from the PX4. MavLinkTest -server:127.0.0.1:14550 -logdir:d:\\temp Sometimes you will also want to log the \"output\" from the Simulator that is being sent to the PX4. Specifically this will capture the \"hilgps\" and \"hilsensor\" messages that are generated by the Simulator. To do that run this as well: MavLinkTest -server:127.0.0.1:14389 -logdir:d:\\temp\\output You will then see log files organized by date in d:\\temp\\logs, specifically input.mavlink and output.mavlink files.","title":"Logging MavLink Messages"},{"location":"px4_logging/#mavlink-logviewer","text":"For MavLink enabled drones, you can also use our Log Viewer to visualize the streams of data.","title":"MavLink LogViewer"},{"location":"px4_logging/#px4-log-in-sitl-mode","text":"In SITL mode, please a log file is produced when drone is armed. The SITL terminal will contain the path to the log file, it should look something like this INFO [logger] Opened log file: rootfs/fs/microsd/log/2017-03-27/20_02_49.ulg","title":"PX4 Log in SITL Mode"},{"location":"px4_logging/#px4-log-in-sitl-mode_1","text":"If you are using Pixhawk hardware in HIL mode, then set parameter SYS_LOGGER=1 using QGroundControl. PX4 will write log file on device which you can download at later date using QGroundControl.","title":"PX4 Log in SITL Mode"},{"location":"px4_setup/","text":"PX4 Setup for AirSim # The PX4 software stack is an open source very popular flight controller with support for wide variety of boards and sensors as well as built-in capability for higher level tasks such as mission planning. Please visit px4.io for more information. Warning : While all releases of AirSim are always tested with PX4 to ensure the support, setting up PX4 is not a trivial task. Unless you have at least intermediate level of experience with PX4 stack, we recommend you use simple_flight , which is now a default in AirSim. Supported Hardware # The following Pixhawk hardware has been tested with AirSim: 3DR Pixhawk v2 3DR Pixhawk mini Pixhawk PX4 2.4.8 PixFalcon PixRacer Pixhawk 2.1 (using PX4 Flight Stack) The 3DR Pixhawk Mini works out of the box, the others you may need to re-flash with the latest firmware. Setting up PX4 Hardware-in-Loop # For this you will need one of the supported device listed above. For manual flight you will also need RC + transmitter. Make sure your RC receiver is bound with its RC transmitter. Connect the RC transmitter to the flight controller's RC port. Refer to your RC manual and PX4 docs for more information. Download QGroundControl , launch it and connect your flight controller to the USB port. Use QGroundControl to flash the latest PX4 Flight Stack. See also initial firmware setup video . In QGroundControl, configure your Pixhawk for HIL simulation by selecting the HIL Quadrocopter X airframe. After PX4 reboots, check that \"HIL Quadrocopter X\" is indeed selected. In QGroundControl, go to Radio tab and calibrate (make sure the remote control is on and the receiver is showing the indicator for the binding). Go to the Flight Mode tab and chose one of the remote control switches as \"Mode Channel\". Then set (for example) Stabilized and Attitude flight modes for two positions of the switch. Go to the Tuning section of QGroundControl and set appropriate values. For example, for Fly Sky's FS-TH9X remote control, the following settings give a more realistic feel: Hover Throttle = mid+1 mark, Roll and pitch sensitivity = mid-3 mark, Altitude and position control sensitivity = mid-2 mark. In AirSim settings file, specify PX4 for your vehicle config like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\" } } } After above setup you should be able to use RC to fly in the AirSim. You can usually arm the vehicle by lowering and bringing two sticks of RC together in-wards. You don't need QGroundControl after the initial setup. Typically the Stabilized (instead of Manual) mode gives better experience for beginners. You can also control the drone from Python APIs . See Walkthrough Demo Video and Unreal AirSim Setup Video that shows you all the setup steps in this document. Setting up PX4 Software-in-Loop # The PX4 SITL mode doesn't require you to have separate device such as a Pixhawk or Pixracer. This is in fact the recommended way to use PX4 with simulators by PX4 team. However, this is indeed harder to set up. Please see this dedicated page for setting up PX4 in SITL mode. FAQ # Drone doesn't fly properly, it just goes \"crazy\". # There are a few reasons that can cause this. First, make sure your drone doesn't fall down large distance when starting the simulator. This might happen if you have created a custom Unreal environment and Player Start is placed too high above the ground. It seems that when this happens internal calibration in PX4 gets confused. You should also use QGroundControl and make sure you can arm and takeoff in QGroundControl properly. Finally, this also can be a machine performance issue in some rare cases, check your hard drive performance . Can I use Arducopter or other MavLink implementations? # Our code is tested with the PX4 firmware . We have not tested Arducopter or other mavlink implementations. Some of the flight API's do use the PX4 custom modes in the MAV_CMD_DO_SET_MODE messages (like PX4_CUSTOM_MAIN_MODE_AUTO) It is not finding my Pixhawk hardware # Check your settings.json file for this line \"SerialPort\":\"*,115200\". The asterisk here means \"find any serial port that looks like a Pixhawk device, but this doesn't always work for all types of Pixhawk hardware. So on Windows you can find the actual COM port using Device Manager, look under \"Ports (COM & LPT), plug the device in and see what new COM port shows up. Let's say you see a new port named \"USB Serial Port (COM5)\". Well, then change the SerialPort setting to this: \"SerialPort\":\"COM5,115200\". On Linux, the device can be found by running \"ls /dev/serial/by-id\" if you see a device name listed that looks like this usb-3D_Robotics_PX4_FMU_v2.x_0-if00 then you can use that name to connect, like this: \"SerialPort\":\"/dev/serial/by-id/usb-3D_Robotics_PX4_FMU_v2.x_0-if00\". Note this long name is actually a symbolic link to the real name, if you use \"ls -l ...\" you can find that symbolic link, it is usually something like \"/dev/ttyACM0\", so this will also work \"SerialPort\":\"/dev/ttyACM0,115200\". But that mapping is similar to windows, it is automatically assigned and can change, whereas the long name will work even if the actual TTY serial device mapping changes. WARN [commander] Takeoff denied, disarm and re-try # This happens if you try and take off when PX4 still has not computed the home position. PX4 will report the home position once it is happy with the GPS signal, and you will see these messages: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Up until this point in time, however, the PX4 will reject takeoff commands. When I tell the drone to do something it always lands # For example, you use DroneShell moveToPosition -z -20 -x 50 -y 0 which it does, but when it gets to the target location the drone starts to land. This is the default behavior of PX4 when offboard mode completes. To set the drone to hover instead set this PX4 parameter: param set COM_OBL_ACT 1 I get message length mismatches errors # You might need to set MAV_PROTO_VER parameter in QGC to \"Always use version 1\". Please see this issue more details.","title":"PX4 Setup for AirSim"},{"location":"px4_setup/#px4-setup-for-airsim","text":"The PX4 software stack is an open source very popular flight controller with support for wide variety of boards and sensors as well as built-in capability for higher level tasks such as mission planning. Please visit px4.io for more information. Warning : While all releases of AirSim are always tested with PX4 to ensure the support, setting up PX4 is not a trivial task. Unless you have at least intermediate level of experience with PX4 stack, we recommend you use simple_flight , which is now a default in AirSim.","title":"PX4 Setup for AirSim"},{"location":"px4_setup/#supported-hardware","text":"The following Pixhawk hardware has been tested with AirSim: 3DR Pixhawk v2 3DR Pixhawk mini Pixhawk PX4 2.4.8 PixFalcon PixRacer Pixhawk 2.1 (using PX4 Flight Stack) The 3DR Pixhawk Mini works out of the box, the others you may need to re-flash with the latest firmware.","title":"Supported Hardware"},{"location":"px4_setup/#setting-up-px4-hardware-in-loop","text":"For this you will need one of the supported device listed above. For manual flight you will also need RC + transmitter. Make sure your RC receiver is bound with its RC transmitter. Connect the RC transmitter to the flight controller's RC port. Refer to your RC manual and PX4 docs for more information. Download QGroundControl , launch it and connect your flight controller to the USB port. Use QGroundControl to flash the latest PX4 Flight Stack. See also initial firmware setup video . In QGroundControl, configure your Pixhawk for HIL simulation by selecting the HIL Quadrocopter X airframe. After PX4 reboots, check that \"HIL Quadrocopter X\" is indeed selected. In QGroundControl, go to Radio tab and calibrate (make sure the remote control is on and the receiver is showing the indicator for the binding). Go to the Flight Mode tab and chose one of the remote control switches as \"Mode Channel\". Then set (for example) Stabilized and Attitude flight modes for two positions of the switch. Go to the Tuning section of QGroundControl and set appropriate values. For example, for Fly Sky's FS-TH9X remote control, the following settings give a more realistic feel: Hover Throttle = mid+1 mark, Roll and pitch sensitivity = mid-3 mark, Altitude and position control sensitivity = mid-2 mark. In AirSim settings file, specify PX4 for your vehicle config like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\" } } } After above setup you should be able to use RC to fly in the AirSim. You can usually arm the vehicle by lowering and bringing two sticks of RC together in-wards. You don't need QGroundControl after the initial setup. Typically the Stabilized (instead of Manual) mode gives better experience for beginners. You can also control the drone from Python APIs . See Walkthrough Demo Video and Unreal AirSim Setup Video that shows you all the setup steps in this document.","title":"Setting up PX4 Hardware-in-Loop"},{"location":"px4_setup/#setting-up-px4-software-in-loop","text":"The PX4 SITL mode doesn't require you to have separate device such as a Pixhawk or Pixracer. This is in fact the recommended way to use PX4 with simulators by PX4 team. However, this is indeed harder to set up. Please see this dedicated page for setting up PX4 in SITL mode.","title":"Setting up PX4 Software-in-Loop"},{"location":"px4_setup/#faq","text":"","title":"FAQ"},{"location":"px4_setup/#drone-doesnt-fly-properly-it-just-goes-crazy","text":"There are a few reasons that can cause this. First, make sure your drone doesn't fall down large distance when starting the simulator. This might happen if you have created a custom Unreal environment and Player Start is placed too high above the ground. It seems that when this happens internal calibration in PX4 gets confused. You should also use QGroundControl and make sure you can arm and takeoff in QGroundControl properly. Finally, this also can be a machine performance issue in some rare cases, check your hard drive performance .","title":"Drone doesn't fly properly, it just goes \"crazy\"."},{"location":"px4_setup/#can-i-use-arducopter-or-other-mavlink-implementations","text":"Our code is tested with the PX4 firmware . We have not tested Arducopter or other mavlink implementations. Some of the flight API's do use the PX4 custom modes in the MAV_CMD_DO_SET_MODE messages (like PX4_CUSTOM_MAIN_MODE_AUTO)","title":"Can I use Arducopter or other MavLink implementations?"},{"location":"px4_setup/#it-is-not-finding-my-pixhawk-hardware","text":"Check your settings.json file for this line \"SerialPort\":\"*,115200\". The asterisk here means \"find any serial port that looks like a Pixhawk device, but this doesn't always work for all types of Pixhawk hardware. So on Windows you can find the actual COM port using Device Manager, look under \"Ports (COM & LPT), plug the device in and see what new COM port shows up. Let's say you see a new port named \"USB Serial Port (COM5)\". Well, then change the SerialPort setting to this: \"SerialPort\":\"COM5,115200\". On Linux, the device can be found by running \"ls /dev/serial/by-id\" if you see a device name listed that looks like this usb-3D_Robotics_PX4_FMU_v2.x_0-if00 then you can use that name to connect, like this: \"SerialPort\":\"/dev/serial/by-id/usb-3D_Robotics_PX4_FMU_v2.x_0-if00\". Note this long name is actually a symbolic link to the real name, if you use \"ls -l ...\" you can find that symbolic link, it is usually something like \"/dev/ttyACM0\", so this will also work \"SerialPort\":\"/dev/ttyACM0,115200\". But that mapping is similar to windows, it is automatically assigned and can change, whereas the long name will work even if the actual TTY serial device mapping changes.","title":"It is not finding my Pixhawk hardware"},{"location":"px4_setup/#warn-commander-takeoff-denied-disarm-and-re-try","text":"This happens if you try and take off when PX4 still has not computed the home position. PX4 will report the home position once it is happy with the GPS signal, and you will see these messages: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Up until this point in time, however, the PX4 will reject takeoff commands.","title":"WARN [commander] Takeoff denied, disarm and re-try"},{"location":"px4_setup/#when-i-tell-the-drone-to-do-something-it-always-lands","text":"For example, you use DroneShell moveToPosition -z -20 -x 50 -y 0 which it does, but when it gets to the target location the drone starts to land. This is the default behavior of PX4 when offboard mode completes. To set the drone to hover instead set this PX4 parameter: param set COM_OBL_ACT 1","title":"When I tell the drone to do something it always lands"},{"location":"px4_setup/#i-get-message-length-mismatches-errors","text":"You might need to set MAV_PROTO_VER parameter in QGC to \"Always use version 1\". Please see this issue more details.","title":"I get message length mismatches errors"},{"location":"px4_sitl/","text":"Setting up PX4 Software-in-Loop # The PX4 software provides a \"software-in-loop\" simulation (SITL) version of their stack that runs in Linux. If you are on Windows then you must use the Cygwin Toolchain as the Bash On Windows toolchain no longer works for SITL. Note that every time you stop the unreal app you have to restart the px4 app. From your bash terminal follow these steps for Linux and follow all the instructions under NuttX based hardware to install prerequisites. We've also included out own copy of the PX4 build instructions which is a bit more concise about what we need exactly. Get the PX4 source code and build the posix SITL version of PX4: mkdir -p PX4 cd PX4 git clone https://github.com/PX4/Firmware.git cd Firmware git checkout v1.10.1 # recommended version Use following command to build and start PX4 firmware in SITL mode: make px4_sitl_default none_iris If you are using older version v1.8.* use this command instead: make posix_sitl_ekf2 none_iris . You should see a message saying the SITL PX4 app is waiting for the simulator (AirSim) to connect. You will also see information about which ports are configured for mavlink connection to the PX4 app. The default ports have changed recently, so check them closely to make sure AirSim settings are correct. INFO [simulator] Waiting for simulator to connect on TCP port 4560 INFO [init] Mixer: etc/mixers/quad_w.main.mix on /dev/pwm_output0 INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 14570 remote port 14550 INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540 Note: this is also an interactive PX4 console, type help to see the list of commands you can enter here. They are mostly low level PX4 commands, but some of them can be useful for debugging. Now edit AirSim settings file to make sure you have matching UDP and TCP port settings: json { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"UseSerial\": false, \"UseTcp\": true, \"TcpPort\": 4560, \"ControlPort\": 14580, \"params\": { \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, \"COM_OBL_ACT\": 1 } } } } Notice the PX4 [simulator] is using TCP, which is why we need to add: \"UseTcp\": true, . Now run your Unreal AirSim environment and it should connect to SITL PX4 via TCP. You should see a bunch of messages from the SITL PX4 window. Specifically, the following messages tell you that AirSim is connected properly and GPS fusion is stable: INFO [simulator] Simulator connected on UDP port 14560 INFO [mavlink] partner IP: 127.0.0.1 INFO [ecl/EKF] EKF GPS checks passed (WGS-84 origin set) INFO [ecl/EKF] EKF commencing GPS fusion If you do not see these messages then check your port settings. You should also be able to use QGroundControl with SITL mode. Make sure there is no Pixhawk hardware plugged in, otherwise QGroundControl will choose to use that instead. Note that as we don't have a physical board, an RC cannot be connected directly to it. So the alternatives are either use XBox 360 Controller or connect your RC using USB (for example, in case of FrSky Taranis X9D Plus) or using trainer USB cable to your PC. This makes your RC look like a joystick. You will need to do extra set up in QGroundControl to use virtual joystick for RC control. You do not need to do this unless you plan to fly a drone manually in AirSim. Autonomous flight using the Python API does not require RC, see No Remote Control below. Setting GPS origin # Notice the above settings are provided in the params section of the settings.json file: \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, PX4 SITL mode needs to be configured to get the home location correct. There is a bug in AirSim that makes it such that flight does not work unless the home location is set to the same coordinates defined in AVehiclePawnBase::HomeLatitude and HomeLongitude. You can also run the following in the SITL PX4 console window to check that these values are set correctly. param show LPE_LAT param show LPE_LON Smooth Offboard Transitions # Notice the above setting is provided in the params section of the settings.json file: \"COM_OBL_ACT\": 1 This tells the drone automatically hover after each offboard control command finishes (the default setting is to land). Hovering is a smoother transition between multiple offboard commands. You can check this setting by running the following PX4 console command: param show COM_OBL_ACT Check the Home Position # If you are using DroneShell to execute commands (arm, takeoff, etc) then you should wait until the Home position is set. You will see the PX4 SITL console output this message: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Now DroneShell 'pos' command should report this position and the commands should be accepted by PX4. If you attempt to takeoff without a home position you will see the message: WARN [commander] Takeoff denied, disarm and re-try After home position is set check the local position reported by 'pos' command : Local position: x=-0.0326988, y=0.00656854, z=5.48506 If the z coordinate is large like this then takeoff might not work as expected. Resetting the SITL and simulation should fix that problem. No Remote Control # Notice the above setting is provided in the params section of the settings.json file: \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, This is required if you plan to fly the SITL mode PX4 with no remote control, just using python scripts, for example. These parameters stop the PX4 from triggering \"failsafe mode on\" every time a move command is finished. You can use the following PX4 command to check these values are set correctly: param show NAV_RCL_ACT param show NAV_DLL_ACT NOTE: Do NOT do this on a real drone as it is too dangerous to fly without these failsafe measures. Manually set parameters # You can also run the following in the PX4 console to set all these parameters: param set LPE_LAT 47.641468 param set LPE_LON -122.140165 param set COM_OBL_ACT 1 param set NAV_RCL_ACT 0 param set NAV_DLL_ACT 0 Using VirtualBox Ubuntu # If you want to run the above posix_sitl in a VirtualBox Ubuntu machine then it will have a different ip address from localhost. So in this case you need to edit the settings file and change the UdpIp and SitlIp to the ip address of your virtual machine set the LocalIpAddress to the address of your host machine running the Unreal engine. Remote Controller # There are several options for flying the simulated drone using a remote control or joystick like xbox gamepad. See remote controllers","title":"PX4 in SITL"},{"location":"px4_sitl/#setting-up-px4-software-in-loop","text":"The PX4 software provides a \"software-in-loop\" simulation (SITL) version of their stack that runs in Linux. If you are on Windows then you must use the Cygwin Toolchain as the Bash On Windows toolchain no longer works for SITL. Note that every time you stop the unreal app you have to restart the px4 app. From your bash terminal follow these steps for Linux and follow all the instructions under NuttX based hardware to install prerequisites. We've also included out own copy of the PX4 build instructions which is a bit more concise about what we need exactly. Get the PX4 source code and build the posix SITL version of PX4: mkdir -p PX4 cd PX4 git clone https://github.com/PX4/Firmware.git cd Firmware git checkout v1.10.1 # recommended version Use following command to build and start PX4 firmware in SITL mode: make px4_sitl_default none_iris If you are using older version v1.8.* use this command instead: make posix_sitl_ekf2 none_iris . You should see a message saying the SITL PX4 app is waiting for the simulator (AirSim) to connect. You will also see information about which ports are configured for mavlink connection to the PX4 app. The default ports have changed recently, so check them closely to make sure AirSim settings are correct. INFO [simulator] Waiting for simulator to connect on TCP port 4560 INFO [init] Mixer: etc/mixers/quad_w.main.mix on /dev/pwm_output0 INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 14570 remote port 14550 INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540 Note: this is also an interactive PX4 console, type help to see the list of commands you can enter here. They are mostly low level PX4 commands, but some of them can be useful for debugging. Now edit AirSim settings file to make sure you have matching UDP and TCP port settings: json { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"UseSerial\": false, \"UseTcp\": true, \"TcpPort\": 4560, \"ControlPort\": 14580, \"params\": { \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, \"COM_OBL_ACT\": 1 } } } } Notice the PX4 [simulator] is using TCP, which is why we need to add: \"UseTcp\": true, . Now run your Unreal AirSim environment and it should connect to SITL PX4 via TCP. You should see a bunch of messages from the SITL PX4 window. Specifically, the following messages tell you that AirSim is connected properly and GPS fusion is stable: INFO [simulator] Simulator connected on UDP port 14560 INFO [mavlink] partner IP: 127.0.0.1 INFO [ecl/EKF] EKF GPS checks passed (WGS-84 origin set) INFO [ecl/EKF] EKF commencing GPS fusion If you do not see these messages then check your port settings. You should also be able to use QGroundControl with SITL mode. Make sure there is no Pixhawk hardware plugged in, otherwise QGroundControl will choose to use that instead. Note that as we don't have a physical board, an RC cannot be connected directly to it. So the alternatives are either use XBox 360 Controller or connect your RC using USB (for example, in case of FrSky Taranis X9D Plus) or using trainer USB cable to your PC. This makes your RC look like a joystick. You will need to do extra set up in QGroundControl to use virtual joystick for RC control. You do not need to do this unless you plan to fly a drone manually in AirSim. Autonomous flight using the Python API does not require RC, see No Remote Control below.","title":"Setting up PX4 Software-in-Loop"},{"location":"px4_sitl/#setting-gps-origin","text":"Notice the above settings are provided in the params section of the settings.json file: \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, PX4 SITL mode needs to be configured to get the home location correct. There is a bug in AirSim that makes it such that flight does not work unless the home location is set to the same coordinates defined in AVehiclePawnBase::HomeLatitude and HomeLongitude. You can also run the following in the SITL PX4 console window to check that these values are set correctly. param show LPE_LAT param show LPE_LON","title":"Setting GPS origin"},{"location":"px4_sitl/#smooth-offboard-transitions","text":"Notice the above setting is provided in the params section of the settings.json file: \"COM_OBL_ACT\": 1 This tells the drone automatically hover after each offboard control command finishes (the default setting is to land). Hovering is a smoother transition between multiple offboard commands. You can check this setting by running the following PX4 console command: param show COM_OBL_ACT","title":"Smooth Offboard Transitions"},{"location":"px4_sitl/#check-the-home-position","text":"If you are using DroneShell to execute commands (arm, takeoff, etc) then you should wait until the Home position is set. You will see the PX4 SITL console output this message: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Now DroneShell 'pos' command should report this position and the commands should be accepted by PX4. If you attempt to takeoff without a home position you will see the message: WARN [commander] Takeoff denied, disarm and re-try After home position is set check the local position reported by 'pos' command : Local position: x=-0.0326988, y=0.00656854, z=5.48506 If the z coordinate is large like this then takeoff might not work as expected. Resetting the SITL and simulation should fix that problem.","title":"Check the Home Position"},{"location":"px4_sitl/#no-remote-control","text":"Notice the above setting is provided in the params section of the settings.json file: \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, This is required if you plan to fly the SITL mode PX4 with no remote control, just using python scripts, for example. These parameters stop the PX4 from triggering \"failsafe mode on\" every time a move command is finished. You can use the following PX4 command to check these values are set correctly: param show NAV_RCL_ACT param show NAV_DLL_ACT NOTE: Do NOT do this on a real drone as it is too dangerous to fly without these failsafe measures.","title":"No Remote Control"},{"location":"px4_sitl/#manually-set-parameters","text":"You can also run the following in the PX4 console to set all these parameters: param set LPE_LAT 47.641468 param set LPE_LON -122.140165 param set COM_OBL_ACT 1 param set NAV_RCL_ACT 0 param set NAV_DLL_ACT 0","title":"Manually set parameters"},{"location":"px4_sitl/#using-virtualbox-ubuntu","text":"If you want to run the above posix_sitl in a VirtualBox Ubuntu machine then it will have a different ip address from localhost. So in this case you need to edit the settings file and change the UdpIp and SitlIp to the ip address of your virtual machine set the LocalIpAddress to the address of your host machine running the Unreal engine.","title":"Using VirtualBox Ubuntu"},{"location":"px4_sitl/#remote-controller","text":"There are several options for flying the simulated drone using a remote control or joystick like xbox gamepad. See remote controllers","title":"Remote Controller"},{"location":"reinforcement_learning/","text":"Reinforcement Learning in AirSim # We below describe how we can implement DQN in AirSim using CNTK. The easiest way is to first install python only CNTK ( instructions ). CNTK provides several demo examples of deep RL . We will modify the DeepQNeuralNetwork.py to work with AirSim. We can utilize most of the classes and methods corresponding to the DQN algorithm. However, there are certain additions we need to make for AirSim. Disclaimer # This is still in active development. What we share below is a framework that can be extended and tweaked to obtain better performance. RL with Car # Source code This example works with AirSimNeighborhood environment available in releases . First, we need to get the images from simulation and transform them appropriately. Below, we show how a depth image can be obtained from the ego camera and transformed to an 84X84 input to the network. (you can use other sensor modalities, and sensor inputs as well \u2013 of course you\u2019ll have to modify the code accordingly). responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthPerspective, True, False)]) current_state = transform_input(responses) We further define the six actions (breaking, straight with throttle, full-left with throttle, full-right with throttle, half-left with throttle, half-right with throttle) that an agent can execute. This is done via the function interpret_action : def interpret_action(action): car_controls.brake = 0 car_controls.throttle = 1 if action == 0: car_controls.throttle = 0 car_controls.brake = 1 elif action == 1: car_controls.steering = 0 elif action == 2: car_controls.steering = 0.5 elif action == 3: car_controls.steering = -0.5 elif action == 4: car_controls.steering = 0.25 else: car_controls.steering = -0.25 return car_controls We then define the reward function in compute_reward as a convex combination of how fast the vehicle is travelling and how much it deviates from the center line. The agent gets a high reward when its moving fast and staying in the center of the lane. def compute_reward(car_state): MAX_SPEED = 300 MIN_SPEED = 10 thresh_dist = 3.5 beta = 3 z = 0 pts = [np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, -1, z])] pd = car_state.position car_pt = np.array(list(pd.values())) dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((car_pt - pts[i]), (car_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) #print(dist) if dist > thresh_dist: reward = -3 else: reward_dist = (math.exp(-beta*dist) - 0.5) reward_speed = (((car_state.speed - MIN_SPEED)/(MAX_SPEED - MIN_SPEED)) - 0.5) reward = reward_dist + reward_speed return reward The function isDone determines if the episode has terminated (e.g. due to collision). We look at the speed of the vehicle and if it is less than a threshold than the episode is considered to be terminated. def isDone(car_state, car_controls, reward): done = 0 if reward < -1: done = 1 if car_controls.brake == 0: if car_state.speed <= 5: done = 1 return done The main loop then sequences through obtaining the image, computing the action to take according to the current policy, getting a reward and so forth. If the episode terminates then we reset the vehicle to the original state via: client.reset() client.enableApiControl(True) client.armDisarm(True) car_control = interpret_action(1) // Reset position and drive straight for one second client.setCarControls(car_control) time.sleep(1) Note that the simulation needs to be up and running before you execute DQNcar.py. The video below shows first few episodes of DQN training. RL with Quadrotor # Source code This example works with AirSimMountainLandscape environment available in releases . We can similarly apply RL for various autonomous flight scenarios with quadrotors. Below is an example on how RL could be used to train quadrotors to follow high tension power lines (e.g. application for energy infrastructure inspection). There are seven actions here that correspond to different directions in which the quadrotor can move in (six directions + one hovering action). def interpret_action(action): if action == 0: quad_offset = (0, 0, 0) elif action == 1: quad_offset = (1, 0, 0) elif action == 2: quad_offset = (0, 1, 0) elif action == 3: quad_offset = (0, 0, 1) elif action == 4: quad_offset = (-1, 0, 0) elif action == 5: quad_offset = (0, -1, 0) elif action == 6: quad_offset = (0, 0, -1) return quad_offset The reward again is a function how how fast the quad travels in conjunction with how far it gets from the known powerlines. def compute_reward(quad_state, quad_vel, collision_info): thresh_dist = 10 beta = 1 z = -10 pts = [np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, 0, z])] quad_pt = np.array(list((quad_state.x_val, quad_state.y_val, quad_state.z_val))) if collision_info.has_collided: reward = -100 else: dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((quad_pt - pts[i]), (quad_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) if dist > thresh_dist: reward = -10 else: reward = 2*(0.5 - math.exp(-beta*dist)) + np.linalg.norm([quad_vel.x_val, quad_vel.y_val, quad_vel.z_val]) return reward We consider an episode to terminate if it drifts too much away from the known power line coordinates. The reset function here flies the quadrotor to the initial starting point: if done: client.moveToZAsync(clearZ, 2).join() client.moveToPositionAsync(initX, initY, clearZ, 2).join() client.moveToPositionAsync(initZ, initY, initZ, 2).join() current_step +=1 Here is the video of first few episodes during the training. Related # Please also see The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter.","title":"Reinforcement Learning"},{"location":"reinforcement_learning/#reinforcement-learning-in-airsim","text":"We below describe how we can implement DQN in AirSim using CNTK. The easiest way is to first install python only CNTK ( instructions ). CNTK provides several demo examples of deep RL . We will modify the DeepQNeuralNetwork.py to work with AirSim. We can utilize most of the classes and methods corresponding to the DQN algorithm. However, there are certain additions we need to make for AirSim.","title":"Reinforcement Learning in AirSim"},{"location":"reinforcement_learning/#disclaimer","text":"This is still in active development. What we share below is a framework that can be extended and tweaked to obtain better performance.","title":"Disclaimer"},{"location":"reinforcement_learning/#rl-with-car","text":"Source code This example works with AirSimNeighborhood environment available in releases . First, we need to get the images from simulation and transform them appropriately. Below, we show how a depth image can be obtained from the ego camera and transformed to an 84X84 input to the network. (you can use other sensor modalities, and sensor inputs as well \u2013 of course you\u2019ll have to modify the code accordingly). responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthPerspective, True, False)]) current_state = transform_input(responses) We further define the six actions (breaking, straight with throttle, full-left with throttle, full-right with throttle, half-left with throttle, half-right with throttle) that an agent can execute. This is done via the function interpret_action : def interpret_action(action): car_controls.brake = 0 car_controls.throttle = 1 if action == 0: car_controls.throttle = 0 car_controls.brake = 1 elif action == 1: car_controls.steering = 0 elif action == 2: car_controls.steering = 0.5 elif action == 3: car_controls.steering = -0.5 elif action == 4: car_controls.steering = 0.25 else: car_controls.steering = -0.25 return car_controls We then define the reward function in compute_reward as a convex combination of how fast the vehicle is travelling and how much it deviates from the center line. The agent gets a high reward when its moving fast and staying in the center of the lane. def compute_reward(car_state): MAX_SPEED = 300 MIN_SPEED = 10 thresh_dist = 3.5 beta = 3 z = 0 pts = [np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, -1, z])] pd = car_state.position car_pt = np.array(list(pd.values())) dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((car_pt - pts[i]), (car_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) #print(dist) if dist > thresh_dist: reward = -3 else: reward_dist = (math.exp(-beta*dist) - 0.5) reward_speed = (((car_state.speed - MIN_SPEED)/(MAX_SPEED - MIN_SPEED)) - 0.5) reward = reward_dist + reward_speed return reward The function isDone determines if the episode has terminated (e.g. due to collision). We look at the speed of the vehicle and if it is less than a threshold than the episode is considered to be terminated. def isDone(car_state, car_controls, reward): done = 0 if reward < -1: done = 1 if car_controls.brake == 0: if car_state.speed <= 5: done = 1 return done The main loop then sequences through obtaining the image, computing the action to take according to the current policy, getting a reward and so forth. If the episode terminates then we reset the vehicle to the original state via: client.reset() client.enableApiControl(True) client.armDisarm(True) car_control = interpret_action(1) // Reset position and drive straight for one second client.setCarControls(car_control) time.sleep(1) Note that the simulation needs to be up and running before you execute DQNcar.py. The video below shows first few episodes of DQN training.","title":"RL with Car"},{"location":"reinforcement_learning/#rl-with-quadrotor","text":"Source code This example works with AirSimMountainLandscape environment available in releases . We can similarly apply RL for various autonomous flight scenarios with quadrotors. Below is an example on how RL could be used to train quadrotors to follow high tension power lines (e.g. application for energy infrastructure inspection). There are seven actions here that correspond to different directions in which the quadrotor can move in (six directions + one hovering action). def interpret_action(action): if action == 0: quad_offset = (0, 0, 0) elif action == 1: quad_offset = (1, 0, 0) elif action == 2: quad_offset = (0, 1, 0) elif action == 3: quad_offset = (0, 0, 1) elif action == 4: quad_offset = (-1, 0, 0) elif action == 5: quad_offset = (0, -1, 0) elif action == 6: quad_offset = (0, 0, -1) return quad_offset The reward again is a function how how fast the quad travels in conjunction with how far it gets from the known powerlines. def compute_reward(quad_state, quad_vel, collision_info): thresh_dist = 10 beta = 1 z = -10 pts = [np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, 0, z])] quad_pt = np.array(list((quad_state.x_val, quad_state.y_val, quad_state.z_val))) if collision_info.has_collided: reward = -100 else: dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((quad_pt - pts[i]), (quad_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) if dist > thresh_dist: reward = -10 else: reward = 2*(0.5 - math.exp(-beta*dist)) + np.linalg.norm([quad_vel.x_val, quad_vel.y_val, quad_vel.z_val]) return reward We consider an episode to terminate if it drifts too much away from the known power line coordinates. The reset function here flies the quadrotor to the initial starting point: if done: client.moveToZAsync(clearZ, 2).join() client.moveToPositionAsync(initX, initY, clearZ, 2).join() client.moveToPositionAsync(initZ, initY, initZ, 2).join() current_step +=1 Here is the video of first few episodes during the training.","title":"RL with Quadrotor"},{"location":"reinforcement_learning/#related","text":"Please also see The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter.","title":"Related"},{"location":"remote_control/","text":"Remote Control # To fly manually, you need remote control or RC. If you don't have one then you can use APIs to fly programmatically or use so-called Computer Vision mode to move around using keyboard. RC Setup for Default Config # By default AirSim uses simple_flight as its flight controller which connects to RC via USB port to your computer. You can either use XBox controller or FrSky Taranis X9D Plus . Note that XBox 360 controller is not precise enough and is not recommended if you wanted more real world experience. See FAQ below if things are not working. Other Devices # AirSim can detect large variety of devices however devices other than above might need extra configuration. In future we will add ability to set this config through settings.json. For now, if things are not working then you might want to try workarounds such as x360ce or change code in SimJoystick.cpp file . Note on FrSky Taranis X9D Plus # FrSky Taranis X9D Plus is real UAV remote control with an advantage that it has USB port so it can be directly connected to PC. You can download AirSim config file and follow this tutorial to import it in your RC. You should then see \"sim\" model in RC with all channels configured properly. Note on Linux # Currently default config on Linux is for using Xbox controller. This means other devices might not work properly. In future we will add ability to configure RC in settings.json but for now you might have to change code in SimJoystick.cpp file to use other devices. RC Setup for PX4 # AirSim supports PX4 flight controller however it requires different setup. There are many remote control options that you can use with quadrotors. We have successfully used FrSky Taranis X9D Plus, FlySky FS-TH9X and Futaba 14SG with AirSim. Following are the high level steps to configure your RC: If you are going to use Hardware-in-Loop mode, you need transmitter for your specific brand of RC and bind it. You can find this information in RC's user guide. For Hardware-in-Loop mode, you connect transmitter to Pixhawk. Usually you can find online doc or YouTube video tutorial on how to do that. Calibrate your RC in QGroundControl . See PX4 RC configuration and Please see this guide for more information. Using XBox 360 USB Gamepad # You can also use an xbox controller in SITL mode, it just won't be as precise as a real RC controller. See xbox controller for details on how to set that up. Using Playstation 3 controller # A Playstation 3 controller is confirmed to work as an AirSim controller. On Windows, an emulator to make it look like an Xbox 360 controller, is required however. Many different solutions are available online, for example x360ce Xbox 360 Controller Emulator . DJI Controller # Nils Tijtgat wrote an excellent blog on how to get the DJI controller working with AirSim . FAQ # I'm using default config and AirSim says my RC is not detected on USB. # This typically happens if you have multiple RCs and or XBox/Playstation gamepads etc connected. In Windows, hit Windows+S key and search for \"Set up USB Game controllers\" (in older versions of Windows try \"joystick\"). This will show you all game controllers connected to your PC. If you don't see yours than Windows haven't detected it and so you need to first solve that issue. If you do see yours but not at the top of the list (i.e. index 0) than you need to tell AirSim because AirSim by default tries to use RC at index 0. To do this, navigate to your ~/Documents/AirSim folder, open up settings.json and add/modify following setting. Below tells AirSim to use RC at index = 2. { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"RC\": { \"RemoteControlID\": 2 } } } } Vehicle seems unstable when using XBox/PS3 controller # Regular gamepads are not very precise and have lot of random noise. Most of the times you may see significant offsets as well (i.e. output is not zero when sticks are at zero). So this behavior is expected. Where is RC calibration in AirSim? # We haven't implemented it yet. This means your RC firmware will need to have a capability to do calibration for now. My RC is not working with PX4 setup. # First you want to make sure your RC is working in QGroundControl . If it doesn't then it will sure not work in AirSim. The PX4 mode is suitable for folks who have at least intermediate level of experience to deal with various issues related to PX4 and we would generally refer you to get help from PX4 forums.","title":"Remote Control"},{"location":"remote_control/#remote-control","text":"To fly manually, you need remote control or RC. If you don't have one then you can use APIs to fly programmatically or use so-called Computer Vision mode to move around using keyboard.","title":"Remote Control"},{"location":"remote_control/#rc-setup-for-default-config","text":"By default AirSim uses simple_flight as its flight controller which connects to RC via USB port to your computer. You can either use XBox controller or FrSky Taranis X9D Plus . Note that XBox 360 controller is not precise enough and is not recommended if you wanted more real world experience. See FAQ below if things are not working.","title":"RC Setup for Default Config"},{"location":"remote_control/#other-devices","text":"AirSim can detect large variety of devices however devices other than above might need extra configuration. In future we will add ability to set this config through settings.json. For now, if things are not working then you might want to try workarounds such as x360ce or change code in SimJoystick.cpp file .","title":"Other Devices"},{"location":"remote_control/#note-on-frsky-taranis-x9d-plus","text":"FrSky Taranis X9D Plus is real UAV remote control with an advantage that it has USB port so it can be directly connected to PC. You can download AirSim config file and follow this tutorial to import it in your RC. You should then see \"sim\" model in RC with all channels configured properly.","title":"Note on FrSky Taranis X9D Plus"},{"location":"remote_control/#note-on-linux","text":"Currently default config on Linux is for using Xbox controller. This means other devices might not work properly. In future we will add ability to configure RC in settings.json but for now you might have to change code in SimJoystick.cpp file to use other devices.","title":"Note on Linux"},{"location":"remote_control/#rc-setup-for-px4","text":"AirSim supports PX4 flight controller however it requires different setup. There are many remote control options that you can use with quadrotors. We have successfully used FrSky Taranis X9D Plus, FlySky FS-TH9X and Futaba 14SG with AirSim. Following are the high level steps to configure your RC: If you are going to use Hardware-in-Loop mode, you need transmitter for your specific brand of RC and bind it. You can find this information in RC's user guide. For Hardware-in-Loop mode, you connect transmitter to Pixhawk. Usually you can find online doc or YouTube video tutorial on how to do that. Calibrate your RC in QGroundControl . See PX4 RC configuration and Please see this guide for more information.","title":"RC Setup for PX4"},{"location":"remote_control/#using-xbox-360-usb-gamepad","text":"You can also use an xbox controller in SITL mode, it just won't be as precise as a real RC controller. See xbox controller for details on how to set that up.","title":"Using XBox 360 USB Gamepad"},{"location":"remote_control/#using-playstation-3-controller","text":"A Playstation 3 controller is confirmed to work as an AirSim controller. On Windows, an emulator to make it look like an Xbox 360 controller, is required however. Many different solutions are available online, for example x360ce Xbox 360 Controller Emulator .","title":"Using Playstation 3 controller"},{"location":"remote_control/#dji-controller","text":"Nils Tijtgat wrote an excellent blog on how to get the DJI controller working with AirSim .","title":"DJI Controller"},{"location":"remote_control/#faq","text":"","title":"FAQ"},{"location":"remote_control/#im-using-default-config-and-airsim-says-my-rc-is-not-detected-on-usb","text":"This typically happens if you have multiple RCs and or XBox/Playstation gamepads etc connected. In Windows, hit Windows+S key and search for \"Set up USB Game controllers\" (in older versions of Windows try \"joystick\"). This will show you all game controllers connected to your PC. If you don't see yours than Windows haven't detected it and so you need to first solve that issue. If you do see yours but not at the top of the list (i.e. index 0) than you need to tell AirSim because AirSim by default tries to use RC at index 0. To do this, navigate to your ~/Documents/AirSim folder, open up settings.json and add/modify following setting. Below tells AirSim to use RC at index = 2. { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"RC\": { \"RemoteControlID\": 2 } } } }","title":"I'm using default config and AirSim says my RC is not detected on USB."},{"location":"remote_control/#vehicle-seems-unstable-when-using-xboxps3-controller","text":"Regular gamepads are not very precise and have lot of random noise. Most of the times you may see significant offsets as well (i.e. output is not zero when sticks are at zero). So this behavior is expected.","title":"Vehicle seems unstable when using XBox/PS3 controller"},{"location":"remote_control/#where-is-rc-calibration-in-airsim","text":"We haven't implemented it yet. This means your RC firmware will need to have a capability to do calibration for now.","title":"Where is RC calibration in AirSim?"},{"location":"remote_control/#my-rc-is-not-working-with-px4-setup","text":"First you want to make sure your RC is working in QGroundControl . If it doesn't then it will sure not work in AirSim. The PX4 mode is suitable for folks who have at least intermediate level of experience to deal with various issues related to PX4 and we would generally refer you to get help from PX4 forums.","title":"My RC is not working with PX4 setup."},{"location":"retexturing/","text":"Runtime Texture Swapping # How to Make An Actor Retexturable # To be made texture-swappable, an actor must derive from the parent class TextureShuffleActor. The parent class can be set via the settings tab in the actor's blueprint. After setting the parent class to TextureShuffActor, the object gains the member DynamicMaterial. DynamicMaterial needs to be set--on all actor instances in the scene--to TextureSwappableMaterial. Warning: Statically setting the Dynamic Material in the blueprint class may cause rendering errors. It seems to work better to set it on all the actor instances in the scene, using the details panel. How to Define the Set(s) of Textures to Choose From # Typically, certain subsets of actors will share a set of texture options with each other. (e.g. walls that are part of the same building) It's easy to set up these groupings by using Unreal Engine's group editing functionality. Select all the instances that should have the same texture selection, and add the textures to all of them simultaneously via the Details panel. Use the same technique to add descriptive tags to groups of actors, which will be used to address them in the API. It's ideal to work from larger groupings to smaller groupings, simply deselecting actors to narrow down the grouping as you go, and applying any individual actor properties last. How to Swap Textures from the API # The following API is available in C++ and python. (C++ shown) std::vector simSwapTextures(const std::string& tags, int tex_id); The string of \",\" or \", \" delimited tags identifies on which actors to perform the swap. The tex_id indexes the array of textures assigned to each actor undergoing a swap. The function will return the list of objects which matched the provided tags and had the texture swap perfomed. If tex_id is out-of-bounds for some object's texture set, it will be taken modulo the number of textures that were available. Demo (Python): import airsim import time c = airsim.client.MultirotorClient() print(c.simSwapTextures(\"furniture\", 0)) time.sleep(2) print(c.simSwapTextures(\"chair\", 1)) time.sleep(2) print(c.simSwapTextures(\"table\", 1)) time.sleep(2) print(c.simSwapTextures(\"chair, right\", 0)) Results: ['RetexturableChair', 'RetexturableChair2', 'RetexturableTable'] ['RetexturableChair', 'RetexturableChair2'] ['RetexturableTable'] ['RetexturableChair2'] Note that in this example, different textures were chosen on each actor for the same index value.","title":"Domain Randomization"},{"location":"retexturing/#runtime-texture-swapping","text":"","title":"Runtime Texture Swapping"},{"location":"retexturing/#how-to-make-an-actor-retexturable","text":"To be made texture-swappable, an actor must derive from the parent class TextureShuffleActor. The parent class can be set via the settings tab in the actor's blueprint. After setting the parent class to TextureShuffActor, the object gains the member DynamicMaterial. DynamicMaterial needs to be set--on all actor instances in the scene--to TextureSwappableMaterial. Warning: Statically setting the Dynamic Material in the blueprint class may cause rendering errors. It seems to work better to set it on all the actor instances in the scene, using the details panel.","title":"How to Make An Actor Retexturable"},{"location":"retexturing/#how-to-define-the-sets-of-textures-to-choose-from","text":"Typically, certain subsets of actors will share a set of texture options with each other. (e.g. walls that are part of the same building) It's easy to set up these groupings by using Unreal Engine's group editing functionality. Select all the instances that should have the same texture selection, and add the textures to all of them simultaneously via the Details panel. Use the same technique to add descriptive tags to groups of actors, which will be used to address them in the API. It's ideal to work from larger groupings to smaller groupings, simply deselecting actors to narrow down the grouping as you go, and applying any individual actor properties last.","title":"How to Define the Set(s) of Textures to Choose From"},{"location":"retexturing/#how-to-swap-textures-from-the-api","text":"The following API is available in C++ and python. (C++ shown) std::vector simSwapTextures(const std::string& tags, int tex_id); The string of \",\" or \", \" delimited tags identifies on which actors to perform the swap. The tex_id indexes the array of textures assigned to each actor undergoing a swap. The function will return the list of objects which matched the provided tags and had the texture swap perfomed. If tex_id is out-of-bounds for some object's texture set, it will be taken modulo the number of textures that were available. Demo (Python): import airsim import time c = airsim.client.MultirotorClient() print(c.simSwapTextures(\"furniture\", 0)) time.sleep(2) print(c.simSwapTextures(\"chair\", 1)) time.sleep(2) print(c.simSwapTextures(\"table\", 1)) time.sleep(2) print(c.simSwapTextures(\"chair, right\", 0)) Results: ['RetexturableChair', 'RetexturableChair2', 'RetexturableTable'] ['RetexturableChair', 'RetexturableChair2'] ['RetexturableTable'] ['RetexturableChair2'] Note that in this example, different textures were chosen on each actor for the same index value.","title":"How to Swap Textures from the API"},{"location":"sensors/","text":"Sensors in AirSim # AirSim currently supports the following sensors. Each sensor is associated with a integer enum specifying its sensor type. Camera Barometer = 1 Imu = 2 Gps = 3 Magnetometer = 4 Distance Sensor = 5 Lidar = 6 Note : Cameras are configured differently than the other sensors and do not have an enum associated with them. Look at general settings and image API for camera config and API. Default sensors # If no sensors are specified in the settings.json , the the following sensors are enabled by default based on the sim mode. Multirotor # Imu Magnetometer Gps Barometer Car # Gps ComputerVision # None Behind the scenes, createDefaultSensorSettings method in AirSimSettings.hpp sets up the above sensors with their default parameters, depending on the sim mode specified in the settings.json file. Configuring the default sensor list # The default sensor list can be configured in settings json: \"DefaultSensors\": { \"Barometer\": { \"SensorType\": 1, \"Enabled\" : true }, \"Imu\": { \"SensorType\": 2, \"Enabled\" : true }, \"Gps\": { \"SensorType\": 3, \"Enabled\" : true }, \"Magnetometer\": { \"SensorType\": 4, \"Enabled\" : true }, \"Distance\": { \"SensorType\": 5, \"Enabled\" : true }, \"Lidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000 } }, Configuring vehicle-specific sensor list # If a vehicle provides its sensor list, it must provide the whole list. Selective add/remove/update of the default sensor list is NOT supported. A vehicle specific sensor list can be specified in the vehicle settings part of the json. e.g., \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"AutoCreate\": true, ... \"Sensors\": { \"MyLidar1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true }, \"MyLidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true } } } } Sensor specific settings # Each sensor-type has its own set of settings as well. Please see lidar for example of Lidar specific settings. Distance Sensor # By default, Distance Sensor points to the front of the vehicle. It can be pointed in any direction by modifying the settings Configurable Parameters - Parameter Description X Y Z Position of the sensor relative to the vehicle (in NED, in meters) (Default (0,0,0)-Multirotor, (0,0,-1)-Car) Yaw Pitch Roll Orientation of the sensor relative to the vehicle (degrees) (Default (0,0,0)) MinDistance Minimum distance measured by distance sensor (metres, only used to fill Mavlink message for PX4) (Default 0.2m) MaxDistance Maximum distance measured by distance sensor (metres) (Default 40.0m) For example, to make the sensor point towards the ground (for altitude measurement similar to barometer), the orientation can be modified as follows - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, \"Yaw\": 0, \"Pitch\": -90, \"Roll\": 0 } Note: For Cars, the sensor is placed 1 meter above the vehicle center by default. This is required since otherwise the sensor gives strange data due it being inside the vehicle. This doesn't affect the sensor values say when measuring the distance between 2 cars. See PythonClient/car/distance_sensor_multi.py for an example usage. Server side visualization for debugging # Be default, the points hit by distance sensor are not drawn on the viewport. To enable the drawing of hit points on the viewport, please enable setting DrawDebugPoints via settings json. E.g. - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, ... \"DrawDebugPoints\": true } Sensor APIs # Jump straight to hello_drone.py or hello_drone.cpp for example usage, or see follow below for the full API. Barometer C++ cpp msr::airlib::BarometerBase::Output getBarometerData(const std::string& barometer_name, const std::string& vehicle_name); Python python barometer_data = client.getBarometerData(barometer_name = \"\", vehicle_name = \"\") IMU C++ cpp msr::airlib::ImuBase::Output getImuData(const std::string& imu_name = \"\", const std::string& vehicle_name = \"\"); Python python imu_data = client.getImuData(imu_name = \"\", vehicle_name = \"\") GPS C++ cpp msr::airlib::GpsBase::Output getGpsData(const std::string& gps_name = \"\", const std::string& vehicle_name = \"\"); Python python gps_data = client.getGpsData(gps_name = \"\", vehicle_name = \"\") Magnetometer C++ cpp msr::airlib::MagnetometerBase::Output getMagnetometerData(const std::string& magnetometer_name = \"\", const std::string& vehicle_name = \"\"); Python python magnetometer_data = client.getMagnetometerData(magnetometer_name = \"\", vehicle_name = \"\") Distance sensor C++ cpp msr::airlib::DistanceSensorData getDistanceSensorData(const std::string& distance_sensor_name = \"\", const std::string& vehicle_name = \"\"); Python python distance_sensor_data = client.getDistanceSensorData(distance_sensor_name = \"\", vehicle_name = \"\") Lidar See lidar for Lidar API.","title":"Sensors"},{"location":"sensors/#sensors-in-airsim","text":"AirSim currently supports the following sensors. Each sensor is associated with a integer enum specifying its sensor type. Camera Barometer = 1 Imu = 2 Gps = 3 Magnetometer = 4 Distance Sensor = 5 Lidar = 6 Note : Cameras are configured differently than the other sensors and do not have an enum associated with them. Look at general settings and image API for camera config and API.","title":"Sensors in AirSim"},{"location":"sensors/#default-sensors","text":"If no sensors are specified in the settings.json , the the following sensors are enabled by default based on the sim mode.","title":"Default sensors"},{"location":"sensors/#multirotor","text":"Imu Magnetometer Gps Barometer","title":"Multirotor"},{"location":"sensors/#car","text":"Gps","title":"Car"},{"location":"sensors/#computervision","text":"None Behind the scenes, createDefaultSensorSettings method in AirSimSettings.hpp sets up the above sensors with their default parameters, depending on the sim mode specified in the settings.json file.","title":"ComputerVision"},{"location":"sensors/#configuring-the-default-sensor-list","text":"The default sensor list can be configured in settings json: \"DefaultSensors\": { \"Barometer\": { \"SensorType\": 1, \"Enabled\" : true }, \"Imu\": { \"SensorType\": 2, \"Enabled\" : true }, \"Gps\": { \"SensorType\": 3, \"Enabled\" : true }, \"Magnetometer\": { \"SensorType\": 4, \"Enabled\" : true }, \"Distance\": { \"SensorType\": 5, \"Enabled\" : true }, \"Lidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000 } },","title":"Configuring the default sensor list"},{"location":"sensors/#configuring-vehicle-specific-sensor-list","text":"If a vehicle provides its sensor list, it must provide the whole list. Selective add/remove/update of the default sensor list is NOT supported. A vehicle specific sensor list can be specified in the vehicle settings part of the json. e.g., \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"AutoCreate\": true, ... \"Sensors\": { \"MyLidar1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true }, \"MyLidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true } } } }","title":"Configuring vehicle-specific sensor list"},{"location":"sensors/#sensor-specific-settings","text":"Each sensor-type has its own set of settings as well. Please see lidar for example of Lidar specific settings.","title":"Sensor specific settings"},{"location":"sensors/#distance-sensor","text":"By default, Distance Sensor points to the front of the vehicle. It can be pointed in any direction by modifying the settings Configurable Parameters - Parameter Description X Y Z Position of the sensor relative to the vehicle (in NED, in meters) (Default (0,0,0)-Multirotor, (0,0,-1)-Car) Yaw Pitch Roll Orientation of the sensor relative to the vehicle (degrees) (Default (0,0,0)) MinDistance Minimum distance measured by distance sensor (metres, only used to fill Mavlink message for PX4) (Default 0.2m) MaxDistance Maximum distance measured by distance sensor (metres) (Default 40.0m) For example, to make the sensor point towards the ground (for altitude measurement similar to barometer), the orientation can be modified as follows - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, \"Yaw\": 0, \"Pitch\": -90, \"Roll\": 0 } Note: For Cars, the sensor is placed 1 meter above the vehicle center by default. This is required since otherwise the sensor gives strange data due it being inside the vehicle. This doesn't affect the sensor values say when measuring the distance between 2 cars. See PythonClient/car/distance_sensor_multi.py for an example usage.","title":"Distance Sensor"},{"location":"sensors/#server-side-visualization-for-debugging","text":"Be default, the points hit by distance sensor are not drawn on the viewport. To enable the drawing of hit points on the viewport, please enable setting DrawDebugPoints via settings json. E.g. - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, ... \"DrawDebugPoints\": true }","title":"Server side visualization for debugging"},{"location":"sensors/#sensor-apis","text":"Jump straight to hello_drone.py or hello_drone.cpp for example usage, or see follow below for the full API. Barometer C++ cpp msr::airlib::BarometerBase::Output getBarometerData(const std::string& barometer_name, const std::string& vehicle_name); Python python barometer_data = client.getBarometerData(barometer_name = \"\", vehicle_name = \"\") IMU C++ cpp msr::airlib::ImuBase::Output getImuData(const std::string& imu_name = \"\", const std::string& vehicle_name = \"\"); Python python imu_data = client.getImuData(imu_name = \"\", vehicle_name = \"\") GPS C++ cpp msr::airlib::GpsBase::Output getGpsData(const std::string& gps_name = \"\", const std::string& vehicle_name = \"\"); Python python gps_data = client.getGpsData(gps_name = \"\", vehicle_name = \"\") Magnetometer C++ cpp msr::airlib::MagnetometerBase::Output getMagnetometerData(const std::string& magnetometer_name = \"\", const std::string& vehicle_name = \"\"); Python python magnetometer_data = client.getMagnetometerData(magnetometer_name = \"\", vehicle_name = \"\") Distance sensor C++ cpp msr::airlib::DistanceSensorData getDistanceSensorData(const std::string& distance_sensor_name = \"\", const std::string& vehicle_name = \"\"); Python python distance_sensor_data = client.getDistanceSensorData(distance_sensor_name = \"\", vehicle_name = \"\") Lidar See lidar for Lidar API.","title":"Sensor APIs"},{"location":"settings/","text":"AirSim Settings # Where are Settings Stored? # AirSim is searching for the settings definition in 4 different ways in the following order. The first match will be used: Looking at the (absolute) path specified by the --settings command line argument. For example, in Windows: AirSim.exe --settings 'C:\\path\\to\\settings.json' In Linux ./Blocks.sh --settings '/home/$USER/path/to/settings.json' Looking for a json document passed as a command line argument by the --settings argument. For example, in Windows: AirSim.exe --settings '{\"foo\" : \"bar\"}' In Linux ./Blocks.sh --settings '{\"foo\" : \"bar\"}' Looking in the folder of the executable for a file called settings.json . Looking in the users home folder for a file called settings.json . The AirSim subfolder is located at Documents\\AirSim on Windows and ~/Documents/AirSim on Linux systems. The file is in usual json format . On first startup AirSim would create settings.json file with no settings at the users home folder. To avoid problems, always use ASCII format to save json file. How to Chose Between Car and Multirotor? # The default is to use multirotor. To use car simple set \"SimMode\": \"Car\" like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } To choose multirotor, set \"SimMode\": \"Multirotor\" . If you want to prompt user to select vehicle type then use \"SimMode\": \"\" . Available Settings and Their Defaults # Below are complete list of settings available along with their default values. If any of the settings is missing from json file, then default value is used. Some default values are simply specified as \"\" which means actual value may be chosen based on the vehicle you are using. For example, ViewMode setting has default value \"\" which translates to \"FlyWithMe\" for drones and \"SpringArmChase\" for cars. WARNING: Do not copy paste all of below in your settings.json. We strongly recommend adding only those settings that you don't want default values. Only required element is \"SettingsVersion\" . { \"SimMode\": \"\", \"ClockType\": \"\", \"ClockSpeed\": 1, \"LocalHostIp\": \"127.0.0.1\", \"RecordUIVisible\": true, \"LogMessagesVisible\": true, \"ViewMode\": \"\", \"RpcEnabled\": true, \"EngineSound\": true, \"PhysicsEngineName\": \"\", \"SpeedUnitFactor\": 1.0, \"SpeedUnitLabel\": \"m/s\", \"Recording\": { \"RecordOnMove\": false, \"RecordInterval\": 0.05, \"Cameras\": [ { \"CameraName\": \"0\", \"ImageType\": 0, \"PixelsAsFloat\": false, \"Compress\": true } ] }, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"AutoExposureBias\": 0, \"AutoExposureMaxBrightness\": 0.64, \"AutoExposureMinBrightness\": 0.03, \"MotionBlurAmount\": 0, \"TargetGamma\": 1.0, \"ProjectionMode\": \"\", \"OrthoWidth\": 5.12 } ], \"NoiseSettings\": [ { \"Enabled\": false, \"ImageType\": 0, \"RandContrib\": 0.2, \"RandSpeed\": 100000.0, \"RandSize\": 500.0, \"RandDensity\": 2, \"HorzWaveContrib\":0.03, \"HorzWaveStrength\": 0.08, \"HorzWaveVertSize\": 1.0, \"HorzWaveScreenSize\": 1.0, \"HorzNoiseLinesContrib\": 1.0, \"HorzNoiseLinesDensityY\": 0.01, \"HorzNoiseLinesDensityXY\": 0.5, \"HorzDistortionContrib\": 1.0, \"HorzDistortionStrength\": 0.002 } ], \"Gimbal\": { \"Stabilization\": 0, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"OriginGeopoint\": { \"Latitude\": 47.641468, \"Longitude\": -122.140165, \"Altitude\": 122 }, \"TimeOfDay\": { \"Enabled\": false, \"StartDateTime\": \"\", \"CelestialClockSpeed\": 1, \"StartDateTimeDst\": false, \"UpdateIntervalSecs\": 60 }, \"SubWindows\": [ {\"WindowID\": 0, \"CameraName\": \"0\", \"ImageType\": 3, \"Visible\": false}, {\"WindowID\": 1, \"CameraName\": \"0\", \"ImageType\": 5, \"Visible\": false}, {\"WindowID\": 2, \"CameraName\": \"0\", \"ImageType\": 0, \"Visible\": false} ], \"SegmentationSettings\": { \"InitMethod\": \"\", \"MeshNamingMethod\": \"\", \"OverrideExisting\": false }, \"PawnPaths\": { \"BareboneCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/Vehicle/VehicleAdvPawn.VehicleAdvPawn_C'\"}, \"DefaultCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/SUV/SuvCarPawn.SuvCarPawn_C'\"}, \"DefaultQuadrotor\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_FlyingPawn.BP_FlyingPawn_C'\"}, \"DefaultComputerVision\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_ComputerVisionPawn.BP_ComputerVisionPawn_C'\"} }, \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Armed\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": false }, \"Cameras\": { //same elements as CameraDefaults above, key as name }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"PhysXCar\": { \"VehicleType\": \"PhysXCar\", \"DefaultVehicleState\": \"\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"RC\": { \"RemoteControlID\": -1 }, \"Cameras\": { \"MyCamera1\": { //same elements as elements inside CameraDefaults above }, \"MyCamera2\": { //same elements as elements inside CameraDefaults above }, }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } } } SimMode # SimMode determines which simulation mode will be used. Below are currently supported values: - \"\" : prompt user to select vehicle type multirotor or car - \"Multirotor\" : Use multirotor simulation - \"Car\" : Use car simulation - \"ComputerVision\" : Use only camera, no vehicle or physics ViewMode # The ViewMode determines which camera to use as default and how camera will follow the vehicle. For multirotors, the default ViewMode is \"FlyWithMe\" while for cars the default ViewMode is \"SpringArmChase\" . FlyWithMe : Chase the vehicle from behind with 6 degrees of freedom GroundObserver : Chase the vehicle from 6' above the ground but with full freedom in XY plane. Fpv : View the scene from front camera of vehicle Manual : Don't move camera automatically. Use arrow keys and ASWD keys for move camera manually. SpringArmChase : Chase the vehicle with camera mounted on (invisible) arm that is attached to the vehicle via spring (so it has some latency in movement). NoDisplay : This will freeze rendering for main screen however rendering for subwindows, recording and APIs remain active. This mode is useful to save resources in \"headless\" mode where you are only interested in getting images and don't care about what gets rendered on main screen. This may also improve FPS for recording images. TimeOfDay # This setting controls the position of Sun in the environment. By default Enabled is false which means Sun's position is left at whatever was the default in the environment and it doesn't change over the time. If Enabled is true then Sun position is computed using longitude, latitude and altitude specified in OriginGeopoint section for the date specified in StartDateTime in the string format as %Y-%m-%d %H:%M:%S , for example, 2018-02-12 15:20:00 . If this string is empty then current date and time is used. If StartDateTimeDst is true then we adjust for day light savings time. The Sun's position is then continuously updated at the interval specified in UpdateIntervalSecs . In some cases, it might be desirable to have celestial clock run faster or slower than simulation clock. This can be specified using CelestialClockSpeed , for example, value 100 means for every 1 second of simulation clock, Sun's position is advanced by 100 seconds so Sun will move in sky much faster. Also see Time of Day API . OriginGeopoint # This setting specifies the latitude, longitude and altitude of the Player Start component placed in the Unreal environment. The vehicle's home point is computed using this transformation. Note that all coordinates exposed via APIs are using NED system in SI units which means each vehicle starts at (0, 0, 0) in NED system. Time of Day settings are computed for geographical coordinates specified in OriginGeopoint . SubWindows # This setting determines what is shown in each of 3 subwindows which are visible when you press 0 key. The WindowsID can be 0 to 2, CameraName is any available camera on the vehicle. ImageType integer value determines what kind of image gets shown according to ImageType enum . For example, for car vehicles below shows driver view, front bumper view and rear view as scene, depth and surface normals respectively. \"SubWindows\": [ {\"WindowID\": 0, \"ImageType\": 0, \"CameraName\": \"3\", \"Visible\": true}, {\"WindowID\": 1, \"ImageType\": 3, \"CameraName\": \"0\", \"Visible\": true}, {\"WindowID\": 2, \"ImageType\": 6, \"CameraName\": \"4\", \"Visible\": true} ] Recording # The recording feature allows you to record data such as position, orientation, velocity along with the captured image at specified intervals. You can start recording by pressing red Record button on lower right or the R key. The data is stored in the Documents\\AirSim folder, in a time stamped subfolder for each recording session, as tab separated file. RecordInterval : specifies minimal interval in seconds between capturing two images. RecordOnMove : specifies that do not record frame if there was vehicle's position or orientation hasn't changed. Cameras : this element controls which cameras are used to capture images. By default scene image from camera 0 is recorded as compressed png format. This setting is json array so you can specify multiple cameras to capture images, each with potentially different image types . When PixelsAsFloat is true, image is saved as pfm file instead of png file. ClockSpeed # This setting allows you to set the speed of simulation clock with respect to wall clock. For example, value of 5.0 would mean simulation clock has 5 seconds elapsed when wall clock has 1 second elapsed (i.e. simulation is running faster). The value of 0.1 means that simulation clock is 10X slower than wall clock. The value of 1 means simulation is running in real time. It is important to realize that quality of simulation may decrease as the simulation clock runs faster. You might see artifacts like object moving past obstacles because collision is not detected. However slowing down simulation clock (i.e. values < 1.0) generally improves the quality of simulation. Segmentation Settings # The InitMethod determines how object IDs are initialized at startup to generate segmentation . The value \"\" or \"CommonObjectsRandomIDs\" (default) means assign random IDs to each object at startup. This will generate segmentation view with random colors assign to each object. The value \"None\" means don't initialize object IDs. This will cause segmentation view to have single solid colors. This mode is useful if you plan to set up object IDs using APIs and it can save lot of delay at startup for large environments like CityEnviron. If OverrideExisting is false then initialization does not alter non-zero object IDs already assigned otherwise it does. If MeshNamingMethod is \"\" or \"OwnerName\" then we use mesh's owner name to generate random hash as object IDs. If it is \"StaticMeshName\" then we use static mesh's name to generate random hash as object IDs. Note that it is not possible to tell individual instances of the same static mesh apart this way, but the names are often more intuitive. Camera Settings # The CameraDefaults element at root level specifies defaults used for all cameras. These defaults can be overridden for individual camera in Cameras element inside Vehicles as described later. Note on ImageType element # The ImageType element in JSON array determines which image type that settings applies to. The valid values are described in ImageType section . In addition, we also support special value ImageType: -1 to apply the settings to external camera (i.e. what you are looking at on the screen). For example, CaptureSettings element is json array so you can add settings for multiple image types easily. CaptureSettings # The CaptureSettings determines how different image types such as scene, depth, disparity, surface normals and segmentation views are rendered. The Width, Height and FOV settings should be self explanatory. The AutoExposureSpeed decides how fast eye adaptation works. We set to generally high value such as 100 to avoid artifacts in image capture. Similarly we set MotionBlurAmount to 0 by default to avoid artifacts in ground truth images. The ProjectionMode decides the projection used by the capture camera and can take value \"perspective\" (default) or \"orthographic\". If projection mode is \"orthographic\" then OrthoWidth determines width of projected area captured in meters. For explanation of other settings, please see this article . NoiseSettings # The NoiseSettings allows to add noise to the specified image type with a goal of simulating camera sensor noise, interference and other artifacts. By default no noise is added, i.e., Enabled: false . If you set Enabled: true then following different types of noise and interference artifacts are enabled, each can be further tuned using setting. The noise effects are implemented as shader created as post processing material in Unreal Engine called CameraSensorNoise . Demo of camera noise and interference simulation: Random noise # This adds random noise blobs with following parameters. * RandContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * RandSpeed : This determines how fast noise fluctuates, 1 means no fluctuation and higher values like 1E6 means full fluctuation. * RandSize : This determines how coarse noise is, 1 means every pixel has its own noise while higher value means more than 1 pixels share same noise value. * RandDensity : This determines how many pixels out of total will have noise, 1 means all pixels while higher value means lesser number of pixels (exponentially). Horizontal bump distortion # This adds horizontal bumps / flickering / ghosting effect. * HorzWaveContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzWaveStrength : This determines overall strength of the effect. * HorzWaveVertSize : This determines how many vertical pixels would be effected by the effect. * HorzWaveScreenSize : This determines how much of the screen is effected by the effect. Horizontal noise lines # This adds regions of noise on horizontal lines. * HorzNoiseLinesContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzNoiseLinesDensityY : This determines how many pixels in horizontal line gets affected. * HorzNoiseLinesDensityXY : This determines how many lines on screen gets affected. Horizontal line distortion # This adds fluctuations on horizontal line. * HorzDistortionContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzDistortionStrength : This determines how large is the distortion. Gimbal # The Gimbal element allows to freeze camera orientation for pitch, roll and/or yaw. This setting is ignored unless ImageType is -1. The Stabilization is defaulted to 0 meaning no gimbal i.e. camera orientation changes with body orientation on all axis. The value of 1 means full stabilization. The value between 0 to 1 acts as a weight for fixed angles specified (in degrees, in world-frame) in Pitch , Roll and Yaw elements and orientation of the vehicle body. When any of the angles is omitted from json or set to NaN, that angle is not stabilized (i.e. it moves along with vehicle body). Vehicles Settings # Each simulation mode will go through the list of vehicles specified in this setting and create the ones that has \"AutoCreate\": true . Each vehicle specified in this setting has key which becomes the name of the vehicle. If \"Vehicles\" element is missing then this list is populated with default car named \"PhysXCar\" and default multirotor named \"SimpleFlight\". Common Vehicle Setting # VehicleType : This could be either PhysXCar , SimpleFlight , PX4Multirotor or ComputerVision . There is no default value therefore this element must be specified. PawnPath : This allows to override the pawn blueprint to use for the vehicle. For example, you may create new pawn blueprint derived from ACarPawn for a warehouse robot in your own project outside the AirSim code and then specify its path here. See also PawnPaths . DefaultVehicleState : Possible value for multirotors is Armed or Disarmed . AutoCreate : If true then this vehicle would be spawned (if supported by selected sim mode). RC : This sub-element allows to specify which remote controller to use for vehicle using RemoteControlID . The value of -1 means use keyboard (not supported yet for multirotors). The value >= 0 specifies one of many remote controllers connected to the system. The list of available RCs can be seen in Game Controllers panel in Windows, for example. X, Y, Z, Yaw, Roll, Pitch : These elements allows you to specify the initial position and orientation of the vehicle. Position is in NED coordinates in SI units with origin set to Player Start location in Unreal environment. The orientation is specified in degrees. IsFpvVehicle : This setting allows to specify which vehicle camera will follow and the view that will be shown when ViewMode is set to Fpv. By default, AirSim selects the first vehicle in settings as FPV vehicle. Cameras : This element specifies camera settings for vehicle. The key in this element is name of the available camera and the value is same as CameraDefaults as described above. For example, to change FOV for the front center camera to 120 degrees, you can use this for Vehicles setting: \"Vehicles\": { \"FishEyeDrone\": { \"VehicleType\": \"SimpleFlight\", \"Cameras\": { \"front-center\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"FOV_Degrees\": 120 } ] } } } } Using PX4 # By default we use simple_flight so you don't have to do separate HITL or SITL setups. We also support \"PX4\" for advanced users. To use PX4 with AirSim, you can use the following for Vehicles setting: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", } } Additional PX4 Settings # The defaults for PX4 is to enable hardware-in-loop setup. There are various other settings available for PX4 as follows with their default values: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"ControlIp\": \"127.0.0.1\", \"ControlPort\": 14580, \"LogViewerHostIp\": \"127.0.0.1\", \"LogViewerPort\": 14388, \"OffboardCompID\": 1, \"OffboardSysID\": 134, \"QgcHostIp\": \"127.0.0.1\", \"QgcPort\": 14550, \"SerialBaudRate\": 115200, \"SerialPort\": \"*\", \"SimCompID\": 42, \"SimSysID\": 142, \"TcpPort\": 4560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14560, \"UseSerial\": true, \"UseTcp\": false, \"VehicleCompID\": 1, \"VehicleSysID\": 135, \"Model\": \"Generic\", \"LocalHostIp\": \"127.0.0.1\" } } These settings define the MavLink SystemId and ComponentId for the Simulator (SimSysID, SimCompID), and for the vehicle (VehicleSysID, VehicleCompID) and the node that allows remote control of the drone from another app this is called the offboard node (OffboardSysID, OffboardCompID). If you want the simulator to also talk to your ground control app (like QGroundControl) you can also set the UDP address for that in case you want to run that on a different machine (QgcHostIp, QgcPort). The default is local host so QGroundControl should \"just work\" if it is running on the same machine. You can connect the simulator to the LogViewer app, provided in this repo, by setting the UDP address for that (LogViewerHostIp, LogViewerPort). And for each flying drone added to the simulator there is a named block of additional settings. In the above you see the default name \"PX4\". You can change this name from the Unreal Editor when you add a new BP_FlyingPawn asset. You will see these properties grouped under the category \"MavLink\". The MavLink node for this pawn can be remote over UDP or it can be connected to a local serial port. If serial then set UseSerial to true, otherwise set UseSerial to false. For serial connections you also need to set the appropriate SerialBaudRate. The default of 115200 works with Pixhawk version 2 over USB. When communicating with the PX4 drone over serial port both the HIL_ messages and vehicle control messages share the same serial port. When communicating over UDP or TCP PX4 requires two separate channels. If UseTcp is false, then UdpIp, UdpPort are used to send HIL_ messages, otherwise the TcpPort is used. TCP support in PX4 was added in 1.9.2 with the lockstep feature because the guarantee of message delivery that TCP provides is required for the proper functioning of lockstep. AirSim becomes a TCP server in that case, and waits for a connection from the PX4 app. The second channel for controlling the vehicle is defined by (ControlIp, ControlPort) and is always a UDP channel. Other Settings # EngineSound # To turn off the engine sound use setting \"EngineSound\": false . Currently this setting applies only to car. PawnPaths # This allows you to specify your own vehicle pawn blueprints, for example, you can replace the default car in AirSim with your own car. Your vehicle BP can reside in Content folder of your own Unreal project (i.e. outside of AirSim plugin folder). For example, if you have a car BP located in file Content\\MyCar\\MySedanBP.uasset in your project then you can set \"DefaultCar\": {\"PawnBP\":\"Class'/Game/MyCar/MySedanBP.MySedanBP_C'\"} . The XYZ.XYZ_C is a special notation required to specify class for BP XYZ . Please note that your BP must be derived from CarPawn class. By default this is not the case but you can re-parent the BP using the \"Class Settings\" button in toolbar in UE editor after you open the BP and then choosing \"Car Pawn\" for Parent Class settings in Class Options. It is also a good idea to disable \"Auto Possess Player\" and \"Auto Possess AI\" as well as set AI Controller Class to None in BP details. Please make sure your asset is included for cooking in packaging options if you are creating binary. PhysicsEngineName # For cars, we support only PhysX for now (regardless of value in this setting). For multirotors, we support \"FastPhysicsEngine\" only. LocalHostIp Setting # Now when connecting to remote machines you may need to pick a specific Ethernet adapter to reach those machines, for example, it might be over Ethernet or over Wi-Fi, or some other special virtual adapter or a VPN. Your PC may have multiple networks, and those networks might not be allowed to talk to each other, in which case the UDP messages from one network will not get through to the others. So the LocalHostIp allows you to configure how you are reaching those machines. The default of 127.0.0.1 is not able to reach external machines, this default is only used when everything you are talking to is contained on a single PC. SpeedUnitFactor # Unit conversion factor for speed related to m/s , default is 1. Used in conjunction with SpeedUnitLabel. This may be only used for display purposes for example on-display speed when car is being driven. For example, to get speed in miles/hr use factor 2.23694. SpeedUnitLabel # Unit label for speed, default is m/s . Used in conjunction with SpeedUnitFactor.","title":"Settings"},{"location":"settings/#airsim-settings","text":"","title":"AirSim Settings"},{"location":"settings/#where-are-settings-stored","text":"AirSim is searching for the settings definition in 4 different ways in the following order. The first match will be used: Looking at the (absolute) path specified by the --settings command line argument. For example, in Windows: AirSim.exe --settings 'C:\\path\\to\\settings.json' In Linux ./Blocks.sh --settings '/home/$USER/path/to/settings.json' Looking for a json document passed as a command line argument by the --settings argument. For example, in Windows: AirSim.exe --settings '{\"foo\" : \"bar\"}' In Linux ./Blocks.sh --settings '{\"foo\" : \"bar\"}' Looking in the folder of the executable for a file called settings.json . Looking in the users home folder for a file called settings.json . The AirSim subfolder is located at Documents\\AirSim on Windows and ~/Documents/AirSim on Linux systems. The file is in usual json format . On first startup AirSim would create settings.json file with no settings at the users home folder. To avoid problems, always use ASCII format to save json file.","title":"Where are Settings Stored?"},{"location":"settings/#how-to-chose-between-car-and-multirotor","text":"The default is to use multirotor. To use car simple set \"SimMode\": \"Car\" like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } To choose multirotor, set \"SimMode\": \"Multirotor\" . If you want to prompt user to select vehicle type then use \"SimMode\": \"\" .","title":"How to Chose Between Car and Multirotor?"},{"location":"settings/#available-settings-and-their-defaults","text":"Below are complete list of settings available along with their default values. If any of the settings is missing from json file, then default value is used. Some default values are simply specified as \"\" which means actual value may be chosen based on the vehicle you are using. For example, ViewMode setting has default value \"\" which translates to \"FlyWithMe\" for drones and \"SpringArmChase\" for cars. WARNING: Do not copy paste all of below in your settings.json. We strongly recommend adding only those settings that you don't want default values. Only required element is \"SettingsVersion\" . { \"SimMode\": \"\", \"ClockType\": \"\", \"ClockSpeed\": 1, \"LocalHostIp\": \"127.0.0.1\", \"RecordUIVisible\": true, \"LogMessagesVisible\": true, \"ViewMode\": \"\", \"RpcEnabled\": true, \"EngineSound\": true, \"PhysicsEngineName\": \"\", \"SpeedUnitFactor\": 1.0, \"SpeedUnitLabel\": \"m/s\", \"Recording\": { \"RecordOnMove\": false, \"RecordInterval\": 0.05, \"Cameras\": [ { \"CameraName\": \"0\", \"ImageType\": 0, \"PixelsAsFloat\": false, \"Compress\": true } ] }, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"AutoExposureBias\": 0, \"AutoExposureMaxBrightness\": 0.64, \"AutoExposureMinBrightness\": 0.03, \"MotionBlurAmount\": 0, \"TargetGamma\": 1.0, \"ProjectionMode\": \"\", \"OrthoWidth\": 5.12 } ], \"NoiseSettings\": [ { \"Enabled\": false, \"ImageType\": 0, \"RandContrib\": 0.2, \"RandSpeed\": 100000.0, \"RandSize\": 500.0, \"RandDensity\": 2, \"HorzWaveContrib\":0.03, \"HorzWaveStrength\": 0.08, \"HorzWaveVertSize\": 1.0, \"HorzWaveScreenSize\": 1.0, \"HorzNoiseLinesContrib\": 1.0, \"HorzNoiseLinesDensityY\": 0.01, \"HorzNoiseLinesDensityXY\": 0.5, \"HorzDistortionContrib\": 1.0, \"HorzDistortionStrength\": 0.002 } ], \"Gimbal\": { \"Stabilization\": 0, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"OriginGeopoint\": { \"Latitude\": 47.641468, \"Longitude\": -122.140165, \"Altitude\": 122 }, \"TimeOfDay\": { \"Enabled\": false, \"StartDateTime\": \"\", \"CelestialClockSpeed\": 1, \"StartDateTimeDst\": false, \"UpdateIntervalSecs\": 60 }, \"SubWindows\": [ {\"WindowID\": 0, \"CameraName\": \"0\", \"ImageType\": 3, \"Visible\": false}, {\"WindowID\": 1, \"CameraName\": \"0\", \"ImageType\": 5, \"Visible\": false}, {\"WindowID\": 2, \"CameraName\": \"0\", \"ImageType\": 0, \"Visible\": false} ], \"SegmentationSettings\": { \"InitMethod\": \"\", \"MeshNamingMethod\": \"\", \"OverrideExisting\": false }, \"PawnPaths\": { \"BareboneCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/Vehicle/VehicleAdvPawn.VehicleAdvPawn_C'\"}, \"DefaultCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/SUV/SuvCarPawn.SuvCarPawn_C'\"}, \"DefaultQuadrotor\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_FlyingPawn.BP_FlyingPawn_C'\"}, \"DefaultComputerVision\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_ComputerVisionPawn.BP_ComputerVisionPawn_C'\"} }, \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Armed\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": false }, \"Cameras\": { //same elements as CameraDefaults above, key as name }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"PhysXCar\": { \"VehicleType\": \"PhysXCar\", \"DefaultVehicleState\": \"\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"RC\": { \"RemoteControlID\": -1 }, \"Cameras\": { \"MyCamera1\": { //same elements as elements inside CameraDefaults above }, \"MyCamera2\": { //same elements as elements inside CameraDefaults above }, }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } } }","title":"Available Settings and Their Defaults"},{"location":"settings/#simmode","text":"SimMode determines which simulation mode will be used. Below are currently supported values: - \"\" : prompt user to select vehicle type multirotor or car - \"Multirotor\" : Use multirotor simulation - \"Car\" : Use car simulation - \"ComputerVision\" : Use only camera, no vehicle or physics","title":"SimMode"},{"location":"settings/#viewmode","text":"The ViewMode determines which camera to use as default and how camera will follow the vehicle. For multirotors, the default ViewMode is \"FlyWithMe\" while for cars the default ViewMode is \"SpringArmChase\" . FlyWithMe : Chase the vehicle from behind with 6 degrees of freedom GroundObserver : Chase the vehicle from 6' above the ground but with full freedom in XY plane. Fpv : View the scene from front camera of vehicle Manual : Don't move camera automatically. Use arrow keys and ASWD keys for move camera manually. SpringArmChase : Chase the vehicle with camera mounted on (invisible) arm that is attached to the vehicle via spring (so it has some latency in movement). NoDisplay : This will freeze rendering for main screen however rendering for subwindows, recording and APIs remain active. This mode is useful to save resources in \"headless\" mode where you are only interested in getting images and don't care about what gets rendered on main screen. This may also improve FPS for recording images.","title":"ViewMode"},{"location":"settings/#timeofday","text":"This setting controls the position of Sun in the environment. By default Enabled is false which means Sun's position is left at whatever was the default in the environment and it doesn't change over the time. If Enabled is true then Sun position is computed using longitude, latitude and altitude specified in OriginGeopoint section for the date specified in StartDateTime in the string format as %Y-%m-%d %H:%M:%S , for example, 2018-02-12 15:20:00 . If this string is empty then current date and time is used. If StartDateTimeDst is true then we adjust for day light savings time. The Sun's position is then continuously updated at the interval specified in UpdateIntervalSecs . In some cases, it might be desirable to have celestial clock run faster or slower than simulation clock. This can be specified using CelestialClockSpeed , for example, value 100 means for every 1 second of simulation clock, Sun's position is advanced by 100 seconds so Sun will move in sky much faster. Also see Time of Day API .","title":"TimeOfDay"},{"location":"settings/#origingeopoint","text":"This setting specifies the latitude, longitude and altitude of the Player Start component placed in the Unreal environment. The vehicle's home point is computed using this transformation. Note that all coordinates exposed via APIs are using NED system in SI units which means each vehicle starts at (0, 0, 0) in NED system. Time of Day settings are computed for geographical coordinates specified in OriginGeopoint .","title":"OriginGeopoint"},{"location":"settings/#subwindows","text":"This setting determines what is shown in each of 3 subwindows which are visible when you press 0 key. The WindowsID can be 0 to 2, CameraName is any available camera on the vehicle. ImageType integer value determines what kind of image gets shown according to ImageType enum . For example, for car vehicles below shows driver view, front bumper view and rear view as scene, depth and surface normals respectively. \"SubWindows\": [ {\"WindowID\": 0, \"ImageType\": 0, \"CameraName\": \"3\", \"Visible\": true}, {\"WindowID\": 1, \"ImageType\": 3, \"CameraName\": \"0\", \"Visible\": true}, {\"WindowID\": 2, \"ImageType\": 6, \"CameraName\": \"4\", \"Visible\": true} ]","title":"SubWindows"},{"location":"settings/#recording","text":"The recording feature allows you to record data such as position, orientation, velocity along with the captured image at specified intervals. You can start recording by pressing red Record button on lower right or the R key. The data is stored in the Documents\\AirSim folder, in a time stamped subfolder for each recording session, as tab separated file. RecordInterval : specifies minimal interval in seconds between capturing two images. RecordOnMove : specifies that do not record frame if there was vehicle's position or orientation hasn't changed. Cameras : this element controls which cameras are used to capture images. By default scene image from camera 0 is recorded as compressed png format. This setting is json array so you can specify multiple cameras to capture images, each with potentially different image types . When PixelsAsFloat is true, image is saved as pfm file instead of png file.","title":"Recording"},{"location":"settings/#clockspeed","text":"This setting allows you to set the speed of simulation clock with respect to wall clock. For example, value of 5.0 would mean simulation clock has 5 seconds elapsed when wall clock has 1 second elapsed (i.e. simulation is running faster). The value of 0.1 means that simulation clock is 10X slower than wall clock. The value of 1 means simulation is running in real time. It is important to realize that quality of simulation may decrease as the simulation clock runs faster. You might see artifacts like object moving past obstacles because collision is not detected. However slowing down simulation clock (i.e. values < 1.0) generally improves the quality of simulation.","title":"ClockSpeed"},{"location":"settings/#segmentation-settings","text":"The InitMethod determines how object IDs are initialized at startup to generate segmentation . The value \"\" or \"CommonObjectsRandomIDs\" (default) means assign random IDs to each object at startup. This will generate segmentation view with random colors assign to each object. The value \"None\" means don't initialize object IDs. This will cause segmentation view to have single solid colors. This mode is useful if you plan to set up object IDs using APIs and it can save lot of delay at startup for large environments like CityEnviron. If OverrideExisting is false then initialization does not alter non-zero object IDs already assigned otherwise it does. If MeshNamingMethod is \"\" or \"OwnerName\" then we use mesh's owner name to generate random hash as object IDs. If it is \"StaticMeshName\" then we use static mesh's name to generate random hash as object IDs. Note that it is not possible to tell individual instances of the same static mesh apart this way, but the names are often more intuitive.","title":"Segmentation Settings"},{"location":"settings/#camera-settings","text":"The CameraDefaults element at root level specifies defaults used for all cameras. These defaults can be overridden for individual camera in Cameras element inside Vehicles as described later.","title":"Camera Settings"},{"location":"settings/#note-on-imagetype-element","text":"The ImageType element in JSON array determines which image type that settings applies to. The valid values are described in ImageType section . In addition, we also support special value ImageType: -1 to apply the settings to external camera (i.e. what you are looking at on the screen). For example, CaptureSettings element is json array so you can add settings for multiple image types easily.","title":"Note on ImageType element"},{"location":"settings/#capturesettings","text":"The CaptureSettings determines how different image types such as scene, depth, disparity, surface normals and segmentation views are rendered. The Width, Height and FOV settings should be self explanatory. The AutoExposureSpeed decides how fast eye adaptation works. We set to generally high value such as 100 to avoid artifacts in image capture. Similarly we set MotionBlurAmount to 0 by default to avoid artifacts in ground truth images. The ProjectionMode decides the projection used by the capture camera and can take value \"perspective\" (default) or \"orthographic\". If projection mode is \"orthographic\" then OrthoWidth determines width of projected area captured in meters. For explanation of other settings, please see this article .","title":"CaptureSettings"},{"location":"settings/#noisesettings","text":"The NoiseSettings allows to add noise to the specified image type with a goal of simulating camera sensor noise, interference and other artifacts. By default no noise is added, i.e., Enabled: false . If you set Enabled: true then following different types of noise and interference artifacts are enabled, each can be further tuned using setting. The noise effects are implemented as shader created as post processing material in Unreal Engine called CameraSensorNoise . Demo of camera noise and interference simulation:","title":"NoiseSettings"},{"location":"settings/#random-noise","text":"This adds random noise blobs with following parameters. * RandContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * RandSpeed : This determines how fast noise fluctuates, 1 means no fluctuation and higher values like 1E6 means full fluctuation. * RandSize : This determines how coarse noise is, 1 means every pixel has its own noise while higher value means more than 1 pixels share same noise value. * RandDensity : This determines how many pixels out of total will have noise, 1 means all pixels while higher value means lesser number of pixels (exponentially).","title":"Random noise"},{"location":"settings/#horizontal-bump-distortion","text":"This adds horizontal bumps / flickering / ghosting effect. * HorzWaveContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzWaveStrength : This determines overall strength of the effect. * HorzWaveVertSize : This determines how many vertical pixels would be effected by the effect. * HorzWaveScreenSize : This determines how much of the screen is effected by the effect.","title":"Horizontal bump distortion"},{"location":"settings/#horizontal-noise-lines","text":"This adds regions of noise on horizontal lines. * HorzNoiseLinesContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzNoiseLinesDensityY : This determines how many pixels in horizontal line gets affected. * HorzNoiseLinesDensityXY : This determines how many lines on screen gets affected.","title":"Horizontal noise lines"},{"location":"settings/#horizontal-line-distortion","text":"This adds fluctuations on horizontal line. * HorzDistortionContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzDistortionStrength : This determines how large is the distortion.","title":"Horizontal line distortion"},{"location":"settings/#gimbal","text":"The Gimbal element allows to freeze camera orientation for pitch, roll and/or yaw. This setting is ignored unless ImageType is -1. The Stabilization is defaulted to 0 meaning no gimbal i.e. camera orientation changes with body orientation on all axis. The value of 1 means full stabilization. The value between 0 to 1 acts as a weight for fixed angles specified (in degrees, in world-frame) in Pitch , Roll and Yaw elements and orientation of the vehicle body. When any of the angles is omitted from json or set to NaN, that angle is not stabilized (i.e. it moves along with vehicle body).","title":"Gimbal"},{"location":"settings/#vehicles-settings","text":"Each simulation mode will go through the list of vehicles specified in this setting and create the ones that has \"AutoCreate\": true . Each vehicle specified in this setting has key which becomes the name of the vehicle. If \"Vehicles\" element is missing then this list is populated with default car named \"PhysXCar\" and default multirotor named \"SimpleFlight\".","title":"Vehicles Settings"},{"location":"settings/#common-vehicle-setting","text":"VehicleType : This could be either PhysXCar , SimpleFlight , PX4Multirotor or ComputerVision . There is no default value therefore this element must be specified. PawnPath : This allows to override the pawn blueprint to use for the vehicle. For example, you may create new pawn blueprint derived from ACarPawn for a warehouse robot in your own project outside the AirSim code and then specify its path here. See also PawnPaths . DefaultVehicleState : Possible value for multirotors is Armed or Disarmed . AutoCreate : If true then this vehicle would be spawned (if supported by selected sim mode). RC : This sub-element allows to specify which remote controller to use for vehicle using RemoteControlID . The value of -1 means use keyboard (not supported yet for multirotors). The value >= 0 specifies one of many remote controllers connected to the system. The list of available RCs can be seen in Game Controllers panel in Windows, for example. X, Y, Z, Yaw, Roll, Pitch : These elements allows you to specify the initial position and orientation of the vehicle. Position is in NED coordinates in SI units with origin set to Player Start location in Unreal environment. The orientation is specified in degrees. IsFpvVehicle : This setting allows to specify which vehicle camera will follow and the view that will be shown when ViewMode is set to Fpv. By default, AirSim selects the first vehicle in settings as FPV vehicle. Cameras : This element specifies camera settings for vehicle. The key in this element is name of the available camera and the value is same as CameraDefaults as described above. For example, to change FOV for the front center camera to 120 degrees, you can use this for Vehicles setting: \"Vehicles\": { \"FishEyeDrone\": { \"VehicleType\": \"SimpleFlight\", \"Cameras\": { \"front-center\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"FOV_Degrees\": 120 } ] } } } }","title":"Common Vehicle Setting"},{"location":"settings/#using-px4","text":"By default we use simple_flight so you don't have to do separate HITL or SITL setups. We also support \"PX4\" for advanced users. To use PX4 with AirSim, you can use the following for Vehicles setting: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", } }","title":"Using PX4"},{"location":"settings/#additional-px4-settings","text":"The defaults for PX4 is to enable hardware-in-loop setup. There are various other settings available for PX4 as follows with their default values: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"ControlIp\": \"127.0.0.1\", \"ControlPort\": 14580, \"LogViewerHostIp\": \"127.0.0.1\", \"LogViewerPort\": 14388, \"OffboardCompID\": 1, \"OffboardSysID\": 134, \"QgcHostIp\": \"127.0.0.1\", \"QgcPort\": 14550, \"SerialBaudRate\": 115200, \"SerialPort\": \"*\", \"SimCompID\": 42, \"SimSysID\": 142, \"TcpPort\": 4560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14560, \"UseSerial\": true, \"UseTcp\": false, \"VehicleCompID\": 1, \"VehicleSysID\": 135, \"Model\": \"Generic\", \"LocalHostIp\": \"127.0.0.1\" } } These settings define the MavLink SystemId and ComponentId for the Simulator (SimSysID, SimCompID), and for the vehicle (VehicleSysID, VehicleCompID) and the node that allows remote control of the drone from another app this is called the offboard node (OffboardSysID, OffboardCompID). If you want the simulator to also talk to your ground control app (like QGroundControl) you can also set the UDP address for that in case you want to run that on a different machine (QgcHostIp, QgcPort). The default is local host so QGroundControl should \"just work\" if it is running on the same machine. You can connect the simulator to the LogViewer app, provided in this repo, by setting the UDP address for that (LogViewerHostIp, LogViewerPort). And for each flying drone added to the simulator there is a named block of additional settings. In the above you see the default name \"PX4\". You can change this name from the Unreal Editor when you add a new BP_FlyingPawn asset. You will see these properties grouped under the category \"MavLink\". The MavLink node for this pawn can be remote over UDP or it can be connected to a local serial port. If serial then set UseSerial to true, otherwise set UseSerial to false. For serial connections you also need to set the appropriate SerialBaudRate. The default of 115200 works with Pixhawk version 2 over USB. When communicating with the PX4 drone over serial port both the HIL_ messages and vehicle control messages share the same serial port. When communicating over UDP or TCP PX4 requires two separate channels. If UseTcp is false, then UdpIp, UdpPort are used to send HIL_ messages, otherwise the TcpPort is used. TCP support in PX4 was added in 1.9.2 with the lockstep feature because the guarantee of message delivery that TCP provides is required for the proper functioning of lockstep. AirSim becomes a TCP server in that case, and waits for a connection from the PX4 app. The second channel for controlling the vehicle is defined by (ControlIp, ControlPort) and is always a UDP channel.","title":"Additional PX4 Settings"},{"location":"settings/#other-settings","text":"","title":"Other Settings"},{"location":"settings/#enginesound","text":"To turn off the engine sound use setting \"EngineSound\": false . Currently this setting applies only to car.","title":"EngineSound"},{"location":"settings/#pawnpaths","text":"This allows you to specify your own vehicle pawn blueprints, for example, you can replace the default car in AirSim with your own car. Your vehicle BP can reside in Content folder of your own Unreal project (i.e. outside of AirSim plugin folder). For example, if you have a car BP located in file Content\\MyCar\\MySedanBP.uasset in your project then you can set \"DefaultCar\": {\"PawnBP\":\"Class'/Game/MyCar/MySedanBP.MySedanBP_C'\"} . The XYZ.XYZ_C is a special notation required to specify class for BP XYZ . Please note that your BP must be derived from CarPawn class. By default this is not the case but you can re-parent the BP using the \"Class Settings\" button in toolbar in UE editor after you open the BP and then choosing \"Car Pawn\" for Parent Class settings in Class Options. It is also a good idea to disable \"Auto Possess Player\" and \"Auto Possess AI\" as well as set AI Controller Class to None in BP details. Please make sure your asset is included for cooking in packaging options if you are creating binary.","title":"PawnPaths"},{"location":"settings/#physicsenginename","text":"For cars, we support only PhysX for now (regardless of value in this setting). For multirotors, we support \"FastPhysicsEngine\" only.","title":"PhysicsEngineName"},{"location":"settings/#localhostip-setting","text":"Now when connecting to remote machines you may need to pick a specific Ethernet adapter to reach those machines, for example, it might be over Ethernet or over Wi-Fi, or some other special virtual adapter or a VPN. Your PC may have multiple networks, and those networks might not be allowed to talk to each other, in which case the UDP messages from one network will not get through to the others. So the LocalHostIp allows you to configure how you are reaching those machines. The default of 127.0.0.1 is not able to reach external machines, this default is only used when everything you are talking to is contained on a single PC.","title":"LocalHostIp Setting"},{"location":"settings/#speedunitfactor","text":"Unit conversion factor for speed related to m/s , default is 1. Used in conjunction with SpeedUnitLabel. This may be only used for display purposes for example on-display speed when car is being driven. For example, to get speed in miles/hr use factor 2.23694.","title":"SpeedUnitFactor"},{"location":"settings/#speedunitlabel","text":"Unit label for speed, default is m/s . Used in conjunction with SpeedUnitFactor.","title":"SpeedUnitLabel"},{"location":"simple_flight/","text":"simple_flight # If you don't know what flight controller does than see What is Flight Controller? . AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In future, we also plan to support ROSFlight and Hackflight . Advantages # The advantage of using simple_flight is zero additional setup you need to do and it \"just works\". Also, simple_flight uses steppable clock which means you can pause the simulation and things are not at mercy of high variance low precision clock that operating system provides. Further, simple_flight is simple, cross platform and 100% header-only dependency-free C++ code which means you can literally step through from simulator to inside flight controller code within same code base! Design # Normally flight controllers are designed to run on actual hardware on vehicles and their support for running in simulator varies widely. They are often fairly difficult to configure for non-expert users and typically have complex build usually lacking cross platform support. All these problems have played significant part in design of simple_flight. simple_flight is designed from ground up as library with clean interface that can work onboard the vehicle as well as simulator. The core principle is that flight controller has no way to specify special simulation mode and there for it has no way to know if it is running under simulation or real vehicle. We thus view flight controller simply as collection of algorithms packaged in a library. Another key emphasis is to develop this code as dependency free header-only pure standard C++11 code. This means there is no special build required to compile simple_flight. You just copy its source code to any project you wish and it just works. Control # simple_flight can control vehicle by taking in desired input as angle rate, angle level, velocity or position. Each axis of control can be specified with one of these modes. Internally simple_flight uses cascade of PID controllers to finally generate actuator signals. This means position PID drives velocity PID which drives angle level PID which finally drives angle rate PID. State Estimation # In current release we are using ground truth from simulator for our state estimation. We plan to add complimentary filter based state estimation for angular velocity and orientation using 2 sensors (gyroscope, accelerometer) in near future. In more longer term, we plan to integrate another library to do velocity and position estimation using 4 sensors (gyroscope, accelerometer, magnetometer and barometer) using EKF. If you have experience this area than we encourage you to engage with us and contribute! Supported Boards # Currently we have implemented simple_flight interfaces for simulated board. We plan to implement it for Pixhawk V2 board and possibly Naze32 board. We expect all our code to remain unchanged and the implementation would mainly involve adding drivers for various sensors, handling ISRs and managing other board specific details. If you have experience this area than we encourage you to engage with us and contribute! Configuration # To have AirSim use simple_flight, you can specify it in settings.json as shown below. Note that this is default so you don't have to do it explicitly. \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", } } By default, vehicle using simple_flight is already armed which is why you would see propellers spinning. However if you don't want that than set DefaultVehicleState to Inactive like this: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Inactive\" } } In this case, you will need to either manually arm using RC sticks in down inward position or using APIs. For safety reasons, flight controllers would disallow API control unless human operator has consented using a switch on RC. Also, when RC control is lost, vehicle should disable API control and enter hover mode for safety reasons. To simplify things a bit, simple_flight enables API control without human consent using RC and even when RC is not detected by default however you can change this using following setting: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": true } } } Finally, simple_flight uses steppable clock by default which means clock advances when simulator tells it to advance (unlike wall clock which advances strictly according to passage of time). This means clock can be paused, for example, if code hits the break point and there is zero variance in clock (clock APIs provides by operating system might have significant variance unless its \"real time\" OS). If you want simple_flight to use wall clock instead than use following settings: \"ClockType\": \"ScalableClock\"","title":"Simple Flight"},{"location":"simple_flight/#simple_flight","text":"If you don't know what flight controller does than see What is Flight Controller? . AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In future, we also plan to support ROSFlight and Hackflight .","title":"simple_flight"},{"location":"simple_flight/#advantages","text":"The advantage of using simple_flight is zero additional setup you need to do and it \"just works\". Also, simple_flight uses steppable clock which means you can pause the simulation and things are not at mercy of high variance low precision clock that operating system provides. Further, simple_flight is simple, cross platform and 100% header-only dependency-free C++ code which means you can literally step through from simulator to inside flight controller code within same code base!","title":"Advantages"},{"location":"simple_flight/#design","text":"Normally flight controllers are designed to run on actual hardware on vehicles and their support for running in simulator varies widely. They are often fairly difficult to configure for non-expert users and typically have complex build usually lacking cross platform support. All these problems have played significant part in design of simple_flight. simple_flight is designed from ground up as library with clean interface that can work onboard the vehicle as well as simulator. The core principle is that flight controller has no way to specify special simulation mode and there for it has no way to know if it is running under simulation or real vehicle. We thus view flight controller simply as collection of algorithms packaged in a library. Another key emphasis is to develop this code as dependency free header-only pure standard C++11 code. This means there is no special build required to compile simple_flight. You just copy its source code to any project you wish and it just works.","title":"Design"},{"location":"simple_flight/#control","text":"simple_flight can control vehicle by taking in desired input as angle rate, angle level, velocity or position. Each axis of control can be specified with one of these modes. Internally simple_flight uses cascade of PID controllers to finally generate actuator signals. This means position PID drives velocity PID which drives angle level PID which finally drives angle rate PID.","title":"Control"},{"location":"simple_flight/#state-estimation","text":"In current release we are using ground truth from simulator for our state estimation. We plan to add complimentary filter based state estimation for angular velocity and orientation using 2 sensors (gyroscope, accelerometer) in near future. In more longer term, we plan to integrate another library to do velocity and position estimation using 4 sensors (gyroscope, accelerometer, magnetometer and barometer) using EKF. If you have experience this area than we encourage you to engage with us and contribute!","title":"State Estimation"},{"location":"simple_flight/#supported-boards","text":"Currently we have implemented simple_flight interfaces for simulated board. We plan to implement it for Pixhawk V2 board and possibly Naze32 board. We expect all our code to remain unchanged and the implementation would mainly involve adding drivers for various sensors, handling ISRs and managing other board specific details. If you have experience this area than we encourage you to engage with us and contribute!","title":"Supported Boards"},{"location":"simple_flight/#configuration","text":"To have AirSim use simple_flight, you can specify it in settings.json as shown below. Note that this is default so you don't have to do it explicitly. \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", } } By default, vehicle using simple_flight is already armed which is why you would see propellers spinning. However if you don't want that than set DefaultVehicleState to Inactive like this: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Inactive\" } } In this case, you will need to either manually arm using RC sticks in down inward position or using APIs. For safety reasons, flight controllers would disallow API control unless human operator has consented using a switch on RC. Also, when RC control is lost, vehicle should disable API control and enter hover mode for safety reasons. To simplify things a bit, simple_flight enables API control without human consent using RC and even when RC is not detected by default however you can change this using following setting: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": true } } } Finally, simple_flight uses steppable clock by default which means clock advances when simulator tells it to advance (unlike wall clock which advances strictly according to passage of time). This means clock can be paused, for example, if code hits the break point and there is zero variance in clock (clock APIs provides by operating system might have significant variance unless its \"real time\" OS). If you want simple_flight to use wall clock instead than use following settings: \"ClockType\": \"ScalableClock\"","title":"Configuration"},{"location":"steering_wheel_installation/","text":"Logitech G920 Steering Wheel Installation # To use Logitech G920 steering wheel with AirSim follow these steps: Connect the steering wheel to the computer and wait until drivers installation complete. Install Logitech Gaming Software from here Before debug, you\u2019ll have to normalize the values in AirSim code. Perform this changes in CarPawn.cpp (according to the current update in the git): In line 382, change \u201cVal\u201d to \u201c1 \u2013 Val\u201d. (the complementary value in the range [0.0,1.0]). In line 388, change \u201cVal\u201d to \u201c5Val - 2.5\u201d (Change the range of the given input from [0.0,1.0] to [-1.0,1.0]). In line 404, change \u201cVal\u201d to \u201c4(1 \u2013 Val)\u201d. (the complementary value in the range [0.0,1.0]). Debug AirSim project (while the steering wheel is connected \u2013 it\u2019s important). On Unreal Editor, go to Edit->plugins->input devices and enable \u201cWindows RawInput\u201d. Go to Edit->Project Settings->Raw Input, and add new device configuration: Vendor ID: 0x046d (In case of Logitech G920, otherwise you might need to check it). Product ID: 0xc261 (In case of Logitech G920, otherwise you might need to check it). Under \u201cAxis Properties\u201d, make sure that \u201cGenericUSBController Axis 2\u201d, \u201cGenericUSBController Axis 4\u201d and \u201cGenericUSBController Axis 5\u201d are all enabled with an offset of 1.0. Explanation: axis 2 is responsible for steering movement, axis 4 is for brake and axis 5 is for gas. If you need to configure the clutch, it\u2019s on axis 3. Go to Edit->Project Settings->Input, Under Bindings in \u201cAxis Mappings\u201d: Remove existing mappings from the groups \u201cMoveRight\u201d and \u201cMoveForward\u201d. Add new axis mapping to the group \u201cMoveRight\u201d, use GenericUSBController axis 2 with a scale of 1.0. Add new axis mapping to the group \u201cMoveForward\u201d, use GenericUSBController axis 5 with a scale of 1.0. Add a new group of axis mappings, name it \u201cFootBrake\u201d and add new axis mapping to this group, use GenericUSBController axis 4 with a scale of 1.0. Play and drive ! Pay Attention # Notice that in the first time we \"play\" after debug, we need to touch the wheel to \u201creset\u201d the values. Tip # In the gaming software, you can configure buttons as keyboard shortcuts, we used it to configure a shortcut to record dataset or to play in full screen.","title":"Steering Wheel"},{"location":"steering_wheel_installation/#logitech-g920-steering-wheel-installation","text":"To use Logitech G920 steering wheel with AirSim follow these steps: Connect the steering wheel to the computer and wait until drivers installation complete. Install Logitech Gaming Software from here Before debug, you\u2019ll have to normalize the values in AirSim code. Perform this changes in CarPawn.cpp (according to the current update in the git): In line 382, change \u201cVal\u201d to \u201c1 \u2013 Val\u201d. (the complementary value in the range [0.0,1.0]). In line 388, change \u201cVal\u201d to \u201c5Val - 2.5\u201d (Change the range of the given input from [0.0,1.0] to [-1.0,1.0]). In line 404, change \u201cVal\u201d to \u201c4(1 \u2013 Val)\u201d. (the complementary value in the range [0.0,1.0]). Debug AirSim project (while the steering wheel is connected \u2013 it\u2019s important). On Unreal Editor, go to Edit->plugins->input devices and enable \u201cWindows RawInput\u201d. Go to Edit->Project Settings->Raw Input, and add new device configuration: Vendor ID: 0x046d (In case of Logitech G920, otherwise you might need to check it). Product ID: 0xc261 (In case of Logitech G920, otherwise you might need to check it). Under \u201cAxis Properties\u201d, make sure that \u201cGenericUSBController Axis 2\u201d, \u201cGenericUSBController Axis 4\u201d and \u201cGenericUSBController Axis 5\u201d are all enabled with an offset of 1.0. Explanation: axis 2 is responsible for steering movement, axis 4 is for brake and axis 5 is for gas. If you need to configure the clutch, it\u2019s on axis 3. Go to Edit->Project Settings->Input, Under Bindings in \u201cAxis Mappings\u201d: Remove existing mappings from the groups \u201cMoveRight\u201d and \u201cMoveForward\u201d. Add new axis mapping to the group \u201cMoveRight\u201d, use GenericUSBController axis 2 with a scale of 1.0. Add new axis mapping to the group \u201cMoveForward\u201d, use GenericUSBController axis 5 with a scale of 1.0. Add a new group of axis mappings, name it \u201cFootBrake\u201d and add new axis mapping to this group, use GenericUSBController axis 4 with a scale of 1.0. Play and drive !","title":"Logitech G920 Steering Wheel Installation"},{"location":"steering_wheel_installation/#pay-attention","text":"Notice that in the first time we \"play\" after debug, we need to touch the wheel to \u201creset\u201d the values.","title":"Pay Attention"},{"location":"steering_wheel_installation/#tip","text":"In the gaming software, you can configure buttons as keyboard shortcuts, we used it to configure a shortcut to record dataset or to play in full screen.","title":"Tip"},{"location":"unity_api_support/","text":"Api Car Multirotor reset Supported Supported enableApiControl Supported Supported armDisarm Supported Supported confirmConnection Supported Supported isApiControlEnabled Supported Supported ping Supported Supported simGetObjectPose Supported Supported simSetObjectPose Supported Supported pause Supported Supported continueForTime Supported Supported simGetCollisionInfo Supported Supported setCarControls Supported NA getCarState Supported NA moveByAngleThrottleAsync NA Supported getMultirotorState NA Supported takeoffAsync NA Supported waitOnLastTask Supported Supported cancelLastTask Supported Supported simGetGroundTruthKinematics Supported Supported getCameraInfo Supported Supported getClientVersion Supported Supported getCollisionInfo Supported Supported getGpsLocation TODO TODO getHomeGeoPoint TODO TODO getLandedState TODO TODO getLidarData TODO TODO getMinRequiredClientVersion Supported Supported getMinRequiredServerVersion Supported Supported getOrientation TODO TODO getPosition TODO TODO getServerVersion Supported Supported getVelocity TODO TODO goHome TODO TODO hover TODO TODO land TODO TODO moveByAngleThrottle TODO TODO moveByAngleZ TODO TODO moveByManual TODO TODO moveByVelocity TODO TODO moveByVelocityZ TODO TODO moveOnPath TODO TODO moveToPosition TODO TODO moveToZ TODO TODO rotateByYawRate TODO TODO rotateToYaw TODO TODO setCameraOrientation TODO TODO setRCData TODO TODO simContinueForTime Supported Supported simGetCameraInfo TODO TODO simGetGroundTruthEnvironment TODO TODO simGetImage TODO TODO simGetImages Supported Supported simGetPose TODO TODO simGetSegmentationObjectID TODO TODO simGetVehiclePose TODO TODO simIsPause TODO TODO simPause TODO TODO simPrintLogMessage TODO TODO simSetCameraOrientation TODO TODO simSetPose TODO TODO simSetSegmentationObjectID TODO TODO simSetVehiclePose TODO TODO takeoff TODO TODO goHomeAsync NA TODO hoverAsync NA TODO landAsync NA TODO moveByAngleZAsync NA TODO moveByManualAsync NA TODO moveByRC NA TODO moveByVelocityAsync NA Supported moveByVelocityZAsync NA TODO moveOnPathAsync NA TODO moveToPositionAsync NA TODO moveToZAsync NA TODO rotateByYawRateAsync NA TODO rotateToYawAsync NA TODO","title":"Unity APIs"},{"location":"unreal_blocks/","text":"Setup Blocks Environment for AirSim # Blocks environment is available in repo in folder Unreal/Environments/Blocks and is designed to be lightweight in size. That means its very basic but fast. Here are quick steps to get Blocks environment up and running: Windows # Make sure you have installed Unreal and built AirSim . Navigate to folder AirSim\\Unreal\\Environments\\Blocks and run update_from_git.bat . Double click on generated .sln file to open in Visual Studio 2019 or newer. Make sure Blocks project is the startup project, build configuration is set to DebugGame_Editor and Win64 . Hit F5 to run. Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim . Changing Code and Rebuilding # For Windows, you can just change the code in Visual Studio, press F5 and re-run. There are few batch files available in folder AirSim\\Unreal\\Environments\\Blocks that lets you sync code, clean etc. Linux # Make sure you have built the Unreal Engine and AirSim . Navigate to your UnrealEngine repo folder and run Engine/Binaries/Linux/UE4Editor which will start Unreal Editor. On first start you might not see any projects in UE4 editor. Click on Projects tab, Browse button and then navigate to AirSim/Unreal/Environments/Blocks/Blocks.uproject . If you get prompted for incompatible version and conversion, select In-place conversion which is usually under \"More\" options. If you get prompted for missing modules, make sure to select No so you don't exit. Finally, when prompted with building AirSim, select Yes. Now it might take a while so go get some coffee :). Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim . Changing Code and Rebuilding # For Linux, make code changes in AirLib or Unreal/Plugins folder and then run ./build.sh to rebuild. This step also copies the build output to Blocks sample project. You can then follow above steps again to re-run. Chosing Your Vehicle: Car or Multirotor # By default AirSim spawns multirotor. You can easily change this to car and use all of AirSim goodies. Please see using car guide. FAQ # I see warnings about like \"_BuitData\" file is missing. # These are intermediate files and you can safely ignore it.","title":"Blocks Environment"},{"location":"unreal_blocks/#setup-blocks-environment-for-airsim","text":"Blocks environment is available in repo in folder Unreal/Environments/Blocks and is designed to be lightweight in size. That means its very basic but fast. Here are quick steps to get Blocks environment up and running:","title":"Setup Blocks Environment for AirSim"},{"location":"unreal_blocks/#windows","text":"Make sure you have installed Unreal and built AirSim . Navigate to folder AirSim\\Unreal\\Environments\\Blocks and run update_from_git.bat . Double click on generated .sln file to open in Visual Studio 2019 or newer. Make sure Blocks project is the startup project, build configuration is set to DebugGame_Editor and Win64 . Hit F5 to run. Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim .","title":"Windows"},{"location":"unreal_blocks/#changing-code-and-rebuilding","text":"For Windows, you can just change the code in Visual Studio, press F5 and re-run. There are few batch files available in folder AirSim\\Unreal\\Environments\\Blocks that lets you sync code, clean etc.","title":"Changing Code and Rebuilding"},{"location":"unreal_blocks/#linux","text":"Make sure you have built the Unreal Engine and AirSim . Navigate to your UnrealEngine repo folder and run Engine/Binaries/Linux/UE4Editor which will start Unreal Editor. On first start you might not see any projects in UE4 editor. Click on Projects tab, Browse button and then navigate to AirSim/Unreal/Environments/Blocks/Blocks.uproject . If you get prompted for incompatible version and conversion, select In-place conversion which is usually under \"More\" options. If you get prompted for missing modules, make sure to select No so you don't exit. Finally, when prompted with building AirSim, select Yes. Now it might take a while so go get some coffee :). Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim .","title":"Linux"},{"location":"unreal_blocks/#changing-code-and-rebuilding_1","text":"For Linux, make code changes in AirLib or Unreal/Plugins folder and then run ./build.sh to rebuild. This step also copies the build output to Blocks sample project. You can then follow above steps again to re-run.","title":"Changing Code and Rebuilding"},{"location":"unreal_blocks/#chosing-your-vehicle-car-or-multirotor","text":"By default AirSim spawns multirotor. You can easily change this to car and use all of AirSim goodies. Please see using car guide.","title":"Chosing Your Vehicle: Car or Multirotor"},{"location":"unreal_blocks/#faq","text":"","title":"FAQ"},{"location":"unreal_blocks/#i-see-warnings-about-like-_buitdata-file-is-missing","text":"These are intermediate files and you can safely ignore it.","title":"I see warnings about like \"_BuitData\" file is missing."},{"location":"unreal_custenv/","text":"Creating and Setting Up Unreal Environment # This page contains the complete instructions start to finish for setting up Unreal environment with AirSim. The Unreal Marketplace has several environment available that you can start using in just few minutes. It is also possible to use environments available on websites such as turbosquid.com or cgitrader.com with bit more effort (here's tutorial video ). In addition there also several free environments available. Below we will use a freely downloadable environment from Unreal Marketplace called Landscape Mountain but the steps are same for any other environments. You can also view these steps performed in Unreal AirSim Setup Video . Note for Linux Users # There is no Epic Games Launcher for Linux which means that if you need to create custom environment, you will need Windows machine to do that. Once you have Unreal project folder, just copy it over to your Linux machine. Step by Step Instructions # Make sure AirSim is built and Unreal 4.24 is installed as described in build instructions . In Epic Games Launcher click the Learn tab then scroll down and find Landscape Mountains . Click the Create Project and download this content (~2GB download). Open LandscapeMountains.uproject , it should launch the Unreal Editor. From the File menu select New C++ class , leave default None on the type of class, click Next , leave default name MyClass , and click Create Class . We need to do this because Unreal requires at least one source file in project. It should trigger compile and open up Visual Studio solution LandscapeMountains.sln . Go to your folder for AirSim repo and copy Unreal\\Plugins folder in to your LandscapeMountains folder. This way now your own Unreal project has AirSim plugin. Edit the LandscapeMountains.uproject so that it looks like this { \"FileVersion\": 3, \"EngineAssociation\": \"4.24\", \"Category\": \"Samples\", \"Description\": \"\", \"Modules\": [ { \"Name\": \"LandscapeMountains\", \"Type\": \"Runtime\", \"LoadingPhase\": \"Default\", \"AdditionalDependencies\": [ \"AirSim\" ] } ], \"TargetPlatforms\": [ \"MacNoEditor\", \"WindowsNoEditor\" ], \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ] } Close Visual Studio and the Unreal Editor and right click the LandscapeMountains.uproject in Windows Explorer and select Generate Visual Studio Project Files . This step detects all plugins and source files in your Unreal project and generates .sln file for Visual Studio. Tip: If the Generate Visual Studio Project Files option is missing you may need to reboot your machine for the Unreal Shell extensions to take effect. If it is still missing then open the LandscapeMountains.uproject in the Unreal Editor and select Refresh Visual Studio Project from the File menu. Reopen LandscapeMountains.sln in Visual Studio, and make sure \"DebugGame Editor\" and \"Win64\" build configuration is the active build configuration. Press F5 to run . This will start the Unreal Editor. The Unreal Editor allows you to edit the environment, assets and other game related settings. First thing you want to do in your environment is set up PlayerStart object. In Landscape Mountains environment, PlayerStart object already exist and you can find it in the World Outliner . Make sure its location is setup as shown. This is where AirSim plugin will create and place the vehicle. If its too high up then vehicle will fall down as soon as you press play giving potentially random behavior In Window/World Settings as shown below, set the GameMode Override to AirSimGameMode : Go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. If you don't do this then UE will be slowed down dramatically when UE window loses focus. Be sure to Save these edits. Hit the Play button in the Unreal Editor. See how to use AirSim . Congratulations! You are now running AirSim in your own Unreal environment. Choosing Your Vehicle: Car or Multirotor # By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . Please see using car guide. Updating Your Environment to Latest Version of AirSim # Once you have your environment using above instructions, you should frequently update your local AirSim code to latest version from GitHub. Below are the instructions to do this: First put clean.bat (or clean.sh for Linux users) in the root folder of your environment. Run this file to clean up all intermediate files in your Unreal project. Do git pull in your AirSim repo followed by build.cmd (or ./build.sh for Linux users). Replace [your project]/Plugins folder with AirSim/Unreal/Plugins folder. Right click on your .uproject file and chose \"Generate Visual Studio project files\" option. This is not required for Linux. FAQ # What are other cool environments? # Unreal Marketplace has dozens of prebuilt extra-ordinarily detailed environments ranging from Moon to Mars and everything in between. The one we have used for testing is called Modular Neighborhood Pack but you can use any environment. Another free environment is Infinity Blade series . Alternatively, if you look under the Learn tab in Epic Game Launcher, you will find many free samples that you can use. One of our favorites is \"A Boy and His Kite\" which is a 100 square miles of highly detailed environment (caution: you will need very beefy PC to run it!). When I press Play button some kind of video starts instead of my vehicle. # If the environment comes with MatineeActor, delete it to avoid any startup demo sequences. There might be other ways to remove it as well, for example, click on Blueprints button, then Level Blueprint and then look at Begin Play event in Event Graph. You might want to disconnect any connections that may be starting \"matinee\". Is there easy way to sync code in my Unreal project with code in AirSim repo? # Sure, there is! You can find bunch of .bat files (for linux, .sh ) in AirSim\\Unreal\\Environments\\Blocks . Just copy them over to your own Unreal project. Most of these are quite simple and self explanatory. I get some error about map. # You might have to set default map for your project. For example, if you are using Modular Neighborhood Pack, set the Editor Starter Map as well as Game Default Map to Demo_Map in Project Settings > Maps & Modes. I see \"Add to project\" option for environment but not \"Create project\" option. # In this case, create a new blank C++ project with no Starter Content and add your environment in to it. I already have my own Unreal project. How do I use AirSim with it? # Copy the Unreal\\Plugins folder from the build you did in the above section into the root of your Unreal project's folder. In your Unreal project's .uproject file, add the key AdditionalDependencies to the \"Modules\" object as we showed in the LandscapeMountains.uproject above. \"AdditionalDependencies\": [ \"AirSim\" ] and the Plugins section to the top level object: \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ]","title":"Custom Unreal Environment"},{"location":"unreal_custenv/#creating-and-setting-up-unreal-environment","text":"This page contains the complete instructions start to finish for setting up Unreal environment with AirSim. The Unreal Marketplace has several environment available that you can start using in just few minutes. It is also possible to use environments available on websites such as turbosquid.com or cgitrader.com with bit more effort (here's tutorial video ). In addition there also several free environments available. Below we will use a freely downloadable environment from Unreal Marketplace called Landscape Mountain but the steps are same for any other environments. You can also view these steps performed in Unreal AirSim Setup Video .","title":"Creating and Setting Up Unreal Environment"},{"location":"unreal_custenv/#note-for-linux-users","text":"There is no Epic Games Launcher for Linux which means that if you need to create custom environment, you will need Windows machine to do that. Once you have Unreal project folder, just copy it over to your Linux machine.","title":"Note for Linux Users"},{"location":"unreal_custenv/#step-by-step-instructions","text":"Make sure AirSim is built and Unreal 4.24 is installed as described in build instructions . In Epic Games Launcher click the Learn tab then scroll down and find Landscape Mountains . Click the Create Project and download this content (~2GB download). Open LandscapeMountains.uproject , it should launch the Unreal Editor. From the File menu select New C++ class , leave default None on the type of class, click Next , leave default name MyClass , and click Create Class . We need to do this because Unreal requires at least one source file in project. It should trigger compile and open up Visual Studio solution LandscapeMountains.sln . Go to your folder for AirSim repo and copy Unreal\\Plugins folder in to your LandscapeMountains folder. This way now your own Unreal project has AirSim plugin. Edit the LandscapeMountains.uproject so that it looks like this { \"FileVersion\": 3, \"EngineAssociation\": \"4.24\", \"Category\": \"Samples\", \"Description\": \"\", \"Modules\": [ { \"Name\": \"LandscapeMountains\", \"Type\": \"Runtime\", \"LoadingPhase\": \"Default\", \"AdditionalDependencies\": [ \"AirSim\" ] } ], \"TargetPlatforms\": [ \"MacNoEditor\", \"WindowsNoEditor\" ], \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ] } Close Visual Studio and the Unreal Editor and right click the LandscapeMountains.uproject in Windows Explorer and select Generate Visual Studio Project Files . This step detects all plugins and source files in your Unreal project and generates .sln file for Visual Studio. Tip: If the Generate Visual Studio Project Files option is missing you may need to reboot your machine for the Unreal Shell extensions to take effect. If it is still missing then open the LandscapeMountains.uproject in the Unreal Editor and select Refresh Visual Studio Project from the File menu. Reopen LandscapeMountains.sln in Visual Studio, and make sure \"DebugGame Editor\" and \"Win64\" build configuration is the active build configuration. Press F5 to run . This will start the Unreal Editor. The Unreal Editor allows you to edit the environment, assets and other game related settings. First thing you want to do in your environment is set up PlayerStart object. In Landscape Mountains environment, PlayerStart object already exist and you can find it in the World Outliner . Make sure its location is setup as shown. This is where AirSim plugin will create and place the vehicle. If its too high up then vehicle will fall down as soon as you press play giving potentially random behavior In Window/World Settings as shown below, set the GameMode Override to AirSimGameMode : Go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. If you don't do this then UE will be slowed down dramatically when UE window loses focus. Be sure to Save these edits. Hit the Play button in the Unreal Editor. See how to use AirSim . Congratulations! You are now running AirSim in your own Unreal environment.","title":"Step by Step Instructions"},{"location":"unreal_custenv/#choosing-your-vehicle-car-or-multirotor","text":"By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . Please see using car guide.","title":"Choosing Your Vehicle: Car or Multirotor"},{"location":"unreal_custenv/#updating-your-environment-to-latest-version-of-airsim","text":"Once you have your environment using above instructions, you should frequently update your local AirSim code to latest version from GitHub. Below are the instructions to do this: First put clean.bat (or clean.sh for Linux users) in the root folder of your environment. Run this file to clean up all intermediate files in your Unreal project. Do git pull in your AirSim repo followed by build.cmd (or ./build.sh for Linux users). Replace [your project]/Plugins folder with AirSim/Unreal/Plugins folder. Right click on your .uproject file and chose \"Generate Visual Studio project files\" option. This is not required for Linux.","title":"Updating Your Environment to Latest Version of AirSim"},{"location":"unreal_custenv/#faq","text":"","title":"FAQ"},{"location":"unreal_custenv/#what-are-other-cool-environments","text":"Unreal Marketplace has dozens of prebuilt extra-ordinarily detailed environments ranging from Moon to Mars and everything in between. The one we have used for testing is called Modular Neighborhood Pack but you can use any environment. Another free environment is Infinity Blade series . Alternatively, if you look under the Learn tab in Epic Game Launcher, you will find many free samples that you can use. One of our favorites is \"A Boy and His Kite\" which is a 100 square miles of highly detailed environment (caution: you will need very beefy PC to run it!).","title":"What are other cool environments?"},{"location":"unreal_custenv/#when-i-press-play-button-some-kind-of-video-starts-instead-of-my-vehicle","text":"If the environment comes with MatineeActor, delete it to avoid any startup demo sequences. There might be other ways to remove it as well, for example, click on Blueprints button, then Level Blueprint and then look at Begin Play event in Event Graph. You might want to disconnect any connections that may be starting \"matinee\".","title":"When I press Play button some kind of video starts instead of my vehicle."},{"location":"unreal_custenv/#is-there-easy-way-to-sync-code-in-my-unreal-project-with-code-in-airsim-repo","text":"Sure, there is! You can find bunch of .bat files (for linux, .sh ) in AirSim\\Unreal\\Environments\\Blocks . Just copy them over to your own Unreal project. Most of these are quite simple and self explanatory.","title":"Is there easy way to sync code in my Unreal project with code in AirSim repo?"},{"location":"unreal_custenv/#i-get-some-error-about-map","text":"You might have to set default map for your project. For example, if you are using Modular Neighborhood Pack, set the Editor Starter Map as well as Game Default Map to Demo_Map in Project Settings > Maps & Modes.","title":"I get some error about map."},{"location":"unreal_custenv/#i-see-add-to-project-option-for-environment-but-not-create-project-option","text":"In this case, create a new blank C++ project with no Starter Content and add your environment in to it.","title":"I see \"Add to project\" option for environment but not \"Create project\" option."},{"location":"unreal_custenv/#i-already-have-my-own-unreal-project-how-do-i-use-airsim-with-it","text":"Copy the Unreal\\Plugins folder from the build you did in the above section into the root of your Unreal project's folder. In your Unreal project's .uproject file, add the key AdditionalDependencies to the \"Modules\" object as we showed in the LandscapeMountains.uproject above. \"AdditionalDependencies\": [ \"AirSim\" ] and the Plugins section to the top level object: \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ]","title":"I already have my own Unreal project. How do I use AirSim with it?"},{"location":"unreal_proj/","text":"Unreal Environment # Setting Up the Unreal Project # Option 1: Built-in Blocks Environment # To get up and running fast, you can use the Blocks project that already comes with AirSim. This is not very highly detailed environment to keep the repo size reasonable but we use it for various testing all the times and it is the easiest way to get your feet wet in this strange land. Follow these quick steps . Option 2: Create Your Own Unreal Environment # If you want to setup photo-realistic high quality environments, then you will need to create your own Unreal project. This is little bit more involved but worthwhile! Follow this step-by-step guide . Changing Code and Development Workflow # To see how you can change and test AirSim code, please read our recommended development workflow .","title":"Setting up Unreal Environment"},{"location":"unreal_proj/#unreal-environment","text":"","title":"Unreal Environment"},{"location":"unreal_proj/#setting-up-the-unreal-project","text":"","title":"Setting Up the Unreal Project"},{"location":"unreal_proj/#option-1-built-in-blocks-environment","text":"To get up and running fast, you can use the Blocks project that already comes with AirSim. This is not very highly detailed environment to keep the repo size reasonable but we use it for various testing all the times and it is the easiest way to get your feet wet in this strange land. Follow these quick steps .","title":"Option 1: Built-in Blocks Environment"},{"location":"unreal_proj/#option-2-create-your-own-unreal-environment","text":"If you want to setup photo-realistic high quality environments, then you will need to create your own Unreal project. This is little bit more involved but worthwhile! Follow this step-by-step guide .","title":"Option 2: Create Your Own Unreal Environment"},{"location":"unreal_proj/#changing-code-and-development-workflow","text":"To see how you can change and test AirSim code, please read our recommended development workflow .","title":"Changing Code and Development Workflow"},{"location":"unreal_upgrade/","text":"Upgrading to Unreal Engine 4.24 # These instructions apply if you are already using AirSim on Unreal Engine 4.16. If you have never installed AirSim, please see How to get it . Caution: The below steps will delete any of your unsaved work in AirSim or Unreal folder. Do this first # For Windows Users # Install Visual Studio 2019 with VC++, Python and C#. Install UE 4.24 through Epic Games Launcher. Start x64 Native Tools Command Prompt for VS 2019 and navigate to AirSim repo. Run clean_rebuild.bat to remove all unchecked/extra stuff and rebuild everything. See also Build AirSim on Windows for more information. For Linux Users # From your AirSim repo folder, run 'clean_rebuild.sh`. Rename or delete your existing folder for Unreal Engine. Follow step 1 and 2 to install Unreal Engine 4.24 . See also Build AirSim on Linux for more information. Upgrading Your Custom Unreal Project # If you have your own Unreal project created in an older version of Unreal Engine then you need to upgrade your project to Unreal 4.24. To do this, Open .uproject file and look for the line \"EngineAssociation\" and make sure it reads like \"EngineAssociation\": \"4.24\" . Delete Plugins/AirSim folder in your Unreal project's folder. Go to your AirSim repo folder and copy Unreal\\Plugins folder to your Unreal project's folder. Copy .bat (or .sh for Linux) from Unreal\\Environments\\Blocks to your project's folder. Run clean.bat (or clean.sh for Linux) followed by GenerateProjectFiles.bat (only for Windows). FAQ # I have an Unreal project that is older than 4.16. How do I upgrade it? # Option 1: Just Recreate Project # If your project doesn't have any code or assets other than environment you downloaded then you can also simply recreate the project in Unreal 4.24 Editor and then copy Plugins folder from AirSim/Unreal/Plugins . Option 2: Modify Few Files # Unreal versions newer than Unreal 4.15 has breaking changes. So you need to modify your .Build.cs and .Target.cs which you can find in the Source folder of your Unreal project. So what are those changes? Below is the gist of it but you should really refer to Unreal's official 4.16 transition post . In your project's *.Target.cs # Change the contructor from, public MyProjectTarget(TargetInfo Target) to public MyProjectTarget(TargetInfo Target) : base(Target) Remove SetupBinaries method if you have one and instead add following line in contructor above: ExtraModuleNames.AddRange(new string[] { \"MyProject\" }); In your project's *.Build.cs # Change the constructor from public MyProject(TargetInfo Target) to public MyProject(ReadOnlyTargetRules Target) : base(Target) . And finally... # Follow above steps to continue the upgrade. The warning box might show only \"Open Copy\" button. Don't click that. Instead, click on More Options which will reveal more buttons. Choose Convert-In-Place option . Caution: Always keep backup of your project first! If you don't have anything nasty, in place conversion should go through and you are now on the new version of Unreal.","title":"Upgrading Unreal"},{"location":"unreal_upgrade/#upgrading-to-unreal-engine-424","text":"These instructions apply if you are already using AirSim on Unreal Engine 4.16. If you have never installed AirSim, please see How to get it . Caution: The below steps will delete any of your unsaved work in AirSim or Unreal folder.","title":"Upgrading to Unreal Engine 4.24"},{"location":"unreal_upgrade/#do-this-first","text":"","title":"Do this first"},{"location":"unreal_upgrade/#for-windows-users","text":"Install Visual Studio 2019 with VC++, Python and C#. Install UE 4.24 through Epic Games Launcher. Start x64 Native Tools Command Prompt for VS 2019 and navigate to AirSim repo. Run clean_rebuild.bat to remove all unchecked/extra stuff and rebuild everything. See also Build AirSim on Windows for more information.","title":"For Windows Users"},{"location":"unreal_upgrade/#for-linux-users","text":"From your AirSim repo folder, run 'clean_rebuild.sh`. Rename or delete your existing folder for Unreal Engine. Follow step 1 and 2 to install Unreal Engine 4.24 . See also Build AirSim on Linux for more information.","title":"For Linux Users"},{"location":"unreal_upgrade/#upgrading-your-custom-unreal-project","text":"If you have your own Unreal project created in an older version of Unreal Engine then you need to upgrade your project to Unreal 4.24. To do this, Open .uproject file and look for the line \"EngineAssociation\" and make sure it reads like \"EngineAssociation\": \"4.24\" . Delete Plugins/AirSim folder in your Unreal project's folder. Go to your AirSim repo folder and copy Unreal\\Plugins folder to your Unreal project's folder. Copy .bat (or .sh for Linux) from Unreal\\Environments\\Blocks to your project's folder. Run clean.bat (or clean.sh for Linux) followed by GenerateProjectFiles.bat (only for Windows).","title":"Upgrading Your Custom Unreal Project"},{"location":"unreal_upgrade/#faq","text":"","title":"FAQ"},{"location":"unreal_upgrade/#i-have-an-unreal-project-that-is-older-than-416-how-do-i-upgrade-it","text":"","title":"I have an Unreal project that is older than 4.16. How do I upgrade it?"},{"location":"unreal_upgrade/#option-1-just-recreate-project","text":"If your project doesn't have any code or assets other than environment you downloaded then you can also simply recreate the project in Unreal 4.24 Editor and then copy Plugins folder from AirSim/Unreal/Plugins .","title":"Option 1: Just Recreate Project"},{"location":"unreal_upgrade/#option-2-modify-few-files","text":"Unreal versions newer than Unreal 4.15 has breaking changes. So you need to modify your .Build.cs and .Target.cs which you can find in the Source folder of your Unreal project. So what are those changes? Below is the gist of it but you should really refer to Unreal's official 4.16 transition post .","title":"Option 2: Modify Few Files"},{"location":"unreal_upgrade/#in-your-projects-targetcs","text":"Change the contructor from, public MyProjectTarget(TargetInfo Target) to public MyProjectTarget(TargetInfo Target) : base(Target) Remove SetupBinaries method if you have one and instead add following line in contructor above: ExtraModuleNames.AddRange(new string[] { \"MyProject\" });","title":"In your project's *.Target.cs"},{"location":"unreal_upgrade/#in-your-projects-buildcs","text":"Change the constructor from public MyProject(TargetInfo Target) to public MyProject(ReadOnlyTargetRules Target) : base(Target) .","title":"In your project's *.Build.cs"},{"location":"unreal_upgrade/#and-finally","text":"Follow above steps to continue the upgrade. The warning box might show only \"Open Copy\" button. Don't click that. Instead, click on More Options which will reveal more buttons. Choose Convert-In-Place option . Caution: Always keep backup of your project first! If you don't have anything nasty, in place conversion should go through and you are now on the new version of Unreal.","title":"And finally..."},{"location":"upgrade_apis/","text":"Upgrading API Client Code # There have been several API changes in AirSim v1.2 that we hope removes inconsistency, adds future extensibility and presents cleaner interface. Many of these changes are however breaking changes which means you will need to modify your client code that talks to AirSim. Quicker Way # While most changes you need to do in your client code are fairly easy, a quicker way is simply to take a look at the example code such as Hello Drone or Hello Car to get gist of changes. Importing AirSim # Instead of, from AirSimClient import * use this: import airsim Above assumes you have installed AirSim module using, pip install --user airsim If you are running you code from PythonClient folder in repo then you can also do this: import setup_path import airsim Here setup_path.py should exist in your folder and it will set the path of airsim package in PythonClient repo folder. All examples in PythonClient folder uses this method. Using AirSim Classes # As we have everything now in package, you will need to use explicit namespace for AirSim classes like shown below. Instead of, client1 = CarClient() use this: client1 = airsim.CarClient() AirSim Types # We have moved all types in airsim namespace. Instead of, image_type = AirSimImageType.DepthVis d = DrivetrainType.MaxDegreeOfFreedom use this: image_type = airsim.ImageType.DepthVis d = airsim.DrivetrainType.MaxDegreeOfFreedom Getting Images # Nothing new below, it's just combination of above. Note that all APIs that previously took camera_id , now takes camera_name instead. You can take a look at available cameras here. Instead of, responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthVis)]) use this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis)]) Utility Methods # In earlier version, we provided several utility methods as part of AirSimClientBase . These methods are now moved to airsim namespace for more pythonic interface. Instead of, AirSimClientBase.write_png(my_path, img_rgba) AirSimClientBase.wait_key('Press any key') use this: airsim.write_png(my_path, img_rgba) airsim.wait_key('Press any key') Camera Names # AirSim now uses names to reference cameras instead of index numbers. However to retain backward compatibility, these names are aliased with old index numbers as string. Instead of, client.simGetCameraInfo(0) use this: client.simGetCameraInfo(\"0\") # or client.simGetCameraInfo(\"front-center\") Async Methods # For multirotors, AirSim had various methods such as takeoff or moveByVelocityZ that would take long time to complete. All of such methods are now renamed by adding the suffix Async as shown below. Instead of, client.takeoff() client.moveToPosition(-10, 10, -10, 5) use this: client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() Here .join() is a call on Python's Future class to wait for the async call to complete. You can also choose to do some other computation instead while the call is in progress. Simulation-Only Methods # Now we have clear distinction between methods that are only available in simulation from the ones that may be available on actual vehicle. The simulation only methods are prefixed with sim as shown below. getCollisionInfo() is renamed to simGetCollisionInfo() getCameraInfo() is renamed to simGetCameraInfo() setCameraOrientation() is renamed to simSetCameraOrientation() State Information # Previously CarState mixed simulation-only information like kinematics_true . Moving forward, CarState will only contain information that can be obtained in real world. k = car_state.kinematics_true use this: k = car_state.kinematics_estimated # or k = client.simGetGroundTruthKinematics()","title":"Upgrading APIs"},{"location":"upgrade_apis/#upgrading-api-client-code","text":"There have been several API changes in AirSim v1.2 that we hope removes inconsistency, adds future extensibility and presents cleaner interface. Many of these changes are however breaking changes which means you will need to modify your client code that talks to AirSim.","title":"Upgrading API Client Code"},{"location":"upgrade_apis/#quicker-way","text":"While most changes you need to do in your client code are fairly easy, a quicker way is simply to take a look at the example code such as Hello Drone or Hello Car to get gist of changes.","title":"Quicker Way"},{"location":"upgrade_apis/#importing-airsim","text":"Instead of, from AirSimClient import * use this: import airsim Above assumes you have installed AirSim module using, pip install --user airsim If you are running you code from PythonClient folder in repo then you can also do this: import setup_path import airsim Here setup_path.py should exist in your folder and it will set the path of airsim package in PythonClient repo folder. All examples in PythonClient folder uses this method.","title":"Importing AirSim"},{"location":"upgrade_apis/#using-airsim-classes","text":"As we have everything now in package, you will need to use explicit namespace for AirSim classes like shown below. Instead of, client1 = CarClient() use this: client1 = airsim.CarClient()","title":"Using AirSim Classes"},{"location":"upgrade_apis/#airsim-types","text":"We have moved all types in airsim namespace. Instead of, image_type = AirSimImageType.DepthVis d = DrivetrainType.MaxDegreeOfFreedom use this: image_type = airsim.ImageType.DepthVis d = airsim.DrivetrainType.MaxDegreeOfFreedom","title":"AirSim Types"},{"location":"upgrade_apis/#getting-images","text":"Nothing new below, it's just combination of above. Note that all APIs that previously took camera_id , now takes camera_name instead. You can take a look at available cameras here. Instead of, responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthVis)]) use this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis)])","title":"Getting Images"},{"location":"upgrade_apis/#utility-methods","text":"In earlier version, we provided several utility methods as part of AirSimClientBase . These methods are now moved to airsim namespace for more pythonic interface. Instead of, AirSimClientBase.write_png(my_path, img_rgba) AirSimClientBase.wait_key('Press any key') use this: airsim.write_png(my_path, img_rgba) airsim.wait_key('Press any key')","title":"Utility Methods"},{"location":"upgrade_apis/#camera-names","text":"AirSim now uses names to reference cameras instead of index numbers. However to retain backward compatibility, these names are aliased with old index numbers as string. Instead of, client.simGetCameraInfo(0) use this: client.simGetCameraInfo(\"0\") # or client.simGetCameraInfo(\"front-center\")","title":"Camera Names"},{"location":"upgrade_apis/#async-methods","text":"For multirotors, AirSim had various methods such as takeoff or moveByVelocityZ that would take long time to complete. All of such methods are now renamed by adding the suffix Async as shown below. Instead of, client.takeoff() client.moveToPosition(-10, 10, -10, 5) use this: client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() Here .join() is a call on Python's Future class to wait for the async call to complete. You can also choose to do some other computation instead while the call is in progress.","title":"Async Methods"},{"location":"upgrade_apis/#simulation-only-methods","text":"Now we have clear distinction between methods that are only available in simulation from the ones that may be available on actual vehicle. The simulation only methods are prefixed with sim as shown below. getCollisionInfo() is renamed to simGetCollisionInfo() getCameraInfo() is renamed to simGetCameraInfo() setCameraOrientation() is renamed to simSetCameraOrientation()","title":"Simulation-Only Methods"},{"location":"upgrade_apis/#state-information","text":"Previously CarState mixed simulation-only information like kinematics_true . Moving forward, CarState will only contain information that can be obtained in real world. k = car_state.kinematics_true use this: k = car_state.kinematics_estimated # or k = client.simGetGroundTruthKinematics()","title":"State Information"},{"location":"upgrade_settings/","text":"Upgrading Settings # The settings schema in AirSim 1.2 is changed for more flexibility and cleaner interface. If you have older settings.json file then you can either delete it and restart AirSim or use this guide to make manual upgrade. Quicker Way # We recommend simply deleting the settings.json and add back the settings you need. Please see the doc for complete information on available settings. Changes # UsageScenario # Previously we used UsageScenario to specify the ComputerVision mode. Now we use \"SimMode\": \"ComputerVision\" instead. CameraDefaults and Changing Camera Settings # Previously we had CaptureSettings and NoiseSettings in root. Now these are combined in new CameraDefaults element. The schema for this element is later used to configure cameras on vehicle. Gimbal # The Gimbal element (instead of old Gimble element) is now moved out of CaptureSettings . CameraID to CameraName # All settings now reference cameras by name instead of ID. Using PX4 # The new Vehicles element allows to specify which vehicles to create. To use PX4, please see this section . AdditionalCameras # The old AdditionalCameras setting is now replaced by Cameras element within vehicle setting.","title":"Upgrading Settings"},{"location":"upgrade_settings/#upgrading-settings","text":"The settings schema in AirSim 1.2 is changed for more flexibility and cleaner interface. If you have older settings.json file then you can either delete it and restart AirSim or use this guide to make manual upgrade.","title":"Upgrading Settings"},{"location":"upgrade_settings/#quicker-way","text":"We recommend simply deleting the settings.json and add back the settings you need. Please see the doc for complete information on available settings.","title":"Quicker Way"},{"location":"upgrade_settings/#changes","text":"","title":"Changes"},{"location":"upgrade_settings/#usagescenario","text":"Previously we used UsageScenario to specify the ComputerVision mode. Now we use \"SimMode\": \"ComputerVision\" instead.","title":"UsageScenario"},{"location":"upgrade_settings/#cameradefaults-and-changing-camera-settings","text":"Previously we had CaptureSettings and NoiseSettings in root. Now these are combined in new CameraDefaults element. The schema for this element is later used to configure cameras on vehicle.","title":"CameraDefaults and Changing Camera Settings"},{"location":"upgrade_settings/#gimbal","text":"The Gimbal element (instead of old Gimble element) is now moved out of CaptureSettings .","title":"Gimbal"},{"location":"upgrade_settings/#cameraid-to-cameraname","text":"All settings now reference cameras by name instead of ID.","title":"CameraID to CameraName"},{"location":"upgrade_settings/#using-px4","text":"The new Vehicles element allows to specify which vehicles to create. To use PX4, please see this section .","title":"Using PX4"},{"location":"upgrade_settings/#additionalcameras","text":"The old AdditionalCameras setting is now replaced by Cameras element within vehicle setting.","title":"AdditionalCameras"},{"location":"use_precompiled/","text":"Download Binaries # You can simply download precompiled binaries and run to get started immediately. If you want to set up your own Unreal environment then please see these instructions . Unreal Engine # Windows, Linux : Download the binaries for the environment of your choice from the latest release . macOS : You will need to build it yourself Unity (Experimental) # A free environment called Windridge City is available at Unity Asset Store as an experimental release of AirSim on Unity. Note : This is an old release, and many of the features and APIs might not work. Controlling Vehicles # Most of our users typically use APIs to control the vehicles. However if you can also control vehicles manually. You can drive the car using keyboard, gamepad or steering wheel . To fly drone manually, you will need either XBox controller or a remote control (feel free to contribute keyboard support). Please see remote control setup for more details. Alternatively you can use APIs for programmatic control or use so-called Computer Vision mode to move around in environment using the keyboard. Don't Have Good GPU? # The AirSim binaries, like CityEnviron, requires a beefy GPU to run smoothly. You can run them in low resolution mode by editing the run.bat file on Windows like this: start CityEnviron -ResX=640 -ResY=480 -windowed For Linux binaries, use the Blocks.sh or corresponding shell script as follows - ./Blocks.sh -ResX=640 -ResY=480 -windowed UE 4.24 uses Vulkan drivers by default, but they can consume more GPU memory. If you get memory allocation errors, then you can try switching to OpenGL using -opengl","title":"Download Binaries"},{"location":"use_precompiled/#download-binaries","text":"You can simply download precompiled binaries and run to get started immediately. If you want to set up your own Unreal environment then please see these instructions .","title":"Download Binaries"},{"location":"use_precompiled/#unreal-engine","text":"Windows, Linux : Download the binaries for the environment of your choice from the latest release . macOS : You will need to build it yourself","title":"Unreal Engine"},{"location":"use_precompiled/#unity-experimental","text":"A free environment called Windridge City is available at Unity Asset Store as an experimental release of AirSim on Unity. Note : This is an old release, and many of the features and APIs might not work.","title":"Unity (Experimental)"},{"location":"use_precompiled/#controlling-vehicles","text":"Most of our users typically use APIs to control the vehicles. However if you can also control vehicles manually. You can drive the car using keyboard, gamepad or steering wheel . To fly drone manually, you will need either XBox controller or a remote control (feel free to contribute keyboard support). Please see remote control setup for more details. Alternatively you can use APIs for programmatic control or use so-called Computer Vision mode to move around in environment using the keyboard.","title":"Controlling Vehicles"},{"location":"use_precompiled/#dont-have-good-gpu","text":"The AirSim binaries, like CityEnviron, requires a beefy GPU to run smoothly. You can run them in low resolution mode by editing the run.bat file on Windows like this: start CityEnviron -ResX=640 -ResY=480 -windowed For Linux binaries, use the Blocks.sh or corresponding shell script as follows - ./Blocks.sh -ResX=640 -ResY=480 -windowed UE 4.24 uses Vulkan drivers by default, but they can consume more GPU memory. If you get memory allocation errors, then you can try switching to OpenGL using -opengl","title":"Don't Have Good GPU?"},{"location":"using_car/","text":"How to Use Car in AirSim # By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . For example, if you want to use car instead then just set the SimMode in your settings.json which you can find in your ~/Documents/AirSim folder, like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } Now when you restart AirSim, you should see the car spawned automatically. Manual Driving # Please use the keyboard arrow keys to drive manually. Spacebar for the handbrake. In manual drive mode, gears are set in \"auto\". Using APIs # You can control the car, get state and images by calling APIs in variety of client languages including C++ and Python. Please see APIs doc for more details. Changing Views # By default camera will chase the car from the back. You can get the FPV view by pressing F key and switch back to chasing from back view by pressing / key. More keyboard shortcuts can be seen by pressing F1. Cameras # By default car is installed with 5 cameras: center, left and right, driver and reverse. You can chose the images from these camera by specifying the name .","title":"Car Mode"},{"location":"using_car/#how-to-use-car-in-airsim","text":"By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . For example, if you want to use car instead then just set the SimMode in your settings.json which you can find in your ~/Documents/AirSim folder, like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } Now when you restart AirSim, you should see the car spawned automatically.","title":"How to Use Car in AirSim"},{"location":"using_car/#manual-driving","text":"Please use the keyboard arrow keys to drive manually. Spacebar for the handbrake. In manual drive mode, gears are set in \"auto\".","title":"Manual Driving"},{"location":"using_car/#using-apis","text":"You can control the car, get state and images by calling APIs in variety of client languages including C++ and Python. Please see APIs doc for more details.","title":"Using APIs"},{"location":"using_car/#changing-views","text":"By default camera will chase the car from the back. You can get the FPV view by pressing F key and switch back to chasing from back view by pressing / key. More keyboard shortcuts can be seen by pressing F1.","title":"Changing Views"},{"location":"using_car/#cameras","text":"By default car is installed with 5 cameras: center, left and right, driver and reverse. You can chose the images from these camera by specifying the name .","title":"Cameras"},{"location":"who_is_using/","text":"Who is Using AirSim? # Would you like to see your own group or project here? # Just add a GitHub issue with quick details and link to your website or paper or send us email at msrair at microsoft.com. NASA Ames Research Center \u2013 Systems Analysis Office Astrobotic GRASP Lab, Univ of Pennsylvania Department of Aeronautics and Astronautics, Stanford University Formula Technion Ghent University ICARUS UC, Santa Barbara WISE Lab, Univ of Waterloo HAMS project, MSR India Washington and Lee University University of Oklahoma Robotics Institute, Carnegie Mellon University Texas A&M Robotics and Perception Group, University of Zurich National University of Ireland, Galway (NUIG) Soda Mobility Technologies","title":"Who is Using AirSim"},{"location":"who_is_using/#who-is-using-airsim","text":"","title":"Who is Using AirSim?"},{"location":"who_is_using/#would-you-like-to-see-your-own-group-or-project-here","text":"Just add a GitHub issue with quick details and link to your website or paper or send us email at msrair at microsoft.com. NASA Ames Research Center \u2013 Systems Analysis Office Astrobotic GRASP Lab, Univ of Pennsylvania Department of Aeronautics and Astronautics, Stanford University Formula Technion Ghent University ICARUS UC, Santa Barbara WISE Lab, Univ of Waterloo HAMS project, MSR India Washington and Lee University University of Oklahoma Robotics Institute, Carnegie Mellon University Texas A&M Robotics and Perception Group, University of Zurich National University of Ireland, Galway (NUIG) Soda Mobility Technologies","title":"Would you like to see your own group or project here?"},{"location":"working_with_plugin_contents/","text":"How to use plugin contents # Plugin contents are not shown in Unreal projects by default. To view plugin content, you need to click on few semi-hidden buttons: Causion Changes you make in content folder are changes to binary files so be careful.","title":"Working with UE Plugin Contents"},{"location":"working_with_plugin_contents/#how-to-use-plugin-contents","text":"Plugin contents are not shown in Unreal projects by default. To view plugin content, you need to click on few semi-hidden buttons: Causion Changes you make in content folder are changes to binary files so be careful.","title":"How to use plugin contents"},{"location":"xbox_controller/","text":"XBox Controller # To use an XBox controller with AirSim follow these steps: Connect XBox controller so it shows up in your PC Game Controllers: Launch QGroundControl and you should see a new Joystick tab under settings: Now calibrate the radio, and setup some handy button actions. For example, I set mine so that the 'A' button arms the drone, 'B' put it in manual flight mode, 'X' puts it in altitude hold mode and 'Y' puts it in position hold mode. I also prefer the feel of the controller when I check the box labelled \"Use exponential curve on roll,pitch, yaw\" because this gives me more sensitivity for small movements. QGroundControl will find your Pixhawk via the UDP proxy port 14550 setup by MavLinkTest above. AirSim will find your Pixhawk via the other UDP server port 14570 also setup by MavLinkTest above. You can also use all the QGroundControl controls for autonomous flying at this point too. Connect to Pixhawk serial port using MavLinkTest.exe like this: MavLinkTest.exe -serial:*,115200 -proxy:127.0.0.1:14550 -server:127.0.0.1:14570 Run AirSim Unreal simulator with these ~/Documents/AirSim/settings.json settings: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"SitlIp\": \"\", \"SitlPort\": 14560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14570, \"UseSerial\": false } } Advanced # If the Joystick tab doesn't show up in QGroundControl then Click on the purple \"Q\" icon on left in tool bar to reveal the Preferences panel. Go to General tab and check the Virtual Joystick checkbox. Go back to settings screen (gears icon), click on Parameters tab, type COM_RC_IN_MODE in search box and change its value to either Joystick/No RC Checks or Virtual RC by Joystick . Other Options # See remote controller options","title":"XBox Controller"},{"location":"xbox_controller/#xbox-controller","text":"To use an XBox controller with AirSim follow these steps: Connect XBox controller so it shows up in your PC Game Controllers: Launch QGroundControl and you should see a new Joystick tab under settings: Now calibrate the radio, and setup some handy button actions. For example, I set mine so that the 'A' button arms the drone, 'B' put it in manual flight mode, 'X' puts it in altitude hold mode and 'Y' puts it in position hold mode. I also prefer the feel of the controller when I check the box labelled \"Use exponential curve on roll,pitch, yaw\" because this gives me more sensitivity for small movements. QGroundControl will find your Pixhawk via the UDP proxy port 14550 setup by MavLinkTest above. AirSim will find your Pixhawk via the other UDP server port 14570 also setup by MavLinkTest above. You can also use all the QGroundControl controls for autonomous flying at this point too. Connect to Pixhawk serial port using MavLinkTest.exe like this: MavLinkTest.exe -serial:*,115200 -proxy:127.0.0.1:14550 -server:127.0.0.1:14570 Run AirSim Unreal simulator with these ~/Documents/AirSim/settings.json settings: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"SitlIp\": \"\", \"SitlPort\": 14560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14570, \"UseSerial\": false } }","title":"XBox Controller"},{"location":"xbox_controller/#advanced","text":"If the Joystick tab doesn't show up in QGroundControl then Click on the purple \"Q\" icon on left in tool bar to reveal the Preferences panel. Go to General tab and check the Virtual Joystick checkbox. Go back to settings screen (gears icon), click on Parameters tab, type COM_RC_IN_MODE in search box and change its value to either Joystick/No RC Checks or Virtual RC by Joystick .","title":"Advanced"},{"location":"xbox_controller/#other-options","text":"See remote controller options","title":"Other Options"}]} \ No newline at end of file +{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to AirSim # AirSim is a simulator for drones, cars and more, built on Unreal Engine (we now also have an experimental Unity release). It is open-source, cross platform, and supports hardware-in-loop with popular flight controllers such as PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. Similarly, we have an experimental release for a Unity plugin. Our goal is to develop AirSim as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles. For this purpose, AirSim also exposes APIs to retrieve data and control vehicles in a platform independent way. Check out the quick 1.5 minute demo Drones in AirSim Cars in AirSim What's New # Latest release v1.3.1 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added support for docker in ubuntu For complete list of changes, view our Changelog How to Get It # Windows # Download binaries Build it Linux # Download binaries Build it macOS # Build it How to Use It # Documentation # View our detailed documentation on all aspects of AirSim. Manual drive # If you have remote control (RC) as shown below, you can manually control the drone in the simulator. For cars, you can use arrow keys to drive manually. More details Programmatic control # AirSim exposes APIs so you can interact with the vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on. The APIs are exposed through the RPC, and are accessible via a variety of languages, including C++, Python, C# and Java. These APIs are also available as part of a separate, independent cross-platform library, so you can deploy them on a companion computer on your vehicle. This way you can write and test your code in the simulator, and later execute it on the real vehicles. Transfer learning and related research is one of our focus areas. Note that you can use SimMode setting to specify the default vehicle or the new ComputerVision mode so you don't get prompted each time you start AirSim. More details Gathering training data # There are two ways you can generate training data from AirSim for deep learning. The easiest way is to simply press the record button in the lower right corner. This will start writing pose and images for each frame. The data logging code is pretty simple and you can modify it to your heart's content. A better way to generate training data exactly the way you want is by accessing the APIs. This allows you to be in full control of how, what, where and when you want to log data. Computer Vision mode # Yet another way to use AirSim is the so-called \"Computer Vision\" mode. In this mode, you don't have vehicles or physics. You can use the keyboard to move around the scene, or use APIs to position available cameras in any arbitrary pose, and collect images such as depth, disparity, surface normals or object segmentation. More details Weather Effects # Press F10 to see various options available for weather effects. You can also control the weather using APIs . Press F1 to see other options available. Tutorials # Video - Setting up AirSim with Pixhawk Tutorial by Chris Lovett Video - Using AirSim with Pixhawk Tutorial by Chris Lovett Video - Using off-the-self environments with AirSim by Jim Piavis Reinforcement Learning with AirSim by Ashish Kapoor The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter Using TensorFlow for simple collision avoidance by Simon Levy and WLU team Participate # Paper # More technical details are available in AirSim paper (FSR 2017 Conference) . Please cite this as: @inproceedings{airsim2017fsr, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, title = {AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles}, year = {2017}, booktitle = {Field and Service Robotics}, eprint = {arXiv:1705.05065}, url = {https://arxiv.org/abs/1705.05065} } Contribute # Please take a look at open issues if you are looking for areas to contribute to. More on AirSim design More on code structure Contribution Guidelines Trello Board Who is Using AirSim? # We are maintaining a list of a few projects, people and groups that we are aware of. If you would like to be featured in this list please make a request here . Contact # Join the AirSim group on Facebook to stay up to date or ask any questions. FAQ # If you run into problems, check the FAQ and feel free to post issues in the AirSim repository. License # This project is released under the MIT License. Please review the License file for more details.","title":"Home"},{"location":"#welcome-to-airsim","text":"AirSim is a simulator for drones, cars and more, built on Unreal Engine (we now also have an experimental Unity release). It is open-source, cross platform, and supports hardware-in-loop with popular flight controllers such as PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. Similarly, we have an experimental release for a Unity plugin. Our goal is to develop AirSim as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles. For this purpose, AirSim also exposes APIs to retrieve data and control vehicles in a platform independent way. Check out the quick 1.5 minute demo Drones in AirSim Cars in AirSim","title":"Welcome to AirSim"},{"location":"#whats-new","text":"Latest release v1.3.1 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added support for docker in ubuntu For complete list of changes, view our Changelog","title":"What's New"},{"location":"#how-to-get-it","text":"","title":"How to Get It"},{"location":"#windows","text":"Download binaries Build it","title":"Windows"},{"location":"#linux","text":"Download binaries Build it","title":"Linux"},{"location":"#macos","text":"Build it","title":"macOS"},{"location":"#how-to-use-it","text":"","title":"How to Use It"},{"location":"#documentation","text":"View our detailed documentation on all aspects of AirSim.","title":"Documentation"},{"location":"#manual-drive","text":"If you have remote control (RC) as shown below, you can manually control the drone in the simulator. For cars, you can use arrow keys to drive manually. More details","title":"Manual drive"},{"location":"#programmatic-control","text":"AirSim exposes APIs so you can interact with the vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on. The APIs are exposed through the RPC, and are accessible via a variety of languages, including C++, Python, C# and Java. These APIs are also available as part of a separate, independent cross-platform library, so you can deploy them on a companion computer on your vehicle. This way you can write and test your code in the simulator, and later execute it on the real vehicles. Transfer learning and related research is one of our focus areas. Note that you can use SimMode setting to specify the default vehicle or the new ComputerVision mode so you don't get prompted each time you start AirSim. More details","title":"Programmatic control"},{"location":"#gathering-training-data","text":"There are two ways you can generate training data from AirSim for deep learning. The easiest way is to simply press the record button in the lower right corner. This will start writing pose and images for each frame. The data logging code is pretty simple and you can modify it to your heart's content. A better way to generate training data exactly the way you want is by accessing the APIs. This allows you to be in full control of how, what, where and when you want to log data.","title":"Gathering training data"},{"location":"#computer-vision-mode","text":"Yet another way to use AirSim is the so-called \"Computer Vision\" mode. In this mode, you don't have vehicles or physics. You can use the keyboard to move around the scene, or use APIs to position available cameras in any arbitrary pose, and collect images such as depth, disparity, surface normals or object segmentation. More details","title":"Computer Vision mode"},{"location":"#weather-effects","text":"Press F10 to see various options available for weather effects. You can also control the weather using APIs . Press F1 to see other options available.","title":"Weather Effects"},{"location":"#tutorials","text":"Video - Setting up AirSim with Pixhawk Tutorial by Chris Lovett Video - Using AirSim with Pixhawk Tutorial by Chris Lovett Video - Using off-the-self environments with AirSim by Jim Piavis Reinforcement Learning with AirSim by Ashish Kapoor The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter Using TensorFlow for simple collision avoidance by Simon Levy and WLU team","title":"Tutorials"},{"location":"#participate","text":"","title":"Participate"},{"location":"#paper","text":"More technical details are available in AirSim paper (FSR 2017 Conference) . Please cite this as: @inproceedings{airsim2017fsr, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, title = {AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles}, year = {2017}, booktitle = {Field and Service Robotics}, eprint = {arXiv:1705.05065}, url = {https://arxiv.org/abs/1705.05065} }","title":"Paper"},{"location":"#contribute","text":"Please take a look at open issues if you are looking for areas to contribute to. More on AirSim design More on code structure Contribution Guidelines Trello Board","title":"Contribute"},{"location":"#who-is-using-airsim","text":"We are maintaining a list of a few projects, people and groups that we are aware of. If you would like to be featured in this list please make a request here .","title":"Who is Using AirSim?"},{"location":"#contact","text":"Join the AirSim group on Facebook to stay up to date or ask any questions.","title":"Contact"},{"location":"#faq","text":"If you run into problems, check the FAQ and feel free to post issues in the AirSim repository.","title":"FAQ"},{"location":"#license","text":"This project is released under the MIT License. Please review the License file for more details.","title":"License"},{"location":"CHANGELOG/","text":"What's new # Below is summarized list of important changes. This does not include minor/less important changes or bug fixes or documentation update. This list updated every few months. For complete detailed changes, please review commit history . April 2020 # Fix issues with PX4 latest master branch Fix Lidar DrawDebugPoints causing crash Add docstrings for Python API Add missing noise, weather texture materials Update AirSim.uplugin version to 1.3.1 Camera Roll angle control using Q,E keys in CV mode, manual camera Remove broken GCC build New API - simSetTraceLine() ROS package compilation fixes and updates Latest release v1.3.1 for Windows and Linux APIs added and fixed - simSetCameraFov , rotateToYaw airsim Python package update to 1.2.8 NoDisplay ViewMode render state fix March 2020 # Latest release v1.3.0 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging Low-level Multirotor APIs Updated Eigen version to 3.3.7 Distance Sensor API fix Add simSwapTextures API Fix simContinueForTime , simPause APIs Lidar Sensor Trace Casting fix Fix rare reset() bug which causes Unreal crash Lidar sensor improvements, add simGetLidarSegmentation API Add RpcLibPort in settings Recording thread deadlock fix Prevent environment crash when Sun is not present Africa Tracking feautre, add simListSceneObjects() API, fix camera projection matrix ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added sensor APIs for Barometer, IMU, GPS, Magnetometer, Distance Sensor Added support for docker in ubuntu November, 2018 # Added Weather Effects and APIs Added Time of Day API An experimental integration of AirSim on Unity is now available. Learn more in Unity blog post . New environments : Forest, Plains (windmill farm), TalkingHeads (human head simulation), TrapCam (animal detection via camera) Highly efficient NoDisplay view mode to turn off main screen rendering so you can capture images at high rate Enable/disable sensors via settings Lidar Sensor Support for Flysky FS-SM100 RC USB adapter Case Study: Formula Student Technion Driverless Multi-Vehicle Capability Custom speed units ROS publisher simSetObjectPose API Character Control APIs (works on TalkingHeads binaries in release) Arducopter Solo Support Linux install without sudo access Kinect like ROS publisher June, 2018 # Development workflow doc Better Python 2 compatibility OSX setup fixes Almost complete rewrite of our APIs with new threading model, merging old APIs and creating few newer ones April, 2018 # Upgraded to Unreal Engine 4.18 and Visual Studio 2017 API framework refactoring to support world-level APIs Latest PX4 firmware now supported CarState with more information ThrustMaster wheel support pause and continueForTime APIs for drone as well as car Allow drone simulation run at higher clock rate without any degradation Forward-only mode fully functional for drone (do orbits while looking at center) Better PID tuning to reduce wobble for drones Ability to set arbitrary vehicle blueprint for drone as well as car gimbal stabilization via settings Ability to segment skinned and skeletal meshes by their name moveByAngleThrottle API Car physics tuning for better maneuverability Configure additional cameras via settings Time of day with geographically computed sun position Better car steering via keyboard Added MeshNamingMethod in segmentation setting gimbal API getCameraParameters API Ability turn off main rendering to save GPU resources Projection mode for capture settings getRCData, setRCData APIs Ability to turn off segmentation using negative IDs OSX build improvements Segmentation working for very large environments with initial IDs Better and extensible hash calculation for segmentation IDs Extensible PID controller for custom integration methods Sensor architecture now enables renderer specific features like ray casting Laser altimeter sensor Jan 2018 # Config system rewrite, enable flexible config we are targeting in future Multi-Vehicle support Phase 1, core infrastructure changes MacOS support Infrared view 5 types of noise and interference for cameras WYSIWIG capture settings for cameras, preview recording settings in main view Azure support Phase 1, enable configurability of instances for headless mode Full kinematics APIs, ability to get pose, linear and angular velocities + accelerations via APIs Record multiple images from multiple cameras New segmentation APIs, ability to set configure object IDs, search via regex New object pose APIs, ability to get pose of objects (like animals) in environment Camera infrastructure enhancements, ability to add new image types like IR with just few lines Clock speed APIs for drone as well as car, simulation can be run with speed factor of 0 < x < infinity Support for Logitech G920 wheel Physics tuning of the car, Car doesn\u2019t roll over, responds to steering with better curve, releasing gas paddle behavior more realistic Debugging APIs Stress tested to 24+ hours of continuous runs Support for Landscape and sky segmentation Manual navigation with accelerated controls in CV mode, user can explore environment much more easily Collison APIs Recording enhancements, log several new data points including ground truth, multiple images, controls state Planner and Perspective Depth views Disparity view New Image APIs supports float, png or numpy formats 6 config settings for image capture, ability to set auto-exposure, motion blur, gamma etc Full multi-camera support through out including sub-windows, recording, APIs etc Command line script to build all environments in one shot Remove submodules, use rpclib as download Nov 2017 # We now have the car model . No need to build the code. Just download binaries and you are good to go! The reinforcement learning example with AirSim New built-in flight controller called simple_flight that \"just works\" without any additional setup. It is also now default . AirSim now also generates depth as well as disparity images that is in camera plan. We also have official Linux build now! Sep 2017 # We have added car model ! Aug 2017 # simple_flight is now default flight controller for drones. If you want to use PX4, you will need to modify settings.json as per PX4 setup doc . Linux build is official and currently uses Unreal 4.17 due to various bug fixes required ImageType enum has breaking changes with several new additions and clarifying existing ones SubWindows are now configurable from settings.json PythonClient is now complete and has parity with C++ APIs. Some of these would have breaking changes. Feb 2017 # First release!","title":"Changelog"},{"location":"CHANGELOG/#whats-new","text":"Below is summarized list of important changes. This does not include minor/less important changes or bug fixes or documentation update. This list updated every few months. For complete detailed changes, please review commit history .","title":"What's new"},{"location":"CHANGELOG/#april-2020","text":"Fix issues with PX4 latest master branch Fix Lidar DrawDebugPoints causing crash Add docstrings for Python API Add missing noise, weather texture materials Update AirSim.uplugin version to 1.3.1 Camera Roll angle control using Q,E keys in CV mode, manual camera Remove broken GCC build New API - simSetTraceLine() ROS package compilation fixes and updates Latest release v1.3.1 for Windows and Linux APIs added and fixed - simSetCameraFov , rotateToYaw airsim Python package update to 1.2.8 NoDisplay ViewMode render state fix","title":"April 2020"},{"location":"CHANGELOG/#march-2020","text":"Latest release v1.3.0 for Windows and Linux Upgraded to Unreal Engine 4.24, Visual Studio 2019, Clang 8, C++ 17 standard Mac OSX Catalina support Updated airsim Python package, with lots of new APIs Removed legacy API wrappers Support for latest PX4 stable release Support for ArduPilot - Copter, Rover vehicles Updated Unity support Removed simChar* APIs Plotting APIs for Debugging Low-level Multirotor APIs Updated Eigen version to 3.3.7 Distance Sensor API fix Add simSwapTextures API Fix simContinueForTime , simPause APIs Lidar Sensor Trace Casting fix Fix rare reset() bug which causes Unreal crash Lidar sensor improvements, add simGetLidarSegmentation API Add RpcLibPort in settings Recording thread deadlock fix Prevent environment crash when Sun is not present Africa Tracking feautre, add simListSceneObjects() API, fix camera projection matrix ROS wrapper for multirotors is available. See airsim_ros_pkgs for the ROS API, and airsim_tutorial_pkgs for tutorials. Added sensor APIs for Barometer, IMU, GPS, Magnetometer, Distance Sensor Added support for docker in ubuntu","title":"March 2020"},{"location":"CHANGELOG/#november-2018","text":"Added Weather Effects and APIs Added Time of Day API An experimental integration of AirSim on Unity is now available. Learn more in Unity blog post . New environments : Forest, Plains (windmill farm), TalkingHeads (human head simulation), TrapCam (animal detection via camera) Highly efficient NoDisplay view mode to turn off main screen rendering so you can capture images at high rate Enable/disable sensors via settings Lidar Sensor Support for Flysky FS-SM100 RC USB adapter Case Study: Formula Student Technion Driverless Multi-Vehicle Capability Custom speed units ROS publisher simSetObjectPose API Character Control APIs (works on TalkingHeads binaries in release) Arducopter Solo Support Linux install without sudo access Kinect like ROS publisher","title":"November, 2018"},{"location":"CHANGELOG/#june-2018","text":"Development workflow doc Better Python 2 compatibility OSX setup fixes Almost complete rewrite of our APIs with new threading model, merging old APIs and creating few newer ones","title":"June, 2018"},{"location":"CHANGELOG/#april-2018","text":"Upgraded to Unreal Engine 4.18 and Visual Studio 2017 API framework refactoring to support world-level APIs Latest PX4 firmware now supported CarState with more information ThrustMaster wheel support pause and continueForTime APIs for drone as well as car Allow drone simulation run at higher clock rate without any degradation Forward-only mode fully functional for drone (do orbits while looking at center) Better PID tuning to reduce wobble for drones Ability to set arbitrary vehicle blueprint for drone as well as car gimbal stabilization via settings Ability to segment skinned and skeletal meshes by their name moveByAngleThrottle API Car physics tuning for better maneuverability Configure additional cameras via settings Time of day with geographically computed sun position Better car steering via keyboard Added MeshNamingMethod in segmentation setting gimbal API getCameraParameters API Ability turn off main rendering to save GPU resources Projection mode for capture settings getRCData, setRCData APIs Ability to turn off segmentation using negative IDs OSX build improvements Segmentation working for very large environments with initial IDs Better and extensible hash calculation for segmentation IDs Extensible PID controller for custom integration methods Sensor architecture now enables renderer specific features like ray casting Laser altimeter sensor","title":"April, 2018"},{"location":"CHANGELOG/#jan-2018","text":"Config system rewrite, enable flexible config we are targeting in future Multi-Vehicle support Phase 1, core infrastructure changes MacOS support Infrared view 5 types of noise and interference for cameras WYSIWIG capture settings for cameras, preview recording settings in main view Azure support Phase 1, enable configurability of instances for headless mode Full kinematics APIs, ability to get pose, linear and angular velocities + accelerations via APIs Record multiple images from multiple cameras New segmentation APIs, ability to set configure object IDs, search via regex New object pose APIs, ability to get pose of objects (like animals) in environment Camera infrastructure enhancements, ability to add new image types like IR with just few lines Clock speed APIs for drone as well as car, simulation can be run with speed factor of 0 < x < infinity Support for Logitech G920 wheel Physics tuning of the car, Car doesn\u2019t roll over, responds to steering with better curve, releasing gas paddle behavior more realistic Debugging APIs Stress tested to 24+ hours of continuous runs Support for Landscape and sky segmentation Manual navigation with accelerated controls in CV mode, user can explore environment much more easily Collison APIs Recording enhancements, log several new data points including ground truth, multiple images, controls state Planner and Perspective Depth views Disparity view New Image APIs supports float, png or numpy formats 6 config settings for image capture, ability to set auto-exposure, motion blur, gamma etc Full multi-camera support through out including sub-windows, recording, APIs etc Command line script to build all environments in one shot Remove submodules, use rpclib as download","title":"Jan 2018"},{"location":"CHANGELOG/#nov-2017","text":"We now have the car model . No need to build the code. Just download binaries and you are good to go! The reinforcement learning example with AirSim New built-in flight controller called simple_flight that \"just works\" without any additional setup. It is also now default . AirSim now also generates depth as well as disparity images that is in camera plan. We also have official Linux build now!","title":"Nov 2017"},{"location":"CHANGELOG/#sep-2017","text":"We have added car model !","title":"Sep 2017"},{"location":"CHANGELOG/#aug-2017","text":"simple_flight is now default flight controller for drones. If you want to use PX4, you will need to modify settings.json as per PX4 setup doc . Linux build is official and currently uses Unreal 4.17 due to various bug fixes required ImageType enum has breaking changes with several new additions and clarifying existing ones SubWindows are now configurable from settings.json PythonClient is now complete and has parity with C++ APIs. Some of these would have breaking changes.","title":"Aug 2017"},{"location":"CHANGELOG/#feb-2017","text":"First release!","title":"Feb 2017"},{"location":"CONTRIBUTING/","text":"Contributing # Quick Start # Please read our short and sweet coding guidelines . For big changes such as adding new feature or refactoring, file an issue first . Use our recommended development workflow to make changes and test it. Use usual steps to make contributions just like other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first . Checklist # Use same style and formatting as rest of code even if it's not your preferred one. Change any documentation that goes with code changes. Do not include OS specific header files or new 3rd party dependencies. Keep your pull request small, ideally under 10 files. Make sure you don't include large binary files. When adding new includes, make dependency is absolutely necessary. Rebase your branch frequently with master (once every 2-3 days is ideal). Make sure your code would compile on Windows, Linux and OSX.","title":"Contribute"},{"location":"CONTRIBUTING/#contributing","text":"","title":"Contributing"},{"location":"CONTRIBUTING/#quick-start","text":"Please read our short and sweet coding guidelines . For big changes such as adding new feature or refactoring, file an issue first . Use our recommended development workflow to make changes and test it. Use usual steps to make contributions just like other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first .","title":"Quick Start"},{"location":"CONTRIBUTING/#checklist","text":"Use same style and formatting as rest of code even if it's not your preferred one. Change any documentation that goes with code changes. Do not include OS specific header files or new 3rd party dependencies. Keep your pull request small, ideally under 10 files. Make sure you don't include large binary files. When adding new includes, make dependency is absolutely necessary. Rebase your branch frequently with master (once every 2-3 days is ideal). Make sure your code would compile on Windows, Linux and OSX.","title":"Checklist"},{"location":"InfraredCamera/","text":"This is a tutorial for generating simulated thermal infrared (IR) images using AirSim and the AirSim Africa environment. Pre-compiled Africa Environment can be downloaded from the Releases tab of this Github repo: Windows Pre-compiled binary To generate your own data, you may use two python files: create_ir_segmentation_map.py and capture_ir_segmentation.py . create_ir_segmentation_map.py uses temperature, emissivity, and camera response information to estimate the thermal digital count that could be expected for the objects in the environment, and then reassigns the segmentation IDs in AirSim to match these digital counts. It should be run before starting to capture thermal IR data. Otherwise, digital counts in the IR images will be incorrect. The camera response, temperature, and emissivity data are all included for the Africa environment. capture_ir_segmentation.py is run after the segmentation IDs have been reassigned. It tracks objects of interest and records the infrared and scene images from the multirotor. It uses Computer Vision mode. Finally, the details about how temperatures were estimated for plants and animals in the Africa environment, etc. can be found in this paper: @inproceedings{bondi2018airsim, title={AirSim-W: A Simulation Environment for Wildlife Conservation with UAVs}, author={Bondi, Elizabeth and Dey, Debadeepta and Kapoor, Ashish and Piavis, Jim and Shah, Shital and Fang, Fei and Dilkina, Bistra and Hannaford, Robert and Iyer, Arvind and Joppa, Lucas and others}, booktitle={Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies}, pages={40}, year={2018}, organization={ACM} } nb","title":"Infrared Camera"},{"location":"SUPPORT/","text":"Support # We highly recommend to take a look at source code and contribute to the project. Due to large number of incoming feature request we may not be able to get to your request in your desired timeframe. So please contribute :). Join AirSim Facebook Group File GitHub Issue","title":"Support"},{"location":"SUPPORT/#support","text":"We highly recommend to take a look at source code and contribute to the project. Due to large number of incoming feature request we may not be able to get to your request in your desired timeframe. So please contribute :). Join AirSim Facebook Group File GitHub Issue","title":"Support"},{"location":"Unity/","text":"AirSim on Unity # AirSim on Unity allows you to run your simulators in the Unity Engine . This project comes with some sample Unity projects and a wrapper around the AirLib library to run as a native plugin in Unity. Included are two basic Unity Projects, one for a Car simulator and another for a Drone simulator. They are meant to be lightweight, and can be used to verify your setup is correct. Check out the Unity blogpost for overview on the release. Warning: Experimental Release # This project is still in early development, expect some rough edges. We are working to fully support the full AirLib API and feature set, but some things may be missing. Click here for the list of currently supported APIs. Windows # Building from source # Install Unity # Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub from here . Detailed instructions here . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs. Build Airsim # Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start x64 Native Tools Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line. Build Unity Project # Go inside the AirSim\\Unity directory: cd Unity . Build the unity project: build.cmd . Additionally, there is a free environment Windridge City which you can download from Unity Asset Store . And, of course, you can always create your own environment. Linux # Dependencies # sudo apt-get install libboost-all-dev Download and Install Unity for Linux # Warning: Unity Editor for Linux is still in Beta. Expect some rough edges. Install Unity # Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs. Build Airsim # git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh Generate AirsimWrapper Shared Library # cd AirSim/Unity ./build.sh This will generate the necessary shared library and copy it to the UnityDemo Plugins folder. Usage # Start Unity Hub, click on Projects on left pane, and then click on the Add button Select the folder AirSim\\Unity\\UnityDemo , and then hit the OK button. Click on the new project which showed up in the Unity Hub menu to open it in Unity. In the bottom pane, click on Projects -> Assets -> Scenes . Then, Double-click on SimModeSelector . This will load the SimModeSelector scene into the scene hierarchy pane. DO NOT add CarDemo or DroneDemo scene into the scene hierarchy pane. Hit the play button to start the simulation (and hit play again to stop the simulation. . Alternatively, you can change the SimMode in your Settings.json file. (You can read more about Settings.json here ) Controlling the car: Use WASD or the Arrow keys or the AirSim client. Controlling the drone: Keyboard control is not currently available for drone flight. Changing camera views: Keys 0 , 1 , 2 , 3 are used to toggle windows of different camera views. Recording simulation data: Press Record button(Red button) located at the right bottom corner of the screen, to toggle recording of the simulation data. The recorded data can be found at Documents\\AirSim\\(Date of recording) on Windows and ~/Documents/AirSim/(Date of recording) on Linux. Building Custom Environments For AirSim # To use environments other than UnityDemo , follow the instructions written here Cross-Compiling to Linux # Unity Editor supports compiling projects to Linux systems. After following the steps to build AirSim and Unity on Windows, do the following: Linux Pre-Requisites # Before being able to run Unity Binaries with the Airsim plugin, be sure have airsim and airsim unity built on your linux machine by following the Linux build steps above. Package UnityDemo Binary On Windows # Install Necessary Components # In order to package your project for linux, the Linux Build Support Unity add-on must be installed. * Open Unity Hub , and click the Add component button in the dropdown window under more options to the right of your Unity 2018.2.15f1 tab. * Make sure the Linux Build Support Platform is selected Once this component is successfully installed, you are ready to build Unity Projects for Linux! Build the Project # On your Windows machine, build the Unity Demo by navigating to the build settings option in the toolbar File -> Build Settings Make sure the following scenes are set to be built: SimModeSelector CarDemo DroneDemo Set the target operating system to linux, and choose the version appropriate for your system (x86 vs x86_64) Click Build Transport the built project as well as the generated folder \"{project_name}_Data\" to your linux machine Copy The AirsimWrapper Library to the Project Plugins folder # On your linux machine, navigate to your AirSim repository, and run the following commands in a terminal window: cp Unity/linux-build/libAirsimWrapper.so path/to/your/project/{project_name}_Data/Plugins/{os_version} This will generate the necessary shared library to allow Airsim to communicate with Unity and copy it to the plugins folder of your project binary. Run the Project Binary # Open a terminal and navigate to your project directory Set your project binary as an executable file: chmod +x \"{project_name}.{configuration}\" Run the binary file ./{project_name}.{configuration} Using Airsim API # For quickstart with the Python APIs for the car or the drone, simply run the hello_car.py or the hello_drone.py script accordingly. Details of the AirSim C++ and Python APIs are here . Acknowledgements # The drone object was provided by user 31415926 on sketchfab . It is licensed under the CC License .","title":"AirSim with Unity"},{"location":"Unity/#airsim-on-unity","text":"AirSim on Unity allows you to run your simulators in the Unity Engine . This project comes with some sample Unity projects and a wrapper around the AirLib library to run as a native plugin in Unity. Included are two basic Unity Projects, one for a Car simulator and another for a Drone simulator. They are meant to be lightweight, and can be used to verify your setup is correct. Check out the Unity blogpost for overview on the release.","title":"AirSim on Unity"},{"location":"Unity/#warning-experimental-release","text":"This project is still in early development, expect some rough edges. We are working to fully support the full AirLib API and feature set, but some things may be missing. Click here for the list of currently supported APIs.","title":"Warning: Experimental Release"},{"location":"Unity/#windows","text":"","title":"Windows"},{"location":"Unity/#building-from-source","text":"","title":"Building from source"},{"location":"Unity/#install-unity","text":"Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub from here . Detailed instructions here . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs.","title":"Install Unity"},{"location":"Unity/#build-airsim","text":"Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start x64 Native Tools Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line.","title":"Build Airsim"},{"location":"Unity/#build-unity-project","text":"Go inside the AirSim\\Unity directory: cd Unity . Build the unity project: build.cmd . Additionally, there is a free environment Windridge City which you can download from Unity Asset Store . And, of course, you can always create your own environment.","title":"Build Unity Project"},{"location":"Unity/#linux","text":"","title":"Linux"},{"location":"Unity/#dependencies","text":"sudo apt-get install libboost-all-dev","title":"Dependencies"},{"location":"Unity/#download-and-install-unity-for-linux","text":"Warning: Unity Editor for Linux is still in Beta. Expect some rough edges.","title":"Download and Install Unity for Linux"},{"location":"Unity/#install-unity_1","text":"Download Unity Hub from this page . Install Unity 2019.3.12 using the Unity Hub . Note: If you are using Unity for the first time, check out the Getting started guide . The Unity User Manual has additional tips, resources, and FAQs.","title":"Install Unity"},{"location":"Unity/#build-airsim_1","text":"git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh","title":"Build Airsim"},{"location":"Unity/#generate-airsimwrapper-shared-library","text":"cd AirSim/Unity ./build.sh This will generate the necessary shared library and copy it to the UnityDemo Plugins folder.","title":"Generate AirsimWrapper Shared Library"},{"location":"Unity/#usage","text":"Start Unity Hub, click on Projects on left pane, and then click on the Add button Select the folder AirSim\\Unity\\UnityDemo , and then hit the OK button. Click on the new project which showed up in the Unity Hub menu to open it in Unity. In the bottom pane, click on Projects -> Assets -> Scenes . Then, Double-click on SimModeSelector . This will load the SimModeSelector scene into the scene hierarchy pane. DO NOT add CarDemo or DroneDemo scene into the scene hierarchy pane. Hit the play button to start the simulation (and hit play again to stop the simulation. . Alternatively, you can change the SimMode in your Settings.json file. (You can read more about Settings.json here ) Controlling the car: Use WASD or the Arrow keys or the AirSim client. Controlling the drone: Keyboard control is not currently available for drone flight. Changing camera views: Keys 0 , 1 , 2 , 3 are used to toggle windows of different camera views. Recording simulation data: Press Record button(Red button) located at the right bottom corner of the screen, to toggle recording of the simulation data. The recorded data can be found at Documents\\AirSim\\(Date of recording) on Windows and ~/Documents/AirSim/(Date of recording) on Linux.","title":"Usage"},{"location":"Unity/#building-custom-environments-for-airsim","text":"To use environments other than UnityDemo , follow the instructions written here","title":"Building Custom Environments For AirSim"},{"location":"Unity/#cross-compiling-to-linux","text":"Unity Editor supports compiling projects to Linux systems. After following the steps to build AirSim and Unity on Windows, do the following:","title":"Cross-Compiling to Linux"},{"location":"Unity/#linux-pre-requisites","text":"Before being able to run Unity Binaries with the Airsim plugin, be sure have airsim and airsim unity built on your linux machine by following the Linux build steps above.","title":"Linux Pre-Requisites"},{"location":"Unity/#package-unitydemo-binary-on-windows","text":"","title":"Package UnityDemo Binary On Windows"},{"location":"Unity/#install-necessary-components","text":"In order to package your project for linux, the Linux Build Support Unity add-on must be installed. * Open Unity Hub , and click the Add component button in the dropdown window under more options to the right of your Unity 2018.2.15f1 tab. * Make sure the Linux Build Support Platform is selected Once this component is successfully installed, you are ready to build Unity Projects for Linux!","title":"Install Necessary Components"},{"location":"Unity/#build-the-project","text":"On your Windows machine, build the Unity Demo by navigating to the build settings option in the toolbar File -> Build Settings Make sure the following scenes are set to be built: SimModeSelector CarDemo DroneDemo Set the target operating system to linux, and choose the version appropriate for your system (x86 vs x86_64) Click Build Transport the built project as well as the generated folder \"{project_name}_Data\" to your linux machine","title":"Build the Project"},{"location":"Unity/#copy-the-airsimwrapper-library-to-the-project-plugins-folder","text":"On your linux machine, navigate to your AirSim repository, and run the following commands in a terminal window: cp Unity/linux-build/libAirsimWrapper.so path/to/your/project/{project_name}_Data/Plugins/{os_version} This will generate the necessary shared library to allow Airsim to communicate with Unity and copy it to the plugins folder of your project binary.","title":"Copy The AirsimWrapper Library to the Project Plugins folder"},{"location":"Unity/#run-the-project-binary","text":"Open a terminal and navigate to your project directory Set your project binary as an executable file: chmod +x \"{project_name}.{configuration}\" Run the binary file ./{project_name}.{configuration}","title":"Run the Project Binary"},{"location":"Unity/#using-airsim-api","text":"For quickstart with the Python APIs for the car or the drone, simply run the hello_car.py or the hello_drone.py script accordingly. Details of the AirSim C++ and Python APIs are here .","title":"Using Airsim API"},{"location":"Unity/#acknowledgements","text":"The drone object was provided by user 31415926 on sketchfab . It is licensed under the CC License .","title":"Acknowledgements"},{"location":"airsim_ros_pkgs/","text":"airsim_ros_pkgs # A ROS wrapper over the AirSim C++ client library. Setup # Install gcc >= 8.0.0 sudo apt-get install gcc-8 g++-8 Verify version by gcc --version Ubuntu 16.04 Install ROS kinetic Install tf2 sensor and mavros packages: sudo apt-get install ros-kinetic-tf2-sensor-msgs ros-kinetic-mavros* Ubuntu 18.04 Install ROS melodic Install tf2 sensor and mavros packages: sudo apt-get install ros-melodic-tf2-sensor-msgs ros-melodic-mavros* Install catkin_tools sudo apt-get install python-catkin-tools or pip install catkin_tools Build # Build AirSim git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh; Build ROS package cd ros; catkin build; # or catkin_make If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 Running # source devel/setup.bash; roslaunch airsim_ros_pkgs airsim_node.launch; roslaunch airsim_ros_pkgs rviz.launch; Using AirSim ROS wrapper # The ROS wrapper is composed of two ROS nodes - the first is a wrapper over AirSim's multirotor C++ client library, and the second is a simple PD position controller. Let's look at the ROS API for both nodes: AirSim ROS Wrapper Node # Publishers: # /airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw GPS coordinates corresponding to global NED frame. This is set in the airsim's settings.json file under the OriginGeopoint key. /airsim_node/VEHICLE_NAME/global_gps sensor_msgs/NavSatFix This the current GPS coordinates of the drone in airsim. /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Odometry in NED frame (default name: odom_local_ned, launch name and frame type are configurable) wrt take-off point. /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE/camera_info sensor_msgs/CameraInfo /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE sensor_msgs/Image RGB or float image depending on image type requested in settings.json. /tf tf2_msgs/TFMessage /airsim_node/VEHICLE_NAME/altimeter/SENSOR_NAME [airsim_ros_pkgs::Altimeter] This the current altimeter reading for altitude, pressure, and QNH (https://en.wikipedia.org/wiki/QNH) /airsim_node/VEHICLE_NAME/imu/SENSOR_NAME [sensor_msgs::Imu] (http://docs.ros.org/api/sensor_msgs/html/msg/Imu.html) IMU sensor data /airsim_node/VEHICLE_NAME/magnetometer/SENSOR_NAME [sensor_msgs::MagneticField] (http://docs.ros.org/api/sensor_msgs/html/msg/MagneticField.html) Meausrement of magnetic field vector/compass /airsim_node/VEHICLE_NAME/distance/SENSOR_NAME [sensor_msgs::Range] (http://docs.ros.org/api/sensor_msgs/html/msg/Range.html) Meausrement of distance from an active ranger, such as infrared or IR /airsim_node/VEHICLE_NAME/lidar/SENSOR_NAME [sensor_msgs::PointCloud2] (http://docs.ros.org/api/sensor_msgs/html/msg/PointCloud2.html) LIDAR pointcloud Subscribers: # /airsim_node/vel_cmd_body_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /airsim_node/vel_cmd_world_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /gimbal_angle_euler_cmd airsim_ros_pkgs/GimbalAngleEulerCmd Gimbal set point in euler angles. /gimbal_angle_quat_cmd airsim_ros_pkgs/GimbalAngleQuatCmd Gimbal set point in quaternion. /airsim_node/VEHICLE_NAME/car_cmd [airsim_ros_pkgs/CarControls] Throttle, brake, steering and gear selections for control. Both automatic and manual transmission control possible, see the car_joy.py script for use. Services: # /airsim_node/VEHICLE_NAME/land airsim_ros_pkgs/Takeoff /airsim_node/takeoff airsim_ros_pkgs/Takeoff /airsim_node/reset airsim_ros_pkgs/Reset Resets all drones Parameters: # /airsim_node/world_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: world_ned Set to \"world_enu\" to switch to ENU frames automatically /airsim_node/odom_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: odom_local_ned If you set world_frame_id to \"world_enu\", the default odom name will instead default to \"odom_local_enu\" /airsim_node/coordinate_system_enu [boolean] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false If you set world_frame_id to \"world_enu\", this setting will instead default to true /airsim_node/update_airsim_control_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for updating drone odom and state from airsim, and sending in control commands. The current RPClib interface to unreal engine maxes out at 50 Hz. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/update_airsim_img_response_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for receiving images from all cameras in airsim. The speed will depend on number of images requested and their resolution. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/publish_clock [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false Will publish the ros /clock topic if set to true. Simple PID Position Controller Node # Parameters: # PD controller parameters: /pd_position_node/kd_x [double], /pd_position_node/kp_y [double], /pd_position_node/kp_z [double], /pd_position_node/kp_yaw [double] Proportional gains /pd_position_node/kd_x [double], /pd_position_node/kd_y [double], /pd_position_node/kd_z [double], /pd_position_node/kd_yaw [double] Derivative gains /pd_position_node/reached_thresh_xyz [double] Threshold euler distance (meters) from current position to setpoint position /pd_position_node/reached_yaw_degrees [double] Threshold yaw distance (degrees) from current position to setpoint position /pd_position_node/update_control_every_n_sec [double] Default: 0.01 seconds Services: # /airsim_node/VEHICLE_NAME/gps_goal [Request: srv/SetGPSPosition ] Target gps position + yaw. In absolute altitude. /airsim_node/VEHICLE_NAME/local_position_goal [Request: srv/SetLocalPosition Target local position + yaw in global NED frame. Subscribers: # /airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw Listens to home geo coordinates published by airsim_node . /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Listens to odometry published by airsim_node Publishers: # /vel_cmd_world_frame airsim_ros_pkgs/VelCmd Sends velocity command to airsim_node Global params # Dynamic constraints. These can be changed in dynamic_constraints.launch : /max_vel_horz_abs [double] Maximum horizontal velocity of the drone (meters/second) /max_vel_vert_abs [double] Maximum vertical velocity of the drone (meters/second) /max_yaw_rate_degree [double] Maximum yaw rate (degrees/second) Misc # Windows Subsytem for Linux on Windows 10 # WSL setup: Get Windows Subsystem for Linux Get Ubuntu 16.04 or Ubuntu 18.04 Go to Ubuntu 16 / 18 instructions! Setup for X apps (like RViz, rqt_image_view, terminator) in Windows + WSL Install Xming X Server . Find and run XLaunch from the Windows start menu. Select Multiple Windows in first popup, Start no client in second popup, only Clipboard in third popup. Do not select Native Opengl . Open Ubuntu 16.04 / 18.04 session by typing Ubuntu 16.04 / Ubuntu 18.04 in Windows start menu. Recommended: Install terminator : $ sudo apt-get install terminator. You can open terminator in a new window by entering $ DISPLAY=:0 terminator -u .","title":"ROS: AirSim ROS Wrapper"},{"location":"airsim_ros_pkgs/#airsim_ros_pkgs","text":"A ROS wrapper over the AirSim C++ client library.","title":"airsim_ros_pkgs"},{"location":"airsim_ros_pkgs/#setup","text":"Install gcc >= 8.0.0 sudo apt-get install gcc-8 g++-8 Verify version by gcc --version Ubuntu 16.04 Install ROS kinetic Install tf2 sensor and mavros packages: sudo apt-get install ros-kinetic-tf2-sensor-msgs ros-kinetic-mavros* Ubuntu 18.04 Install ROS melodic Install tf2 sensor and mavros packages: sudo apt-get install ros-melodic-tf2-sensor-msgs ros-melodic-mavros* Install catkin_tools sudo apt-get install python-catkin-tools or pip install catkin_tools","title":"Setup"},{"location":"airsim_ros_pkgs/#build","text":"Build AirSim git clone https://github.com/Microsoft/AirSim.git; cd AirSim; ./setup.sh; ./build.sh; Build ROS package cd ros; catkin build; # or catkin_make If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8","title":"Build"},{"location":"airsim_ros_pkgs/#running","text":"source devel/setup.bash; roslaunch airsim_ros_pkgs airsim_node.launch; roslaunch airsim_ros_pkgs rviz.launch;","title":"Running"},{"location":"airsim_ros_pkgs/#using-airsim-ros-wrapper","text":"The ROS wrapper is composed of two ROS nodes - the first is a wrapper over AirSim's multirotor C++ client library, and the second is a simple PD position controller. Let's look at the ROS API for both nodes:","title":"Using AirSim ROS wrapper"},{"location":"airsim_ros_pkgs/#airsim-ros-wrapper-node","text":"","title":"AirSim ROS Wrapper Node"},{"location":"airsim_ros_pkgs/#publishers","text":"/airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw GPS coordinates corresponding to global NED frame. This is set in the airsim's settings.json file under the OriginGeopoint key. /airsim_node/VEHICLE_NAME/global_gps sensor_msgs/NavSatFix This the current GPS coordinates of the drone in airsim. /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Odometry in NED frame (default name: odom_local_ned, launch name and frame type are configurable) wrt take-off point. /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE/camera_info sensor_msgs/CameraInfo /airsim_node/VEHICLE_NAME/CAMERA_NAME/IMAGE_TYPE sensor_msgs/Image RGB or float image depending on image type requested in settings.json. /tf tf2_msgs/TFMessage /airsim_node/VEHICLE_NAME/altimeter/SENSOR_NAME [airsim_ros_pkgs::Altimeter] This the current altimeter reading for altitude, pressure, and QNH (https://en.wikipedia.org/wiki/QNH) /airsim_node/VEHICLE_NAME/imu/SENSOR_NAME [sensor_msgs::Imu] (http://docs.ros.org/api/sensor_msgs/html/msg/Imu.html) IMU sensor data /airsim_node/VEHICLE_NAME/magnetometer/SENSOR_NAME [sensor_msgs::MagneticField] (http://docs.ros.org/api/sensor_msgs/html/msg/MagneticField.html) Meausrement of magnetic field vector/compass /airsim_node/VEHICLE_NAME/distance/SENSOR_NAME [sensor_msgs::Range] (http://docs.ros.org/api/sensor_msgs/html/msg/Range.html) Meausrement of distance from an active ranger, such as infrared or IR /airsim_node/VEHICLE_NAME/lidar/SENSOR_NAME [sensor_msgs::PointCloud2] (http://docs.ros.org/api/sensor_msgs/html/msg/PointCloud2.html) LIDAR pointcloud","title":"Publishers:"},{"location":"airsim_ros_pkgs/#subscribers","text":"/airsim_node/vel_cmd_body_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /airsim_node/vel_cmd_world_frame airsim_ros_pkgs/VelCmd Ignore vehicle_name field, leave it to blank. We will use vehicle_name in future for multiple drones. /gimbal_angle_euler_cmd airsim_ros_pkgs/GimbalAngleEulerCmd Gimbal set point in euler angles. /gimbal_angle_quat_cmd airsim_ros_pkgs/GimbalAngleQuatCmd Gimbal set point in quaternion. /airsim_node/VEHICLE_NAME/car_cmd [airsim_ros_pkgs/CarControls] Throttle, brake, steering and gear selections for control. Both automatic and manual transmission control possible, see the car_joy.py script for use.","title":"Subscribers:"},{"location":"airsim_ros_pkgs/#services","text":"/airsim_node/VEHICLE_NAME/land airsim_ros_pkgs/Takeoff /airsim_node/takeoff airsim_ros_pkgs/Takeoff /airsim_node/reset airsim_ros_pkgs/Reset Resets all drones","title":"Services:"},{"location":"airsim_ros_pkgs/#parameters","text":"/airsim_node/world_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: world_ned Set to \"world_enu\" to switch to ENU frames automatically /airsim_node/odom_frame_id [string] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: odom_local_ned If you set world_frame_id to \"world_enu\", the default odom name will instead default to \"odom_local_enu\" /airsim_node/coordinate_system_enu [boolean] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false If you set world_frame_id to \"world_enu\", this setting will instead default to true /airsim_node/update_airsim_control_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for updating drone odom and state from airsim, and sending in control commands. The current RPClib interface to unreal engine maxes out at 50 Hz. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/update_airsim_img_response_every_n_sec [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: 0.01 seconds. Timer callback frequency for receiving images from all cameras in airsim. The speed will depend on number of images requested and their resolution. Timer callbacks in ROS run at maximum rate possible, so it's best to not touch this parameter. /airsim_node/publish_clock [double] Set in: $(airsim_ros_pkgs)/launch/airsim_node.launch Default: false Will publish the ros /clock topic if set to true.","title":"Parameters:"},{"location":"airsim_ros_pkgs/#simple-pid-position-controller-node","text":"","title":"Simple PID Position Controller Node"},{"location":"airsim_ros_pkgs/#parameters_1","text":"PD controller parameters: /pd_position_node/kd_x [double], /pd_position_node/kp_y [double], /pd_position_node/kp_z [double], /pd_position_node/kp_yaw [double] Proportional gains /pd_position_node/kd_x [double], /pd_position_node/kd_y [double], /pd_position_node/kd_z [double], /pd_position_node/kd_yaw [double] Derivative gains /pd_position_node/reached_thresh_xyz [double] Threshold euler distance (meters) from current position to setpoint position /pd_position_node/reached_yaw_degrees [double] Threshold yaw distance (degrees) from current position to setpoint position /pd_position_node/update_control_every_n_sec [double] Default: 0.01 seconds","title":"Parameters:"},{"location":"airsim_ros_pkgs/#services_1","text":"/airsim_node/VEHICLE_NAME/gps_goal [Request: srv/SetGPSPosition ] Target gps position + yaw. In absolute altitude. /airsim_node/VEHICLE_NAME/local_position_goal [Request: srv/SetLocalPosition Target local position + yaw in global NED frame.","title":"Services:"},{"location":"airsim_ros_pkgs/#subscribers_1","text":"/airsim_node/origin_geo_point airsim_ros_pkgs/GPSYaw Listens to home geo coordinates published by airsim_node . /airsim_node/VEHICLE_NAME/odom_local_ned nav_msgs/Odometry Listens to odometry published by airsim_node","title":"Subscribers:"},{"location":"airsim_ros_pkgs/#publishers_1","text":"/vel_cmd_world_frame airsim_ros_pkgs/VelCmd Sends velocity command to airsim_node","title":"Publishers:"},{"location":"airsim_ros_pkgs/#global-params","text":"Dynamic constraints. These can be changed in dynamic_constraints.launch : /max_vel_horz_abs [double] Maximum horizontal velocity of the drone (meters/second) /max_vel_vert_abs [double] Maximum vertical velocity of the drone (meters/second) /max_yaw_rate_degree [double] Maximum yaw rate (degrees/second)","title":"Global params"},{"location":"airsim_ros_pkgs/#misc","text":"","title":"Misc"},{"location":"airsim_ros_pkgs/#windows-subsytem-for-linux-on-windows-10","text":"WSL setup: Get Windows Subsystem for Linux Get Ubuntu 16.04 or Ubuntu 18.04 Go to Ubuntu 16 / 18 instructions! Setup for X apps (like RViz, rqt_image_view, terminator) in Windows + WSL Install Xming X Server . Find and run XLaunch from the Windows start menu. Select Multiple Windows in first popup, Start no client in second popup, only Clipboard in third popup. Do not select Native Opengl . Open Ubuntu 16.04 / 18.04 session by typing Ubuntu 16.04 / Ubuntu 18.04 in Windows start menu. Recommended: Install terminator : $ sudo apt-get install terminator. You can open terminator in a new window by entering $ DISPLAY=:0 terminator -u .","title":"Windows Subsytem for Linux on Windows 10"},{"location":"airsim_tutorial_pkgs/","text":"AirSim ROS Tutorials # This is a set of sample AirSim settings.json s, roslaunch and rviz files to give a starting point for using AirSim with ROS. See airsim_ros_pkgs for the ROS API. Setup # Make sure that the airsim_ros_pkgs Setup has been completed and the prerequisites installed. $ cd PATH_TO/AirSim/ros $ catkin build airsim_tutorial_pkgs If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build airsim_tutorial_pkgs -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 Note: For running examples, and also whenever a new terminal is opened, sourcing the setup.bash file is necessary. If you're using the ROS wrapper frequently, it might be helpful to add the source PATH_TO/AirSim/ros/devel/setup.bash to your ~/.profile or ~/.bashrc to avoid the need to run the command every time a new terminal is opened Examples # Single drone with monocular and depth cameras, and lidar # Settings.json - front_stereo_and_center_mono.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/front_stereo_and_center_mono.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; # in a new pane / terminal $ source PATH_TO/AirSim/ros/devel/setup.bash $ roslaunch airsim_tutorial_pkgs front_stereo_and_center_mono.launch `` The above would start rviz with tf's, registered RGBD cloud using [depth_image_proc](https://wiki.ros.org/depth_image_proc) using the [ depth_to_pointcloud` launch file](https://github.com/microsoft/AirSim/master/ros/src/airsim_tutorial_pkgs/launch/front_stereo_and_center_mono/depth_to_pointcloud.launch), and the lidar point cloud. Two drones, with cameras, lidar, IMU each # Settings.json - two_drones_camera_lidar_imu.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/two_drones_camera_lidar_imu.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable. Twenty-five drones in a square pattern # Settings.json - twenty_five_drones.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/twenty_five_drones.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable.","title":"ROS: AirSim Tutorial Packages"},{"location":"airsim_tutorial_pkgs/#airsim-ros-tutorials","text":"This is a set of sample AirSim settings.json s, roslaunch and rviz files to give a starting point for using AirSim with ROS. See airsim_ros_pkgs for the ROS API.","title":"AirSim ROS Tutorials"},{"location":"airsim_tutorial_pkgs/#setup","text":"Make sure that the airsim_ros_pkgs Setup has been completed and the prerequisites installed. $ cd PATH_TO/AirSim/ros $ catkin build airsim_tutorial_pkgs If your default GCC isn't 8 or greater (check using gcc --version ), then compilation will fail. In that case, use gcc-8 explicitly as follows- catkin build airsim_tutorial_pkgs -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 Note: For running examples, and also whenever a new terminal is opened, sourcing the setup.bash file is necessary. If you're using the ROS wrapper frequently, it might be helpful to add the source PATH_TO/AirSim/ros/devel/setup.bash to your ~/.profile or ~/.bashrc to avoid the need to run the command every time a new terminal is opened","title":"Setup"},{"location":"airsim_tutorial_pkgs/#examples","text":"","title":"Examples"},{"location":"airsim_tutorial_pkgs/#single-drone-with-monocular-and-depth-cameras-and-lidar","text":"Settings.json - front_stereo_and_center_mono.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/front_stereo_and_center_mono.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; # in a new pane / terminal $ source PATH_TO/AirSim/ros/devel/setup.bash $ roslaunch airsim_tutorial_pkgs front_stereo_and_center_mono.launch `` The above would start rviz with tf's, registered RGBD cloud using [depth_image_proc](https://wiki.ros.org/depth_image_proc) using the [ depth_to_pointcloud` launch file](https://github.com/microsoft/AirSim/master/ros/src/airsim_tutorial_pkgs/launch/front_stereo_and_center_mono/depth_to_pointcloud.launch), and the lidar point cloud.","title":"Single drone with monocular and depth cameras, and lidar"},{"location":"airsim_tutorial_pkgs/#two-drones-with-cameras-lidar-imu-each","text":"Settings.json - two_drones_camera_lidar_imu.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/two_drones_camera_lidar_imu.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable.","title":"Two drones, with cameras, lidar, IMU each"},{"location":"airsim_tutorial_pkgs/#twenty-five-drones-in-a-square-pattern","text":"Settings.json - twenty_five_drones.json ```shell $ source PATH_TO/AirSim/ros/devel/setup.bash $ roscd airsim_tutorial_pkgs $ cp settings/twenty_five_drones.json ~/Documents/AirSim/settings.json ## Start your unreal package or binary here $ roslaunch airsim_ros_pkgs airsim_node.launch; $ roslaunch airsim_ros_pkgs rviz.launch `` You can view the tfs in rviz. And do a rostopic list and rosservice list` to inspect the services avaiable.","title":"Twenty-five drones in a square pattern"},{"location":"apis/","text":"AirSim APIs # Introduction # AirSim exposes APIs so you can interact with vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on. Python Quickstart # If you want to use Python to call AirSim APIs, we recommend using Anaconda with Python 3.5 or later versions however some code may also work with Python 2.7 ( help us improve compatibility!). First install this package: pip install msgpack-rpc-python You can either get AirSim binaries from releases or compile from the source ( Windows , Linux ). Once you can run AirSim, choose Car as vehicle and then navigate to PythonClient\\car\\ folder and run: python hello_car.py If you are using Visual Studio 2019 then just open AirSim.sln, set PythonClient as startup project and choose car\\hello_car.py as your startup script. Installing AirSim Package # You can also install airsim package simply by, pip install airsim You can find source code and samples for this package in PythonClient folder in your repo. Notes 1. You may notice a file setup_path.py in our example folders. This file has simple code to detect if airsim package is available in parent folder and in that case we use that instead of pip installed package so you always use latest code. 2. AirSim is still under heavy development which means you might frequently need to update the package to use new APIs. C++ Users # If you want to use C++ APIs and examples, please see C++ APIs Guide . Hello Car # Here's how to use AirSim APIs using Python to control simulated car (see also C++ example ): # ready to run example: PythonClient/car/hello_car.py import airsim import time # connect to the AirSim simulator client = airsim.CarClient() client.confirmConnection() client.enableApiControl(True) car_controls = airsim.CarControls() while True: # get state of the car car_state = client.getCarState() print(\"Speed %d, Gear %d\" % (car_state.speed, car_state.gear)) # set the controls for car car_controls.throttle = 1 car_controls.steering = 1 client.setCarControls(car_controls) # let car drive a bit time.sleep(1) # get camera images from the car responses = client.simGetImages([ airsim.ImageRequest(0, airsim.ImageType.DepthVis), airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm('py1.pfm', airsim.get_pfm_array(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file('py1.png', response.image_data_uint8) Hello Drone # Here's how to use AirSim APIs using Python to control simulated quadrotor (see also C++ example ): # ready to run example: PythonClient/multirotor/hello_drone.py import airsim # connect to the AirSim simulator client = airsim.MultirotorClient() client.confirmConnection() client.enableApiControl(True) client.armDisarm(True) # Async methods returns Future. Call join() to wait for task to complete. client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() # take images responses = client.simGetImages([ airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis), airsim.ImageRequest(\"1\", airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with the images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm(os.path.normpath('/temp/py1.pfm'), airsim.getPfmArray(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file(os.path.normpath('/temp/py1.png'), response.image_data_uint8) Common APIs # reset : This resets the vehicle to its original starting state. Note that you must call enableApiControl and armDisarm again after the call to reset . confirmConnection : Checks state of connection every 1 sec and reports it in Console so user can see the progress for connection. enableApiControl : For safety reasons, by default API control for autonomous vehicle is not enabled and human operator has full control (usually via RC or joystick in simulator). The client must make this call to request control via API. It is likely that human operator of vehicle might have disallowed API control which would mean that enableApiControl has no effect. This can be checked by isApiControlEnabled . isApiControlEnabled : Returns true if API control is established. If false (which is default) then API calls would be ignored. After a successful call to enableApiControl , the isApiControlEnabled should return true. ping : If connection is established then this call will return true otherwise it will be blocked until timeout. simPrintLogMessage : Prints the specified message in the simulator's window. If message_param is also supplied then its printed next to the message and in that case if this API is called with same message value but different message_param again then previous line is overwritten with new line (instead of API creating new line on display). For example, simPrintLogMessage(\"Iteration: \", to_string(i)) keeps updating same line on display when API is called with different values of i. The valid values of severity parameter is 0 to 3 inclusive that corresponds to different colors. simGetObjectPose , simSetObjectPose : Gets and sets the pose of specified object in Unreal environment. Here the object means \"actor\" in Unreal terminology. They are searched by tag as well as name. Please note that the names shown in UE Editor are auto-generated in each run and are not permanent. So if you want to refer to actor by name, you must change its auto-generated name in UE Editor. Alternatively you can add a tag to actor which can be done by clicking on that actor in Unreal Editor and then going to Tags property , click \"+\" sign and add some string value. If multiple actors have same tag then the first match is returned. If no matches are found then NaN pose is returned. The returned pose is in NED coordinates in SI units with its origin at Player Start. For simSetObjectPose , the specified actor must have Mobility set to Movable or otherwise you will get undefined behavior. The simSetObjectPose has parameter teleport which means object is moved through other objects in its way and it returns true if move was successful Image / Computer Vision APIs # AirSim offers comprehensive images APIs to retrieve synchronized images from multiple cameras along with ground truth including depth, disparity, surface normals and vision. You can set the resolution, FOV, motion blur etc parameters in settings.json . There is also API for detecting collision state. See also complete code that generates specified number of stereo images and ground truth depth with normalization to camera plan, computation of disparity image and saving it to pfm format . More on image APIs and Computer Vision mode . For vision problems that can benefit from domain randomization, there is also an object retexturing API , which can be used in supported scenes. Pause and Continue APIs # AirSim allows to pause and continue the simulation through pause(is_paused) API. To pause the simulation call pause(True) and to continue the simulation call pause(False) . You may have scenario, especially while using reinforcement learning, to run the simulation for specified amount of time and then automatically pause. While simulation is paused, you may then do some expensive computation, send a new command and then again run the simulation for specified amount of time. This can be achieved by API continueForTime(seconds) . This API runs the simulation for the specified number of seconds and then pauses the simulation. For example usage, please see pause_continue_car.py and pause_continue_drone.py . Collision API # The collision information can be obtained using simGetCollisionInfo API. This call returns a struct that has information not only whether collision occurred but also collision position, surface normal, penetration depth and so on. Time of Day API # AirSim assumes there exist sky sphere of class EngineSky/BP_Sky_Sphere in your environment with ADirectionalLight actor . By default, the position of the sun in the scene doesn't move with time. You can use settings to set up latitude, longitude, date and time which AirSim uses to compute the position of sun in the scene. You can also use following API call to set the sun position according to given date time: simSetTimeOfDay(self, is_enabled, start_datetime = \"\", is_start_datetime_dst = False, celestial_clock_speed = 1, update_interval_secs = 60, move_sun = True) The is_enabled parameter must be True to enable time of day effect. If it is False then sun position is reset to its original in the environment. Other parameters are same as in settings . Weather APIs # By default all weather effects are disabled. To enable weather effect, first call: simEnableWeather(True) Various weather effects can be enabled by using simSetWeatherParameter method which takes WeatherParameter , for example, client.simSetWeatherParameter(airsim.WeatherParameter.Rain, 0.25); The second parameter value is from 0 to 1. The first parameter provides following options: class WeatherParameter: Rain = 0 Roadwetness = 1 Snow = 2 RoadSnow = 3 MapleLeaf = 4 RoadLeaf = 5 Dust = 6 Fog = 7 Please note that Roadwetness , RoadSnow and RoadLeaf effects requires adding materials to your scene. Please see example code for more details. Recording APIs # Recording APIs can be used to start recording data through APIs. Data to be recorded can be specified using settings . To start recording, use - client.startRecording() Similarly, to stop recording, use client.stopRecording() . To check whether Recording is running, call client.isRecording() , returns a bool . This API works alongwith toggling Recording using R button, therefore if it's enabled using R key, isRecording() will return True , and recording can be stopped via API using stopRecording() . Similarly, recording started using API will be stopped if R key is pressed in Viewport. LogMessage will also appear in the top-left of the viewport if recording is started or stopped using API. Note that this will only save the data as specfied in the settings. For full freedom in storing data such as certain sensor information, or in a different format or layout, use the other APIs to fetch the data and save as desired. Lidar APIs # AirSim offers API to retrieve point cloud data from Lidar sensors on vehicles. You can set the number of channels, points per second, horizontal and vertical FOV, etc parameters in settings.json . More on lidar APIs and settings and sensor settings Multiple Vehicles # AirSim supports multiple vehicles and control them through APIs. Please Multiple Vehicles doc. Coordinate System # All AirSim API uses NED coordinate system, i.e., +X is North, +Y is East and +Z is Down. All units are in SI system. Please note that this is different from coordinate system used internally by Unreal Engine. In Unreal Engine, +Z is up instead of down and length unit is in centimeters instead of meters. AirSim APIs takes care of the appropriate conversions. The starting point of the vehicle is always coordinates (0, 0, 0) in NED system. Thus when converting from Unreal coordinates to NED, we first subtract the starting offset and then scale by 100 for cm to m conversion. The vehicle is spawned in Unreal environment where the Player Start component is placed. There is a setting called OriginGeopoint in settings.json which assigns geographic longitude, longitude and altitude to the Player Start component. Vehicle Specific APIs # APIs for Car # Car has followings APIs available: setCarControls : This allows you to set throttle, steering, handbrake and auto or manual gear. getCarState : This retrieves the state information including speed, current gear and 6 kinematics quantities: position, orientation, linear and angular velocity, linear and angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame. Image APIs . APIs for Multirotor # Multirotor can be controlled by specifying angles, velocity vector, destination position or some combination of these. There are corresponding move* APIs for this purpose. When doing position control, we need to use some path following algorithm. By default AirSim uses carrot following algorithm. This is often referred to as \"high level control\" because you just need to specify high level goal and the firmware takes care of the rest. Currently lowest level control available in AirSim is moveByAngleThrottleAsync API. getMultirotorState # This API returns the state of the vehicle in one call. The state includes, collision, estimated kinematics (i.e. kinematics computed by fusing sensors), and timestamp (nano seconds since epoch). The kinematics here means 6 quantities: position, orientation, linear and angular velocity, linear and angular acceleration. Please note that simple_slight currently doesn't support state estimator which means estimated and ground truth kinematics values would be same for simple_flight. Estimated kinematics are however available for PX4 except for angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame. Async methods, duration and max_wait_seconds # Many API methods has parameters named duration or max_wait_seconds and they have Async as suffix, for example, takeoffAsync . These methods will return immediately after starting the task in AirSim so that your client code can do something else while that task is being executed. If you want to wait for this task to complete then you can call waitOnLastTask like this: //C++ client.takeoffAsync()->waitOnLastTask(); # Python client.takeoffAsync().join() If you start another command then it automatically cancels the previous task and starts new command. This allows to use pattern where your coded continuously does the sensing, computes a new trajectory to follow and issues that path to vehicle in AirSim. Each newly issued trajectory cancels the previous trajectory allowing your code to continuously do the update as new sensor data arrives. All Async method returns concurrent.futures.Future in Python ( std::future in C++). Please note that these future classes currently do not allow to check status or cancel the task; they only allow to wait for task to complete. AirSim does provide API cancelLastTask , however. drivetrain # There are two modes you can fly vehicle: drivetrain parameter is set to airsim.DrivetrainType.ForwardOnly or airsim.DrivetrainType.MaxDegreeOfFreedom . When you specify ForwardOnly, you are saying that vehicle's front should always point in the direction of travel. So if you want drone to take left turn then it would first rotate so front points to left. This mode is useful when you have only front camera and you are operating vehicle using FPV view. This is more or less like travelling in car where you always have front view. The MaxDegreeOfFreedom means you don't care where the front points to. So when you take left turn, you just start going left like crab. Quadrotors can go in any direction regardless of where front points to. The MaxDegreeOfFreedom enables this mode. yaw_mode # yaw_mode is a struct YawMode with two fields, yaw_or_rate and is_rate . If is_rate field is True then yaw_or_rate field is interpreted as angular velocity in degrees/sec which means you want vehicle to rotate continuously around its axis at that angular velocity while moving. If is_rate is False then yaw_or_rate is interpreted as angle in degrees which means you want vehicle to rotate to specific angle (i.e. yaw) and keep that angle while moving. You can probably see that when yaw_mode.is_rate == true , the drivetrain parameter shouldn't be set to ForwardOnly because you are contradicting by saying that keep front pointing ahead but also rotate continuously. However if you have yaw_mode.is_rate = false in ForwardOnly mode then you can do some funky stuff. For example, you can have drone do circles and have yaw_or_rate set to 90 so camera is always pointed to center (\"super cool selfie mode\"). In MaxDegreeofFreedom also you can get some funky stuff by setting yaw_mode.is_rate = true and say yaw_mode.yaw_or_rate = 20 . This will cause drone to go in its path while rotating which may allow to do 360 scanning. In most cases, you just don't want yaw to change which you can do by setting yaw rate of 0. The shorthand for this is airsim.YawMode.Zero() (or in C++: YawMode::Zero() ). lookahead and adaptive_lookahead # When you ask vehicle to follow a path, AirSim uses \"carrot following\" algorithm. This algorithm operates by looking ahead on path and adjusting its velocity vector. The parameters for this algorithm is specified by lookahead and adaptive_lookahead . For most of the time you want algorithm to auto-decide the values by simply setting lookahead = -1 and adaptive_lookahead = 0 . Using APIs on Real Vehicles # We want to be able to run same code that runs in simulation as on real vehicle. This allows you to test your code in simulator and deploy to real vehicle. Generally speaking, APIs therefore shouldn't allow you to do something that cannot be done on real vehicle (for example, getting the ground truth). But, of course, simulator has much more information and it would be useful in applications that may not care about running things on real vehicle. For this reason, we clearly delineate between sim-only APIs by attaching sim prefix, for example, simGetGroundTruthKinematics . This way you can avoid using these simulation-only APIs if you care about running your code on real vehicles. The AirLib is self-contained library that you can put on an offboard computing module such as the Gigabyte barebone Mini PC. This module then can talk to the flight controllers such as PX4 using exact same code and flight controller protocol. The code you write for testing in the simulator remains unchanged. See AirLib on custom drones . Adding New APIs to AirSim # Adding new APIs requires modifying the source code. Much of the changes are mechanical and required for various levels of abstractions that AirSim supports. This commit demonstrates how to add a simple API simPrintLogMessage that prints message in simulator window. Some Internals # The APIs use msgpack-rpc protocol over TCP/IP through rpclib developed by Tam\u00c3\u00a1s Szelei which allows you to use variety of programming languages including C++, C#, Python, Java etc. When AirSim starts, it opens port 41451 (this can be changed via settings ) and listens for incoming request. The Python or C++ client code connects to this port and sends RPC calls using msgpack serialization format . References and Examples # C++ API Examples Car Examples Multirotor Examples Computer Vision Examples Move on Path demo showing video of fast multirotor flight through Modular Neighborhood environment Building a Hexacopter Building Point Clouds FAQ # Unreal is slowed down dramatically when I run API # If you see Unreal getting slowed down dramatically when Unreal Engine window loses focus then go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. Do I need anything else on Windows? # You should install VS2019 with VC++, Windows SDK 10.0 and Python. To use Python APIs you will need Python 3.5 or later (install it using Anaconda). Which version of Python should I use? # We recommend Anaconda to get Python tools and libraries. Our code is tested with Python 3.5.3 :: Anaconda 4.4.0. This is important because older version have been known to have problems . I get error on import cv2 # You can install OpenCV using: conda install opencv pip install opencv-python TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float' # This error happens if you install Jupyter, which somehow breaks the msgpackrpc library. Create a new python environment which the minimal required packages.","title":"Core APIs"},{"location":"apis/#airsim-apis","text":"","title":"AirSim APIs"},{"location":"apis/#introduction","text":"AirSim exposes APIs so you can interact with vehicle in the simulation programmatically. You can use these APIs to retrieve images, get state, control the vehicle and so on.","title":"Introduction"},{"location":"apis/#python-quickstart","text":"If you want to use Python to call AirSim APIs, we recommend using Anaconda with Python 3.5 or later versions however some code may also work with Python 2.7 ( help us improve compatibility!). First install this package: pip install msgpack-rpc-python You can either get AirSim binaries from releases or compile from the source ( Windows , Linux ). Once you can run AirSim, choose Car as vehicle and then navigate to PythonClient\\car\\ folder and run: python hello_car.py If you are using Visual Studio 2019 then just open AirSim.sln, set PythonClient as startup project and choose car\\hello_car.py as your startup script.","title":"Python Quickstart"},{"location":"apis/#installing-airsim-package","text":"You can also install airsim package simply by, pip install airsim You can find source code and samples for this package in PythonClient folder in your repo. Notes 1. You may notice a file setup_path.py in our example folders. This file has simple code to detect if airsim package is available in parent folder and in that case we use that instead of pip installed package so you always use latest code. 2. AirSim is still under heavy development which means you might frequently need to update the package to use new APIs.","title":"Installing AirSim Package"},{"location":"apis/#c-users","text":"If you want to use C++ APIs and examples, please see C++ APIs Guide .","title":"C++ Users"},{"location":"apis/#hello-car","text":"Here's how to use AirSim APIs using Python to control simulated car (see also C++ example ): # ready to run example: PythonClient/car/hello_car.py import airsim import time # connect to the AirSim simulator client = airsim.CarClient() client.confirmConnection() client.enableApiControl(True) car_controls = airsim.CarControls() while True: # get state of the car car_state = client.getCarState() print(\"Speed %d, Gear %d\" % (car_state.speed, car_state.gear)) # set the controls for car car_controls.throttle = 1 car_controls.steering = 1 client.setCarControls(car_controls) # let car drive a bit time.sleep(1) # get camera images from the car responses = client.simGetImages([ airsim.ImageRequest(0, airsim.ImageType.DepthVis), airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm('py1.pfm', airsim.get_pfm_array(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file('py1.png', response.image_data_uint8)","title":"Hello Car"},{"location":"apis/#hello-drone","text":"Here's how to use AirSim APIs using Python to control simulated quadrotor (see also C++ example ): # ready to run example: PythonClient/multirotor/hello_drone.py import airsim # connect to the AirSim simulator client = airsim.MultirotorClient() client.confirmConnection() client.enableApiControl(True) client.armDisarm(True) # Async methods returns Future. Call join() to wait for task to complete. client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() # take images responses = client.simGetImages([ airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis), airsim.ImageRequest(\"1\", airsim.ImageType.DepthPlanner, True)]) print('Retrieved images: %d', len(responses)) # do something with the images for response in responses: if response.pixels_as_float: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_float))) airsim.write_pfm(os.path.normpath('/temp/py1.pfm'), airsim.getPfmArray(response)) else: print(\"Type %d, size %d\" % (response.image_type, len(response.image_data_uint8))) airsim.write_file(os.path.normpath('/temp/py1.png'), response.image_data_uint8)","title":"Hello Drone"},{"location":"apis/#common-apis","text":"reset : This resets the vehicle to its original starting state. Note that you must call enableApiControl and armDisarm again after the call to reset . confirmConnection : Checks state of connection every 1 sec and reports it in Console so user can see the progress for connection. enableApiControl : For safety reasons, by default API control for autonomous vehicle is not enabled and human operator has full control (usually via RC or joystick in simulator). The client must make this call to request control via API. It is likely that human operator of vehicle might have disallowed API control which would mean that enableApiControl has no effect. This can be checked by isApiControlEnabled . isApiControlEnabled : Returns true if API control is established. If false (which is default) then API calls would be ignored. After a successful call to enableApiControl , the isApiControlEnabled should return true. ping : If connection is established then this call will return true otherwise it will be blocked until timeout. simPrintLogMessage : Prints the specified message in the simulator's window. If message_param is also supplied then its printed next to the message and in that case if this API is called with same message value but different message_param again then previous line is overwritten with new line (instead of API creating new line on display). For example, simPrintLogMessage(\"Iteration: \", to_string(i)) keeps updating same line on display when API is called with different values of i. The valid values of severity parameter is 0 to 3 inclusive that corresponds to different colors. simGetObjectPose , simSetObjectPose : Gets and sets the pose of specified object in Unreal environment. Here the object means \"actor\" in Unreal terminology. They are searched by tag as well as name. Please note that the names shown in UE Editor are auto-generated in each run and are not permanent. So if you want to refer to actor by name, you must change its auto-generated name in UE Editor. Alternatively you can add a tag to actor which can be done by clicking on that actor in Unreal Editor and then going to Tags property , click \"+\" sign and add some string value. If multiple actors have same tag then the first match is returned. If no matches are found then NaN pose is returned. The returned pose is in NED coordinates in SI units with its origin at Player Start. For simSetObjectPose , the specified actor must have Mobility set to Movable or otherwise you will get undefined behavior. The simSetObjectPose has parameter teleport which means object is moved through other objects in its way and it returns true if move was successful","title":"Common APIs"},{"location":"apis/#image-computer-vision-apis","text":"AirSim offers comprehensive images APIs to retrieve synchronized images from multiple cameras along with ground truth including depth, disparity, surface normals and vision. You can set the resolution, FOV, motion blur etc parameters in settings.json . There is also API for detecting collision state. See also complete code that generates specified number of stereo images and ground truth depth with normalization to camera plan, computation of disparity image and saving it to pfm format . More on image APIs and Computer Vision mode . For vision problems that can benefit from domain randomization, there is also an object retexturing API , which can be used in supported scenes.","title":"Image / Computer Vision APIs"},{"location":"apis/#pause-and-continue-apis","text":"AirSim allows to pause and continue the simulation through pause(is_paused) API. To pause the simulation call pause(True) and to continue the simulation call pause(False) . You may have scenario, especially while using reinforcement learning, to run the simulation for specified amount of time and then automatically pause. While simulation is paused, you may then do some expensive computation, send a new command and then again run the simulation for specified amount of time. This can be achieved by API continueForTime(seconds) . This API runs the simulation for the specified number of seconds and then pauses the simulation. For example usage, please see pause_continue_car.py and pause_continue_drone.py .","title":"Pause and Continue APIs"},{"location":"apis/#collision-api","text":"The collision information can be obtained using simGetCollisionInfo API. This call returns a struct that has information not only whether collision occurred but also collision position, surface normal, penetration depth and so on.","title":"Collision API"},{"location":"apis/#time-of-day-api","text":"AirSim assumes there exist sky sphere of class EngineSky/BP_Sky_Sphere in your environment with ADirectionalLight actor . By default, the position of the sun in the scene doesn't move with time. You can use settings to set up latitude, longitude, date and time which AirSim uses to compute the position of sun in the scene. You can also use following API call to set the sun position according to given date time: simSetTimeOfDay(self, is_enabled, start_datetime = \"\", is_start_datetime_dst = False, celestial_clock_speed = 1, update_interval_secs = 60, move_sun = True) The is_enabled parameter must be True to enable time of day effect. If it is False then sun position is reset to its original in the environment. Other parameters are same as in settings .","title":"Time of Day API"},{"location":"apis/#weather-apis","text":"By default all weather effects are disabled. To enable weather effect, first call: simEnableWeather(True) Various weather effects can be enabled by using simSetWeatherParameter method which takes WeatherParameter , for example, client.simSetWeatherParameter(airsim.WeatherParameter.Rain, 0.25); The second parameter value is from 0 to 1. The first parameter provides following options: class WeatherParameter: Rain = 0 Roadwetness = 1 Snow = 2 RoadSnow = 3 MapleLeaf = 4 RoadLeaf = 5 Dust = 6 Fog = 7 Please note that Roadwetness , RoadSnow and RoadLeaf effects requires adding materials to your scene. Please see example code for more details.","title":"Weather APIs"},{"location":"apis/#recording-apis","text":"Recording APIs can be used to start recording data through APIs. Data to be recorded can be specified using settings . To start recording, use - client.startRecording() Similarly, to stop recording, use client.stopRecording() . To check whether Recording is running, call client.isRecording() , returns a bool . This API works alongwith toggling Recording using R button, therefore if it's enabled using R key, isRecording() will return True , and recording can be stopped via API using stopRecording() . Similarly, recording started using API will be stopped if R key is pressed in Viewport. LogMessage will also appear in the top-left of the viewport if recording is started or stopped using API. Note that this will only save the data as specfied in the settings. For full freedom in storing data such as certain sensor information, or in a different format or layout, use the other APIs to fetch the data and save as desired.","title":"Recording APIs"},{"location":"apis/#lidar-apis","text":"AirSim offers API to retrieve point cloud data from Lidar sensors on vehicles. You can set the number of channels, points per second, horizontal and vertical FOV, etc parameters in settings.json . More on lidar APIs and settings and sensor settings","title":"Lidar APIs"},{"location":"apis/#multiple-vehicles","text":"AirSim supports multiple vehicles and control them through APIs. Please Multiple Vehicles doc.","title":"Multiple Vehicles"},{"location":"apis/#coordinate-system","text":"All AirSim API uses NED coordinate system, i.e., +X is North, +Y is East and +Z is Down. All units are in SI system. Please note that this is different from coordinate system used internally by Unreal Engine. In Unreal Engine, +Z is up instead of down and length unit is in centimeters instead of meters. AirSim APIs takes care of the appropriate conversions. The starting point of the vehicle is always coordinates (0, 0, 0) in NED system. Thus when converting from Unreal coordinates to NED, we first subtract the starting offset and then scale by 100 for cm to m conversion. The vehicle is spawned in Unreal environment where the Player Start component is placed. There is a setting called OriginGeopoint in settings.json which assigns geographic longitude, longitude and altitude to the Player Start component.","title":"Coordinate System"},{"location":"apis/#vehicle-specific-apis","text":"","title":"Vehicle Specific APIs"},{"location":"apis/#apis-for-car","text":"Car has followings APIs available: setCarControls : This allows you to set throttle, steering, handbrake and auto or manual gear. getCarState : This retrieves the state information including speed, current gear and 6 kinematics quantities: position, orientation, linear and angular velocity, linear and angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame. Image APIs .","title":"APIs for Car"},{"location":"apis/#apis-for-multirotor","text":"Multirotor can be controlled by specifying angles, velocity vector, destination position or some combination of these. There are corresponding move* APIs for this purpose. When doing position control, we need to use some path following algorithm. By default AirSim uses carrot following algorithm. This is often referred to as \"high level control\" because you just need to specify high level goal and the firmware takes care of the rest. Currently lowest level control available in AirSim is moveByAngleThrottleAsync API.","title":"APIs for Multirotor"},{"location":"apis/#getmultirotorstate","text":"This API returns the state of the vehicle in one call. The state includes, collision, estimated kinematics (i.e. kinematics computed by fusing sensors), and timestamp (nano seconds since epoch). The kinematics here means 6 quantities: position, orientation, linear and angular velocity, linear and angular acceleration. Please note that simple_slight currently doesn't support state estimator which means estimated and ground truth kinematics values would be same for simple_flight. Estimated kinematics are however available for PX4 except for angular acceleration. All quantities are in NED coordinate system, SI units in world frame except for angular velocity and accelerations which are in body frame.","title":"getMultirotorState"},{"location":"apis/#async-methods-duration-and-max_wait_seconds","text":"Many API methods has parameters named duration or max_wait_seconds and they have Async as suffix, for example, takeoffAsync . These methods will return immediately after starting the task in AirSim so that your client code can do something else while that task is being executed. If you want to wait for this task to complete then you can call waitOnLastTask like this: //C++ client.takeoffAsync()->waitOnLastTask(); # Python client.takeoffAsync().join() If you start another command then it automatically cancels the previous task and starts new command. This allows to use pattern where your coded continuously does the sensing, computes a new trajectory to follow and issues that path to vehicle in AirSim. Each newly issued trajectory cancels the previous trajectory allowing your code to continuously do the update as new sensor data arrives. All Async method returns concurrent.futures.Future in Python ( std::future in C++). Please note that these future classes currently do not allow to check status or cancel the task; they only allow to wait for task to complete. AirSim does provide API cancelLastTask , however.","title":"Async methods, duration and max_wait_seconds"},{"location":"apis/#drivetrain","text":"There are two modes you can fly vehicle: drivetrain parameter is set to airsim.DrivetrainType.ForwardOnly or airsim.DrivetrainType.MaxDegreeOfFreedom . When you specify ForwardOnly, you are saying that vehicle's front should always point in the direction of travel. So if you want drone to take left turn then it would first rotate so front points to left. This mode is useful when you have only front camera and you are operating vehicle using FPV view. This is more or less like travelling in car where you always have front view. The MaxDegreeOfFreedom means you don't care where the front points to. So when you take left turn, you just start going left like crab. Quadrotors can go in any direction regardless of where front points to. The MaxDegreeOfFreedom enables this mode.","title":"drivetrain"},{"location":"apis/#yaw_mode","text":"yaw_mode is a struct YawMode with two fields, yaw_or_rate and is_rate . If is_rate field is True then yaw_or_rate field is interpreted as angular velocity in degrees/sec which means you want vehicle to rotate continuously around its axis at that angular velocity while moving. If is_rate is False then yaw_or_rate is interpreted as angle in degrees which means you want vehicle to rotate to specific angle (i.e. yaw) and keep that angle while moving. You can probably see that when yaw_mode.is_rate == true , the drivetrain parameter shouldn't be set to ForwardOnly because you are contradicting by saying that keep front pointing ahead but also rotate continuously. However if you have yaw_mode.is_rate = false in ForwardOnly mode then you can do some funky stuff. For example, you can have drone do circles and have yaw_or_rate set to 90 so camera is always pointed to center (\"super cool selfie mode\"). In MaxDegreeofFreedom also you can get some funky stuff by setting yaw_mode.is_rate = true and say yaw_mode.yaw_or_rate = 20 . This will cause drone to go in its path while rotating which may allow to do 360 scanning. In most cases, you just don't want yaw to change which you can do by setting yaw rate of 0. The shorthand for this is airsim.YawMode.Zero() (or in C++: YawMode::Zero() ).","title":"yaw_mode"},{"location":"apis/#lookahead-and-adaptive_lookahead","text":"When you ask vehicle to follow a path, AirSim uses \"carrot following\" algorithm. This algorithm operates by looking ahead on path and adjusting its velocity vector. The parameters for this algorithm is specified by lookahead and adaptive_lookahead . For most of the time you want algorithm to auto-decide the values by simply setting lookahead = -1 and adaptive_lookahead = 0 .","title":"lookahead and adaptive_lookahead"},{"location":"apis/#using-apis-on-real-vehicles","text":"We want to be able to run same code that runs in simulation as on real vehicle. This allows you to test your code in simulator and deploy to real vehicle. Generally speaking, APIs therefore shouldn't allow you to do something that cannot be done on real vehicle (for example, getting the ground truth). But, of course, simulator has much more information and it would be useful in applications that may not care about running things on real vehicle. For this reason, we clearly delineate between sim-only APIs by attaching sim prefix, for example, simGetGroundTruthKinematics . This way you can avoid using these simulation-only APIs if you care about running your code on real vehicles. The AirLib is self-contained library that you can put on an offboard computing module such as the Gigabyte barebone Mini PC. This module then can talk to the flight controllers such as PX4 using exact same code and flight controller protocol. The code you write for testing in the simulator remains unchanged. See AirLib on custom drones .","title":"Using APIs on Real Vehicles"},{"location":"apis/#adding-new-apis-to-airsim","text":"Adding new APIs requires modifying the source code. Much of the changes are mechanical and required for various levels of abstractions that AirSim supports. This commit demonstrates how to add a simple API simPrintLogMessage that prints message in simulator window.","title":"Adding New APIs to AirSim"},{"location":"apis/#some-internals","text":"The APIs use msgpack-rpc protocol over TCP/IP through rpclib developed by Tam\u00c3\u00a1s Szelei which allows you to use variety of programming languages including C++, C#, Python, Java etc. When AirSim starts, it opens port 41451 (this can be changed via settings ) and listens for incoming request. The Python or C++ client code connects to this port and sends RPC calls using msgpack serialization format .","title":"Some Internals"},{"location":"apis/#references-and-examples","text":"C++ API Examples Car Examples Multirotor Examples Computer Vision Examples Move on Path demo showing video of fast multirotor flight through Modular Neighborhood environment Building a Hexacopter Building Point Clouds","title":"References and Examples"},{"location":"apis/#faq","text":"","title":"FAQ"},{"location":"apis/#unreal-is-slowed-down-dramatically-when-i-run-api","text":"If you see Unreal getting slowed down dramatically when Unreal Engine window loses focus then go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked.","title":"Unreal is slowed down dramatically when I run API"},{"location":"apis/#do-i-need-anything-else-on-windows","text":"You should install VS2019 with VC++, Windows SDK 10.0 and Python. To use Python APIs you will need Python 3.5 or later (install it using Anaconda).","title":"Do I need anything else on Windows?"},{"location":"apis/#which-version-of-python-should-i-use","text":"We recommend Anaconda to get Python tools and libraries. Our code is tested with Python 3.5.3 :: Anaconda 4.4.0. This is important because older version have been known to have problems .","title":"Which version of Python should I use?"},{"location":"apis/#i-get-error-on-import-cv2","text":"You can install OpenCV using: conda install opencv pip install opencv-python","title":"I get error on import cv2"},{"location":"apis/#typeerror-unsupported-operand-types-for-asyncioloop-and-float","text":"This error happens if you install Jupyter, which somehow breaks the msgpackrpc library. Create a new python environment which the minimal required packages.","title":"TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float'"},{"location":"apis_cpp/","text":"Using C++ APIs for AirSim # Please read general API doc first if you haven't already. This document describes C++ examples and other C++ specific details. Quick Start # Fastest way to get started is to open AirSim.sln in Visual Studio 2019. You will see Hello Car and Hello Drone examples in the solution. These examples will show you the include paths and lib paths you will need to setup in your VC++ projects. If you are using Linux then you will specify these paths either in your cmake file or on compiler command line. Include and Lib Folders # Include folders: $(ProjectDir)..\\AirLib\\deps\\rpclib\\include;include;$(ProjectDir)..\\AirLib\\deps\\eigen3;$(ProjectDir)..\\AirLib\\include Dependencies: rpc.lib Lib folders: $(ProjectDir)\\..\\AirLib\\deps\\MavLinkCom\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\deps\\rpclib\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\lib\\$(Platform)\\$(Configuration) Hello Car # Here's how to use AirSim APIs using C++ to control simulated car (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloCar/main.cpp #include #include \"vehicles/car/api/CarRpcLibClient.hpp\" int main() { msr::airlib::CarRpcLibClient client; client.enableApiControl(true); //this disables manual control CarControllerBase::CarControls controls; std::cout << \"Press enter to drive forward\" << std::endl; std::cin.get(); controls.throttle = 1; client.setCarControls(controls); std::cout << \"Press Enter to activate handbrake\" << std::endl; std::cin.get(); controls.handbrake = true; client.setCarControls(controls); std::cout << \"Press Enter to take turn and drive backward\" << std::endl; std::cin.get(); controls.handbrake = false; controls.throttle = -1; controls.steering = 1; client.setCarControls(controls); std::cout << \"Press Enter to stop\" << std::endl; std::cin.get(); client.setCarControls(CarControllerBase::CarControls()); return 0; } Hello Drone # Here's how to use AirSim APIs using C++ to control simulated quadrotor (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloDrone/main.cpp #include #include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int main() { using namespace std; msr::airlib::MultirotorRpcLibClient client; cout << \"Press Enter to enable API control\" << endl; cin.get(); client.enableApiControl(true); cout << \"Press Enter to arm the drone\" << endl; cin.get(); client.armDisarm(true); cout << \"Press Enter to takeoff\" << endl; cin.get(); client.takeoffAsync(5)->waitOnLastTask(); cout << \"Press Enter to move 5 meters in x direction with 1 m/s velocity\" << endl; cin.get(); auto position = client.getMultirotorState().getPosition(); // from current location client.moveToPositionAsync(position.x() + 5, position.y(), position.z(), 1)->waitOnLastTask(); cout << \"Press Enter to land\" << endl; cin.get(); client.landAsync()->waitOnLastTask(); return 0; } See Also # Examples of how to use internal infrastructure in AirSim in your other projects DroneShell app shows how to make simple interface using C++ APIs to control drones Python APIs","title":"C++ APIs"},{"location":"apis_cpp/#using-c-apis-for-airsim","text":"Please read general API doc first if you haven't already. This document describes C++ examples and other C++ specific details.","title":"Using C++ APIs for AirSim"},{"location":"apis_cpp/#quick-start","text":"Fastest way to get started is to open AirSim.sln in Visual Studio 2019. You will see Hello Car and Hello Drone examples in the solution. These examples will show you the include paths and lib paths you will need to setup in your VC++ projects. If you are using Linux then you will specify these paths either in your cmake file or on compiler command line.","title":"Quick Start"},{"location":"apis_cpp/#include-and-lib-folders","text":"Include folders: $(ProjectDir)..\\AirLib\\deps\\rpclib\\include;include;$(ProjectDir)..\\AirLib\\deps\\eigen3;$(ProjectDir)..\\AirLib\\include Dependencies: rpc.lib Lib folders: $(ProjectDir)\\..\\AirLib\\deps\\MavLinkCom\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\deps\\rpclib\\lib\\$(Platform)\\$(Configuration);$(ProjectDir)\\..\\AirLib\\lib\\$(Platform)\\$(Configuration)","title":"Include and Lib Folders"},{"location":"apis_cpp/#hello-car","text":"Here's how to use AirSim APIs using C++ to control simulated car (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloCar/main.cpp #include #include \"vehicles/car/api/CarRpcLibClient.hpp\" int main() { msr::airlib::CarRpcLibClient client; client.enableApiControl(true); //this disables manual control CarControllerBase::CarControls controls; std::cout << \"Press enter to drive forward\" << std::endl; std::cin.get(); controls.throttle = 1; client.setCarControls(controls); std::cout << \"Press Enter to activate handbrake\" << std::endl; std::cin.get(); controls.handbrake = true; client.setCarControls(controls); std::cout << \"Press Enter to take turn and drive backward\" << std::endl; std::cin.get(); controls.handbrake = false; controls.throttle = -1; controls.steering = 1; client.setCarControls(controls); std::cout << \"Press Enter to stop\" << std::endl; std::cin.get(); client.setCarControls(CarControllerBase::CarControls()); return 0; }","title":"Hello Car"},{"location":"apis_cpp/#hello-drone","text":"Here's how to use AirSim APIs using C++ to control simulated quadrotor (see also Python example ): // ready to run example: https://github.com/Microsoft/AirSim/blob/master/HelloDrone/main.cpp #include #include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int main() { using namespace std; msr::airlib::MultirotorRpcLibClient client; cout << \"Press Enter to enable API control\" << endl; cin.get(); client.enableApiControl(true); cout << \"Press Enter to arm the drone\" << endl; cin.get(); client.armDisarm(true); cout << \"Press Enter to takeoff\" << endl; cin.get(); client.takeoffAsync(5)->waitOnLastTask(); cout << \"Press Enter to move 5 meters in x direction with 1 m/s velocity\" << endl; cin.get(); auto position = client.getMultirotorState().getPosition(); // from current location client.moveToPositionAsync(position.x() + 5, position.y(), position.z(), 1)->waitOnLastTask(); cout << \"Press Enter to land\" << endl; cin.get(); client.landAsync()->waitOnLastTask(); return 0; }","title":"Hello Drone"},{"location":"apis_cpp/#see-also","text":"Examples of how to use internal infrastructure in AirSim in your other projects DroneShell app shows how to make simple interface using C++ APIs to control drones Python APIs","title":"See Also"},{"location":"azure/","text":"AirSim Development Environment on Azure # This document explains how to automate the creation of a development environment on Azure and code and debug a Python application connected to AirSim using Visual Studio Code Automatically Deploy Your Azure VM # Use this template to create, deploy and configure an Azure VM to work with AirSim Note: the VM deployment and configuration process may take 20+ minutes to complete Regarding the deployment of the Azure VM # When using an Azure Trial account, the default vCPU quota may not be enough to allocate the required VM to run AirSim. If that's the case, you will see an error when trying to create the VM and will have to make a support request to increase the default quota. To avoid charges for the Virtual Machine usage while not in use, remember to deallocate its resources from the Azure Portal or use the following command from the Azure CLI: bash az vm deallocate --resource-group MyResourceGroup --name MyVMName Code and debug from Visual Studio Code and Remote SSH # Install Visual Studio Code Install the Remote - SSH extension Press F1 and run the Remote - SSH: Connect to host... command Add the recently create VM details. For instance, AzureUser@11.22.33.44 Run the Remote - SSH: Connect to host... command again, and now select the newly added connection. Once connected, click on the Clone Repository button in Visual Studio Code, and either clone this repository in the remote VM and open just the azure folder , or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. It is important to open that directory so Visual Studio Code can use the specific .vscode directory for the scenario and not the general AirSim .vscode directory. It contains the recommended extensions to install, the task to start AirSim remotely and the launch configuration for the Python application. Install all the recommended extensions Press F1 and select the Tasks: Run Task option. Then, select the Start AirSim task from Visual Studio Code to execute the start-airsim.ps1 script from Visual Studio Code. Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges Code and debug from a local Visual Studio Code and connect to AirSim via forwarded ports # Note: this scenario, will be using two Visual Studio Code instances. The first one will be used as a bridge to forward ports via SSH to the Azure VM and execute remote processes, and the second one will be used for local Python development. To be able to reach the VM from the local Python code, it is required to keep the Remote - SSH instance of Visual Studio Code opened, while working with the local Python environment on the second instance Open the first Visual Studio Code instance Follow the steps in the previous section to connect via Remote - SSH In the Remote Explorer , add the port 41451 as a forwarded port to localhost Either run the Start AirSim task on the Visual Studio Code with the remote session as explained in the previous scenario or manually start the AirSim binary in the VM Open a second Visual Studio Code instance, without disconnecting or closing the first one Either clone this repository locally and open just the azure folder in Visual Studio Code, or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. Run pip install -r requirements.txt inside the app directory Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges Running with Docker # Once both the AirSim environment and the Python application are ready, you can package everything as a Docker image. The sample project inside the azure directory is already prepared to run a prebuilt AirSim binary and Python code using Docker. This would be a perfect scenario when you want to run the simulation at scale. For instance, you could have several different configurations for the same simulation and execute them in a parallel, unattended way using a Docker image on Azure Container Services Since AirSim requires access to the host GPU, it is required to use a Docker runtime that supports it. For more information about running AirSim in Docker, click here . When using Azure Container Services to run this image, the only extra-requirement is to add GPU support to the Container Group where it will be deployed. It can use either public docker images from DockerHub or images deployed to a private Azure Container Registry Building the docker image # docker build -t / -f ./docker/Dockerfile .` Using a different AirSim binary # To use a different AirSim binary, first check the official documentation on How to Build AirSim on Windows and How to Build AirSim on Linux if you also want to run it with Docker Once you have a zip file with the new AirSim environment (or prefer to use one from the Official Releases ), you need to modify some of the scripts in the azure directory of the repository to point to the new environment: - In azure/azure-env-creation/configure-vm.ps1 , modify all the parameters starting with $airSimBinary with the new values - In azure/start-airsim.ps1 , modify $airSimExecutable and $airSimProcessName with the new values If you are using the docker image, you also need a linux binary zip file and modify the following Docker-related files: - In azure/docker/Dockerfile , modify the AIRSIM_BINARY_ZIP_URL and AIRSIM_BINARY_ZIP_FILENAME ENV declarations with the new values - In azure/docker/docker-entrypoint.sh , modify AIRSIM_EXECUTABLE with the new value Maintaining this development environment # Several components of this development environment (ARM templates, initialization scripts and VSCode tasks) directly depend on the current directory structures file names and repository locations. When planning to modify/fork any of those, make sure to check every script and template to make any required adjustment.","title":"AirSim on Azure"},{"location":"azure/#airsim-development-environment-on-azure","text":"This document explains how to automate the creation of a development environment on Azure and code and debug a Python application connected to AirSim using Visual Studio Code","title":"AirSim Development Environment on Azure"},{"location":"azure/#automatically-deploy-your-azure-vm","text":"Use this template to create, deploy and configure an Azure VM to work with AirSim Note: the VM deployment and configuration process may take 20+ minutes to complete","title":"Automatically Deploy Your Azure VM"},{"location":"azure/#regarding-the-deployment-of-the-azure-vm","text":"When using an Azure Trial account, the default vCPU quota may not be enough to allocate the required VM to run AirSim. If that's the case, you will see an error when trying to create the VM and will have to make a support request to increase the default quota. To avoid charges for the Virtual Machine usage while not in use, remember to deallocate its resources from the Azure Portal or use the following command from the Azure CLI: bash az vm deallocate --resource-group MyResourceGroup --name MyVMName","title":"Regarding the deployment of the Azure VM"},{"location":"azure/#code-and-debug-from-visual-studio-code-and-remote-ssh","text":"Install Visual Studio Code Install the Remote - SSH extension Press F1 and run the Remote - SSH: Connect to host... command Add the recently create VM details. For instance, AzureUser@11.22.33.44 Run the Remote - SSH: Connect to host... command again, and now select the newly added connection. Once connected, click on the Clone Repository button in Visual Studio Code, and either clone this repository in the remote VM and open just the azure folder , or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. It is important to open that directory so Visual Studio Code can use the specific .vscode directory for the scenario and not the general AirSim .vscode directory. It contains the recommended extensions to install, the task to start AirSim remotely and the launch configuration for the Python application. Install all the recommended extensions Press F1 and select the Tasks: Run Task option. Then, select the Start AirSim task from Visual Studio Code to execute the start-airsim.ps1 script from Visual Studio Code. Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges","title":"Code and debug from Visual Studio Code and Remote SSH"},{"location":"azure/#code-and-debug-from-a-local-visual-studio-code-and-connect-to-airsim-via-forwarded-ports","text":"Note: this scenario, will be using two Visual Studio Code instances. The first one will be used as a bridge to forward ports via SSH to the Azure VM and execute remote processes, and the second one will be used for local Python development. To be able to reach the VM from the local Python code, it is required to keep the Remote - SSH instance of Visual Studio Code opened, while working with the local Python environment on the second instance Open the first Visual Studio Code instance Follow the steps in the previous section to connect via Remote - SSH In the Remote Explorer , add the port 41451 as a forwarded port to localhost Either run the Start AirSim task on the Visual Studio Code with the remote session as explained in the previous scenario or manually start the AirSim binary in the VM Open a second Visual Studio Code instance, without disconnecting or closing the first one Either clone this repository locally and open just the azure folder in Visual Studio Code, or create a brand new repository, clone it and copy the contents of the azure folder from this repository in it. Run pip install -r requirements.txt inside the app directory Open the multirotor.py file inside the app directory Start debugging with Python When finished, remember to stop an deallocate the Azure VM to avoid extra charges","title":"Code and debug from a local Visual Studio Code and connect to AirSim via forwarded ports"},{"location":"azure/#running-with-docker","text":"Once both the AirSim environment and the Python application are ready, you can package everything as a Docker image. The sample project inside the azure directory is already prepared to run a prebuilt AirSim binary and Python code using Docker. This would be a perfect scenario when you want to run the simulation at scale. For instance, you could have several different configurations for the same simulation and execute them in a parallel, unattended way using a Docker image on Azure Container Services Since AirSim requires access to the host GPU, it is required to use a Docker runtime that supports it. For more information about running AirSim in Docker, click here . When using Azure Container Services to run this image, the only extra-requirement is to add GPU support to the Container Group where it will be deployed. It can use either public docker images from DockerHub or images deployed to a private Azure Container Registry","title":"Running with Docker"},{"location":"azure/#building-the-docker-image","text":"docker build -t / -f ./docker/Dockerfile .`","title":"Building the docker image"},{"location":"azure/#using-a-different-airsim-binary","text":"To use a different AirSim binary, first check the official documentation on How to Build AirSim on Windows and How to Build AirSim on Linux if you also want to run it with Docker Once you have a zip file with the new AirSim environment (or prefer to use one from the Official Releases ), you need to modify some of the scripts in the azure directory of the repository to point to the new environment: - In azure/azure-env-creation/configure-vm.ps1 , modify all the parameters starting with $airSimBinary with the new values - In azure/start-airsim.ps1 , modify $airSimExecutable and $airSimProcessName with the new values If you are using the docker image, you also need a linux binary zip file and modify the following Docker-related files: - In azure/docker/Dockerfile , modify the AIRSIM_BINARY_ZIP_URL and AIRSIM_BINARY_ZIP_FILENAME ENV declarations with the new values - In azure/docker/docker-entrypoint.sh , modify AIRSIM_EXECUTABLE with the new value","title":"Using a different AirSim binary"},{"location":"azure/#maintaining-this-development-environment","text":"Several components of this development environment (ARM templates, initialization scripts and VSCode tasks) directly depend on the current directory structures file names and repository locations. When planning to modify/fork any of those, make sure to check every script and template to make any required adjustment.","title":"Maintaining this development environment"},{"location":"build_linux/","text":"Build AirSim on Linux & MacOS # The current recommended and tested environment is Ubuntu 18.04 LTS . Theoretically, you can build on other distros as well, but we haven't tested it. Only macOS Catalina (10.15) is supported. We've two options - you can either build inside docker containers or your host machine. Docker # Please see instructions here Host machine # Pre-build Setup # Linux - Build Unreal Engine # Make sure you are registered with Epic Games . This is required to get source code access for Unreal Engine. Clone Unreal in your favorite folder and build it (this may take a while!). Note : We only support Unreal >= 4.22 at present. We recommend using 4.24. # go to the folder where you clone GitHub projects git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git cd UnrealEngine ./Setup.sh ./GenerateProjectFiles.sh make macOS - Download Unreal Engine # Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects. Build AirSim # Clone AirSim and build it: # go to the folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git cd AirSim By default AirSim uses clang 8 to build for compatibility with UE 4.24. The setup script will install the right version of cmake, llvm, and eigen. ./setup.sh ./build.sh # use ./build.sh --debug to build in debug mode Build Unreal Environment # Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment if you'd like to setup your own environment. How to Use AirSim # Linux # Once AirSim is setup: Go to UnrealEngine installation folder and start Unreal by running ./Engine/Binaries/Linux/UE4Editor . When Unreal Engine prompts for opening or creating project, select Browse and choose AirSim/Unreal/Environments/Blocks (or your custom Unreal project). Alternatively, the project file can be passed as a commandline argument. For Blocks: ./Engine/Binaries/Linux/UE4Editor /Unreal/Environments/Blocks/Blocks.uproject If you get prompts to convert project, look for More Options or Convert-In-Place option. If you get prompted to build, choose Yes. If you get prompted to disable AirSim plugin, choose No. After Unreal Editor loads, press Play button. Mac # Browse to AirSim/Unreal/Environments/Blocks . Run ./GenerateProjectFiles.sh from the terminal, where UE_PATH is the path to the Unreal installation folder. (By default, this is /Users/Shared/Epic\\ Games/UE_4.24/ ) The script creates an XCode workspace by the name Blocks.xcworkspace. Open the XCode workspace, and press the Build and run button in the top left. After Unreal Editor loads, press Play button. See Using APIs and settings.json for various options available for AirSim usage. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. [Optional] Setup Remote Control (Multirotor Only) # A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard. FAQs # I'm getting error \" could not be compiled. Try rebuilding from source manually\". This could either happen because of compile error or the fact that your gch files are outdated. Look in to your console window. Do you see something like below? fatal error: file '/usr/include/linux/version.h''/usr/include/linux/version.h' has been modified since the precompiled header If this is the case then look for *.gch file(s) that follows after that message, delete them and try again. Here's relevant thread on Unreal Engine forums. If you see other compile errors in console then open up those source files and see if it is due to changes you made. If not, then report it as issue on GitHub. Unreal crashed! How do I know what went wrong? Go to the MyUnrealProject/Saved/Crashes folder and search for the file MyProject.log within its subdirectories. At the end of this file you will see the stack trace and messages. You can also take a look at the Diagnostics.txt file. How do I use an IDE on Linux? You can use Qt Creator or CodeLite. Instructions for Qt Creator are available here . Can I cross compile for Linux from a Windows machine? Yes, you can, but we haven't tested it. You can find the instructions here . What compiler and stdlib does AirSim use? We use the same compiler that Unreal Engine uses, Clang 8 , and stdlib, libc++ . AirSim's setup.sh will automatically download them. What version of CMake does the AirSim build use? 3.10.0 or higher. This is not the default in Ubuntu 16.04 so setup.sh installs it for you. You can check your CMake version using cmake --version . If you have an older version, follow these instructions or see the CMake website . Can I compile AirSim in BashOnWindows? Yes, however, you can't run Unreal from BashOnWindows. So this is kind of useful to check a Linux compile, but not for an end-to-end run. See the BashOnWindows install guide . Make sure to have the latest version (Windows 10 Creators Edition) as previous versions had various issues. Also, don't invoke bash from Visual Studio Command Prompt , otherwise CMake might find VC++ and try and use that! Where can I find more info on running Unreal on Linux? Start here: Unreal on Linux Building Unreal on Linux Unreal Linux Support Unreal Cross Compilation","title":"Build on Linux"},{"location":"build_linux/#build-airsim-on-linux-macos","text":"The current recommended and tested environment is Ubuntu 18.04 LTS . Theoretically, you can build on other distros as well, but we haven't tested it. Only macOS Catalina (10.15) is supported. We've two options - you can either build inside docker containers or your host machine.","title":"Build AirSim on Linux & MacOS"},{"location":"build_linux/#docker","text":"Please see instructions here","title":"Docker"},{"location":"build_linux/#host-machine","text":"","title":"Host machine"},{"location":"build_linux/#pre-build-setup","text":"","title":"Pre-build Setup"},{"location":"build_linux/#linux-build-unreal-engine","text":"Make sure you are registered with Epic Games . This is required to get source code access for Unreal Engine. Clone Unreal in your favorite folder and build it (this may take a while!). Note : We only support Unreal >= 4.22 at present. We recommend using 4.24. # go to the folder where you clone GitHub projects git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git cd UnrealEngine ./Setup.sh ./GenerateProjectFiles.sh make","title":"Linux - Build Unreal Engine"},{"location":"build_linux/#macos-download-unreal-engine","text":"Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects.","title":"macOS - Download Unreal Engine"},{"location":"build_linux/#build-airsim","text":"Clone AirSim and build it: # go to the folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git cd AirSim By default AirSim uses clang 8 to build for compatibility with UE 4.24. The setup script will install the right version of cmake, llvm, and eigen. ./setup.sh ./build.sh # use ./build.sh --debug to build in debug mode","title":"Build AirSim"},{"location":"build_linux/#build-unreal-environment","text":"Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment if you'd like to setup your own environment.","title":"Build Unreal Environment"},{"location":"build_linux/#how-to-use-airsim","text":"","title":"How to Use AirSim"},{"location":"build_linux/#linux","text":"Once AirSim is setup: Go to UnrealEngine installation folder and start Unreal by running ./Engine/Binaries/Linux/UE4Editor . When Unreal Engine prompts for opening or creating project, select Browse and choose AirSim/Unreal/Environments/Blocks (or your custom Unreal project). Alternatively, the project file can be passed as a commandline argument. For Blocks: ./Engine/Binaries/Linux/UE4Editor /Unreal/Environments/Blocks/Blocks.uproject If you get prompts to convert project, look for More Options or Convert-In-Place option. If you get prompted to build, choose Yes. If you get prompted to disable AirSim plugin, choose No. After Unreal Editor loads, press Play button.","title":"Linux"},{"location":"build_linux/#mac","text":"Browse to AirSim/Unreal/Environments/Blocks . Run ./GenerateProjectFiles.sh from the terminal, where UE_PATH is the path to the Unreal installation folder. (By default, this is /Users/Shared/Epic\\ Games/UE_4.24/ ) The script creates an XCode workspace by the name Blocks.xcworkspace. Open the XCode workspace, and press the Build and run button in the top left. After Unreal Editor loads, press Play button. See Using APIs and settings.json for various options available for AirSim usage. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked.","title":"Mac"},{"location":"build_linux/#optional-setup-remote-control-multirotor-only","text":"A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard.","title":"[Optional] Setup Remote Control (Multirotor Only)"},{"location":"build_linux/#faqs","text":"I'm getting error \" could not be compiled. Try rebuilding from source manually\". This could either happen because of compile error or the fact that your gch files are outdated. Look in to your console window. Do you see something like below? fatal error: file '/usr/include/linux/version.h''/usr/include/linux/version.h' has been modified since the precompiled header If this is the case then look for *.gch file(s) that follows after that message, delete them and try again. Here's relevant thread on Unreal Engine forums. If you see other compile errors in console then open up those source files and see if it is due to changes you made. If not, then report it as issue on GitHub. Unreal crashed! How do I know what went wrong? Go to the MyUnrealProject/Saved/Crashes folder and search for the file MyProject.log within its subdirectories. At the end of this file you will see the stack trace and messages. You can also take a look at the Diagnostics.txt file. How do I use an IDE on Linux? You can use Qt Creator or CodeLite. Instructions for Qt Creator are available here . Can I cross compile for Linux from a Windows machine? Yes, you can, but we haven't tested it. You can find the instructions here . What compiler and stdlib does AirSim use? We use the same compiler that Unreal Engine uses, Clang 8 , and stdlib, libc++ . AirSim's setup.sh will automatically download them. What version of CMake does the AirSim build use? 3.10.0 or higher. This is not the default in Ubuntu 16.04 so setup.sh installs it for you. You can check your CMake version using cmake --version . If you have an older version, follow these instructions or see the CMake website . Can I compile AirSim in BashOnWindows? Yes, however, you can't run Unreal from BashOnWindows. So this is kind of useful to check a Linux compile, but not for an end-to-end run. See the BashOnWindows install guide . Make sure to have the latest version (Windows 10 Creators Edition) as previous versions had various issues. Also, don't invoke bash from Visual Studio Command Prompt , otherwise CMake might find VC++ and try and use that! Where can I find more info on running Unreal on Linux? Start here: Unreal on Linux Building Unreal on Linux Unreal Linux Support Unreal Cross Compilation","title":"FAQs"},{"location":"build_windows/","text":"Build AirSim on Windows # Install Unreal Engine # Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects. Build AirSim # Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start Developer Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line. This will create ready to use plugin bits in the Unreal\\Plugins folder that can be dropped into any Unreal project. Build Unreal Project # Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment . Setup Remote Control (Multirotor only) # A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard. How to Use AirSim # Once AirSim is set up by following above steps, you can, Double click on .sln file to load the Blocks project in Unreal\\Environments\\Blocks (or .sln file in your own custom Unreal project). If you don't see .sln file then you probably haven't completed steps in Build Unreal Project section above. Select your Unreal project as Start Up project (for example, Blocks project) and make sure Build config is set to \"Develop Editor\" and x64. After Unreal Editor loads, press Play button. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. See Using APIs and settings.json for various options available. AirSim on Unity (Experimental) # Unity is another great game engine platform and we have an experimental integration of AirSim with Unity . Please note that this is work in progress and all features may not work yet. FAQ # How to force Unreal to use Visual Studio 2019? # If the default update_from_git.bat file results in VS 2017 project, then you may need to run the C:\\Program Files\\Epic Games\\UE_4.24\\Engine\\Binaries\\DotNET\\UnrealBuildTool.exe tool manually, with the command line options -projectfiles -project= -game -rocket -progress -2019 . If you are upgrading from 4.18 to 4.24 you may also need to add BuildSettingsVersion.V2 to your *.Target.cs and *Editor.Target.cs build files, like this: public AirSimNHTestTarget(TargetInfo Target) : base(Target) { Type = TargetType.Game; DefaultBuildSettings = BuildSettingsVersion.V2; ExtraModuleNames.AddRange(new string[] { \"AirSimNHTest\" }); } You may also need to edit this file: \"%APPDATA%\\Unreal Engine\\UnrealBuildTool\\BuildConfiguration.xml And add this Compiler version setting: VisualStudio2019 I get error C100 : An internal error has occurred in the compiler when running build.cmd # We have noticed this happening with VS version 15.9.0 and have checked-in a workaround in AirSim code. If you have this VS version, please make sure to pull the latest AirSim code. I get error \"'corecrt.h': No such file or directory\" or \"Windows SDK version 8.1 not found\" # Very likely you don't have Windows SDK installed with Visual Studio. How do I use PX4 firmware with AirSim? # By default, AirSim uses its own built-in firmware called simple_flight . There is no additional setup if you just want to go with it. If you want to switch to using PX4 instead then please see this guide . I made changes in Visual Studio but there is no effect # Sometimes the Unreal + VS build system doesn't recompile if you make changes to only header files. To ensure a recompile, make some Unreal based cpp file \"dirty\" like AirSimGameMode.cpp. Unreal still uses VS2015 or I'm getting some link error # Running several versions of VS can lead to issues when compiling UE projects. One problem that may arise is that UE will try to compile with an older version of VS which may or may not work. There are two settings in Unreal, one for for the engine and one for the project, to adjust the version of VS to be used. 1. Edit -> Editor preferences -> General -> Source code 2. Edit -> Project Settings -> Platforms -> Windows -> Toolchain ->CompilerVersion In some cases, these settings will still not lead to the desired result and errors such as the following might be produced: LINK : fatal error LNK1181: cannot open input file 'ws2_32.lib' To resolve such issues the following procedure can be applied: 1. Uninstall all old versions of VS using the VisualStudioUninstaller 2. Repair/Install VS 2019 3. Restart machine and install Epic launcher and desired version of the engine","title":"Build on Windows"},{"location":"build_windows/#build-airsim-on-windows","text":"","title":"Build AirSim on Windows"},{"location":"build_windows/#install-unreal-engine","text":"Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required. Run the Epic Games Launcher, open the Library tab on the left pane. Click on the Add Versions which should show the option to download Unreal 4.24 as shown below. If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version. Note : AirSim also works with UE >= 4.22, however, we recommend you update to 4.24. Note : If you have UE 4.16 or older projects, please see the upgrade guide to upgrade your projects.","title":"Install Unreal Engine"},{"location":"build_windows/#build-airsim","text":"Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019. Start Developer Command Prompt for VS 2019 . Clone the repo: git clone https://github.com/Microsoft/AirSim.git , and go the AirSim directory by cd AirSim . Run build.cmd from the command line. This will create ready to use plugin bits in the Unreal\\Plugins folder that can be dropped into any Unreal project.","title":"Build AirSim"},{"location":"build_windows/#build-unreal-project","text":"Finally, you will need an Unreal project that hosts the environment for your vehicles. AirSim comes with a built-in \"Blocks Environment\" which you can use, or you can create your own. Please see setting up Unreal Environment .","title":"Build Unreal Project"},{"location":"build_windows/#setup-remote-control-multirotor-only","text":"A remote control is required if you want to fly manually. See the remote control setup for more details. Alternatively, you can use APIs for programmatic control or use the so-called Computer Vision mode to move around using the keyboard.","title":"Setup Remote Control (Multirotor only)"},{"location":"build_windows/#how-to-use-airsim","text":"Once AirSim is set up by following above steps, you can, Double click on .sln file to load the Blocks project in Unreal\\Environments\\Blocks (or .sln file in your own custom Unreal project). If you don't see .sln file then you probably haven't completed steps in Build Unreal Project section above. Select your Unreal project as Start Up project (for example, Blocks project) and make sure Build config is set to \"Develop Editor\" and x64. After Unreal Editor loads, press Play button. Tip: go to 'Edit->Editor Preferences', in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. See Using APIs and settings.json for various options available.","title":"How to Use AirSim"},{"location":"build_windows/#airsim-on-unity-experimental","text":"Unity is another great game engine platform and we have an experimental integration of AirSim with Unity . Please note that this is work in progress and all features may not work yet.","title":"AirSim on Unity (Experimental)"},{"location":"build_windows/#faq","text":"","title":"FAQ"},{"location":"build_windows/#how-to-force-unreal-to-use-visual-studio-2019","text":"If the default update_from_git.bat file results in VS 2017 project, then you may need to run the C:\\Program Files\\Epic Games\\UE_4.24\\Engine\\Binaries\\DotNET\\UnrealBuildTool.exe tool manually, with the command line options -projectfiles -project= -game -rocket -progress -2019 . If you are upgrading from 4.18 to 4.24 you may also need to add BuildSettingsVersion.V2 to your *.Target.cs and *Editor.Target.cs build files, like this: public AirSimNHTestTarget(TargetInfo Target) : base(Target) { Type = TargetType.Game; DefaultBuildSettings = BuildSettingsVersion.V2; ExtraModuleNames.AddRange(new string[] { \"AirSimNHTest\" }); } You may also need to edit this file: \"%APPDATA%\\Unreal Engine\\UnrealBuildTool\\BuildConfiguration.xml And add this Compiler version setting: VisualStudio2019 ","title":"How to force Unreal to use Visual Studio 2019?"},{"location":"build_windows/#i-get-error-c100-an-internal-error-has-occurred-in-the-compiler-when-running-buildcmd","text":"We have noticed this happening with VS version 15.9.0 and have checked-in a workaround in AirSim code. If you have this VS version, please make sure to pull the latest AirSim code.","title":"I get error C100 : An internal error has occurred in the compiler when running build.cmd"},{"location":"build_windows/#i-get-error-corecrth-no-such-file-or-directory-or-windows-sdk-version-81-not-found","text":"Very likely you don't have Windows SDK installed with Visual Studio.","title":"I get error \"'corecrt.h': No such file or directory\" or \"Windows SDK version 8.1 not found\""},{"location":"build_windows/#how-do-i-use-px4-firmware-with-airsim","text":"By default, AirSim uses its own built-in firmware called simple_flight . There is no additional setup if you just want to go with it. If you want to switch to using PX4 instead then please see this guide .","title":"How do I use PX4 firmware with AirSim?"},{"location":"build_windows/#i-made-changes-in-visual-studio-but-there-is-no-effect","text":"Sometimes the Unreal + VS build system doesn't recompile if you make changes to only header files. To ensure a recompile, make some Unreal based cpp file \"dirty\" like AirSimGameMode.cpp.","title":"I made changes in Visual Studio but there is no effect"},{"location":"build_windows/#unreal-still-uses-vs2015-or-im-getting-some-link-error","text":"Running several versions of VS can lead to issues when compiling UE projects. One problem that may arise is that UE will try to compile with an older version of VS which may or may not work. There are two settings in Unreal, one for for the engine and one for the project, to adjust the version of VS to be used. 1. Edit -> Editor preferences -> General -> Source code 2. Edit -> Project Settings -> Platforms -> Windows -> Toolchain ->CompilerVersion In some cases, these settings will still not lead to the desired result and errors such as the following might be produced: LINK : fatal error LNK1181: cannot open input file 'ws2_32.lib' To resolve such issues the following procedure can be applied: 1. Uninstall all old versions of VS using the VisualStudioUninstaller 2. Repair/Install VS 2019 3. Restart machine and install Epic launcher and desired version of the engine","title":"Unreal still uses VS2015 or I'm getting some link error"},{"location":"camera_views/","text":"Camera Views # The camera views that are shown on screen are the camera views you can fetch via the simGetImages API . From left to right is the depth view, segmentation view and the FPV view. See Image APIs for description of various available views. Turning ON/OFF Views # Press F1 key to see keyboard shortcuts for turning on/off any or all views. You can also select various view modes there, such as \"Fly with Me\" mode, FPV mode and \"Ground View\" mode. Configuring Sub-Windows # Now you can select what is shown by each of above sub windows. For instance, you can chose to show surface normals in first window (instead of depth) and disparity in second window (instead of segmentation). Below is the settings value you can use in settings.json : { \"SubWindows\": [ {\"Index\": 1, \"ImageType\": 5}, {\"Index\": 2, \"ImageType\": 3} ] } Performance Impact # Note : This section is outdated and has not been updated for new performance enhancement changes. Now rendering these views does impact the FPS performance of the game, since this is additional work for the GPU. The following shows the impact on FPS when you open these views. This is measured on Intel core i7 computer with 32 gb RAM and a GeForce GTX 1080 graphics card running the Modular Neighborhood map, using cooked debug bits, no debugger or GameEditor open. The normal state with no subviews open is measuring around 16 ms per frame, which means it is keeping a nice steady 60 FPS (which is the target FPS). As it climbs up to 35ms the FPS drops to around 28 frames per second, spiking to 40ms means a few drops to 25 fps. The simulator can still function and fly correctly when all this is going on even in the worse case because the physics is decoupled from the rendering. However if the delay gets too high such that the communication with PX4 hardware is interrupted due to overly busy CPU then the flight can stall due to timeout in the offboard control messages. On the computer where this was measured the drone could fly the path.py program without any problems with all views open, and with 3 python scripts running to capture each view type. But there was one stall during this flight, but it recovered gracefully and completed the path. So it was right on the limit. The following shows the impact on CPU, perhaps a bit surprisingly, the CPU impact is also non trivial.","title":"Camera Views"},{"location":"camera_views/#camera-views","text":"The camera views that are shown on screen are the camera views you can fetch via the simGetImages API . From left to right is the depth view, segmentation view and the FPV view. See Image APIs for description of various available views.","title":"Camera Views"},{"location":"camera_views/#turning-onoff-views","text":"Press F1 key to see keyboard shortcuts for turning on/off any or all views. You can also select various view modes there, such as \"Fly with Me\" mode, FPV mode and \"Ground View\" mode.","title":"Turning ON/OFF Views"},{"location":"camera_views/#configuring-sub-windows","text":"Now you can select what is shown by each of above sub windows. For instance, you can chose to show surface normals in first window (instead of depth) and disparity in second window (instead of segmentation). Below is the settings value you can use in settings.json : { \"SubWindows\": [ {\"Index\": 1, \"ImageType\": 5}, {\"Index\": 2, \"ImageType\": 3} ] }","title":"Configuring Sub-Windows"},{"location":"camera_views/#performance-impact","text":"Note : This section is outdated and has not been updated for new performance enhancement changes. Now rendering these views does impact the FPS performance of the game, since this is additional work for the GPU. The following shows the impact on FPS when you open these views. This is measured on Intel core i7 computer with 32 gb RAM and a GeForce GTX 1080 graphics card running the Modular Neighborhood map, using cooked debug bits, no debugger or GameEditor open. The normal state with no subviews open is measuring around 16 ms per frame, which means it is keeping a nice steady 60 FPS (which is the target FPS). As it climbs up to 35ms the FPS drops to around 28 frames per second, spiking to 40ms means a few drops to 25 fps. The simulator can still function and fly correctly when all this is going on even in the worse case because the physics is decoupled from the rendering. However if the delay gets too high such that the communication with PX4 hardware is interrupted due to overly busy CPU then the flight can stall due to timeout in the offboard control messages. On the computer where this was measured the drone could fly the path.py program without any problems with all views open, and with 3 python scripts running to capture each view type. But there was one stall during this flight, but it recovered gracefully and completed the path. So it was right on the limit. The following shows the impact on CPU, perhaps a bit surprisingly, the CPU impact is also non trivial.","title":"Performance Impact"},{"location":"cmake_linux/","text":"Installing cmake on Linux # If you don't have cmake version 3.10 (for example, 3.2.2 is the default on Ubuntu 14) you can run the following: mkdir ~/cmake-3.10.2 cd ~/cmake-3.10.2 wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.sh Now you have to run this command by itself (it is interactive) sh cmake-3.10.2-Linux-x86_64.sh --prefix ~/cmake-3.10.2 Answer 'n' to the question about creating another cmake-3.10.2-Linux-x86_64 folder and then sudo update-alternatives --install /usr/bin/cmake cmake ~/cmake-3.10.2/bin/cmake 60 Now type cmake --version to make sure your cmake version is 3.10.2.","title":"Installing cmake on Linux"},{"location":"cmake_linux/#installing-cmake-on-linux","text":"If you don't have cmake version 3.10 (for example, 3.2.2 is the default on Ubuntu 14) you can run the following: mkdir ~/cmake-3.10.2 cd ~/cmake-3.10.2 wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.sh Now you have to run this command by itself (it is interactive) sh cmake-3.10.2-Linux-x86_64.sh --prefix ~/cmake-3.10.2 Answer 'n' to the question about creating another cmake-3.10.2-Linux-x86_64 folder and then sudo update-alternatives --install /usr/bin/cmake cmake ~/cmake-3.10.2/bin/cmake 60 Now type cmake --version to make sure your cmake version is 3.10.2.","title":"Installing cmake on Linux"},{"location":"code_structure/","text":"AirLib # Majority of the code is located in AirLib. This is a self-contained library that you should be able to compile with any C++11 compiler. AirLib consists of the following components: 1. Physics engine: This is header-only physics engine. It is designed to be fast and extensible to implement different vehicles. 2. Sensor models: This is header-only models for Barometer, IMU, GPS and Magnetometer 3. Vehicle models: This is header-only models for vehicle configurations and models. Currently we have implemented model for a MultiRotor and a configuration for PX4 QuadRotor in the X config. 4. Control library: This part of AirLib provides abstract base class for our APIs and concrete implementation for specific vehicle platforms such as MavLink. It also has classes for the RPC client and server. Unreal/Plugins/AirSim # This is the only portion of project which is dependent on Unreal engine. We have kept it isolated so we can implement simulator for other platforms as well (for example, Unity). The Unreal code takes advantage of its UObject based classes including Blueprints. 1. SimMode_ classes : We wish to support various simulator modes such as pure Computer Vision mode where there is no drone. The SimMode classes help implement many different modes. 2. VehiclePawnBase : This is the base class for all vehicle pawn visualizations. 3. VehicleBase : This class provides abstract interface to implement a combination of rendering component (i.e. Unreal pawn), physics component (i.e. MultiRotor) and controller (i.e. MavLinkHelper). MavLinkCom # This is the library developed by our own team member Chris Lovett that provides C++ classes to talk to the MavLink devices. This library is stand alone and can be used in any project. See MavLinkCom for more info. Sample Programs # We have created a few sample programs to demonstrate how to use the API. See HelloDrone and DroneShell. DroneShell demonstrates how to connect to the simulator using UDP. The simulator is running a server (similar to DroneServer). Contributing # See Contribution Guidelines Unreal Framework # The following picture illustrates how AirSim is loaded and invoked by the Unreal Game Engine:","title":"Code Structure"},{"location":"code_structure/#airlib","text":"Majority of the code is located in AirLib. This is a self-contained library that you should be able to compile with any C++11 compiler. AirLib consists of the following components: 1. Physics engine: This is header-only physics engine. It is designed to be fast and extensible to implement different vehicles. 2. Sensor models: This is header-only models for Barometer, IMU, GPS and Magnetometer 3. Vehicle models: This is header-only models for vehicle configurations and models. Currently we have implemented model for a MultiRotor and a configuration for PX4 QuadRotor in the X config. 4. Control library: This part of AirLib provides abstract base class for our APIs and concrete implementation for specific vehicle platforms such as MavLink. It also has classes for the RPC client and server.","title":"AirLib"},{"location":"code_structure/#unrealpluginsairsim","text":"This is the only portion of project which is dependent on Unreal engine. We have kept it isolated so we can implement simulator for other platforms as well (for example, Unity). The Unreal code takes advantage of its UObject based classes including Blueprints. 1. SimMode_ classes : We wish to support various simulator modes such as pure Computer Vision mode where there is no drone. The SimMode classes help implement many different modes. 2. VehiclePawnBase : This is the base class for all vehicle pawn visualizations. 3. VehicleBase : This class provides abstract interface to implement a combination of rendering component (i.e. Unreal pawn), physics component (i.e. MultiRotor) and controller (i.e. MavLinkHelper).","title":"Unreal/Plugins/AirSim"},{"location":"code_structure/#mavlinkcom","text":"This is the library developed by our own team member Chris Lovett that provides C++ classes to talk to the MavLink devices. This library is stand alone and can be used in any project. See MavLinkCom for more info.","title":"MavLinkCom"},{"location":"code_structure/#sample-programs","text":"We have created a few sample programs to demonstrate how to use the API. See HelloDrone and DroneShell. DroneShell demonstrates how to connect to the simulator using UDP. The simulator is running a server (similar to DroneServer).","title":"Sample Programs"},{"location":"code_structure/#contributing","text":"See Contribution Guidelines","title":"Contributing"},{"location":"code_structure/#unreal-framework","text":"The following picture illustrates how AirSim is loaded and invoked by the Unreal Game Engine:","title":"Unreal Framework"},{"location":"coding_guidelines/","text":"Modern C++ Coding Guidelines # We are using Modern C++11. Smart pointers, Lambdas, and C++11 multithreading primitives are your friend. Quick Note # The great thing about \"standards\" is that there are many to chose from: ISO , Sutter & Stroustrup , ROS , LINUX , Google's , Microsoft's , CERN's , GCC's , ARM's , LLVM's and probably thousands of others. Unfortunately most of these can't even agree on something as basic as how to name a class or a constant. This is probably due to the fact that these standards often carry lots of legacy issues due to supporting existing code bases. The intention behind this document is to create guidance that remains as close to ISO, Sutter & Stroustrup and ROS while resolving as many conflicts, disadvantages and inconsistencies as possible among them. Naming Conventions # Avoid using any sort of Hungarian notation on names and \"_ptr\" on pointers. Code Element Style Comment Namespace under_scored Differentiate from class names Class name CamelCase To differentiate from STL types which ISO recommends (do not use \"C\" or \"T\" prefixes) Function name camelCase Lower case start is almost universal except for .Net world Parameters/Locals under_scored Vast majority of standards recommends this because _ is more readable to C++ crowd (although not much to Java/.Net crowd) Member variables under_scored_with_ The prefix _ is heavily discouraged as ISO has rules around reserving _identifiers, so we recommend suffix instead Enums and its members CamelCase Most except very old standards agree with this one Globals g_under_scored You shouldn't have these in first place! Constants UPPER_CASE Very contentious and we just have to pick one here, unless if is a private constant in class or method, then use naming for Members or Locals File names Match case of class name in file Lot of pro and cons either way but this removes inconsistency in auto generated code (important for ROS) Header Files # Use a namespace qualified #ifdef to protect against multiple inclusion: #ifndef msr_airsim_MyHeader_hpp #define msr_airsim_MyHeader_hpp //--your code #endif The reason we don't use #pragma once is because it's not supported if same header file exists at multiple places (which might be possible under ROS build system!). Bracketing # Inside function or method body place curly bracket on same line. Outside that the Namespace, Class and methods levels use separate line. This is called K&R style and its variants are widely used in C++ vs other styles which are more popular in other languages. Notice that curlies are not required if you have single statement, but complex statements are easier to keep correct with the braces. int main(int argc, char* argv[]) { while (x == y) { f0(); if (cont()) { f1(); } else { f2(); f3(); } if (x > 100) break; } } Const and References # Religiously review all non-scalar parameters you declare to be candidate for const and references. If you are coming from languages such as C#/Java/Python, the most often mistake you would make is to pass parameters by value instead of const T&; Especially most of the strings, vectors and maps you want to pass as const T&; (if they are readonly) or T& (if they are writable). Also add const suffix to methods as much as possible. Overriding # When overriding virtual method, use override suffix. Pointers # This is really about memory management. A simulator has much performance critical code, so we try and avoid overloading the memory manager with lots of calls to new/delete. We also want to avoid too much copying of things on the stack, so we pass things by reference when ever possible. But when the object really needs to live longer than the call stack you often need to allocate that object on the heap, and so you have a pointer. Now, if management of the lifetime of that object is going to be tricky we recommend using C++ 11 smart pointers . But smart pointers do have a cost, so don\u2019t use them blindly everywhere. For private code where performance is paramount, raw pointers can be used. Raw pointers are also often needed when interfacing with legacy systems that only accept pointer types, for example, sockets API. But we try to wrap those legacy interfaces as much as possible and avoid that style of programming from leaking into the larger code base. Religiously check if you can use const everywhere, for example, const float * const xP . Avoid using prefix or suffix to indicate pointer types in variable names, i.e. use my_obj instead of myobj_ptr except in cases where it might make sense to differentiate variables better, for example, int mynum = 5; int* mynum_ptr = mynum; This is Too Short, ye? # Yes, and it's on purpose because no one likes to read 200 page coding guidelines. The goal here is to cover only most significant things which are already not covered by strict mode compilation in GCC and Level 4 warnings-as-errors in VC++. If you had like to know about how to write better code in C++, please see GotW and Effective Modern C++ book.","title":"Coding Guidelines"},{"location":"coding_guidelines/#modern-c-coding-guidelines","text":"We are using Modern C++11. Smart pointers, Lambdas, and C++11 multithreading primitives are your friend.","title":"Modern C++ Coding Guidelines"},{"location":"coding_guidelines/#quick-note","text":"The great thing about \"standards\" is that there are many to chose from: ISO , Sutter & Stroustrup , ROS , LINUX , Google's , Microsoft's , CERN's , GCC's , ARM's , LLVM's and probably thousands of others. Unfortunately most of these can't even agree on something as basic as how to name a class or a constant. This is probably due to the fact that these standards often carry lots of legacy issues due to supporting existing code bases. The intention behind this document is to create guidance that remains as close to ISO, Sutter & Stroustrup and ROS while resolving as many conflicts, disadvantages and inconsistencies as possible among them.","title":"Quick Note"},{"location":"coding_guidelines/#naming-conventions","text":"Avoid using any sort of Hungarian notation on names and \"_ptr\" on pointers. Code Element Style Comment Namespace under_scored Differentiate from class names Class name CamelCase To differentiate from STL types which ISO recommends (do not use \"C\" or \"T\" prefixes) Function name camelCase Lower case start is almost universal except for .Net world Parameters/Locals under_scored Vast majority of standards recommends this because _ is more readable to C++ crowd (although not much to Java/.Net crowd) Member variables under_scored_with_ The prefix _ is heavily discouraged as ISO has rules around reserving _identifiers, so we recommend suffix instead Enums and its members CamelCase Most except very old standards agree with this one Globals g_under_scored You shouldn't have these in first place! Constants UPPER_CASE Very contentious and we just have to pick one here, unless if is a private constant in class or method, then use naming for Members or Locals File names Match case of class name in file Lot of pro and cons either way but this removes inconsistency in auto generated code (important for ROS)","title":"Naming Conventions"},{"location":"coding_guidelines/#header-files","text":"Use a namespace qualified #ifdef to protect against multiple inclusion: #ifndef msr_airsim_MyHeader_hpp #define msr_airsim_MyHeader_hpp //--your code #endif The reason we don't use #pragma once is because it's not supported if same header file exists at multiple places (which might be possible under ROS build system!).","title":"Header Files"},{"location":"coding_guidelines/#bracketing","text":"Inside function or method body place curly bracket on same line. Outside that the Namespace, Class and methods levels use separate line. This is called K&R style and its variants are widely used in C++ vs other styles which are more popular in other languages. Notice that curlies are not required if you have single statement, but complex statements are easier to keep correct with the braces. int main(int argc, char* argv[]) { while (x == y) { f0(); if (cont()) { f1(); } else { f2(); f3(); } if (x > 100) break; } }","title":"Bracketing"},{"location":"coding_guidelines/#const-and-references","text":"Religiously review all non-scalar parameters you declare to be candidate for const and references. If you are coming from languages such as C#/Java/Python, the most often mistake you would make is to pass parameters by value instead of const T&; Especially most of the strings, vectors and maps you want to pass as const T&; (if they are readonly) or T& (if they are writable). Also add const suffix to methods as much as possible.","title":"Const and References"},{"location":"coding_guidelines/#overriding","text":"When overriding virtual method, use override suffix.","title":"Overriding"},{"location":"coding_guidelines/#pointers","text":"This is really about memory management. A simulator has much performance critical code, so we try and avoid overloading the memory manager with lots of calls to new/delete. We also want to avoid too much copying of things on the stack, so we pass things by reference when ever possible. But when the object really needs to live longer than the call stack you often need to allocate that object on the heap, and so you have a pointer. Now, if management of the lifetime of that object is going to be tricky we recommend using C++ 11 smart pointers . But smart pointers do have a cost, so don\u2019t use them blindly everywhere. For private code where performance is paramount, raw pointers can be used. Raw pointers are also often needed when interfacing with legacy systems that only accept pointer types, for example, sockets API. But we try to wrap those legacy interfaces as much as possible and avoid that style of programming from leaking into the larger code base. Religiously check if you can use const everywhere, for example, const float * const xP . Avoid using prefix or suffix to indicate pointer types in variable names, i.e. use my_obj instead of myobj_ptr except in cases where it might make sense to differentiate variables better, for example, int mynum = 5; int* mynum_ptr = mynum;","title":"Pointers"},{"location":"coding_guidelines/#this-is-too-short-ye","text":"Yes, and it's on purpose because no one likes to read 200 page coding guidelines. The goal here is to cover only most significant things which are already not covered by strict mode compilation in GCC and Level 4 warnings-as-errors in VC++. If you had like to know about how to write better code in C++, please see GotW and Effective Modern C++ book.","title":"This is Too Short, ye?"},{"location":"create_issue/","text":"How to Create Issue or Ask Question Effectively # AirSim is open source project and contributors like you keeps it going. It is important to respect contributors time and effort when you are asking a question or filing an issue. Your chances of receiving helpful response would increase if you follow below guidelines: DOs # Search issues to see if someone already has asked it. Chose title that is short and summarizes well. Copy and paste full error message. Precisely describe steps you used that produced the error message or symptom. Describe what vehicle, mode, OS, AirSim version and other settings you are using. Copy and paste minimal version of code that reproduces the problem. Tell us what the goal you want to achieve or expected output. Tell us what you did so far to debug this issue. DONT'S # Do not use \"Please help\" etc in the title. See above. Do not copy and paste screen shot of error message. Copy and paste text. Do not use \"it doesn't work\". Precisely state what is the error message or symptom. Do not ask to write code for you. Contribute !","title":"Create Issue"},{"location":"create_issue/#how-to-create-issue-or-ask-question-effectively","text":"AirSim is open source project and contributors like you keeps it going. It is important to respect contributors time and effort when you are asking a question or filing an issue. Your chances of receiving helpful response would increase if you follow below guidelines:","title":"How to Create Issue or Ask Question Effectively"},{"location":"create_issue/#dos","text":"Search issues to see if someone already has asked it. Chose title that is short and summarizes well. Copy and paste full error message. Precisely describe steps you used that produced the error message or symptom. Describe what vehicle, mode, OS, AirSim version and other settings you are using. Copy and paste minimal version of code that reproduces the problem. Tell us what the goal you want to achieve or expected output. Tell us what you did so far to debug this issue.","title":"DOs"},{"location":"create_issue/#donts","text":"Do not use \"Please help\" etc in the title. See above. Do not copy and paste screen shot of error message. Copy and paste text. Do not use \"it doesn't work\". Precisely state what is the error message or symptom. Do not ask to write code for you. Contribute !","title":"DONT'S"},{"location":"custom_drone/","text":"AirLib on a Real Drone # The AirLib library can be compiled and deployed on the companion computer on a real drone. For our testing, we mounted a Gigabyte Brix BXi7-5500 ultra compact PC on the drone connected to the Pixhawk flight controller over USB. The Gigabyte PC is running Ubuntu, so we are able to SSH into it over Wi-Fi: Once connected you can run MavLinkTest with this command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. And this will produce a log file of the flight which can then be used for playback in the simulator . You can also add -proxy:192.168.1.100:14550 to connect MavLinkTest to a remote computer where you can run QGroundControl or our PX4 Log Viewer which is another handy way to see what is going on with your drone. MavLinkTest then has some simple commands for testing your drone, here's a simple example of some commands: arm takeoff 5 orbit 10 2 This will arm the drone, takeoff of 5 meters, then do an orbit pattern radius 10 meters, at 2 m/s. Type '?' to find all available commands. Note: Some commands (for example, orbit ) are named differently and have different syntax in MavLinkTest and DroneShell (for example, circlebypath -radius 10 -velocity 21 ). When you land the drone you can stop MavLinkTest and copy the *.mavlink log file that was generated. DroneServer and DroneShell # Once you are happy that the MavLinkTest is working, you can also run DroneServer and DroneShell as follows. First, run MavLinkTest with a local proxy to send everything to DroneServer: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. -proxy:127.0.0.1:14560 Change ~/Documents/AirSim/settings.json to say \"serial\":false, because we want DroneServer to look for this UDP connection. DroneServer 0 Lastly, you can now connect DroneShell to this instance of DroneServer and use the DroneShell commands to fly your drone: DroneShell ==||=> Welcome to DroneShell 1.0. Type ? for help. Microsoft Research (c) 2016. Waiting for drone to report a valid GPS location... ==||=> requestcontrol ==||=> arm ==||=> takeoff ==||=> circlebypath -radius 10 -velocity 2 PX4 Specific Tools # You can run the MavlinkCom library and MavLinkTest app to test the connection between your companion computer and flight controller. How Does This Work? # AirSim uses MavLinkCom component developed by @lovettchris. The MavLinkCom has a proxy architecture where you can open a connection to PX4 either using serial or UDP and then other components share this connection. When PX4 sends MavLink message, all components receive that message. If any component sends a message then it's received by PX4 only. This allows you to connect any number of components to PX4 This code opens a connection for LogViewer and QGC. You can add something more if you like. If you want to use QGC + AirSim together than you will need QGC to let own the serial port. QGC opens up TCP connection that acts as a proxy so any other component can connect to QGC and send MavLinkMessage to QGC and then QGC forwards that message to PX4. So you tell AirSim to connect to QGC and let QGC own serial port. For companion board, the way we did it earlier was to have Gigabyte Brix on the drone. This x86 full-fledged computer that will connect to PX4 through USB. We had Ubuntu on Brix and ran DroneServer . The DroneServer created an API endpoint that we can talk to via C++ client code (or Python code) and it translated API calls to MavLink messages. That way you can write your code against the same API, test it in the simulator and then run the same code on an actual vehicle. So the companion computer has DroneServer running along with client code.","title":"AirSim on Real Drones"},{"location":"custom_drone/#airlib-on-a-real-drone","text":"The AirLib library can be compiled and deployed on the companion computer on a real drone. For our testing, we mounted a Gigabyte Brix BXi7-5500 ultra compact PC on the drone connected to the Pixhawk flight controller over USB. The Gigabyte PC is running Ubuntu, so we are able to SSH into it over Wi-Fi: Once connected you can run MavLinkTest with this command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. And this will produce a log file of the flight which can then be used for playback in the simulator . You can also add -proxy:192.168.1.100:14550 to connect MavLinkTest to a remote computer where you can run QGroundControl or our PX4 Log Viewer which is another handy way to see what is going on with your drone. MavLinkTest then has some simple commands for testing your drone, here's a simple example of some commands: arm takeoff 5 orbit 10 2 This will arm the drone, takeoff of 5 meters, then do an orbit pattern radius 10 meters, at 2 m/s. Type '?' to find all available commands. Note: Some commands (for example, orbit ) are named differently and have different syntax in MavLinkTest and DroneShell (for example, circlebypath -radius 10 -velocity 21 ). When you land the drone you can stop MavLinkTest and copy the *.mavlink log file that was generated.","title":"AirLib on a Real Drone"},{"location":"custom_drone/#droneserver-and-droneshell","text":"Once you are happy that the MavLinkTest is working, you can also run DroneServer and DroneShell as follows. First, run MavLinkTest with a local proxy to send everything to DroneServer: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. -proxy:127.0.0.1:14560 Change ~/Documents/AirSim/settings.json to say \"serial\":false, because we want DroneServer to look for this UDP connection. DroneServer 0 Lastly, you can now connect DroneShell to this instance of DroneServer and use the DroneShell commands to fly your drone: DroneShell ==||=> Welcome to DroneShell 1.0. Type ? for help. Microsoft Research (c) 2016. Waiting for drone to report a valid GPS location... ==||=> requestcontrol ==||=> arm ==||=> takeoff ==||=> circlebypath -radius 10 -velocity 2","title":"DroneServer and DroneShell"},{"location":"custom_drone/#px4-specific-tools","text":"You can run the MavlinkCom library and MavLinkTest app to test the connection between your companion computer and flight controller.","title":"PX4 Specific Tools"},{"location":"custom_drone/#how-does-this-work","text":"AirSim uses MavLinkCom component developed by @lovettchris. The MavLinkCom has a proxy architecture where you can open a connection to PX4 either using serial or UDP and then other components share this connection. When PX4 sends MavLink message, all components receive that message. If any component sends a message then it's received by PX4 only. This allows you to connect any number of components to PX4 This code opens a connection for LogViewer and QGC. You can add something more if you like. If you want to use QGC + AirSim together than you will need QGC to let own the serial port. QGC opens up TCP connection that acts as a proxy so any other component can connect to QGC and send MavLinkMessage to QGC and then QGC forwards that message to PX4. So you tell AirSim to connect to QGC and let QGC own serial port. For companion board, the way we did it earlier was to have Gigabyte Brix on the drone. This x86 full-fledged computer that will connect to PX4 through USB. We had Ubuntu on Brix and ran DroneServer . The DroneServer created an API endpoint that we can talk to via C++ client code (or Python code) and it translated API calls to MavLink messages. That way you can write your code against the same API, test it in the simulator and then run the same code on an actual vehicle. So the companion computer has DroneServer running along with client code.","title":"How Does This Work?"},{"location":"custom_unity_environments/","text":"Adding AirSim to Custom Unity Projects # Before completing these steps, make sure you have properly set up AirSim for Unity 1. Open the Containing folder of your custom unity project 2. Copy and paste the following items from Unity demo into the main project folder of your custom environment: Assets ProjectSettings Open your custom environment in Unity Drag your desired scene into the Scene Hierarchy panel Drag CarDemo into the Scene Hierarchy panel Copy the following items from CarDemo into your custom scene: Main Camera Directional Light AirSimHUD Car After removing CarDemo from the Hierarchy panel, save your modified scene as CarDemo . Repeat Steps 6 and 7 with DroneDemo . This time, save your custom scene as DroneDemo . Your custom environment is now ready to interface with AirSim!","title":"Custom Unity Environment"},{"location":"custom_unity_environments/#adding-airsim-to-custom-unity-projects","text":"Before completing these steps, make sure you have properly set up AirSim for Unity 1. Open the Containing folder of your custom unity project 2. Copy and paste the following items from Unity demo into the main project folder of your custom environment: Assets ProjectSettings Open your custom environment in Unity Drag your desired scene into the Scene Hierarchy panel Drag CarDemo into the Scene Hierarchy panel Copy the following items from CarDemo into your custom scene: Main Camera Directional Light AirSimHUD Car After removing CarDemo from the Hierarchy panel, save your modified scene as CarDemo . Repeat Steps 6 and 7 with DroneDemo . This time, save your custom scene as DroneDemo . Your custom environment is now ready to interface with AirSim!","title":"Adding AirSim to Custom Unity Projects"},{"location":"design/","text":"Paper # You can read more about our architecture and design in our paper (work in progress) . You may cite this as, @techreport{MSR-TR-2017-9, title = {{A}erial {I}nformatics and {R}obotics Platform}, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, year = {2017}, institution = {Microsoft Research}, number = {{M}{S}{R}-{T}{R}-2017-9}} } Architecture # Below is high level overview of how different components interact with each other.","title":"Architecture"},{"location":"design/#paper","text":"You can read more about our architecture and design in our paper (work in progress) . You may cite this as, @techreport{MSR-TR-2017-9, title = {{A}erial {I}nformatics and {R}obotics Platform}, author = {Shital Shah and Debadeepta Dey and Chris Lovett and Ashish Kapoor}, year = {2017}, institution = {Microsoft Research}, number = {{M}{S}{R}-{T}{R}-2017-9}} }","title":"Paper"},{"location":"design/#architecture","text":"Below is high level overview of how different components interact with each other.","title":"Architecture"},{"location":"dev_workflow/","text":"Development Workflow # Below is the guide on how to perform different development activities while working with AirSim. If you are new to Unreal Engine based projects and want to contribute to AirSim or make your own forks for your custom requirements, this might save you some time. Development Environment # OS # We highly recommend Windows 10 and Visual Studio 2019 as your development environment. The support for other OSes and IDE is unfortunately not as mature on the Unreal Engine side and you may risk severe loss of productivity trying to do workarounds and jumping through the hoops. Hardware # We recommend GPUs such as NVidia 1080 or NVidia Titan series with powerful desktop such as one with 64GB RAM, 6+ cores, SSDs and 2-3 displays (ideally 4K). We have found HP Z840 work quite well for our needs. The development experience on high-end laptops is generally sub-par compared to powerful desktops however they might be useful in a pinch. You generally want laptops with discrete NVidia GPU (at least M2000 or better) with 64GB RAM, SSDs and hopefully 4K display. We have found models such as Lenovo P50 work well for our needs. Laptops with only integrated graphics might not work well. Updating and Changing AirSim Code # Overview # AirSim is designed as plugin. This means it can't run by itself, you need to put it in an Unreal project (we call it \"environment\"). So building and testing AirSim has two steps: (1) build the plugin (2) deploy plugin in Unreal project and run the project. The first step is accomplished by build.cmd available in AirSim root. This command will update everything you need for the plugin in the Unreal\\Plugins folder. So to deploy the plugin, you just need to copy Unreal\\Plugins folder in to your Unreal project folder. Next you should remove all intermediate files in your Unreal project and then regenerate .sln file for your Unreal project. To do this, we have two handy .bat files in Unreal\\Environments\\Blocks folder: clean.bat and GenerateProjectFiles.bat . So just run these bat files in sequence from root of your Unreal project. Now you are ready to open new .sln in Visual Studio and press F5 to run it. Steps # Below are the steps we use to make changes in AirSim and test them out. The best way to do development in AirSim code is to use Blocks project . This is the light weight project so compile time is relatively faster. Generally the workflow is, REM //Use x64 Native Tools Command Prompt for VS 2019 REM //Navigate to AirSim repo folder git pull build.cmd cd Unreal\\Environments\\Blocks update_from_git.bat start Blocks.sln Above commands first builds the AirSim plugin and then deploys it to Blocks project using handy update_from_git.bat . Now you can work inside Visual Studio solution, make changes to the code and just run F5 to build, run and test your changes. The debugging, break points etc should work as usual. After you are done with you code changes, you might want to push your changes back to AirSim repo or your own fork or you may deploy the new plugin to your custom Unreal project. To do this, go back to command prompt and first update the AirSim repo folder: REM //Use x64 Native Tools Command Prompt for VS 2019 REM //run this from Unreal\\Environments\\Blocks update_to_git.bat build.cmd Above command will transfer your code changes from Unreal project folder back to Unreal\\Plugins folder. Now your changes are ready to be pushed to AirSim repo or your own fork. You can also copy Unreal\\Plugins to your custom Unreal engine project and see if everything works in your custom project as well. Take Away # Once you understand how Unreal Build system and plugin model works as well as why we are doing above steps, you should feel quite comfortable in following this workflow. Don't be afraid of opening up .bat files to peek inside and see what its doing. They are quite minimal and straightforward (except, of course, build.cmd - don't look in to that one). FAQ # I made changes in code in Blocks project but its not working. # When you press F5 or F6 in Visual Studio to start build, the Unreal Build system kicks in and it tries to find out if any files are dirty and what it needs to build. Unfortunately, it often fails to recognize dirty files that is not the code that uses Unreal headers and object hierarchy. So, the trick is to just make some file dirty that Unreal Build system always recognizes. My favorite one is AirSimGameMode.cpp. Just insert a line, delete it and save the file. I made changes in the code outside of Visual Studio but its not working. # Don't do that! Unreal Build system assumes that you are using Visual Studio and it does bunch of things to integrate with Visual Studio. If you do insist on using other editors then look up how to do command line builds in Unreal projects OR see docs on your editor on how it can integrate with Unreal build system OR run clean.bat + GenerateProjectFiles.bat to make sure VS solution is in sync. I'm trying to add new file in the Unreal Project and its not working. # It won't! While you are indeed using Visual Studio solution, remember that this solution was actually generated by Unreal Build system. If you want to add new files in your project, first shut down Visual Studio, add an empty file at desired location and then run GenerateProjectFiles.bat which will scan all files in your project and then re-create the .sln file. Now open this new .sln file and you are in business. I copied Unreal\\Plugins folder but nothing happens in Unreal Project. # First make sure your project's .uproject file is referencing the plugin. Then make sure you have run clean.bat and then GenerateProjectFiles.bat as described in Overview above. I have multiple Unreal projects with AirSim plugin. How do I update them easily? # You are in luck! We have build_all_ue_projects.bat which exactly does that. Don't treat it as black box (at least not yet), open it up and see what it does. It has 4 variables that are being set from command line args. If these args is not supplied they are set to default values in next set of statements. You might want to change default values for the paths. This batch file builds AirSim plugin, deploys it to all listed projects (see CALL statements later in the batch file), runs packaging for those projects and puts final binaries in specified folder - all in one step! This is what we use to create our own binary releases. How do I contribute back to AirSim? # Before making any changes make sure you have created your feature branch. After you test your code changes in Blocks environment, follow the usual steps to make contributions just like any other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first .","title":"Development Workflow"},{"location":"dev_workflow/#development-workflow","text":"Below is the guide on how to perform different development activities while working with AirSim. If you are new to Unreal Engine based projects and want to contribute to AirSim or make your own forks for your custom requirements, this might save you some time.","title":"Development Workflow"},{"location":"dev_workflow/#development-environment","text":"","title":"Development Environment"},{"location":"dev_workflow/#os","text":"We highly recommend Windows 10 and Visual Studio 2019 as your development environment. The support for other OSes and IDE is unfortunately not as mature on the Unreal Engine side and you may risk severe loss of productivity trying to do workarounds and jumping through the hoops.","title":"OS"},{"location":"dev_workflow/#hardware","text":"We recommend GPUs such as NVidia 1080 or NVidia Titan series with powerful desktop such as one with 64GB RAM, 6+ cores, SSDs and 2-3 displays (ideally 4K). We have found HP Z840 work quite well for our needs. The development experience on high-end laptops is generally sub-par compared to powerful desktops however they might be useful in a pinch. You generally want laptops with discrete NVidia GPU (at least M2000 or better) with 64GB RAM, SSDs and hopefully 4K display. We have found models such as Lenovo P50 work well for our needs. Laptops with only integrated graphics might not work well.","title":"Hardware"},{"location":"dev_workflow/#updating-and-changing-airsim-code","text":"","title":"Updating and Changing AirSim Code"},{"location":"dev_workflow/#overview","text":"AirSim is designed as plugin. This means it can't run by itself, you need to put it in an Unreal project (we call it \"environment\"). So building and testing AirSim has two steps: (1) build the plugin (2) deploy plugin in Unreal project and run the project. The first step is accomplished by build.cmd available in AirSim root. This command will update everything you need for the plugin in the Unreal\\Plugins folder. So to deploy the plugin, you just need to copy Unreal\\Plugins folder in to your Unreal project folder. Next you should remove all intermediate files in your Unreal project and then regenerate .sln file for your Unreal project. To do this, we have two handy .bat files in Unreal\\Environments\\Blocks folder: clean.bat and GenerateProjectFiles.bat . So just run these bat files in sequence from root of your Unreal project. Now you are ready to open new .sln in Visual Studio and press F5 to run it.","title":"Overview"},{"location":"dev_workflow/#steps","text":"Below are the steps we use to make changes in AirSim and test them out. The best way to do development in AirSim code is to use Blocks project . This is the light weight project so compile time is relatively faster. Generally the workflow is, REM //Use x64 Native Tools Command Prompt for VS 2019 REM //Navigate to AirSim repo folder git pull build.cmd cd Unreal\\Environments\\Blocks update_from_git.bat start Blocks.sln Above commands first builds the AirSim plugin and then deploys it to Blocks project using handy update_from_git.bat . Now you can work inside Visual Studio solution, make changes to the code and just run F5 to build, run and test your changes. The debugging, break points etc should work as usual. After you are done with you code changes, you might want to push your changes back to AirSim repo or your own fork or you may deploy the new plugin to your custom Unreal project. To do this, go back to command prompt and first update the AirSim repo folder: REM //Use x64 Native Tools Command Prompt for VS 2019 REM //run this from Unreal\\Environments\\Blocks update_to_git.bat build.cmd Above command will transfer your code changes from Unreal project folder back to Unreal\\Plugins folder. Now your changes are ready to be pushed to AirSim repo or your own fork. You can also copy Unreal\\Plugins to your custom Unreal engine project and see if everything works in your custom project as well.","title":"Steps"},{"location":"dev_workflow/#take-away","text":"Once you understand how Unreal Build system and plugin model works as well as why we are doing above steps, you should feel quite comfortable in following this workflow. Don't be afraid of opening up .bat files to peek inside and see what its doing. They are quite minimal and straightforward (except, of course, build.cmd - don't look in to that one).","title":"Take Away"},{"location":"dev_workflow/#faq","text":"","title":"FAQ"},{"location":"dev_workflow/#i-made-changes-in-code-in-blocks-project-but-its-not-working","text":"When you press F5 or F6 in Visual Studio to start build, the Unreal Build system kicks in and it tries to find out if any files are dirty and what it needs to build. Unfortunately, it often fails to recognize dirty files that is not the code that uses Unreal headers and object hierarchy. So, the trick is to just make some file dirty that Unreal Build system always recognizes. My favorite one is AirSimGameMode.cpp. Just insert a line, delete it and save the file.","title":"I made changes in code in Blocks project but its not working."},{"location":"dev_workflow/#i-made-changes-in-the-code-outside-of-visual-studio-but-its-not-working","text":"Don't do that! Unreal Build system assumes that you are using Visual Studio and it does bunch of things to integrate with Visual Studio. If you do insist on using other editors then look up how to do command line builds in Unreal projects OR see docs on your editor on how it can integrate with Unreal build system OR run clean.bat + GenerateProjectFiles.bat to make sure VS solution is in sync.","title":"I made changes in the code outside of Visual Studio but its not working."},{"location":"dev_workflow/#im-trying-to-add-new-file-in-the-unreal-project-and-its-not-working","text":"It won't! While you are indeed using Visual Studio solution, remember that this solution was actually generated by Unreal Build system. If you want to add new files in your project, first shut down Visual Studio, add an empty file at desired location and then run GenerateProjectFiles.bat which will scan all files in your project and then re-create the .sln file. Now open this new .sln file and you are in business.","title":"I'm trying to add new file in the Unreal Project and its not working."},{"location":"dev_workflow/#i-copied-unrealplugins-folder-but-nothing-happens-in-unreal-project","text":"First make sure your project's .uproject file is referencing the plugin. Then make sure you have run clean.bat and then GenerateProjectFiles.bat as described in Overview above.","title":"I copied Unreal\\Plugins folder but nothing happens in Unreal Project."},{"location":"dev_workflow/#i-have-multiple-unreal-projects-with-airsim-plugin-how-do-i-update-them-easily","text":"You are in luck! We have build_all_ue_projects.bat which exactly does that. Don't treat it as black box (at least not yet), open it up and see what it does. It has 4 variables that are being set from command line args. If these args is not supplied they are set to default values in next set of statements. You might want to change default values for the paths. This batch file builds AirSim plugin, deploys it to all listed projects (see CALL statements later in the batch file), runs packaging for those projects and puts final binaries in specified folder - all in one step! This is what we use to create our own binary releases.","title":"I have multiple Unreal projects with AirSim plugin. How do I update them easily?"},{"location":"dev_workflow/#how-do-i-contribute-back-to-airsim","text":"Before making any changes make sure you have created your feature branch. After you test your code changes in Blocks environment, follow the usual steps to make contributions just like any other GitHub projects. If you are not familiar with Git Branch-Rebase-Merge workflow, please read this first .","title":"How do I contribute back to AirSim?"},{"location":"docker_ubuntu/","text":"AirSim on Docker in Linux # We've two options for docker. You can either build an image for running airsim linux binaries , or for compiling Unreal Engine + AirSim from source Binaries # Requirements: # Install nvidia-docker2 Build the docker image # Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on Ubuntu 18.04 with CUDA 10.0. You can specify any NVIDIA cudagl at your own risk. --target_image is the desired name of your docker image. Defaults to airsim_binary with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --base_image=nvidia/cudagl:10.0-devel-ubuntu18.04 \\ --target_image=airsim_binary:10.0-devel-ubuntu18.04 Verify you have an image by: $ docker images | grep airsim Running an unreal binary inside a docker container # Get an unreal binary or package your own project in Ubuntu. Let's take the Blocks binary as an example. You can download it by running $ cd Airsim/docker; $ ./download_blocks_env_binary.sh Running an unreal binary inside a docker container The syntax is: $ ./run_airsim_image_binary.sh DOCKER_IMAGE_NAME UNREAL_BINARY_SHELL_SCRIPT UNREAL_BINARY_ARGUMENTS -- headless For blocks, you can do a $ ./run_airsim_image_binary.sh airsim_binary:10.0-devel-ubuntu18.04 Blocks/Blocks.sh -windowed -ResX=1080 -ResY=720 DOCKER_IMAGE_NAME : Same as target_image parameter in previous step. By default, enter airsim_binary:10.0-devel-ubuntu18.04 UNREAL_BINARY_SHELL_SCRIPT : for Blocks enviroment, it will be Blocks/Blocks.sh UNREAL_BINARY_ARGUMENTS : For airsim, most relevant would be -windowed , -ResX , -ResY . Click on link to see all options. Running in Headless mode: Suffix -- headless at the end: $ ./run_airsim_image_binary.sh Blocks/Blocks.sh -- headless Specifying a settings.json Source # Requirements: # Install nvidia-docker2 Install ue4-docker Build Unreal Engine inside docker: # To get access to Unreal Engine's source code, register on Epic Games' website and link it to your github account, as explained in the Required Steps section here . Note that you don't need to do Step 2: Downloading UE4 on Linux ! Build unreal engine 4.19.2 docker image. We're going to use CUDA 10.0 in our example. $ ue4-docker build 4.19.2 --cuda=10.0 --no-full [optional] $ ue4-docker clean to free up some space. Details here ue4-docker supports all CUDA version listed on NVIDIA's cudagl dockerhub here . Please see this page for advanced configurations using ue4-docker Disk space: The unreal images and containers can take up a lot of space, especially if you try more than one version. Here's a list of useful links to monitor space used by docker and clean up intermediate builds: Large container images primer $ docker system df $ docker container prune $ docker image prune $ docker system prune Building AirSim inside UE4 docker container: # Build AirSim docker image (which lays over the unreal image we just built) Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on adamrehn/ue4-engine:4.19.2-cudagl10.0 . See ue4-docker for other versions. --target_image is the desired name of your docker image. Defaults to airsim_source with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --source \\ ----base_image adamrehn/ue4-engine:4.19.2-cudagl10.0 \\ --target_image=airsim_source:4.19.2-cudagl10.0 Running AirSim container # Run the airsim source image we built by: ./run_airsim_image_source.sh airsim_source:4.19.2-cudagl10.0 Syntax is ./run_airsim_image_source.sh DOCKER_IMAGE_NAME -- headless -- headless : suffix this to run in optional headless mode. Inside the container, you can see UnrealEngine and AirSim under /home/ue4 . Start unreal engine inside the container: ue4@HOSTMACHINE:~$ /home/ue4/UnrealEngine/Engine/Binaries/Linux/UE4Editor Specifying an airsim settings.json Continue with AirSim's Linux docs . [Misc] Packaging Unreal Environments in airsim_source containers # Let's take the Blocks environment as an example. In the following script, specify the full path to your unreal uproject file by project and the directory where you want the binaries to be placed by archivedirectory $ /home/ue4/UnrealEngine/Engine/Build/BatchFiles/RunUAT.sh BuildCookRun -platform=Linux -clientconfig=Shipping -serverconfig=Shipping -noP4 -cook -allmaps -build -stage -prereqs -pak -archive \\ -archivedirectory=/home/ue4/Binaries/Blocks/ \\ -project=/home/ue4/AirSim/Unreal/Environments/Blocks/Blocks.uproject This would create a Blocks binary in /home/ue4/Binaries/Blocks/ . You can test it by running /home/ue4/Binaries/Blocks/LinuxNoEditor/Blocks.sh -windowed Specifying an airsim settings.json # #### airsim_binary docker image: - We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . - Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in * run_airsim_image_binary.sh nvidia-docker run -it \\ -v $PATH_TO_YOUR/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -v $UNREAL_BINARY_PATH:$UNREAL_BINARY_PATH \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME \\ /bin/bash -c \"$UNREAL_BINARY_COMMAND\" #### airsim_source docker image: We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in run_airsim_image_source.sh : nvidia-docker run -it \\ -v $(pwd)/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME","title":"Docker on Linux"},{"location":"docker_ubuntu/#airsim-on-docker-in-linux","text":"We've two options for docker. You can either build an image for running airsim linux binaries , or for compiling Unreal Engine + AirSim from source","title":"AirSim on Docker in Linux"},{"location":"docker_ubuntu/#binaries","text":"","title":"Binaries"},{"location":"docker_ubuntu/#requirements","text":"Install nvidia-docker2","title":"Requirements:"},{"location":"docker_ubuntu/#build-the-docker-image","text":"Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on Ubuntu 18.04 with CUDA 10.0. You can specify any NVIDIA cudagl at your own risk. --target_image is the desired name of your docker image. Defaults to airsim_binary with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --base_image=nvidia/cudagl:10.0-devel-ubuntu18.04 \\ --target_image=airsim_binary:10.0-devel-ubuntu18.04 Verify you have an image by: $ docker images | grep airsim","title":"Build the docker image"},{"location":"docker_ubuntu/#running-an-unreal-binary-inside-a-docker-container","text":"Get an unreal binary or package your own project in Ubuntu. Let's take the Blocks binary as an example. You can download it by running $ cd Airsim/docker; $ ./download_blocks_env_binary.sh Running an unreal binary inside a docker container The syntax is: $ ./run_airsim_image_binary.sh DOCKER_IMAGE_NAME UNREAL_BINARY_SHELL_SCRIPT UNREAL_BINARY_ARGUMENTS -- headless For blocks, you can do a $ ./run_airsim_image_binary.sh airsim_binary:10.0-devel-ubuntu18.04 Blocks/Blocks.sh -windowed -ResX=1080 -ResY=720 DOCKER_IMAGE_NAME : Same as target_image parameter in previous step. By default, enter airsim_binary:10.0-devel-ubuntu18.04 UNREAL_BINARY_SHELL_SCRIPT : for Blocks enviroment, it will be Blocks/Blocks.sh UNREAL_BINARY_ARGUMENTS : For airsim, most relevant would be -windowed , -ResX , -ResY . Click on link to see all options. Running in Headless mode: Suffix -- headless at the end: $ ./run_airsim_image_binary.sh Blocks/Blocks.sh -- headless Specifying a settings.json","title":"Running an unreal binary inside a docker container"},{"location":"docker_ubuntu/#source","text":"","title":"Source"},{"location":"docker_ubuntu/#requirements_1","text":"Install nvidia-docker2 Install ue4-docker","title":"Requirements:"},{"location":"docker_ubuntu/#build-unreal-engine-inside-docker","text":"To get access to Unreal Engine's source code, register on Epic Games' website and link it to your github account, as explained in the Required Steps section here . Note that you don't need to do Step 2: Downloading UE4 on Linux ! Build unreal engine 4.19.2 docker image. We're going to use CUDA 10.0 in our example. $ ue4-docker build 4.19.2 --cuda=10.0 --no-full [optional] $ ue4-docker clean to free up some space. Details here ue4-docker supports all CUDA version listed on NVIDIA's cudagl dockerhub here . Please see this page for advanced configurations using ue4-docker Disk space: The unreal images and containers can take up a lot of space, especially if you try more than one version. Here's a list of useful links to monitor space used by docker and clean up intermediate builds: Large container images primer $ docker system df $ docker container prune $ docker image prune $ docker system prune","title":"Build Unreal Engine inside docker:"},{"location":"docker_ubuntu/#building-airsim-inside-ue4-docker-container","text":"Build AirSim docker image (which lays over the unreal image we just built) Below are the default arguments. --base_image : This is image over which we'll install airsim. We've tested on adamrehn/ue4-engine:4.19.2-cudagl10.0 . See ue4-docker for other versions. --target_image is the desired name of your docker image. Defaults to airsim_source with same tag as the base image $ cd Airsim/docker; $ python build_airsim_image.py \\ --source \\ ----base_image adamrehn/ue4-engine:4.19.2-cudagl10.0 \\ --target_image=airsim_source:4.19.2-cudagl10.0","title":"Building AirSim inside UE4 docker container:"},{"location":"docker_ubuntu/#running-airsim-container","text":"Run the airsim source image we built by: ./run_airsim_image_source.sh airsim_source:4.19.2-cudagl10.0 Syntax is ./run_airsim_image_source.sh DOCKER_IMAGE_NAME -- headless -- headless : suffix this to run in optional headless mode. Inside the container, you can see UnrealEngine and AirSim under /home/ue4 . Start unreal engine inside the container: ue4@HOSTMACHINE:~$ /home/ue4/UnrealEngine/Engine/Binaries/Linux/UE4Editor Specifying an airsim settings.json Continue with AirSim's Linux docs .","title":"Running AirSim container"},{"location":"docker_ubuntu/#misc-packaging-unreal-environments-in-airsim_source-containers","text":"Let's take the Blocks environment as an example. In the following script, specify the full path to your unreal uproject file by project and the directory where you want the binaries to be placed by archivedirectory $ /home/ue4/UnrealEngine/Engine/Build/BatchFiles/RunUAT.sh BuildCookRun -platform=Linux -clientconfig=Shipping -serverconfig=Shipping -noP4 -cook -allmaps -build -stage -prereqs -pak -archive \\ -archivedirectory=/home/ue4/Binaries/Blocks/ \\ -project=/home/ue4/AirSim/Unreal/Environments/Blocks/Blocks.uproject This would create a Blocks binary in /home/ue4/Binaries/Blocks/ . You can test it by running /home/ue4/Binaries/Blocks/LinuxNoEditor/Blocks.sh -windowed","title":"[Misc] Packaging Unreal Environments in airsim_source containers"},{"location":"docker_ubuntu/#specifying-an-airsim-settingsjson","text":"#### airsim_binary docker image: - We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . - Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in * run_airsim_image_binary.sh nvidia-docker run -it \\ -v $PATH_TO_YOUR/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -v $UNREAL_BINARY_PATH:$UNREAL_BINARY_PATH \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME \\ /bin/bash -c \"$UNREAL_BINARY_COMMAND\" #### airsim_source docker image: We're mapping the host machine's PATH/TO/Airsim/docker/settings.json to the docker container's /home/airsim_user/Documents/AirSim/settings.json . Hence, we can load any settings file by simply modifying PATH_TO_YOUR/settings.json by modifying the following snippets in run_airsim_image_source.sh : nvidia-docker run -it \\ -v $(pwd)/settings.json:/home/airsim_user/Documents/AirSim/settings.json \\ -e SDL_VIDEODRIVER=$SDL_VIDEODRIVER_VALUE \\ -e SDL_HINT_CUDA_DEVICE='0' \\ --net=host \\ --env=\"DISPLAY=$DISPLAY\" \\ --env=\"QT_X11_NO_MITSHM=1\" \\ --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \\ -env=\"XAUTHORITY=$XAUTH\" \\ --volume=\"$XAUTH:$XAUTH\" \\ --runtime=nvidia \\ --rm \\ $DOCKER_IMAGE_NAME","title":"Specifying an airsim settings.json"},{"location":"drone_survey/","text":"Implementing a Drone Survey script # Moved here from https://github.com/microsoft/AirSim/wiki/Implementing-a-Drone-Survey-script Ever wanted to capture a bunch of top-down pictures of a certain location? Well, the Python API makes this really simple. See the code available here . Let's assume we want the following variables: Variable Description boxsize The overall size of the square box to survey stripewidth How far apart to drive the swim lanes, this can depend on the type of camera lens, for example. altitude The height to fly the survey. speed The speed of the survey can depend on how fast your camera can snap shots. So with these we can compute a square path box using this code: path = [] distance = 0 while x < self.boxsize: distance += self.boxsize path.append(Vector3r(x, self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, self.boxsize, z)) distance += self.boxsize path.append(Vector3r(x, -self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, -self.boxsize, z)) distance += self.boxsize Assuming we start in the corner of the box, increment x by the stripe width, then fly the full y-dimension of -boxsize to +boxsize , so in this case, boxsize is half the size of the actual box we will be covering. Once we have this list of Vector3r objects, we can fly this path very simply with the following call: result = self.client.moveOnPath(path, self.velocity, trip_time, DrivetrainType.ForwardOnly, YawMode(False,0), lookahead, 1) We can compute an appropriate trip_time timeout by dividing the distance of the path and the speed we are flying. The lookahead needed here for smooth path interpolation can be computed from the velocity using self.velocity + (self.velocity/2) . The more lookahead, the smoother the turns. This is why you see in the screenshot that the ends of each swimland are smooth turns rather than a square box pattern. This can result in a smoother video from your camera also. That's it, pretty simple, eh? Now of course you can add a lot more intelligence to this, make it avoid known obstacles on your map, make it climb up and down a hillside so you can survey a slope, etc. Lots of fun to be had.","title":"Surveying Using Drone"},{"location":"drone_survey/#implementing-a-drone-survey-script","text":"Moved here from https://github.com/microsoft/AirSim/wiki/Implementing-a-Drone-Survey-script Ever wanted to capture a bunch of top-down pictures of a certain location? Well, the Python API makes this really simple. See the code available here . Let's assume we want the following variables: Variable Description boxsize The overall size of the square box to survey stripewidth How far apart to drive the swim lanes, this can depend on the type of camera lens, for example. altitude The height to fly the survey. speed The speed of the survey can depend on how fast your camera can snap shots. So with these we can compute a square path box using this code: path = [] distance = 0 while x < self.boxsize: distance += self.boxsize path.append(Vector3r(x, self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, self.boxsize, z)) distance += self.boxsize path.append(Vector3r(x, -self.boxsize, z)) x += self.stripewidth distance += self.stripewidth path.append(Vector3r(x, -self.boxsize, z)) distance += self.boxsize Assuming we start in the corner of the box, increment x by the stripe width, then fly the full y-dimension of -boxsize to +boxsize , so in this case, boxsize is half the size of the actual box we will be covering. Once we have this list of Vector3r objects, we can fly this path very simply with the following call: result = self.client.moveOnPath(path, self.velocity, trip_time, DrivetrainType.ForwardOnly, YawMode(False,0), lookahead, 1) We can compute an appropriate trip_time timeout by dividing the distance of the path and the speed we are flying. The lookahead needed here for smooth path interpolation can be computed from the velocity using self.velocity + (self.velocity/2) . The more lookahead, the smoother the turns. This is why you see in the screenshot that the ends of each swimland are smooth turns rather than a square box pattern. This can result in a smoother video from your camera also. That's it, pretty simple, eh? Now of course you can add a lot more intelligence to this, make it avoid known obstacles on your map, make it climb up and down a hillside so you can survey a slope, etc. Lots of fun to be had.","title":"Implementing a Drone Survey script"},{"location":"faq/","text":"FAQ # General # Unreal editor is slow when it is not the active window # Go to Edit/Editor Preferences, select \"All Settings\" and type \"CPU\" in the search box. It should find the setting titled \"Use Less CPU when in Background\", and you want to uncheck this checkbox. My mouse disappears in Unreal # Yes, Unreal steals the mouse, and we don't draw one. So to get your mouse back just use Alt+TAB to switch to a different window. To avoid this entirely, go to Project settings in Unreal Editor, go to Input tab and disable all settings for mouse capture. Where is the setting file and how do I modify it? # AirSim will create empty settings file at ~/Documents/AirSim/settings.json . You can view the available settings options . How do I arm my drone? # If you're using simple_flight, your vehicle is already armed and ready to fly. For PX4 you can arm by holding both sticks on remote control down and to the center. When making API call I get error # If you are getting this error, TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float' its probably due to upgraded version of tornado package with version > 5.0 in Python that conflicts with msgpack-rpc-python which requires tornado package < 5.0. To fix this you can update the package like this: pip install --upgrade msgpack-rpc-python But this might break something (for example, PyTorch 0.4+) because it will uninstall newer tornado and re-install older one. To avoid this you should create new conda environment . I'm getting Eigen not found error when compiling Unreal project. # This is most likely because AirSim wasn't built and Plugin folder was copied in Unreal project folder. To fix this make sure you build AirSim first (run build.cmd in Windows). Something went wrong. How do I debug? # First turn on C++ exceptions from the Exceptions window: and copy the stack trace of all exceptions you see there during execution that look relevant (for example, there might be an initial exception from VSPerf140 that you can ignore) then paste these call stacks into a new AirSim GitHub issue, thanks. What do the colors mean in the Segmentation View ? # See Camera Views for information on the camera views and how to change them. Unreal 4.xx doesn't look as good as 4.yy # Unreal 4.15 added the ability for Foliage LOD dithering to be disabled on a case-by-case basis by unchecking the Dithered LOD Transition checkbox in the foliage materials. Note that all materials used on all LODs need to have the checkbox checked in order for dithered LOD transitions to work. When checked the transition of generated foliage will be a lot smoother and will look better than 4.14. Can I use an XBox controller to fly? # See XBox controller for details. Can I build a hexacopter with AirSim? # See how to build a hexacopter . How do I use AirSim with multiple vehicles? # Here is multi-vehicle setup guide . What computer do you need? # It depends on how big your Unreal Environment is. The Blocks environment that comes with AirSim is very basic and works on typical laptops. The Modular Neighborhood Pack that we use ourselves for research requires GPUs with at least 4GB of RAM. The Open World environment needs GPU with 8GB RAM. Our typical development machines have 32GB of RAM and NVIDIA TitanX and a fast hard drive . How do I report issues? # It's a good idea to include your configuration like below. If you can also include logs, that could also expedite the investigation. Operating System: Windows 10 64bit CPU: Intel Core i7 GPU: Nvidia GTX 1080 RAM: 32 GB Flight Controller: Pixhawk v2 Remote Control: Futaba If you have modified the default ~/Document/AirSim/settings.json , please include your settings also. If you are using PX4 then try to capture log from MavLink or PX4 . File an issue through GitHub Issues . Others # Linux Build FAQ Windows Build FAQ PX4 Setup FAQ Remote Control FAQ Unreal Blocks Environment FAQ Unreal Custom Environment FAQ","title":"FAQ"},{"location":"faq/#faq","text":"","title":"FAQ"},{"location":"faq/#general","text":"","title":"General"},{"location":"faq/#unreal-editor-is-slow-when-it-is-not-the-active-window","text":"Go to Edit/Editor Preferences, select \"All Settings\" and type \"CPU\" in the search box. It should find the setting titled \"Use Less CPU when in Background\", and you want to uncheck this checkbox.","title":"Unreal editor is slow when it is not the active window"},{"location":"faq/#my-mouse-disappears-in-unreal","text":"Yes, Unreal steals the mouse, and we don't draw one. So to get your mouse back just use Alt+TAB to switch to a different window. To avoid this entirely, go to Project settings in Unreal Editor, go to Input tab and disable all settings for mouse capture.","title":"My mouse disappears in Unreal"},{"location":"faq/#where-is-the-setting-file-and-how-do-i-modify-it","text":"AirSim will create empty settings file at ~/Documents/AirSim/settings.json . You can view the available settings options .","title":"Where is the setting file and how do I modify it?"},{"location":"faq/#how-do-i-arm-my-drone","text":"If you're using simple_flight, your vehicle is already armed and ready to fly. For PX4 you can arm by holding both sticks on remote control down and to the center.","title":"How do I arm my drone?"},{"location":"faq/#when-making-api-call-i-get-error","text":"If you are getting this error, TypeError: unsupported operand type(s) for *: 'AsyncIOLoop' and 'float' its probably due to upgraded version of tornado package with version > 5.0 in Python that conflicts with msgpack-rpc-python which requires tornado package < 5.0. To fix this you can update the package like this: pip install --upgrade msgpack-rpc-python But this might break something (for example, PyTorch 0.4+) because it will uninstall newer tornado and re-install older one. To avoid this you should create new conda environment .","title":"When making API call I get error"},{"location":"faq/#im-getting-eigen-not-found-error-when-compiling-unreal-project","text":"This is most likely because AirSim wasn't built and Plugin folder was copied in Unreal project folder. To fix this make sure you build AirSim first (run build.cmd in Windows).","title":"I'm getting Eigen not found error when compiling Unreal project."},{"location":"faq/#something-went-wrong-how-do-i-debug","text":"First turn on C++ exceptions from the Exceptions window: and copy the stack trace of all exceptions you see there during execution that look relevant (for example, there might be an initial exception from VSPerf140 that you can ignore) then paste these call stacks into a new AirSim GitHub issue, thanks.","title":"Something went wrong. How do I debug?"},{"location":"faq/#what-do-the-colors-mean-in-the-segmentation-view","text":"See Camera Views for information on the camera views and how to change them.","title":"What do the colors mean in the Segmentation View ?"},{"location":"faq/#unreal-4xx-doesnt-look-as-good-as-4yy","text":"Unreal 4.15 added the ability for Foliage LOD dithering to be disabled on a case-by-case basis by unchecking the Dithered LOD Transition checkbox in the foliage materials. Note that all materials used on all LODs need to have the checkbox checked in order for dithered LOD transitions to work. When checked the transition of generated foliage will be a lot smoother and will look better than 4.14.","title":"Unreal 4.xx doesn't look as good as 4.yy"},{"location":"faq/#can-i-use-an-xbox-controller-to-fly","text":"See XBox controller for details.","title":"Can I use an XBox controller to fly?"},{"location":"faq/#can-i-build-a-hexacopter-with-airsim","text":"See how to build a hexacopter .","title":"Can I build a hexacopter with AirSim?"},{"location":"faq/#how-do-i-use-airsim-with-multiple-vehicles","text":"Here is multi-vehicle setup guide .","title":"How do I use AirSim with multiple vehicles?"},{"location":"faq/#what-computer-do-you-need","text":"It depends on how big your Unreal Environment is. The Blocks environment that comes with AirSim is very basic and works on typical laptops. The Modular Neighborhood Pack that we use ourselves for research requires GPUs with at least 4GB of RAM. The Open World environment needs GPU with 8GB RAM. Our typical development machines have 32GB of RAM and NVIDIA TitanX and a fast hard drive .","title":"What computer do you need?"},{"location":"faq/#how-do-i-report-issues","text":"It's a good idea to include your configuration like below. If you can also include logs, that could also expedite the investigation. Operating System: Windows 10 64bit CPU: Intel Core i7 GPU: Nvidia GTX 1080 RAM: 32 GB Flight Controller: Pixhawk v2 Remote Control: Futaba If you have modified the default ~/Document/AirSim/settings.json , please include your settings also. If you are using PX4 then try to capture log from MavLink or PX4 . File an issue through GitHub Issues .","title":"How do I report issues?"},{"location":"faq/#others","text":"Linux Build FAQ Windows Build FAQ PX4 Setup FAQ Remote Control FAQ Unreal Blocks Environment FAQ Unreal Custom Environment FAQ","title":"Others"},{"location":"flight_controller/","text":"Flight Controller # What is Flight Controller? # \"Wait!\" you ask, \"Why do you need flight controller for a simulator?\". The primary job of flight controller is to take in desired state as input, estimate actual state using sensors data and then drive the actuators in such a way so that actual state comes as close to the desired state. For quadrotors, desired state can be specified as roll, pitch and yaw, for example. It then estimates actual roll, pitch and yaw using gyroscope and accelerometer. Then it generates appropriate motor signals so actual state becomes desired state. You can find more in-depth in our paper . How Simulator uses Flight Controller? # Simulator consumes the motor signals generated by flight controller to figure out force and thrust generated by each actuator (i.e. propellers in case of quadrotor). This is then used by the physics engine to compute the kinetic properties of the vehicle. This in turn generates simulated sensor data and feed it back to the flight controller. You can find more in-depth in our paper . What is Hardware- and Software-in-Loop? # Hardware-in-Loop (HITL or HIL) means flight controller runs in actual hardware such as Naze32 or Pixhawk chip. You then connect this hardware to PC using USB port. Simulator talks to the device to retrieve actuator signals and send it simulated sensor data. This is obviously as close as you can get to real thing. However, it typically requires more steps to set up and usually hard to debug. One big issue is that simulator clock and device clock runs on their own speed and accuracy. Also, USB connection (which is usually only USB 2.0) may not be enough for real-time communication. In \"software-in-loop\" simulation (SITL or SIL) mode the firmware runs in your computer as opposed to separate board. This is generally fine except that now you are not touching any code paths that are specific to your device. Also, none of your code now runs with real-time clock usually provided by specialized hardware board. For well-designed flight controllers with software clock, these are usually not concerning issues. What Flight Controllers are Supported? # AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In the future, we also plan to support ROSFlight and Hackflight . Using AirSim Without Flight Controller # Yes, now it's possible to use AirSim without flight controller. Please see the instructions here for how to use so-called \"Computer Vision\" mode. If you don't need vehicle dynamics, we highly recommend using this mode.","title":"Flight Controller"},{"location":"flight_controller/#flight-controller","text":"","title":"Flight Controller"},{"location":"flight_controller/#what-is-flight-controller","text":"\"Wait!\" you ask, \"Why do you need flight controller for a simulator?\". The primary job of flight controller is to take in desired state as input, estimate actual state using sensors data and then drive the actuators in such a way so that actual state comes as close to the desired state. For quadrotors, desired state can be specified as roll, pitch and yaw, for example. It then estimates actual roll, pitch and yaw using gyroscope and accelerometer. Then it generates appropriate motor signals so actual state becomes desired state. You can find more in-depth in our paper .","title":"What is Flight Controller?"},{"location":"flight_controller/#how-simulator-uses-flight-controller","text":"Simulator consumes the motor signals generated by flight controller to figure out force and thrust generated by each actuator (i.e. propellers in case of quadrotor). This is then used by the physics engine to compute the kinetic properties of the vehicle. This in turn generates simulated sensor data and feed it back to the flight controller. You can find more in-depth in our paper .","title":"How Simulator uses Flight Controller?"},{"location":"flight_controller/#what-is-hardware-and-software-in-loop","text":"Hardware-in-Loop (HITL or HIL) means flight controller runs in actual hardware such as Naze32 or Pixhawk chip. You then connect this hardware to PC using USB port. Simulator talks to the device to retrieve actuator signals and send it simulated sensor data. This is obviously as close as you can get to real thing. However, it typically requires more steps to set up and usually hard to debug. One big issue is that simulator clock and device clock runs on their own speed and accuracy. Also, USB connection (which is usually only USB 2.0) may not be enough for real-time communication. In \"software-in-loop\" simulation (SITL or SIL) mode the firmware runs in your computer as opposed to separate board. This is generally fine except that now you are not touching any code paths that are specific to your device. Also, none of your code now runs with real-time clock usually provided by specialized hardware board. For well-designed flight controllers with software clock, these are usually not concerning issues.","title":"What is Hardware- and Software-in-Loop?"},{"location":"flight_controller/#what-flight-controllers-are-supported","text":"AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In the future, we also plan to support ROSFlight and Hackflight .","title":"What Flight Controllers are Supported?"},{"location":"flight_controller/#using-airsim-without-flight-controller","text":"Yes, now it's possible to use AirSim without flight controller. Please see the instructions here for how to use so-called \"Computer Vision\" mode. If you don't need vehicle dynamics, we highly recommend using this mode.","title":"Using AirSim Without Flight Controller"},{"location":"hard_drive/","text":"Busy Hard Drive # It is not required, but we recommend running your Unreal Environment on a Solid State Drive (SSD). Between debugging, logging, and Unreal asset loading the hard drive can become your bottle neck. It is normal that your hard drive will be slammed while Unreal is loading the environment, but if your hard drive performance looks like this while the Unreal game is running then you will probably not get a good flying experience. In fact, if the hard drive is this busy, chances are the drone will not fly properly at all. For some unknown reason this I/O bottle neck also interferes with the drone control loop and if that loop doesn't run at a high rate (300-500 Hz) then the drone will not fly. Not surprising, the control loop inside the PX4 firmware that runs on a Pixhawk flight controller runs at 1000 Hz. Reducing I/O # If you can't whip off to Fry's Electronics and pick up an overpriced super fast SSD this weekend, then the following steps can be taken to reduce the hard drive I/O: First run the Unreal Environment using Cooked content outside of the UE Editor or any debugging environment, and package the content to your fastest SSD drive. You can do that using this menu option: If you must use the UE editor (because you are actively modifying game assets), then at least don't run that in a debugger. If you are using Visual Studio use start without debugging. If you must debug the app, and you are using Visual Studio debugger, stop then Visual Studio from logging Intellitrace information. Go to Tools/Options/Debugging/Intellitrace, and turn off the main checkbox. Turn off any Unreal Analytics that your environment may have enabled, especially any file logging. I/O from Page Faults # If your system is running out of RAM it may start paging memory to disk. If your operating system has enabled paging to disk, make sure it is paging to your fastest SSD. Or if you have enough RAM disable paging all together. In fact, if you disable paging and the game stops working you will know for sure you are running out of RAM. Obviously, shutting down any other unnecessary apps should also free up memory so you don't run out. Ideal Runtime performance # This is what my slow hard drive looks like when flying from UE editor. You can see it's very busy, but the drone still flies ok: This is what my fast SSD looks like when the drone is flying in an Unreal Cooked app (no UE editor, no debugger). Not surprisingly it is flying perfectly in this case:","title":"Tips for Busy HDD"},{"location":"hard_drive/#busy-hard-drive","text":"It is not required, but we recommend running your Unreal Environment on a Solid State Drive (SSD). Between debugging, logging, and Unreal asset loading the hard drive can become your bottle neck. It is normal that your hard drive will be slammed while Unreal is loading the environment, but if your hard drive performance looks like this while the Unreal game is running then you will probably not get a good flying experience. In fact, if the hard drive is this busy, chances are the drone will not fly properly at all. For some unknown reason this I/O bottle neck also interferes with the drone control loop and if that loop doesn't run at a high rate (300-500 Hz) then the drone will not fly. Not surprising, the control loop inside the PX4 firmware that runs on a Pixhawk flight controller runs at 1000 Hz.","title":"Busy Hard Drive"},{"location":"hard_drive/#reducing-io","text":"If you can't whip off to Fry's Electronics and pick up an overpriced super fast SSD this weekend, then the following steps can be taken to reduce the hard drive I/O: First run the Unreal Environment using Cooked content outside of the UE Editor or any debugging environment, and package the content to your fastest SSD drive. You can do that using this menu option: If you must use the UE editor (because you are actively modifying game assets), then at least don't run that in a debugger. If you are using Visual Studio use start without debugging. If you must debug the app, and you are using Visual Studio debugger, stop then Visual Studio from logging Intellitrace information. Go to Tools/Options/Debugging/Intellitrace, and turn off the main checkbox. Turn off any Unreal Analytics that your environment may have enabled, especially any file logging.","title":"Reducing I/O"},{"location":"hard_drive/#io-from-page-faults","text":"If your system is running out of RAM it may start paging memory to disk. If your operating system has enabled paging to disk, make sure it is paging to your fastest SSD. Or if you have enough RAM disable paging all together. In fact, if you disable paging and the game stops working you will know for sure you are running out of RAM. Obviously, shutting down any other unnecessary apps should also free up memory so you don't run out.","title":"I/O from Page Faults"},{"location":"hard_drive/#ideal-runtime-performance","text":"This is what my slow hard drive looks like when flying from UE editor. You can see it's very busy, but the drone still flies ok: This is what my fast SSD looks like when the drone is flying in an Unreal Cooked app (no UE editor, no debugger). Not surprisingly it is flying perfectly in this case:","title":"Ideal Runtime performance"},{"location":"hello_drone/","text":"Hello Drone # How does Hello Drone work? # Hello Drone uses the RPC client to connect to the RPC server that is automatically started by the AirSim. The RPC server routes all the commands to a class that implements MultirotorApiBase . In essence, MultirotorApiBase defines our abstract interface for getting data from the quadrotor and sending back commands. We currently have concrete implementation for MultirotorApiBase for MavLink based vehicles. The implementation for DJI drone platforms, specifically Matrice, is in works.","title":"Hello Drone"},{"location":"hello_drone/#hello-drone","text":"","title":"Hello Drone"},{"location":"hello_drone/#how-does-hello-drone-work","text":"Hello Drone uses the RPC client to connect to the RPC server that is automatically started by the AirSim. The RPC server routes all the commands to a class that implements MultirotorApiBase . In essence, MultirotorApiBase defines our abstract interface for getting data from the quadrotor and sending back commands. We currently have concrete implementation for MultirotorApiBase for MavLink based vehicles. The implementation for DJI drone platforms, specifically Matrice, is in works.","title":"How does Hello Drone work?"},{"location":"image_apis/","text":"Image APIs # Please read general API doc first if you are not familiar with AirSim APIs. Getting a Single Image # Here's a sample code to get a single image from camera named \"0\". The returned value is bytes of png format image. To get uncompressed and other format as well as available cameras please see next sections. Python # import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() png_image = client.simGetImage(\"0\", airsim.ImageType.Scene) # do something with image C++ # #include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int getOneImage() { using namespace msr::airlib; // for car use CarRpcLibClient MultirotorRpcLibClient client; std::vector png_image = client.simGetImage(\"0\", VehicleCameraBase::ImageType::Scene); // do something with images } Getting Images with More Flexibility # The simGetImages API which is slightly more complex to use than simGetImage API, for example, you can get left camera view, right camera view and depth image from left camera in a single API call. The simGetImages API also allows you to get uncompressed images as well as floating point single channel images (instead of 3 channel (RGB), each 8 bit). Python # import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() responses = client.simGetImages([ # png format airsim.ImageRequest(0, airsim.ImageType.Scene), # uncompressed RGB array bytes airsim.ImageRequest(1, airsim.ImageType.Scene, False, False), # floating point uncompressed image airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) # do something with response which contains image data, pose, timestamp etc Using AirSim Images with NumPy # If you plan to use numpy for image manipulation, you should get uncompressed RGB image and then convert to numpy like this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.Scene, False, False)]) response = responses[0] # get numpy array img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) # reshape array to 4 channel image array H X W X 4 img_rgb = img1d.reshape(response.height, response.width, 3) # original image is fliped vertically img_rgb = np.flipud(img_rgb) # write to png airsim.write_png(os.path.normpath(filename + '.png'), img_rgb) Quick Tips # The API simGetImage returns binary string literal which means you can simply dump it in binary file to create a .png file. However if you want to process it in any other way than you can handy function airsim.string_to_uint8_array . This converts binary string literal to NumPy uint8 array. The API simGetImages can accept request for multiple image types from any cameras in single call. You can specify if image is png compressed, RGB uncompressed or float array. For png compressed images, you get binary string literal . For float array you get Python list of float64. You can convert this float array to NumPy 2D array using airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height) You can also save float array to .pfm file (Portable Float Map format) using airsim.write_pfm() function. C++ # int getStereoAndDepthImages() { using namespace msr::airlib; typedef VehicleCameraBase::ImageRequest ImageRequest; typedef VehicleCameraBase::ImageResponse ImageResponse; typedef VehicleCameraBase::ImageType ImageType; // for car use // CarRpcLibClient client; MultirotorRpcLibClient client; // get right, left and depth images. First two as png, second as float16. std::vector request = { //png format ImageRequest(\"0\", ImageType::Scene), //uncompressed RGB array bytes ImageRequest(\"1\", ImageType::Scene, false, false), //floating point uncompressed image ImageRequest(\"1\", ImageType::DepthPlanner, true) }; const std::vector& response = client.simGetImages(request); // do something with response which contains image data, pose, timestamp etc } Ready to Run Complete Examples # Python # For a more complete ready to run sample code please see sample code in AirSimClient project for multirotors or HelloCar sample . This code also demonstrates simple activities such as saving images in files or using numpy to manipulate images. C++ # For a more complete ready to run sample code please see sample code in HelloDrone project for multirotors or HelloCar project . See also other example code that generates specified number of stereo images along with ground truth depth and disparity and saving it to pfm format . Available Cameras # Car # The cameras on car can be accessed by following names in API calls: front_center , front_right , front_left , fpv and back_center . Here FPV camera is driver's head position in the car. Multirotor # The cameras in CV mode can be accessed by following names in API calls: front_center , front_right , front_left , bottom_center and back_center . Computer Vision Mode # Camera names are same as in multirotor. Backward compatibility for camera names # Before AirSim v1.2, cameras were accessed using ID numbers instead of names. For backward compatibility you can still use following ID numbers for above camera names in same order as above: \"0\" , \"1\" , \"2\" , \"3\" , \"4\" . In addition, camera name \"\" is also available to access the default camera which is generally the camera \"0\" . \"Computer Vision\" Mode # You can use AirSim in so-called \"Computer Vision\" mode. In this mode, physics engine is disabled and there is no vehicle, just cameras. You can move around using keyboard (use F1 to see help on keys). You can press Record button to continuously generate images. Or you can call APIs to move cameras around and take images. To active this mode, edit settings.json that you can find in your Documents\\AirSim folder (or ~/Documents/AirSim on Linux) and make sure following values exist at root level: { \"SettingsVersion\": 1.2, \"SimMode\": \"ComputerVision\" } Here's the Python code example to move camera around and capture images. This mode was inspired from UnrealCV project . Setting Pose in Computer Vision Mode # To move around the environment using APIs you can use simSetVehiclePose API. This API takes position and orientation and sets that on the invisible vehicle where the front-center camera is located. All rest of the cameras move along keeping the relative position. If you don't want to change position (or orientation) then just set components of position (or orientation) to floating point nan values. The simGetVehiclePose allows to retrieve the current pose. You can also use simGetGroundTruthKinematics to get the quantities kinematics quantities for the movement. Many other non-vehicle specific APIs are also available such as segmentation APIs, collision APIs and camera APIs. Camera APIs # The simGetCameraInfo returns the pose (in world frame, NED coordinates, SI units) and FOV (in degrees) for the specified camera. Please see example usage . The simSetCameraPose sets the pose for the specified camera while taking an input pose as a combination of relative position and a quaternion in NED frame. The handy airsim.to_quaternion() function allows to convert pitch, roll, yaw to quaternion. For example, to set camera-0 to 15-degree pitch while maintaining the same position, you can use: camera_pose = airsim.Pose(airsim.Vector3r(0, 0, 0), airsim.to_quaternion(0.261799, 0, 0)) #RPY in radians client.simSetCameraPose(0, camera_pose); Gimbal # You can set stabilization for pitch, roll or yaw for any camera using settings . Please see example usage . Changing Resolution and Camera Parameters # To change resolution, FOV etc, you can use settings.json . For example, below addition in settings.json sets parameters for scene capture and uses \"Computer Vision\" mode described above. If you omit any setting then below default values will be used. For more information see settings doc . If you are using stereo camera, currently the distance between left and right is fixed at 25 cm. { \"SettingsVersion\": 1.2, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"MotionBlurAmount\": 0 } ] }, \"SimMode\": \"ComputerVision\" } What Does Pixel Values Mean in Different Image Types? # Available ImageType Values # Scene = 0, DepthPlanner = 1, DepthPerspective = 2, DepthVis = 3, DisparityNormalized = 4, Segmentation = 5, SurfaceNormals = 6, Infrared = 7 DepthPlanner and DepthPerspective # You normally want to retrieve the depth image as float (i.e. set pixels_as_float = true ) and specify ImageType = DepthPlanner or ImageType = DepthPerspective in ImageRequest . For ImageType = DepthPlanner , you get depth in camera plan, i.e., all points that are in plan parallel to camera have same depth. For ImageType = DepthPerspective , you get depth from camera using a projection ray that hits that pixel. Depending on your use case, planner depth or perspective depth may be the ground truth image that you want. For example, you may be able to feed perspective depth to ROS package such as depth_image_proc to generate a point cloud. Or planner depth may be more compatible with estimated depth image generated by stereo algorithms such as SGM. DepthVis # When you specify ImageType = DepthVis in ImageRequest , you get an image that helps depth visualization. In this case, each pixel value is interpolated from black to white depending on depth in camera plane in meters. The pixels with pure white means depth of 100m or more while pure black means depth of 0 meters. DisparityNormalized # You normally want to retrieve disparity image as float (i.e. set pixels_as_float = true and specify ImageType = DisparityNormalized in ImageRequest ) in which case each pixel is (Xl - Xr)/Xmax , which is thereby normalized to values between 0 to 1. Segmentation # When you specify ImageType = Segmentation in ImageRequest , you get an image that gives you ground truth segmentation of the scene. At the startup, AirSim assigns value 0 to 255 to each mesh available in environment. This value is then mapped to a specific color in the pallet . The RGB values for each object ID can be found in this file . You can assign a specific value (limited to the range 0-255) to a specific mesh using APIs. For example, below Python code sets the object ID for the mesh called \"Ground\" to 20 in Blocks environment and hence changes its color in Segmentation view: success = client.simSetSegmentationObjectID(\"Ground\", 20); The return value is a boolean type that lets you know if the mesh was found. Notice that typical Unreal environments, like Blocks, usually have many other meshes that comprises of same object, for example, \"Ground_2\", \"Ground_3\" and so on. As it is tedious to set object ID for all of these meshes, AirSim also supports regular expressions. For example, the code below sets all meshes which have names starting with \"ground\" (ignoring case) to 21 with just one line: success = client.simSetSegmentationObjectID(\"ground[\\w]*\", 21, True); The return value is true if at least one mesh was found using regular expression matching. It is recommended that you request uncompressed image using this API to ensure you get precise RGB values for segmentation image: responses = client.simGetImages([ImageRequest(0, AirSimImageType.Segmentation, False, False)]) img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) #get numpy array img_rgb = img1d.reshape(response.height, response.width, 3) #reshape array to 3 channel image array H X W X 3 img_rgb = np.flipud(img_rgb) #original image is fliped vertically #find unique colors print(np.unique(img_rgb[:,:,0], return_counts=True)) #red print(np.unique(img_rgb[:,:,1], return_counts=True)) #green print(np.unique(img_rgb[:,:,2], return_counts=True)) #blue A complete ready-to-run example can be found in segmentation.py . Unsetting object ID # An object's ID can be set to -1 to make it not show up on the segmentation image. How to Find Mesh Names? # To get desired ground truth segmentation you will need to know the names of the meshes in your Unreal environment. To do this, you will need to open up Unreal Environment in Unreal Editor and then inspect the names of the meshes you are interested in using the World Outliner. For example, below we see the mesh names for the ground in Blocks environment in right panel in the editor: If you don't know how to open Unreal Environment in Unreal Editor then try following the guide for building from source . Once you decide on the meshes you are interested, note down their names and use above API to set their object IDs. There are few settings available to change object ID generation behavior. Changing Colors for Object IDs # At present the color for each object ID is fixed as in this pallet . We will be adding ability to change colors for object IDs to desired values shortly. In the meantime you can open the segmentation image in your favorite image editor and get the RGB values you are interested in. Startup Object IDs # At the start, AirSim assigns object ID to each object found in environment of type UStaticMeshComponent or ALandscapeProxy . It then either uses mesh name or owner name (depending on settings), lower cases it, removes any chars below ASCII 97 to remove numbers and some punctuations, sums int value of all chars and modulo 255 to generate the object ID. In other words, all object with same alphabet chars would get same object ID. This heuristic is simple and effective for many Unreal environments but may not be what you want. In that case, please use above APIs to change object IDs to your desired values. There are few settings available to change this behavior. Getting Object ID for Mesh # The simGetSegmentationObjectID API allows you get object ID for given mesh name. Infrared # Currently this is just a map from object ID to grey scale 0-255. So any mesh with object ID 42 shows up with color (42, 42, 42). Please see segmentation section for more details on how to set object IDs. Typically noise setting can be applied for this image type to get slightly more realistic effect. We are still working on adding other infrared artifacts and any contributions are welcome. Example Code # A complete example of setting vehicle positions at random locations and orientations and then taking images can be found in GenerateImageGenerator.hpp . This example generates specified number of stereo images and ground truth disparity image and saving it to pfm format .","title":"Image APIs"},{"location":"image_apis/#image-apis","text":"Please read general API doc first if you are not familiar with AirSim APIs.","title":"Image APIs"},{"location":"image_apis/#getting-a-single-image","text":"Here's a sample code to get a single image from camera named \"0\". The returned value is bytes of png format image. To get uncompressed and other format as well as available cameras please see next sections.","title":"Getting a Single Image"},{"location":"image_apis/#python","text":"import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() png_image = client.simGetImage(\"0\", airsim.ImageType.Scene) # do something with image","title":"Python"},{"location":"image_apis/#c","text":"#include \"vehicles/multirotor/api/MultirotorRpcLibClient.hpp\" int getOneImage() { using namespace msr::airlib; // for car use CarRpcLibClient MultirotorRpcLibClient client; std::vector png_image = client.simGetImage(\"0\", VehicleCameraBase::ImageType::Scene); // do something with images }","title":"C++"},{"location":"image_apis/#getting-images-with-more-flexibility","text":"The simGetImages API which is slightly more complex to use than simGetImage API, for example, you can get left camera view, right camera view and depth image from left camera in a single API call. The simGetImages API also allows you to get uncompressed images as well as floating point single channel images (instead of 3 channel (RGB), each 8 bit).","title":"Getting Images with More Flexibility"},{"location":"image_apis/#python_1","text":"import airsim #pip install airsim # for car use CarClient() client = airsim.MultirotorClient() responses = client.simGetImages([ # png format airsim.ImageRequest(0, airsim.ImageType.Scene), # uncompressed RGB array bytes airsim.ImageRequest(1, airsim.ImageType.Scene, False, False), # floating point uncompressed image airsim.ImageRequest(1, airsim.ImageType.DepthPlanner, True)]) # do something with response which contains image data, pose, timestamp etc","title":"Python"},{"location":"image_apis/#using-airsim-images-with-numpy","text":"If you plan to use numpy for image manipulation, you should get uncompressed RGB image and then convert to numpy like this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.Scene, False, False)]) response = responses[0] # get numpy array img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) # reshape array to 4 channel image array H X W X 4 img_rgb = img1d.reshape(response.height, response.width, 3) # original image is fliped vertically img_rgb = np.flipud(img_rgb) # write to png airsim.write_png(os.path.normpath(filename + '.png'), img_rgb)","title":"Using AirSim Images with NumPy"},{"location":"image_apis/#quick-tips","text":"The API simGetImage returns binary string literal which means you can simply dump it in binary file to create a .png file. However if you want to process it in any other way than you can handy function airsim.string_to_uint8_array . This converts binary string literal to NumPy uint8 array. The API simGetImages can accept request for multiple image types from any cameras in single call. You can specify if image is png compressed, RGB uncompressed or float array. For png compressed images, you get binary string literal . For float array you get Python list of float64. You can convert this float array to NumPy 2D array using airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height) You can also save float array to .pfm file (Portable Float Map format) using airsim.write_pfm() function.","title":"Quick Tips"},{"location":"image_apis/#c_1","text":"int getStereoAndDepthImages() { using namespace msr::airlib; typedef VehicleCameraBase::ImageRequest ImageRequest; typedef VehicleCameraBase::ImageResponse ImageResponse; typedef VehicleCameraBase::ImageType ImageType; // for car use // CarRpcLibClient client; MultirotorRpcLibClient client; // get right, left and depth images. First two as png, second as float16. std::vector request = { //png format ImageRequest(\"0\", ImageType::Scene), //uncompressed RGB array bytes ImageRequest(\"1\", ImageType::Scene, false, false), //floating point uncompressed image ImageRequest(\"1\", ImageType::DepthPlanner, true) }; const std::vector& response = client.simGetImages(request); // do something with response which contains image data, pose, timestamp etc }","title":"C++"},{"location":"image_apis/#ready-to-run-complete-examples","text":"","title":"Ready to Run Complete Examples"},{"location":"image_apis/#python_2","text":"For a more complete ready to run sample code please see sample code in AirSimClient project for multirotors or HelloCar sample . This code also demonstrates simple activities such as saving images in files or using numpy to manipulate images.","title":"Python"},{"location":"image_apis/#c_2","text":"For a more complete ready to run sample code please see sample code in HelloDrone project for multirotors or HelloCar project . See also other example code that generates specified number of stereo images along with ground truth depth and disparity and saving it to pfm format .","title":"C++"},{"location":"image_apis/#available-cameras","text":"","title":"Available Cameras"},{"location":"image_apis/#car","text":"The cameras on car can be accessed by following names in API calls: front_center , front_right , front_left , fpv and back_center . Here FPV camera is driver's head position in the car.","title":"Car"},{"location":"image_apis/#multirotor","text":"The cameras in CV mode can be accessed by following names in API calls: front_center , front_right , front_left , bottom_center and back_center .","title":"Multirotor"},{"location":"image_apis/#computer-vision-mode","text":"Camera names are same as in multirotor.","title":"Computer Vision Mode"},{"location":"image_apis/#backward-compatibility-for-camera-names","text":"Before AirSim v1.2, cameras were accessed using ID numbers instead of names. For backward compatibility you can still use following ID numbers for above camera names in same order as above: \"0\" , \"1\" , \"2\" , \"3\" , \"4\" . In addition, camera name \"\" is also available to access the default camera which is generally the camera \"0\" .","title":"Backward compatibility for camera names"},{"location":"image_apis/#computer-vision-mode_1","text":"You can use AirSim in so-called \"Computer Vision\" mode. In this mode, physics engine is disabled and there is no vehicle, just cameras. You can move around using keyboard (use F1 to see help on keys). You can press Record button to continuously generate images. Or you can call APIs to move cameras around and take images. To active this mode, edit settings.json that you can find in your Documents\\AirSim folder (or ~/Documents/AirSim on Linux) and make sure following values exist at root level: { \"SettingsVersion\": 1.2, \"SimMode\": \"ComputerVision\" } Here's the Python code example to move camera around and capture images. This mode was inspired from UnrealCV project .","title":"\"Computer Vision\" Mode"},{"location":"image_apis/#setting-pose-in-computer-vision-mode","text":"To move around the environment using APIs you can use simSetVehiclePose API. This API takes position and orientation and sets that on the invisible vehicle where the front-center camera is located. All rest of the cameras move along keeping the relative position. If you don't want to change position (or orientation) then just set components of position (or orientation) to floating point nan values. The simGetVehiclePose allows to retrieve the current pose. You can also use simGetGroundTruthKinematics to get the quantities kinematics quantities for the movement. Many other non-vehicle specific APIs are also available such as segmentation APIs, collision APIs and camera APIs.","title":"Setting Pose in Computer Vision Mode"},{"location":"image_apis/#camera-apis","text":"The simGetCameraInfo returns the pose (in world frame, NED coordinates, SI units) and FOV (in degrees) for the specified camera. Please see example usage . The simSetCameraPose sets the pose for the specified camera while taking an input pose as a combination of relative position and a quaternion in NED frame. The handy airsim.to_quaternion() function allows to convert pitch, roll, yaw to quaternion. For example, to set camera-0 to 15-degree pitch while maintaining the same position, you can use: camera_pose = airsim.Pose(airsim.Vector3r(0, 0, 0), airsim.to_quaternion(0.261799, 0, 0)) #RPY in radians client.simSetCameraPose(0, camera_pose);","title":"Camera APIs"},{"location":"image_apis/#gimbal","text":"You can set stabilization for pitch, roll or yaw for any camera using settings . Please see example usage .","title":"Gimbal"},{"location":"image_apis/#changing-resolution-and-camera-parameters","text":"To change resolution, FOV etc, you can use settings.json . For example, below addition in settings.json sets parameters for scene capture and uses \"Computer Vision\" mode described above. If you omit any setting then below default values will be used. For more information see settings doc . If you are using stereo camera, currently the distance between left and right is fixed at 25 cm. { \"SettingsVersion\": 1.2, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"MotionBlurAmount\": 0 } ] }, \"SimMode\": \"ComputerVision\" }","title":"Changing Resolution and Camera Parameters"},{"location":"image_apis/#what-does-pixel-values-mean-in-different-image-types","text":"","title":"What Does Pixel Values Mean in Different Image Types?"},{"location":"image_apis/#available-imagetype-values","text":"Scene = 0, DepthPlanner = 1, DepthPerspective = 2, DepthVis = 3, DisparityNormalized = 4, Segmentation = 5, SurfaceNormals = 6, Infrared = 7","title":"Available ImageType Values"},{"location":"image_apis/#depthplanner-and-depthperspective","text":"You normally want to retrieve the depth image as float (i.e. set pixels_as_float = true ) and specify ImageType = DepthPlanner or ImageType = DepthPerspective in ImageRequest . For ImageType = DepthPlanner , you get depth in camera plan, i.e., all points that are in plan parallel to camera have same depth. For ImageType = DepthPerspective , you get depth from camera using a projection ray that hits that pixel. Depending on your use case, planner depth or perspective depth may be the ground truth image that you want. For example, you may be able to feed perspective depth to ROS package such as depth_image_proc to generate a point cloud. Or planner depth may be more compatible with estimated depth image generated by stereo algorithms such as SGM.","title":"DepthPlanner and DepthPerspective"},{"location":"image_apis/#depthvis","text":"When you specify ImageType = DepthVis in ImageRequest , you get an image that helps depth visualization. In this case, each pixel value is interpolated from black to white depending on depth in camera plane in meters. The pixels with pure white means depth of 100m or more while pure black means depth of 0 meters.","title":"DepthVis"},{"location":"image_apis/#disparitynormalized","text":"You normally want to retrieve disparity image as float (i.e. set pixels_as_float = true and specify ImageType = DisparityNormalized in ImageRequest ) in which case each pixel is (Xl - Xr)/Xmax , which is thereby normalized to values between 0 to 1.","title":"DisparityNormalized"},{"location":"image_apis/#segmentation","text":"When you specify ImageType = Segmentation in ImageRequest , you get an image that gives you ground truth segmentation of the scene. At the startup, AirSim assigns value 0 to 255 to each mesh available in environment. This value is then mapped to a specific color in the pallet . The RGB values for each object ID can be found in this file . You can assign a specific value (limited to the range 0-255) to a specific mesh using APIs. For example, below Python code sets the object ID for the mesh called \"Ground\" to 20 in Blocks environment and hence changes its color in Segmentation view: success = client.simSetSegmentationObjectID(\"Ground\", 20); The return value is a boolean type that lets you know if the mesh was found. Notice that typical Unreal environments, like Blocks, usually have many other meshes that comprises of same object, for example, \"Ground_2\", \"Ground_3\" and so on. As it is tedious to set object ID for all of these meshes, AirSim also supports regular expressions. For example, the code below sets all meshes which have names starting with \"ground\" (ignoring case) to 21 with just one line: success = client.simSetSegmentationObjectID(\"ground[\\w]*\", 21, True); The return value is true if at least one mesh was found using regular expression matching. It is recommended that you request uncompressed image using this API to ensure you get precise RGB values for segmentation image: responses = client.simGetImages([ImageRequest(0, AirSimImageType.Segmentation, False, False)]) img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) #get numpy array img_rgb = img1d.reshape(response.height, response.width, 3) #reshape array to 3 channel image array H X W X 3 img_rgb = np.flipud(img_rgb) #original image is fliped vertically #find unique colors print(np.unique(img_rgb[:,:,0], return_counts=True)) #red print(np.unique(img_rgb[:,:,1], return_counts=True)) #green print(np.unique(img_rgb[:,:,2], return_counts=True)) #blue A complete ready-to-run example can be found in segmentation.py .","title":"Segmentation"},{"location":"image_apis/#unsetting-object-id","text":"An object's ID can be set to -1 to make it not show up on the segmentation image.","title":"Unsetting object ID"},{"location":"image_apis/#how-to-find-mesh-names","text":"To get desired ground truth segmentation you will need to know the names of the meshes in your Unreal environment. To do this, you will need to open up Unreal Environment in Unreal Editor and then inspect the names of the meshes you are interested in using the World Outliner. For example, below we see the mesh names for the ground in Blocks environment in right panel in the editor: If you don't know how to open Unreal Environment in Unreal Editor then try following the guide for building from source . Once you decide on the meshes you are interested, note down their names and use above API to set their object IDs. There are few settings available to change object ID generation behavior.","title":"How to Find Mesh Names?"},{"location":"image_apis/#changing-colors-for-object-ids","text":"At present the color for each object ID is fixed as in this pallet . We will be adding ability to change colors for object IDs to desired values shortly. In the meantime you can open the segmentation image in your favorite image editor and get the RGB values you are interested in.","title":"Changing Colors for Object IDs"},{"location":"image_apis/#startup-object-ids","text":"At the start, AirSim assigns object ID to each object found in environment of type UStaticMeshComponent or ALandscapeProxy . It then either uses mesh name or owner name (depending on settings), lower cases it, removes any chars below ASCII 97 to remove numbers and some punctuations, sums int value of all chars and modulo 255 to generate the object ID. In other words, all object with same alphabet chars would get same object ID. This heuristic is simple and effective for many Unreal environments but may not be what you want. In that case, please use above APIs to change object IDs to your desired values. There are few settings available to change this behavior.","title":"Startup Object IDs"},{"location":"image_apis/#getting-object-id-for-mesh","text":"The simGetSegmentationObjectID API allows you get object ID for given mesh name.","title":"Getting Object ID for Mesh"},{"location":"image_apis/#infrared","text":"Currently this is just a map from object ID to grey scale 0-255. So any mesh with object ID 42 shows up with color (42, 42, 42). Please see segmentation section for more details on how to set object IDs. Typically noise setting can be applied for this image type to get slightly more realistic effect. We are still working on adding other infrared artifacts and any contributions are welcome.","title":"Infrared"},{"location":"image_apis/#example-code","text":"A complete example of setting vehicle positions at random locations and orientations and then taking images can be found in GenerateImageGenerator.hpp . This example generates specified number of stereo images and ground truth disparity image and saving it to pfm format .","title":"Example Code"},{"location":"lidar/","text":"How to Use Lidar in AirSim # AirSim supports Lidar for multirotors and cars. The enablement of lidar and the other lidar settings can be configured via AirSimSettings json. Please see general sensors for information on configruation of general/shared sensor settings. Enabling lidar on a vehicle # By default, lidars are not enabled. To enable lidar, set the SensorType and Enabled attributes in settings json. \"Lidar1\": { \"SensorType\": 6, \"Enabled\" : true, Multiple lidars can be enabled on a vehicle. Lidar configuration # The following parameters can be configured right now via settings json. Parameter Description NumberOfChannels Number of channels/lasers of the lidar Range Range, in meters PointsPerSecond Number of points captured per second RotationsPerSecond Rotations per second HorizontalFOVStart Horizontal FOV start for the lidar, in degrees HorizontalFOVEnd Horizontal FOV end for the lidar, in degrees VerticalFOVUpper Vertical FOV upper limit for the lidar, in degrees VerticalFOVLower Vertical FOV lower limit for the lidar, in degrees X Y Z Position of the lidar relative to the vehicle (in NED, in meters) Roll Pitch Yaw Orientation of the lidar relative to the vehicle (in degrees, yaw-pitch-roll order to front vector +X) DataFrame Frame for the points in output (\"VehicleInertialFrame\" or \"SensorLocalFrame\") e.g., { \"SeeDocsAt\": \"https://github.com/Microsoft/AirSim/blob/master/docs/settings_json.md\", \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"simpleflight\", \"AutoCreate\": true, \"Sensors\": { \"LidarSensor1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 100000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"HorizontalFOVStart\": -20, \"HorizontalFOVEnd\": 20, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" }, \"LidarSensor2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" } } } } } Server side visualization for debugging # Be default, the lidar points are not drawn on the viewport. To enable the drawing of hit laser points on the viewport, please enable setting 'DrawDebugPoints' via settings json. e.g., \"Lidar1\": { ... \"DrawDebugPoints\": true }, Client API # Use getLidarData() API to retrieve the Lidar data. * The API returns a Point-Cloud as a flat array of floats along with the timestamp of the capture and lidar pose. * Point-Cloud: * The floats represent [x,y,z] coordinate for each point hit within the range in the last scan. * The frame for the points in the output is configurable using \"DataFrame\" attribute \"\" or \"VehicleInertialFrame\" -- default; returned points are in vehicle inertial frame (in NED, in meters) \"SensorLocalFrame\" -- returned points are in lidar local frame (in NED, in meters) * Lidar Pose: * Lidar pose in the vehicle inertial frame (in NED, in meters) * Can be used to transform points to other frames. Python Examples # drone_lidar.py car_lidar.py Coming soon # Visualization of lidar data on client side.","title":"LIDAR"},{"location":"lidar/#how-to-use-lidar-in-airsim","text":"AirSim supports Lidar for multirotors and cars. The enablement of lidar and the other lidar settings can be configured via AirSimSettings json. Please see general sensors for information on configruation of general/shared sensor settings.","title":"How to Use Lidar in AirSim"},{"location":"lidar/#enabling-lidar-on-a-vehicle","text":"By default, lidars are not enabled. To enable lidar, set the SensorType and Enabled attributes in settings json. \"Lidar1\": { \"SensorType\": 6, \"Enabled\" : true, Multiple lidars can be enabled on a vehicle.","title":"Enabling lidar on a vehicle"},{"location":"lidar/#lidar-configuration","text":"The following parameters can be configured right now via settings json. Parameter Description NumberOfChannels Number of channels/lasers of the lidar Range Range, in meters PointsPerSecond Number of points captured per second RotationsPerSecond Rotations per second HorizontalFOVStart Horizontal FOV start for the lidar, in degrees HorizontalFOVEnd Horizontal FOV end for the lidar, in degrees VerticalFOVUpper Vertical FOV upper limit for the lidar, in degrees VerticalFOVLower Vertical FOV lower limit for the lidar, in degrees X Y Z Position of the lidar relative to the vehicle (in NED, in meters) Roll Pitch Yaw Orientation of the lidar relative to the vehicle (in degrees, yaw-pitch-roll order to front vector +X) DataFrame Frame for the points in output (\"VehicleInertialFrame\" or \"SensorLocalFrame\") e.g., { \"SeeDocsAt\": \"https://github.com/Microsoft/AirSim/blob/master/docs/settings_json.md\", \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"simpleflight\", \"AutoCreate\": true, \"Sensors\": { \"LidarSensor1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 100000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"HorizontalFOVStart\": -20, \"HorizontalFOVEnd\": 20, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" }, \"LidarSensor2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"RotationsPerSecond\": 10, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"Roll\": 0, \"Pitch\": 0, \"Yaw\" : 0, \"VerticalFOVUpper\": -15, \"VerticalFOVLower\": -25, \"DrawDebugPoints\": true, \"DataFrame\": \"SensorLocalFrame\" } } } } }","title":"Lidar configuration"},{"location":"lidar/#server-side-visualization-for-debugging","text":"Be default, the lidar points are not drawn on the viewport. To enable the drawing of hit laser points on the viewport, please enable setting 'DrawDebugPoints' via settings json. e.g., \"Lidar1\": { ... \"DrawDebugPoints\": true },","title":"Server side visualization for debugging"},{"location":"lidar/#client-api","text":"Use getLidarData() API to retrieve the Lidar data. * The API returns a Point-Cloud as a flat array of floats along with the timestamp of the capture and lidar pose. * Point-Cloud: * The floats represent [x,y,z] coordinate for each point hit within the range in the last scan. * The frame for the points in the output is configurable using \"DataFrame\" attribute \"\" or \"VehicleInertialFrame\" -- default; returned points are in vehicle inertial frame (in NED, in meters) \"SensorLocalFrame\" -- returned points are in lidar local frame (in NED, in meters) * Lidar Pose: * Lidar pose in the vehicle inertial frame (in NED, in meters) * Can be used to transform points to other frames.","title":"Client API"},{"location":"lidar/#python-examples","text":"drone_lidar.py car_lidar.py","title":"Python Examples"},{"location":"lidar/#coming-soon","text":"Visualization of lidar data on client side.","title":"Coming soon"},{"location":"log_viewer/","text":"Log Viewer # The LogViewer is a Windows WPF app that presents the MavLink streams that it is getting from the Unreal Simulator. You can use this to monitor what is happening on the drone while it is flying. For example, the picture below shows a real time graph of the x, y an z gyro sensor information being generated by the simulator. Usage # To use this LogViewer, connect the simulator before you run the simulation. Simply press the blue connector button on the top right corner of the window, select the Socket tab , enter the port number 14388, and your localhost network. Then press the record button (triangle on the right hand side of the toolbar). Now start the simulator, pick some mavlink items to graph, you should see something like this: The drone view here is the actual estimated position coming from the PX4, so that is a great way to check whether the PX4 is in sync with the simulator. Sometimes you can see some drift here as the attitude estimation catches up with reality, this is more visible after a bad crash. Installation # If you can't build the LogViewer.sln, there is also a click once installer . Configuration # The magic port number 14388 can be configured in the simulator by editing the settings.json file .","title":"MavLink LogViewer"},{"location":"log_viewer/#log-viewer","text":"The LogViewer is a Windows WPF app that presents the MavLink streams that it is getting from the Unreal Simulator. You can use this to monitor what is happening on the drone while it is flying. For example, the picture below shows a real time graph of the x, y an z gyro sensor information being generated by the simulator.","title":"Log Viewer"},{"location":"log_viewer/#usage","text":"To use this LogViewer, connect the simulator before you run the simulation. Simply press the blue connector button on the top right corner of the window, select the Socket tab , enter the port number 14388, and your localhost network. Then press the record button (triangle on the right hand side of the toolbar). Now start the simulator, pick some mavlink items to graph, you should see something like this: The drone view here is the actual estimated position coming from the PX4, so that is a great way to check whether the PX4 is in sync with the simulator. Sometimes you can see some drift here as the attitude estimation catches up with reality, this is more visible after a bad crash.","title":"Usage"},{"location":"log_viewer/#installation","text":"If you can't build the LogViewer.sln, there is also a click once installer .","title":"Installation"},{"location":"log_viewer/#configuration","text":"The magic port number 14388 can be configured in the simulator by editing the settings.json file .","title":"Configuration"},{"location":"mavlinkcom/","text":"Welcome to MavLinkCom # MavLinkCom is a cross-platform C++ library that helps connect to and communicate with MavLink based vehicles. Specifically this library is designed to work well with PX4 based drones. Design # You can view and edit the Design.dgml diagram in Visual Studio. The following are the most important classes in this library. MavLinkNode # This is the base class for all MavLinkNodes (subclasses include MavLinkVehicle, MavLinkVideoClient and MavLinkVideoServer). The node connects to your mavlink enabled vehicle via a MavLinkConnection and provides methods for sending MavLinkMessages and MavLinkCommands and for subscribing to receive messages. This base class also stores the local system id and component id your app wants to use to identify itself to your remove vehicle. You can also call startHeartbeat to send regular heartbeat messages to keep the connection alive. MavLinkMessage # This is the encoded MavLinkMessage. For those who have used the mavlink.h C API, this is similar to mavlink_message_t. You do not create these manually, they are encoded from a strongly typed MavLinkMessageBase subclass. Strongly typed message and command classes # The MavLinkComGenerator parses the mavlink common.xml message definitions and generates all the MavLink* MavLinkMessageBase subclasses as well as a bunch of handy mavlink enums and a bunch of strongly typed MavLinkCommand subclasses. MavLinkMessageBase # This is the base class for a set of strongly typed message classes that are code generated by the MavLinkComGenerator project. This replaces the C messages defined in the mavlink C API and provides a slightly more object oriented way to send and receive messages via sendMessage on MavLinkNode. These classes have encode/decode methods that convert to and from the MavLinkMessage class. MavLinkCommand # This is the base class for a set of strongly typed command classes that are code generated by the MavLinkComGenerator project. This replaces the C definitions defined in the mavlink C API and provides a more object oriented way to send commands via the sendCommand method on MavLinkNode. The MavLinkNode takes care of turning these into the underlying mavlink COMMAND_LONG message. MavLinkConnection # This class provides static helper methods for creating connections to remote MavLink nodes, over serial ports, as well as UDP, or TCP sockets. This class provides a way to subscribe to receive messages from that node in a pub/sub way so you can have multiple subscribers on the same connection. MavLinkVehicle uses this to track various messages that define the overall vehicle state. MavLinkVehicle # MavLinkVehicle is a MavLinkNode that tracks various messages that define the overall vehicle state and provides a VehicleState struct containing a snapshot of that state, including home position, current orientation, local position, global position, and so on. This class also provides a bunch of helper methods that wrap commonly used commands providing simple method calls to do things like arm, disarm, takeoff, land, go to a local coordinate, and fly under offbaord control either by position or velocity control. MavLinkTcpServer # This helper class provides a way to setup a \"server\" that accepts MavLinkConnections from remote nodes. You can use this class to get a connection that you can then give to MavLinkVideoServer to serve images over MavLink. MavLinkFtpClient # This helper class takes a given MavLinkConnection and provides FTP client support for the MAVLINK_MSG_ID_FILE_TRANSFER_PROTOCOL for vehicles that support the FTP capability. This class provides simple methods to list directory contents, and the get and put files. MavLinkVideoClient # This helper class takes a given MavLinkConnection and provides helper methods for requesting video from remote node and packaging up the MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA messages into simple to use MavLinkVideoFrames. MavLinkVideoServer # This helper class takes a given MavLinkConnection and provides the server side of the MavLinkVideoClient protocol, including helper methods for notifying when there is a video request to process (hasVideoRequest) and a method to send video frames (sendFrame) which will generate the right MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA sequence. Examples # The following code from the UnitTest project shows how to connect to a Pixhawk flight controller over USB serial port, then wait for the first heartbeat message to be received: auto connection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); MavLinkHeartbeat heartbeat; if (!waitForHeartbeat(10000, heartbeat)) { throw std::runtime_error(\"Received no heartbeat from PX4 after 10 seconds\"); } The following code connects to serial port, and then forwards all messages to and from QGroundControl to that drone using another connection that is joined to the drone stream. auto droneConnection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); auto proxyConnection = MavLinkConnection::connectRemoteUdp(\"qgc\", \"127.0.0.1\", \"127.0.0.1\", 14550); droneConnection->join(proxyConnection); The following code then takes that connection and turns on heartBeats and starts tracking vehicle information using local system id 166 and component id 1. auto vehicle = std::make_shared(166, 1); vehicle->connect(connection); vehicle->startHeartbeat(); std::this_thread::sleep_for(std::chrono::seconds(5)); VehicleState state = vehicle->getVehicleState(); printf(\"Home position is %s, %f,%f,%f\\n\", state.home.is_set ? \"set\" : \"not set\", state.home.global_pos.lat, state.home.global_pos.lon, state.home.global_pos.alt); The following code uses the vehicle object to arm the drone and take off and wait for the takeoff altitude to be reached: bool rc = false; if (!vehicle->armDisarm(true).wait(3000, &rc) || !rc) { printf(\"arm command failed\\n\"); return; } if (!vehicle->takeoff(targetAlt).wait(3000, &rc) || !rc) { printf(\"takeoff command failed\\n\"); return; } int version = vehicle->getVehicleStateVersion(); while (true) { int newVersion = vehicle->getVehicleStateVersion(); if (version != newVersion) { VehicleState state = vehicle->getVehicleState(); float alt = state.local_est.pos.z; if (alt >= targetAlt - delta && alt <= targetAlt + delta) { reached = true; printf(\"Target altitude reached\\n\"); break; } } else { std::this_thread::sleep_for(std::chrono::milliseconds(10)); } } The following code uses offboard control to make the drone fly in a circle with camera pointed at the center. Here we use the subscribe method to check each new local position message to indicate so we can compute the new velocity vector as soon as that new position is received. We request a high rate for those messages using setMessageInterval to ensure smooth circular orbit. vehicle->setMessageInterval((int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED, 30); vehicle->requestControl(); int subscription = vehicle->getConnection()->subscribe( [&](std::shared_ptr connection, const MavLinkMessage& m) { if (m.msgid == (int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED) { float x = localPos.x; float y = localPos.y; float dx = x - cx; float dy = y - cy; float angle = atan2(dy, dx); if (angle < 0) angle += M_PI * 2; float tangent = angle + M_PI_2; double newvx = orbitSpeed * cos(tangent); double newvy = orbitSpeed * sin(tangent); float heading = angle + M_PI; vehicle->moveByLocalVelocityWithAltHold(newvx, newvy, altitude, true, heading); } }); The following code stops flying the drone in offboard mode and tells the drone to loiter at its current location. This version of the code shows how to use the AsyncResult without blocking on a wait call. vehicle->releaseControl(); if (vehicle->loiter().then([=](bool rc) { printf(\"loiter command %s\\n\", rc ? \"succeeded\" : \"failed\"); } The following code gets all configurable parameters from the drone and prints them: auto list = vehicle->getParamList(); auto end = list.end(); int count = 0; for (auto iter = list.begin(); iter < end; iter++) { count++; MavLinkParameter p = *iter; if (p.type == MAV_PARAM_TYPE_REAL32 || p.type == MAV_PARAM_TYPE_REAL64) { printf(\"%s=%f\\n\", p.name.c_str(), p.value); } else { printf(\"%s=%d\\n\", p.name.c_str(), static_cast(p.value)); } } The following code sets a parameter on the Pixhawk to disable the USB safety check (this is handy if you are controlling the Pixhawk over USB using another onboard computer that is part of the drone itself). You should NOT do this if you are connecting your PC or laptop to the drone over USB. MavLinkParameter p; p.name = \"CBRK_USB_CHK\"; p.value = 197848; if (!vehicle->setParameter(p).wait(3000,&rc) || !rc) { printf(\"Setting the CBRK_USB_CHK failed\"); } MavLinkVehicle actually has a helper method for this called allowFlightControlOverUsb, so now you know how it is implemented :-) Advanced Connections # You can wire up different configurations of mavlink pipelines using the MavLinkConnection class \"join\" method as shown below. Example 1, we connect to PX4 over serial, and proxy those messages through to QGroundControl and the LogViewer who are listening on remote ports. Example 2: simulation can talk to jMavSim and jMavSim connects to PX4. jMavSim can also manage multiple connections, so it can talk to unreal simulator. Another MavLinkConnection can be joined to proxy connections that jMavSim doesn't support, like the LogViewer or a remote camera node. Example 3: we use MavLinkConnection to connect to PX4 over serial, then join additional connections for all our remote nodes including jMavSim. Example 4: We can also do distributed systems to control the drone remotely:","title":"MavLinkCom"},{"location":"mavlinkcom/#welcome-to-mavlinkcom","text":"MavLinkCom is a cross-platform C++ library that helps connect to and communicate with MavLink based vehicles. Specifically this library is designed to work well with PX4 based drones.","title":"Welcome to MavLinkCom"},{"location":"mavlinkcom/#design","text":"You can view and edit the Design.dgml diagram in Visual Studio. The following are the most important classes in this library.","title":"Design"},{"location":"mavlinkcom/#mavlinknode","text":"This is the base class for all MavLinkNodes (subclasses include MavLinkVehicle, MavLinkVideoClient and MavLinkVideoServer). The node connects to your mavlink enabled vehicle via a MavLinkConnection and provides methods for sending MavLinkMessages and MavLinkCommands and for subscribing to receive messages. This base class also stores the local system id and component id your app wants to use to identify itself to your remove vehicle. You can also call startHeartbeat to send regular heartbeat messages to keep the connection alive.","title":"MavLinkNode"},{"location":"mavlinkcom/#mavlinkmessage","text":"This is the encoded MavLinkMessage. For those who have used the mavlink.h C API, this is similar to mavlink_message_t. You do not create these manually, they are encoded from a strongly typed MavLinkMessageBase subclass.","title":"MavLinkMessage"},{"location":"mavlinkcom/#strongly-typed-message-and-command-classes","text":"The MavLinkComGenerator parses the mavlink common.xml message definitions and generates all the MavLink* MavLinkMessageBase subclasses as well as a bunch of handy mavlink enums and a bunch of strongly typed MavLinkCommand subclasses.","title":"Strongly typed message and command classes"},{"location":"mavlinkcom/#mavlinkmessagebase","text":"This is the base class for a set of strongly typed message classes that are code generated by the MavLinkComGenerator project. This replaces the C messages defined in the mavlink C API and provides a slightly more object oriented way to send and receive messages via sendMessage on MavLinkNode. These classes have encode/decode methods that convert to and from the MavLinkMessage class.","title":"MavLinkMessageBase"},{"location":"mavlinkcom/#mavlinkcommand","text":"This is the base class for a set of strongly typed command classes that are code generated by the MavLinkComGenerator project. This replaces the C definitions defined in the mavlink C API and provides a more object oriented way to send commands via the sendCommand method on MavLinkNode. The MavLinkNode takes care of turning these into the underlying mavlink COMMAND_LONG message.","title":"MavLinkCommand"},{"location":"mavlinkcom/#mavlinkconnection","text":"This class provides static helper methods for creating connections to remote MavLink nodes, over serial ports, as well as UDP, or TCP sockets. This class provides a way to subscribe to receive messages from that node in a pub/sub way so you can have multiple subscribers on the same connection. MavLinkVehicle uses this to track various messages that define the overall vehicle state.","title":"MavLinkConnection"},{"location":"mavlinkcom/#mavlinkvehicle","text":"MavLinkVehicle is a MavLinkNode that tracks various messages that define the overall vehicle state and provides a VehicleState struct containing a snapshot of that state, including home position, current orientation, local position, global position, and so on. This class also provides a bunch of helper methods that wrap commonly used commands providing simple method calls to do things like arm, disarm, takeoff, land, go to a local coordinate, and fly under offbaord control either by position or velocity control.","title":"MavLinkVehicle"},{"location":"mavlinkcom/#mavlinktcpserver","text":"This helper class provides a way to setup a \"server\" that accepts MavLinkConnections from remote nodes. You can use this class to get a connection that you can then give to MavLinkVideoServer to serve images over MavLink.","title":"MavLinkTcpServer"},{"location":"mavlinkcom/#mavlinkftpclient","text":"This helper class takes a given MavLinkConnection and provides FTP client support for the MAVLINK_MSG_ID_FILE_TRANSFER_PROTOCOL for vehicles that support the FTP capability. This class provides simple methods to list directory contents, and the get and put files.","title":"MavLinkFtpClient"},{"location":"mavlinkcom/#mavlinkvideoclient","text":"This helper class takes a given MavLinkConnection and provides helper methods for requesting video from remote node and packaging up the MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA messages into simple to use MavLinkVideoFrames.","title":"MavLinkVideoClient"},{"location":"mavlinkcom/#mavlinkvideoserver","text":"This helper class takes a given MavLinkConnection and provides the server side of the MavLinkVideoClient protocol, including helper methods for notifying when there is a video request to process (hasVideoRequest) and a method to send video frames (sendFrame) which will generate the right MAVLINK_MSG_ID_DATA_TRANSMISSION_HANDSHAKE and MAVLINK_MSG_ID_ENCAPSULATED_DATA sequence.","title":"MavLinkVideoServer"},{"location":"mavlinkcom/#examples","text":"The following code from the UnitTest project shows how to connect to a Pixhawk flight controller over USB serial port, then wait for the first heartbeat message to be received: auto connection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); MavLinkHeartbeat heartbeat; if (!waitForHeartbeat(10000, heartbeat)) { throw std::runtime_error(\"Received no heartbeat from PX4 after 10 seconds\"); } The following code connects to serial port, and then forwards all messages to and from QGroundControl to that drone using another connection that is joined to the drone stream. auto droneConnection = MavLinkConnection::connectSerial(\"drone\", \"/dev/ttyACM0\", 115200, \"sh /etc/init.d/rc.usb\\n\"); auto proxyConnection = MavLinkConnection::connectRemoteUdp(\"qgc\", \"127.0.0.1\", \"127.0.0.1\", 14550); droneConnection->join(proxyConnection); The following code then takes that connection and turns on heartBeats and starts tracking vehicle information using local system id 166 and component id 1. auto vehicle = std::make_shared(166, 1); vehicle->connect(connection); vehicle->startHeartbeat(); std::this_thread::sleep_for(std::chrono::seconds(5)); VehicleState state = vehicle->getVehicleState(); printf(\"Home position is %s, %f,%f,%f\\n\", state.home.is_set ? \"set\" : \"not set\", state.home.global_pos.lat, state.home.global_pos.lon, state.home.global_pos.alt); The following code uses the vehicle object to arm the drone and take off and wait for the takeoff altitude to be reached: bool rc = false; if (!vehicle->armDisarm(true).wait(3000, &rc) || !rc) { printf(\"arm command failed\\n\"); return; } if (!vehicle->takeoff(targetAlt).wait(3000, &rc) || !rc) { printf(\"takeoff command failed\\n\"); return; } int version = vehicle->getVehicleStateVersion(); while (true) { int newVersion = vehicle->getVehicleStateVersion(); if (version != newVersion) { VehicleState state = vehicle->getVehicleState(); float alt = state.local_est.pos.z; if (alt >= targetAlt - delta && alt <= targetAlt + delta) { reached = true; printf(\"Target altitude reached\\n\"); break; } } else { std::this_thread::sleep_for(std::chrono::milliseconds(10)); } } The following code uses offboard control to make the drone fly in a circle with camera pointed at the center. Here we use the subscribe method to check each new local position message to indicate so we can compute the new velocity vector as soon as that new position is received. We request a high rate for those messages using setMessageInterval to ensure smooth circular orbit. vehicle->setMessageInterval((int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED, 30); vehicle->requestControl(); int subscription = vehicle->getConnection()->subscribe( [&](std::shared_ptr connection, const MavLinkMessage& m) { if (m.msgid == (int)MavLinkMessageIds::MAVLINK_MSG_ID_LOCAL_POSITION_NED) { float x = localPos.x; float y = localPos.y; float dx = x - cx; float dy = y - cy; float angle = atan2(dy, dx); if (angle < 0) angle += M_PI * 2; float tangent = angle + M_PI_2; double newvx = orbitSpeed * cos(tangent); double newvy = orbitSpeed * sin(tangent); float heading = angle + M_PI; vehicle->moveByLocalVelocityWithAltHold(newvx, newvy, altitude, true, heading); } }); The following code stops flying the drone in offboard mode and tells the drone to loiter at its current location. This version of the code shows how to use the AsyncResult without blocking on a wait call. vehicle->releaseControl(); if (vehicle->loiter().then([=](bool rc) { printf(\"loiter command %s\\n\", rc ? \"succeeded\" : \"failed\"); } The following code gets all configurable parameters from the drone and prints them: auto list = vehicle->getParamList(); auto end = list.end(); int count = 0; for (auto iter = list.begin(); iter < end; iter++) { count++; MavLinkParameter p = *iter; if (p.type == MAV_PARAM_TYPE_REAL32 || p.type == MAV_PARAM_TYPE_REAL64) { printf(\"%s=%f\\n\", p.name.c_str(), p.value); } else { printf(\"%s=%d\\n\", p.name.c_str(), static_cast(p.value)); } } The following code sets a parameter on the Pixhawk to disable the USB safety check (this is handy if you are controlling the Pixhawk over USB using another onboard computer that is part of the drone itself). You should NOT do this if you are connecting your PC or laptop to the drone over USB. MavLinkParameter p; p.name = \"CBRK_USB_CHK\"; p.value = 197848; if (!vehicle->setParameter(p).wait(3000,&rc) || !rc) { printf(\"Setting the CBRK_USB_CHK failed\"); } MavLinkVehicle actually has a helper method for this called allowFlightControlOverUsb, so now you know how it is implemented :-)","title":"Examples"},{"location":"mavlinkcom/#advanced-connections","text":"You can wire up different configurations of mavlink pipelines using the MavLinkConnection class \"join\" method as shown below. Example 1, we connect to PX4 over serial, and proxy those messages through to QGroundControl and the LogViewer who are listening on remote ports. Example 2: simulation can talk to jMavSim and jMavSim connects to PX4. jMavSim can also manage multiple connections, so it can talk to unreal simulator. Another MavLinkConnection can be joined to proxy connections that jMavSim doesn't support, like the LogViewer or a remote camera node. Example 3: we use MavLinkConnection to connect to PX4 over serial, then join additional connections for all our remote nodes including jMavSim. Example 4: We can also do distributed systems to control the drone remotely:","title":"Advanced Connections"},{"location":"mavlinkcom_mocap/","text":"Welcome to MavLinkMoCap # This folder contains the MavLinkMoCap library which connects to a OptiTrack camera system for accurate indoor location. Dependencies: # OptiTrack Motive . MavLinkCom . Setup RigidBody # First you need to define a RigidBody named 'Quadrocopter' using Motive. See Rigid_Body_Tracking . MavLinkTest # Use MavLinkTest to talk to your PX4 drone, with \"-server:addr:port\", for example, when connected to drone wifi use: MavLinkMoCap -server:10.42.0.228:14590 \"-project:D:\\OptiTrack\\Motive Project 2016-12-19 04.09.42 PM.ttp\" This publishes the ATT_POS_MOCAP messages and you can proxy those through to the PX4 by running MavLinkTest on the dronebrain using: MavLinkTest -serial:/dev/ttyACM0,115200 -proxy:10.42.0.228:14590 Now the drone will get the ATT_POS_MOCAP and you should see the light turn green meaning it is now has a home position and is ready to fly.","title":"MavLink MoCap"},{"location":"mavlinkcom_mocap/#welcome-to-mavlinkmocap","text":"This folder contains the MavLinkMoCap library which connects to a OptiTrack camera system for accurate indoor location.","title":"Welcome to MavLinkMoCap"},{"location":"mavlinkcom_mocap/#dependencies","text":"OptiTrack Motive . MavLinkCom .","title":"Dependencies:"},{"location":"mavlinkcom_mocap/#setup-rigidbody","text":"First you need to define a RigidBody named 'Quadrocopter' using Motive. See Rigid_Body_Tracking .","title":"Setup RigidBody"},{"location":"mavlinkcom_mocap/#mavlinktest","text":"Use MavLinkTest to talk to your PX4 drone, with \"-server:addr:port\", for example, when connected to drone wifi use: MavLinkMoCap -server:10.42.0.228:14590 \"-project:D:\\OptiTrack\\Motive Project 2016-12-19 04.09.42 PM.ttp\" This publishes the ATT_POS_MOCAP messages and you can proxy those through to the PX4 by running MavLinkTest on the dronebrain using: MavLinkTest -serial:/dev/ttyACM0,115200 -proxy:10.42.0.228:14590 Now the drone will get the ATT_POS_MOCAP and you should see the light turn green meaning it is now has a home position and is ready to fly.","title":"MavLinkTest"},{"location":"meshes/","text":"How to Access Meshes in AIRSIM # AirSim supports the ability to access the static meshes that make up the scene Mesh structure # Each mesh is represented with the below struct. struct MeshPositionVertexBuffersResponse { Vector3r position; Quaternionr orientation; std::vector vertices; std::vector indices; std::string name; }; The position and orientation are in the Unreal coordinate system. The mesh itself is a triangular mesh represented by the vertices and the indices. The triangular mesh type is typically called a Face-Vertex Mesh. This means every triplet of indices hold the indexes of the vertices that make up the triangle/face. The x,y,z coordinates of the vertices are all stored in a single vector. This means the vertices vertices is Nx3 where N is number of vertices. The position of the vertices are the global positions in the Unreal coordinate system. This means they have already been Transformed by the position and orientation. How to use # The API to get the meshes in the scene is quite simple. However, one should note that the function call is very expensive and should very rarely be called. In general this is ok because this function only accesses the static meshes which for most applications are not changing during the duration of your program. Note that you will have to use a 3rdparty library or your own custom code to actually interact with the recieved meshes. Below I utilize the python bindings of libigl to visualize the recieved meshes. import airsim AIRSIM_HOST_IP='127.0.0.1' client = airsim.VehicleClient(ip=AIRSIM_HOST_IP) client.confirmConnection() # List of returned meshes are received via this function meshes=client.simGetMeshPositionVertexBuffers() index=0 for m in meshes: # Finds one of the cube meshes in the Blocks environment if 'cube' in m.name: # Code from here on relies on libigl. Libigl uses pybind11 to wrap C++ code. So here the built pyigl.so # library is in the same directory as this example code. # This is here as code for your own mesh library should require something similar from pyigl import * from iglhelpers import * # Convert the lists to numpy arrays vertex_list=np.array(m.vertices,dtype=np.float32) indices=np.array(m.indices,dtype=np.uint32) num_vertices=int(len(vertex_list)/3) num_indices=len(indices) # Libigl requires the shape to be Nx3 where N is number of vertices or indices # It also requires the actual type to be double(float64) for vertices and int64 for the triangles/indices vertices_reshaped=vertex_list.reshape((num_vertices,3)) indices_reshaped=indices.reshape((int(num_indices/3),3)) vertices_reshaped=vertices_reshaped.astype(np.float64) indices_reshaped=indices_reshaped.astype(np.int64) #Libigl function to convert to internal Eigen format v_eig=p2e(vertices_reshaped) i_eig=p2e(indices_reshaped) # View the mesh viewer = igl.glfw.Viewer() viewer.data().set_mesh(v_eig,i_eig) viewer.launch() break","title":"Mesh Vertex Buffers"},{"location":"meshes/#how-to-access-meshes-in-airsim","text":"AirSim supports the ability to access the static meshes that make up the scene","title":"How to Access Meshes in AIRSIM"},{"location":"meshes/#mesh-structure","text":"Each mesh is represented with the below struct. struct MeshPositionVertexBuffersResponse { Vector3r position; Quaternionr orientation; std::vector vertices; std::vector indices; std::string name; }; The position and orientation are in the Unreal coordinate system. The mesh itself is a triangular mesh represented by the vertices and the indices. The triangular mesh type is typically called a Face-Vertex Mesh. This means every triplet of indices hold the indexes of the vertices that make up the triangle/face. The x,y,z coordinates of the vertices are all stored in a single vector. This means the vertices vertices is Nx3 where N is number of vertices. The position of the vertices are the global positions in the Unreal coordinate system. This means they have already been Transformed by the position and orientation.","title":"Mesh structure"},{"location":"meshes/#how-to-use","text":"The API to get the meshes in the scene is quite simple. However, one should note that the function call is very expensive and should very rarely be called. In general this is ok because this function only accesses the static meshes which for most applications are not changing during the duration of your program. Note that you will have to use a 3rdparty library or your own custom code to actually interact with the recieved meshes. Below I utilize the python bindings of libigl to visualize the recieved meshes. import airsim AIRSIM_HOST_IP='127.0.0.1' client = airsim.VehicleClient(ip=AIRSIM_HOST_IP) client.confirmConnection() # List of returned meshes are received via this function meshes=client.simGetMeshPositionVertexBuffers() index=0 for m in meshes: # Finds one of the cube meshes in the Blocks environment if 'cube' in m.name: # Code from here on relies on libigl. Libigl uses pybind11 to wrap C++ code. So here the built pyigl.so # library is in the same directory as this example code. # This is here as code for your own mesh library should require something similar from pyigl import * from iglhelpers import * # Convert the lists to numpy arrays vertex_list=np.array(m.vertices,dtype=np.float32) indices=np.array(m.indices,dtype=np.uint32) num_vertices=int(len(vertex_list)/3) num_indices=len(indices) # Libigl requires the shape to be Nx3 where N is number of vertices or indices # It also requires the actual type to be double(float64) for vertices and int64 for the triangles/indices vertices_reshaped=vertex_list.reshape((num_vertices,3)) indices_reshaped=indices.reshape((int(num_indices/3),3)) vertices_reshaped=vertices_reshaped.astype(np.float64) indices_reshaped=indices_reshaped.astype(np.int64) #Libigl function to convert to internal Eigen format v_eig=p2e(vertices_reshaped) i_eig=p2e(indices_reshaped) # View the mesh viewer = igl.glfw.Viewer() viewer.data().set_mesh(v_eig,i_eig) viewer.launch() break","title":"How to use"},{"location":"multi_vehicle/","text":"Multiple Vehicles in AirSim # Since release 1.2, AirSim is fully enabled for multiple vehicles. This capability allows you to create multiple vehicles easily and use APIs to control them. Creating Multiple Vehicles # It's as easy as specifying them in settings.json . The Vehicles element allows you to specify list of vehicles you want to create along with their initial positions and orientations. The positions are specified in NED coordinates in SI units with origin set at Player Start component in Unreal environment. The orientation is specified as Yaw, Pitch and Roll in degrees. Creating Multiple Cars # { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\", \"Vehicles\": { \"Car1\": { \"VehicleType\": \"PhysXCar\", \"X\": 4, \"Y\": 0, \"Z\": -2 }, \"Car2\": { \"VehicleType\": \"PhysXCar\", \"X\": -4, \"Y\": 0, \"Z\": -2, \"Yaw\": 90 } } } Creating Multiple Drones # { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"X\": 4, \"Y\": 0, \"Z\": -2, \"Yaw\": -180 }, \"Drone2\": { \"VehicleType\": \"SimpleFlight\", \"X\": 8, \"Y\": 0, \"Z\": -2 } } } Using APIs for Multiple Vehicles # The new APIs since AirSim 1.2 allows you to specify vehicle_name . This name corresponds to keys in json settings (for example, Car1 or Drone2 above). Example code for cars Example code for multirotors Demo #","title":"Multiple Vehicles"},{"location":"multi_vehicle/#multiple-vehicles-in-airsim","text":"Since release 1.2, AirSim is fully enabled for multiple vehicles. This capability allows you to create multiple vehicles easily and use APIs to control them.","title":"Multiple Vehicles in AirSim"},{"location":"multi_vehicle/#creating-multiple-vehicles","text":"It's as easy as specifying them in settings.json . The Vehicles element allows you to specify list of vehicles you want to create along with their initial positions and orientations. The positions are specified in NED coordinates in SI units with origin set at Player Start component in Unreal environment. The orientation is specified as Yaw, Pitch and Roll in degrees.","title":"Creating Multiple Vehicles"},{"location":"multi_vehicle/#creating-multiple-cars","text":"{ \"SettingsVersion\": 1.2, \"SimMode\": \"Car\", \"Vehicles\": { \"Car1\": { \"VehicleType\": \"PhysXCar\", \"X\": 4, \"Y\": 0, \"Z\": -2 }, \"Car2\": { \"VehicleType\": \"PhysXCar\", \"X\": -4, \"Y\": 0, \"Z\": -2, \"Yaw\": 90 } } }","title":"Creating Multiple Cars"},{"location":"multi_vehicle/#creating-multiple-drones","text":"{ \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"X\": 4, \"Y\": 0, \"Z\": -2, \"Yaw\": -180 }, \"Drone2\": { \"VehicleType\": \"SimpleFlight\", \"X\": 8, \"Y\": 0, \"Z\": -2 } } }","title":"Creating Multiple Drones"},{"location":"multi_vehicle/#using-apis-for-multiple-vehicles","text":"The new APIs since AirSim 1.2 allows you to specify vehicle_name . This name corresponds to keys in json settings (for example, Car1 or Drone2 above). Example code for cars Example code for multirotors","title":"Using APIs for Multiple Vehicles"},{"location":"multi_vehicle/#demo","text":"","title":"Demo"},{"location":"orbit/","text":"An Orbit Trajectory # Moved here from https://github.com/microsoft/AirSim/wiki/An-Orbit-Trajectory Have you ever wanted to fly a nice smooth circular orbit? This can be handy for capturing 3D objects from all sides especially if you get multiple orbits at different altitudes. So the PythonClient/multirotor folder contains a script named Orbit that will do precisely that. See demo video The demo video was created by running this command line: python orbit.py --radius 10 --altitude 5 --speed 1 --center \"0,1\" --iterations 1 This flies a 10 meter radius orbit around the center location at (startpos + radius * [0,1]), in other words, the center is located radius meters away in the direction of the provided center vector. It also keeps the front-facing camera on the drone always pointing at the center of the circle. If you watch the flight using LogViewer you will see a nice circular pattern get traced out on the GPS map: The core of the algorithm is not that complicated. At each point on the circle, we look ahead by a small delta in degrees, called the lookahead_angle , where that angle is computed based on our desired velocity. We then find that lookahead point on the circle using sin/cosine and make that our \"target point\". Calculating the velocity then is easy, just subtract our current position from that point and feed that into the AirSim method moveByVelocityZ .","title":"Orbit Trajectory"},{"location":"orbit/#an-orbit-trajectory","text":"Moved here from https://github.com/microsoft/AirSim/wiki/An-Orbit-Trajectory Have you ever wanted to fly a nice smooth circular orbit? This can be handy for capturing 3D objects from all sides especially if you get multiple orbits at different altitudes. So the PythonClient/multirotor folder contains a script named Orbit that will do precisely that. See demo video The demo video was created by running this command line: python orbit.py --radius 10 --altitude 5 --speed 1 --center \"0,1\" --iterations 1 This flies a 10 meter radius orbit around the center location at (startpos + radius * [0,1]), in other words, the center is located radius meters away in the direction of the provided center vector. It also keeps the front-facing camera on the drone always pointing at the center of the circle. If you watch the flight using LogViewer you will see a nice circular pattern get traced out on the GPS map: The core of the algorithm is not that complicated. At each point on the circle, we look ahead by a small delta in degrees, called the lookahead_angle , where that angle is computed based on our desired velocity. We then find that lookahead point on the circle using sin/cosine and make that our \"target point\". Calculating the velocity then is easy, just subtract our current position from that point and feed that into the AirSim method moveByVelocityZ .","title":"An Orbit Trajectory"},{"location":"pfm/","text":"pfm Format # Pfm (or Portable FloatMap) image format stores image as floating point pixels and hence are not restricted to usual 0-255 pixel value range. This is useful for HDR images or images that describes something other than colors like depth. One of the good viewer to view this file format is PfmPad . We don't recommend Maverick photo viewer because it doesn't seem to show depth images properly. AirSim has code to write pfm file for C++ and read as well as write for Python .","title":"pfm format"},{"location":"pfm/#pfm-format","text":"Pfm (or Portable FloatMap) image format stores image as floating point pixels and hence are not restricted to usual 0-255 pixel value range. This is useful for HDR images or images that describes something other than colors like depth. One of the good viewer to view this file format is PfmPad . We don't recommend Maverick photo viewer because it doesn't seem to show depth images properly. AirSim has code to write pfm file for C++ and read as well as write for Python .","title":"pfm Format"},{"location":"playback/","text":"Playback # AirSim supports playing back the high level commands in a *.mavlink log file that were recorded using the MavLinkTest app for the purpose of comparing real and simulated flight. The recording.mavlink is an example of a log file captured using a real drone using the following command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. Then the log file contains the commands performed, which included several \"orbit\" commands, the resulting GPS map of the flight looks like this: Side-by-side comparison # Now we can copy the *.mavlink log file recorded by MavLinkTest to the PC running the Unreal simulator with AirSim plugin. When the Simulator is running and the drone is parked in a place in a map that has room to do the same maneuvers we can run this MavLinkTest command line: MavLinkTest -server:127.0.0.1:14550 This should connect to the simulator. Now you can enter this command: PlayLog recording.mavlink The same commands you performed on the real drone will now play again in the simulator. You can then press 't' to see the trace, and it will show you the trace of the real drone and the simulated drone. Every time you press 't' again you can reset the lines so they are sync'd to the current position, this way I was able to capture a side-by-side trace of the \"orbit\" command performed in this recording, which generates the picture below. The pink line is the simulated flight and the red line is the real flight: Note: I'm using the ';' key in the simulator to take control of camera position using keyboard to get this shot. Parameters # It may help to set the simulator up with some of the same flight parameters that your real drone is using, for example, in my case I was using a lower than normal cruise speed, slow takeoff speed, and it helps to tell the simulator to wait a long time before disarming (COM_DISARM_LAND) and to turn off the safety switches NAV_RCL_ACT and NAV_DLL_ACT ( don't do that on a real drone). param MPC_XY_CRUISE 2 param MPC_XY_VEL_MAX 2 param MPC_TKO_SPEED 1 param COM_DISARM_LAND 60 param NAV_RCL_ACT 0 param NAV_DLL_ACT 0","title":"Playing Logs"},{"location":"playback/#playback","text":"AirSim supports playing back the high level commands in a *.mavlink log file that were recorded using the MavLinkTest app for the purpose of comparing real and simulated flight. The recording.mavlink is an example of a log file captured using a real drone using the following command line: MavLinkTest -serial:/dev/ttyACM0,115200 -logdir:. Then the log file contains the commands performed, which included several \"orbit\" commands, the resulting GPS map of the flight looks like this:","title":"Playback"},{"location":"playback/#side-by-side-comparison","text":"Now we can copy the *.mavlink log file recorded by MavLinkTest to the PC running the Unreal simulator with AirSim plugin. When the Simulator is running and the drone is parked in a place in a map that has room to do the same maneuvers we can run this MavLinkTest command line: MavLinkTest -server:127.0.0.1:14550 This should connect to the simulator. Now you can enter this command: PlayLog recording.mavlink The same commands you performed on the real drone will now play again in the simulator. You can then press 't' to see the trace, and it will show you the trace of the real drone and the simulated drone. Every time you press 't' again you can reset the lines so they are sync'd to the current position, this way I was able to capture a side-by-side trace of the \"orbit\" command performed in this recording, which generates the picture below. The pink line is the simulated flight and the red line is the real flight: Note: I'm using the ';' key in the simulator to take control of camera position using keyboard to get this shot.","title":"Side-by-side comparison"},{"location":"playback/#parameters","text":"It may help to set the simulator up with some of the same flight parameters that your real drone is using, for example, in my case I was using a lower than normal cruise speed, slow takeoff speed, and it helps to tell the simulator to wait a long time before disarming (COM_DISARM_LAND) and to turn off the safety switches NAV_RCL_ACT and NAV_DLL_ACT ( don't do that on a real drone). param MPC_XY_CRUISE 2 param MPC_XY_VEL_MAX 2 param MPC_TKO_SPEED 1 param COM_DISARM_LAND 60 param NAV_RCL_ACT 0 param NAV_DLL_ACT 0","title":"Parameters"},{"location":"point_clouds/","text":"Point Clouds # Moved here from https://github.com/microsoft/AirSim/wiki/Point-Clouds A Python script point_cloud.py shows how to convert the depth image returned from AirSim into a point cloud. The following depth image was captured using the Modular Neighborhood environment: And with the appropriate projection matrix, the OpenCV reprojectImageTo3D function can turn this into a point cloud. The following is the result, which is also available here: https://skfb.ly/68r7y . SketchFab can upload the resulting file cloud.asc and render it for you. PS: you may notice the scene is reflected on the Y axis, so I may have a sign wrong in the projection matrix. An exercise for the reader :-)","title":"Building Point Clouds"},{"location":"point_clouds/#point-clouds","text":"Moved here from https://github.com/microsoft/AirSim/wiki/Point-Clouds A Python script point_cloud.py shows how to convert the depth image returned from AirSim into a point cloud. The following depth image was captured using the Modular Neighborhood environment: And with the appropriate projection matrix, the OpenCV reprojectImageTo3D function can turn this into a point cloud. The following is the result, which is also available here: https://skfb.ly/68r7y . SketchFab can upload the resulting file cloud.asc and render it for you. PS: you may notice the scene is reflected on the Y axis, so I may have a sign wrong in the projection matrix. An exercise for the reader :-)","title":"Point Clouds"},{"location":"px4_build/","text":"Building PX4 # Source code # Getting the PX4 source code is easy: git clone https://github.com/PX4/Firmware.git cd Firmware Oh, and if you don't have git yet just run this: sudo apt-get install git We are currently testing using the 1.6.0rc1 version, but the latest master branch should be ok too. Now to build it you will need the right tools. PX4 Build tools # The full instructions are available on the dev.px4.io website, but we've copied the relevant subset of those instructions here for your convenience. (Note that BashOnWindows ) can be used to build the PX4 firmware, just follow the BashOnWindows instructions at the bottom of this page). Build SITL version # Now you can make the SITL version that runs in posix, from the Firmware folder you created above: make posix_sitl_ekf2 Note: this build system is quite special, it knows how to update git submodules (and there's a lot of them), then it runs cmake (if necessary), then it runs the build itself. So in a way the root Makefile is a meta-meta makefile :-) It shouldn't take long, about 2 minutes. If all succeeds, the last line will link the px4 app, which you can then run using the following: make posix_sitl_ekf2 none_iris And you should see output that looks like this: creating new parameters file creating new dataman file ______ __ __ ___ | ___ \\ \\ \\ / / / | | |_/ / \\ V / / /| | | __/ / \\ / /_| | | | / /^\\ \\ \\___ | \\_| \\/ \\/ |_/ px4 starting. 18446744073709551615 WARNING: setRealtimeSched failed (not run as root?) ERROR [param] importing from 'rootfs/eeprom/parameters' failed (-1) Command 'param' failed, returned 1 SYS_AUTOSTART: curr: 0 -> new: 4010 SYS_MC_EST_GROUP: curr: 2 -> new: 1 INFO [dataman] Unkown restart, data manager file 'rootfs/fs/microsd/dataman' size is 11797680 bytes BAT_N_CELLS: curr: 0 -> new: 3 CAL_GYRO0_ID: curr: 0 -> new: 2293768 CAL_ACC0_ID: curr: 0 -> new: 1376264 CAL_ACC1_ID: curr: 0 -> new: 1310728 CAL_MAG0_ID: curr: 0 -> new: 196616 so this is good, first run sets up the px4 parameters for SITL mode. Second run has less output. This app is also an interactive console where you can type commands. Type 'help' to see what they are and just type ctrl-C to kill it. You can do that and restart it any time, that's a great way to reset any wonky state if you need to (it's equivalent to a Pixhawk hardware reboot). ARM embedded tools # If you plan to build the PX4 firmware for real Pixhawk hardware then you will need the gcc cross-compiler for ARM Cortex-M4 chipset. You can get this compiler by PX4 DevGuide, specifically this is in their ubuntu_sim_nuttx.sh setup script. After following those setup instructions you can verify the install by entering this command arm-none-eabi-gcc --version . You should see the following output: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204] Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Build PX4 for ARM hardware # Now you can build the PX4 firmware for running on real pixhawk hardware: make px4fmu-v2_default This build will take a little longer because it is building a lot more including the NuttX real time OS, all the drivers for the sensors in the Pixhawk flight controller, and more. It is also running the compiler in super size-squeezing mode so it can fit all that in a 1 megabyte ROM !! One nice tid bit is you can plug in your pixhawk USB, and type make px4fmu-v2_default upload to flash the hardware with these brand new bits, so you don't need to use QGroundControl for that. Some Useful Parameters # PX4 has many customizable parameters (over 700 of them, in fact) and to get best results with AirSim we have found the following parameters are handy: // be sure to enable the new position estimator module: param set SYS_MC_EST_GROUP 2 // increase default limits on cruise speed so you can move around a large map more quickly. param MPC_XY_CRUISE 10 param MPC_XY_VEL_MAX 10 param MPC_Z_VEL_MAX_DN 2 // increase timeout for auto-disarm on landing so that any long running app doesn't have to worry about it param COM_DISARM_LAND 60 // make it possible to fly without radio control attached (do NOT do this one on a real drone) param NAV_RCL_ACT 0 // enable new syslogger to get more information from PX4 logs param set SYS_LOGGER 1 Using BashOnWindows # See Bash on Windows Toolchain .","title":"Building PX4"},{"location":"px4_build/#building-px4","text":"","title":"Building PX4"},{"location":"px4_build/#source-code","text":"Getting the PX4 source code is easy: git clone https://github.com/PX4/Firmware.git cd Firmware Oh, and if you don't have git yet just run this: sudo apt-get install git We are currently testing using the 1.6.0rc1 version, but the latest master branch should be ok too. Now to build it you will need the right tools.","title":"Source code"},{"location":"px4_build/#px4-build-tools","text":"The full instructions are available on the dev.px4.io website, but we've copied the relevant subset of those instructions here for your convenience. (Note that BashOnWindows ) can be used to build the PX4 firmware, just follow the BashOnWindows instructions at the bottom of this page).","title":"PX4 Build tools"},{"location":"px4_build/#build-sitl-version","text":"Now you can make the SITL version that runs in posix, from the Firmware folder you created above: make posix_sitl_ekf2 Note: this build system is quite special, it knows how to update git submodules (and there's a lot of them), then it runs cmake (if necessary), then it runs the build itself. So in a way the root Makefile is a meta-meta makefile :-) It shouldn't take long, about 2 minutes. If all succeeds, the last line will link the px4 app, which you can then run using the following: make posix_sitl_ekf2 none_iris And you should see output that looks like this: creating new parameters file creating new dataman file ______ __ __ ___ | ___ \\ \\ \\ / / / | | |_/ / \\ V / / /| | | __/ / \\ / /_| | | | / /^\\ \\ \\___ | \\_| \\/ \\/ |_/ px4 starting. 18446744073709551615 WARNING: setRealtimeSched failed (not run as root?) ERROR [param] importing from 'rootfs/eeprom/parameters' failed (-1) Command 'param' failed, returned 1 SYS_AUTOSTART: curr: 0 -> new: 4010 SYS_MC_EST_GROUP: curr: 2 -> new: 1 INFO [dataman] Unkown restart, data manager file 'rootfs/fs/microsd/dataman' size is 11797680 bytes BAT_N_CELLS: curr: 0 -> new: 3 CAL_GYRO0_ID: curr: 0 -> new: 2293768 CAL_ACC0_ID: curr: 0 -> new: 1376264 CAL_ACC1_ID: curr: 0 -> new: 1310728 CAL_MAG0_ID: curr: 0 -> new: 196616 so this is good, first run sets up the px4 parameters for SITL mode. Second run has less output. This app is also an interactive console where you can type commands. Type 'help' to see what they are and just type ctrl-C to kill it. You can do that and restart it any time, that's a great way to reset any wonky state if you need to (it's equivalent to a Pixhawk hardware reboot).","title":"Build SITL version"},{"location":"px4_build/#arm-embedded-tools","text":"If you plan to build the PX4 firmware for real Pixhawk hardware then you will need the gcc cross-compiler for ARM Cortex-M4 chipset. You can get this compiler by PX4 DevGuide, specifically this is in their ubuntu_sim_nuttx.sh setup script. After following those setup instructions you can verify the install by entering this command arm-none-eabi-gcc --version . You should see the following output: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204] Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.","title":"ARM embedded tools"},{"location":"px4_build/#build-px4-for-arm-hardware","text":"Now you can build the PX4 firmware for running on real pixhawk hardware: make px4fmu-v2_default This build will take a little longer because it is building a lot more including the NuttX real time OS, all the drivers for the sensors in the Pixhawk flight controller, and more. It is also running the compiler in super size-squeezing mode so it can fit all that in a 1 megabyte ROM !! One nice tid bit is you can plug in your pixhawk USB, and type make px4fmu-v2_default upload to flash the hardware with these brand new bits, so you don't need to use QGroundControl for that.","title":"Build PX4 for ARM hardware"},{"location":"px4_build/#some-useful-parameters","text":"PX4 has many customizable parameters (over 700 of them, in fact) and to get best results with AirSim we have found the following parameters are handy: // be sure to enable the new position estimator module: param set SYS_MC_EST_GROUP 2 // increase default limits on cruise speed so you can move around a large map more quickly. param MPC_XY_CRUISE 10 param MPC_XY_VEL_MAX 10 param MPC_Z_VEL_MAX_DN 2 // increase timeout for auto-disarm on landing so that any long running app doesn't have to worry about it param COM_DISARM_LAND 60 // make it possible to fly without radio control attached (do NOT do this one on a real drone) param NAV_RCL_ACT 0 // enable new syslogger to get more information from PX4 logs param set SYS_LOGGER 1","title":"Some Useful Parameters"},{"location":"px4_build/#using-bashonwindows","text":"See Bash on Windows Toolchain .","title":"Using BashOnWindows"},{"location":"px4_logging/","text":"PX4/MavLink Logging # Thanks to Chris Lovett for developing various tools for PX4/MavLink logging mentioned on this page! Logging MavLink Messages # The following command will connect MavLinkTest app to the Simulator and enable logging of all mavlink commands to and from the PX4. MavLinkTest -server:127.0.0.1:14550 -logdir:d:\\temp Sometimes you will also want to log the \"output\" from the Simulator that is being sent to the PX4. Specifically this will capture the \"hilgps\" and \"hilsensor\" messages that are generated by the Simulator. To do that run this as well: MavLinkTest -server:127.0.0.1:14389 -logdir:d:\\temp\\output You will then see log files organized by date in d:\\temp\\logs, specifically input.mavlink and output.mavlink files. MavLink LogViewer # For MavLink enabled drones, you can also use our Log Viewer to visualize the streams of data. PX4 Log in SITL Mode # In SITL mode, please a log file is produced when drone is armed. The SITL terminal will contain the path to the log file, it should look something like this INFO [logger] Opened log file: rootfs/fs/microsd/log/2017-03-27/20_02_49.ulg PX4 Log in SITL Mode # If you are using Pixhawk hardware in HIL mode, then set parameter SYS_LOGGER=1 using QGroundControl. PX4 will write log file on device which you can download at later date using QGroundControl.","title":"PX4/MavLink Logging"},{"location":"px4_logging/#px4mavlink-logging","text":"Thanks to Chris Lovett for developing various tools for PX4/MavLink logging mentioned on this page!","title":"PX4/MavLink Logging"},{"location":"px4_logging/#logging-mavlink-messages","text":"The following command will connect MavLinkTest app to the Simulator and enable logging of all mavlink commands to and from the PX4. MavLinkTest -server:127.0.0.1:14550 -logdir:d:\\temp Sometimes you will also want to log the \"output\" from the Simulator that is being sent to the PX4. Specifically this will capture the \"hilgps\" and \"hilsensor\" messages that are generated by the Simulator. To do that run this as well: MavLinkTest -server:127.0.0.1:14389 -logdir:d:\\temp\\output You will then see log files organized by date in d:\\temp\\logs, specifically input.mavlink and output.mavlink files.","title":"Logging MavLink Messages"},{"location":"px4_logging/#mavlink-logviewer","text":"For MavLink enabled drones, you can also use our Log Viewer to visualize the streams of data.","title":"MavLink LogViewer"},{"location":"px4_logging/#px4-log-in-sitl-mode","text":"In SITL mode, please a log file is produced when drone is armed. The SITL terminal will contain the path to the log file, it should look something like this INFO [logger] Opened log file: rootfs/fs/microsd/log/2017-03-27/20_02_49.ulg","title":"PX4 Log in SITL Mode"},{"location":"px4_logging/#px4-log-in-sitl-mode_1","text":"If you are using Pixhawk hardware in HIL mode, then set parameter SYS_LOGGER=1 using QGroundControl. PX4 will write log file on device which you can download at later date using QGroundControl.","title":"PX4 Log in SITL Mode"},{"location":"px4_setup/","text":"PX4 Setup for AirSim # The PX4 software stack is an open source very popular flight controller with support for wide variety of boards and sensors as well as built-in capability for higher level tasks such as mission planning. Please visit px4.io for more information. Warning : While all releases of AirSim are always tested with PX4 to ensure the support, setting up PX4 is not a trivial task. Unless you have at least intermediate level of experience with PX4 stack, we recommend you use simple_flight , which is now a default in AirSim. Supported Hardware # The following Pixhawk hardware has been tested with AirSim: 3DR Pixhawk v2 3DR Pixhawk mini Pixhawk PX4 2.4.8 PixFalcon PixRacer Pixhawk 2.1 (using PX4 Flight Stack) The 3DR Pixhawk Mini works out of the box, the others you may need to re-flash with the latest firmware. Setting up PX4 Hardware-in-Loop # For this you will need one of the supported device listed above. For manual flight you will also need RC + transmitter. Make sure your RC receiver is bound with its RC transmitter. Connect the RC transmitter to the flight controller's RC port. Refer to your RC manual and PX4 docs for more information. Download QGroundControl , launch it and connect your flight controller to the USB port. Use QGroundControl to flash the latest PX4 Flight Stack. See also initial firmware setup video . In QGroundControl, configure your Pixhawk for HIL simulation by selecting the HIL Quadrocopter X airframe. After PX4 reboots, check that \"HIL Quadrocopter X\" is indeed selected. In QGroundControl, go to Radio tab and calibrate (make sure the remote control is on and the receiver is showing the indicator for the binding). Go to the Flight Mode tab and chose one of the remote control switches as \"Mode Channel\". Then set (for example) Stabilized and Attitude flight modes for two positions of the switch. Go to the Tuning section of QGroundControl and set appropriate values. For example, for Fly Sky's FS-TH9X remote control, the following settings give a more realistic feel: Hover Throttle = mid+1 mark, Roll and pitch sensitivity = mid-3 mark, Altitude and position control sensitivity = mid-2 mark. In AirSim settings file, specify PX4 for your vehicle config like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\" } } } After above setup you should be able to use RC to fly in the AirSim. You can usually arm the vehicle by lowering and bringing two sticks of RC together in-wards. You don't need QGroundControl after the initial setup. Typically the Stabilized (instead of Manual) mode gives better experience for beginners. You can also control the drone from Python APIs . See Walkthrough Demo Video and Unreal AirSim Setup Video that shows you all the setup steps in this document. Setting up PX4 Software-in-Loop # The PX4 SITL mode doesn't require you to have separate device such as a Pixhawk or Pixracer. This is in fact the recommended way to use PX4 with simulators by PX4 team. However, this is indeed harder to set up. Please see this dedicated page for setting up PX4 in SITL mode. FAQ # Drone doesn't fly properly, it just goes \"crazy\". # There are a few reasons that can cause this. First, make sure your drone doesn't fall down large distance when starting the simulator. This might happen if you have created a custom Unreal environment and Player Start is placed too high above the ground. It seems that when this happens internal calibration in PX4 gets confused. You should also use QGroundControl and make sure you can arm and takeoff in QGroundControl properly. Finally, this also can be a machine performance issue in some rare cases, check your hard drive performance . Can I use Arducopter or other MavLink implementations? # Our code is tested with the PX4 firmware . We have not tested Arducopter or other mavlink implementations. Some of the flight API's do use the PX4 custom modes in the MAV_CMD_DO_SET_MODE messages (like PX4_CUSTOM_MAIN_MODE_AUTO) It is not finding my Pixhawk hardware # Check your settings.json file for this line \"SerialPort\":\"*,115200\". The asterisk here means \"find any serial port that looks like a Pixhawk device, but this doesn't always work for all types of Pixhawk hardware. So on Windows you can find the actual COM port using Device Manager, look under \"Ports (COM & LPT), plug the device in and see what new COM port shows up. Let's say you see a new port named \"USB Serial Port (COM5)\". Well, then change the SerialPort setting to this: \"SerialPort\":\"COM5,115200\". On Linux, the device can be found by running \"ls /dev/serial/by-id\" if you see a device name listed that looks like this usb-3D_Robotics_PX4_FMU_v2.x_0-if00 then you can use that name to connect, like this: \"SerialPort\":\"/dev/serial/by-id/usb-3D_Robotics_PX4_FMU_v2.x_0-if00\". Note this long name is actually a symbolic link to the real name, if you use \"ls -l ...\" you can find that symbolic link, it is usually something like \"/dev/ttyACM0\", so this will also work \"SerialPort\":\"/dev/ttyACM0,115200\". But that mapping is similar to windows, it is automatically assigned and can change, whereas the long name will work even if the actual TTY serial device mapping changes. WARN [commander] Takeoff denied, disarm and re-try # This happens if you try and take off when PX4 still has not computed the home position. PX4 will report the home position once it is happy with the GPS signal, and you will see these messages: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Up until this point in time, however, the PX4 will reject takeoff commands. When I tell the drone to do something it always lands # For example, you use DroneShell moveToPosition -z -20 -x 50 -y 0 which it does, but when it gets to the target location the drone starts to land. This is the default behavior of PX4 when offboard mode completes. To set the drone to hover instead set this PX4 parameter: param set COM_OBL_ACT 1 I get message length mismatches errors # You might need to set MAV_PROTO_VER parameter in QGC to \"Always use version 1\". Please see this issue more details.","title":"PX4 Setup for AirSim"},{"location":"px4_setup/#px4-setup-for-airsim","text":"The PX4 software stack is an open source very popular flight controller with support for wide variety of boards and sensors as well as built-in capability for higher level tasks such as mission planning. Please visit px4.io for more information. Warning : While all releases of AirSim are always tested with PX4 to ensure the support, setting up PX4 is not a trivial task. Unless you have at least intermediate level of experience with PX4 stack, we recommend you use simple_flight , which is now a default in AirSim.","title":"PX4 Setup for AirSim"},{"location":"px4_setup/#supported-hardware","text":"The following Pixhawk hardware has been tested with AirSim: 3DR Pixhawk v2 3DR Pixhawk mini Pixhawk PX4 2.4.8 PixFalcon PixRacer Pixhawk 2.1 (using PX4 Flight Stack) The 3DR Pixhawk Mini works out of the box, the others you may need to re-flash with the latest firmware.","title":"Supported Hardware"},{"location":"px4_setup/#setting-up-px4-hardware-in-loop","text":"For this you will need one of the supported device listed above. For manual flight you will also need RC + transmitter. Make sure your RC receiver is bound with its RC transmitter. Connect the RC transmitter to the flight controller's RC port. Refer to your RC manual and PX4 docs for more information. Download QGroundControl , launch it and connect your flight controller to the USB port. Use QGroundControl to flash the latest PX4 Flight Stack. See also initial firmware setup video . In QGroundControl, configure your Pixhawk for HIL simulation by selecting the HIL Quadrocopter X airframe. After PX4 reboots, check that \"HIL Quadrocopter X\" is indeed selected. In QGroundControl, go to Radio tab and calibrate (make sure the remote control is on and the receiver is showing the indicator for the binding). Go to the Flight Mode tab and chose one of the remote control switches as \"Mode Channel\". Then set (for example) Stabilized and Attitude flight modes for two positions of the switch. Go to the Tuning section of QGroundControl and set appropriate values. For example, for Fly Sky's FS-TH9X remote control, the following settings give a more realistic feel: Hover Throttle = mid+1 mark, Roll and pitch sensitivity = mid-3 mark, Altitude and position control sensitivity = mid-2 mark. In AirSim settings file, specify PX4 for your vehicle config like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\" } } } After above setup you should be able to use RC to fly in the AirSim. You can usually arm the vehicle by lowering and bringing two sticks of RC together in-wards. You don't need QGroundControl after the initial setup. Typically the Stabilized (instead of Manual) mode gives better experience for beginners. You can also control the drone from Python APIs . See Walkthrough Demo Video and Unreal AirSim Setup Video that shows you all the setup steps in this document.","title":"Setting up PX4 Hardware-in-Loop"},{"location":"px4_setup/#setting-up-px4-software-in-loop","text":"The PX4 SITL mode doesn't require you to have separate device such as a Pixhawk or Pixracer. This is in fact the recommended way to use PX4 with simulators by PX4 team. However, this is indeed harder to set up. Please see this dedicated page for setting up PX4 in SITL mode.","title":"Setting up PX4 Software-in-Loop"},{"location":"px4_setup/#faq","text":"","title":"FAQ"},{"location":"px4_setup/#drone-doesnt-fly-properly-it-just-goes-crazy","text":"There are a few reasons that can cause this. First, make sure your drone doesn't fall down large distance when starting the simulator. This might happen if you have created a custom Unreal environment and Player Start is placed too high above the ground. It seems that when this happens internal calibration in PX4 gets confused. You should also use QGroundControl and make sure you can arm and takeoff in QGroundControl properly. Finally, this also can be a machine performance issue in some rare cases, check your hard drive performance .","title":"Drone doesn't fly properly, it just goes \"crazy\"."},{"location":"px4_setup/#can-i-use-arducopter-or-other-mavlink-implementations","text":"Our code is tested with the PX4 firmware . We have not tested Arducopter or other mavlink implementations. Some of the flight API's do use the PX4 custom modes in the MAV_CMD_DO_SET_MODE messages (like PX4_CUSTOM_MAIN_MODE_AUTO)","title":"Can I use Arducopter or other MavLink implementations?"},{"location":"px4_setup/#it-is-not-finding-my-pixhawk-hardware","text":"Check your settings.json file for this line \"SerialPort\":\"*,115200\". The asterisk here means \"find any serial port that looks like a Pixhawk device, but this doesn't always work for all types of Pixhawk hardware. So on Windows you can find the actual COM port using Device Manager, look under \"Ports (COM & LPT), plug the device in and see what new COM port shows up. Let's say you see a new port named \"USB Serial Port (COM5)\". Well, then change the SerialPort setting to this: \"SerialPort\":\"COM5,115200\". On Linux, the device can be found by running \"ls /dev/serial/by-id\" if you see a device name listed that looks like this usb-3D_Robotics_PX4_FMU_v2.x_0-if00 then you can use that name to connect, like this: \"SerialPort\":\"/dev/serial/by-id/usb-3D_Robotics_PX4_FMU_v2.x_0-if00\". Note this long name is actually a symbolic link to the real name, if you use \"ls -l ...\" you can find that symbolic link, it is usually something like \"/dev/ttyACM0\", so this will also work \"SerialPort\":\"/dev/ttyACM0,115200\". But that mapping is similar to windows, it is automatically assigned and can change, whereas the long name will work even if the actual TTY serial device mapping changes.","title":"It is not finding my Pixhawk hardware"},{"location":"px4_setup/#warn-commander-takeoff-denied-disarm-and-re-try","text":"This happens if you try and take off when PX4 still has not computed the home position. PX4 will report the home position once it is happy with the GPS signal, and you will see these messages: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Up until this point in time, however, the PX4 will reject takeoff commands.","title":"WARN [commander] Takeoff denied, disarm and re-try"},{"location":"px4_setup/#when-i-tell-the-drone-to-do-something-it-always-lands","text":"For example, you use DroneShell moveToPosition -z -20 -x 50 -y 0 which it does, but when it gets to the target location the drone starts to land. This is the default behavior of PX4 when offboard mode completes. To set the drone to hover instead set this PX4 parameter: param set COM_OBL_ACT 1","title":"When I tell the drone to do something it always lands"},{"location":"px4_setup/#i-get-message-length-mismatches-errors","text":"You might need to set MAV_PROTO_VER parameter in QGC to \"Always use version 1\". Please see this issue more details.","title":"I get message length mismatches errors"},{"location":"px4_sitl/","text":"Setting up PX4 Software-in-Loop # The PX4 software provides a \"software-in-loop\" simulation (SITL) version of their stack that runs in Linux. If you are on Windows then you must use the Cygwin Toolchain as the Bash On Windows toolchain no longer works for SITL. Note that every time you stop the unreal app you have to restart the px4 app. From your bash terminal follow these steps for Linux and follow all the instructions under NuttX based hardware to install prerequisites. We've also included out own copy of the PX4 build instructions which is a bit more concise about what we need exactly. Get the PX4 source code and build the posix SITL version of PX4: mkdir -p PX4 cd PX4 git clone https://github.com/PX4/Firmware.git cd Firmware git checkout v1.10.1 # recommended version Use following command to build and start PX4 firmware in SITL mode: make px4_sitl_default none_iris If you are using older version v1.8.* use this command instead: make posix_sitl_ekf2 none_iris . You should see a message saying the SITL PX4 app is waiting for the simulator (AirSim) to connect. You will also see information about which ports are configured for mavlink connection to the PX4 app. The default ports have changed recently, so check them closely to make sure AirSim settings are correct. INFO [simulator] Waiting for simulator to connect on TCP port 4560 INFO [init] Mixer: etc/mixers/quad_w.main.mix on /dev/pwm_output0 INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 14570 remote port 14550 INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540 Note: this is also an interactive PX4 console, type help to see the list of commands you can enter here. They are mostly low level PX4 commands, but some of them can be useful for debugging. Now edit AirSim settings file to make sure you have matching UDP and TCP port settings: json { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"UseSerial\": false, \"UseTcp\": true, \"TcpPort\": 4560, \"ControlPort\": 14580, \"params\": { \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, \"COM_OBL_ACT\": 1 } } } } Notice the PX4 [simulator] is using TCP, which is why we need to add: \"UseTcp\": true, . Now run your Unreal AirSim environment and it should connect to SITL PX4 via TCP. You should see a bunch of messages from the SITL PX4 window. Specifically, the following messages tell you that AirSim is connected properly and GPS fusion is stable: INFO [simulator] Simulator connected on UDP port 14560 INFO [mavlink] partner IP: 127.0.0.1 INFO [ecl/EKF] EKF GPS checks passed (WGS-84 origin set) INFO [ecl/EKF] EKF commencing GPS fusion If you do not see these messages then check your port settings. You should also be able to use QGroundControl with SITL mode. Make sure there is no Pixhawk hardware plugged in, otherwise QGroundControl will choose to use that instead. Note that as we don't have a physical board, an RC cannot be connected directly to it. So the alternatives are either use XBox 360 Controller or connect your RC using USB (for example, in case of FrSky Taranis X9D Plus) or using trainer USB cable to your PC. This makes your RC look like a joystick. You will need to do extra set up in QGroundControl to use virtual joystick for RC control. You do not need to do this unless you plan to fly a drone manually in AirSim. Autonomous flight using the Python API does not require RC, see No Remote Control below. Setting GPS origin # Notice the above settings are provided in the params section of the settings.json file: \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, PX4 SITL mode needs to be configured to get the home location correct. There is a bug in AirSim that makes it such that flight does not work unless the home location is set to the same coordinates defined in AVehiclePawnBase::HomeLatitude and HomeLongitude. You can also run the following in the SITL PX4 console window to check that these values are set correctly. param show LPE_LAT param show LPE_LON Smooth Offboard Transitions # Notice the above setting is provided in the params section of the settings.json file: \"COM_OBL_ACT\": 1 This tells the drone automatically hover after each offboard control command finishes (the default setting is to land). Hovering is a smoother transition between multiple offboard commands. You can check this setting by running the following PX4 console command: param show COM_OBL_ACT Check the Home Position # If you are using DroneShell to execute commands (arm, takeoff, etc) then you should wait until the Home position is set. You will see the PX4 SITL console output this message: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Now DroneShell 'pos' command should report this position and the commands should be accepted by PX4. If you attempt to takeoff without a home position you will see the message: WARN [commander] Takeoff denied, disarm and re-try After home position is set check the local position reported by 'pos' command : Local position: x=-0.0326988, y=0.00656854, z=5.48506 If the z coordinate is large like this then takeoff might not work as expected. Resetting the SITL and simulation should fix that problem. No Remote Control # Notice the above setting is provided in the params section of the settings.json file: \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, This is required if you plan to fly the SITL mode PX4 with no remote control, just using python scripts, for example. These parameters stop the PX4 from triggering \"failsafe mode on\" every time a move command is finished. You can use the following PX4 command to check these values are set correctly: param show NAV_RCL_ACT param show NAV_DLL_ACT NOTE: Do NOT do this on a real drone as it is too dangerous to fly without these failsafe measures. Manually set parameters # You can also run the following in the PX4 console to set all these parameters: param set LPE_LAT 47.641468 param set LPE_LON -122.140165 param set COM_OBL_ACT 1 param set NAV_RCL_ACT 0 param set NAV_DLL_ACT 0 Using VirtualBox Ubuntu # If you want to run the above posix_sitl in a VirtualBox Ubuntu machine then it will have a different ip address from localhost. So in this case you need to edit the settings file and change the UdpIp and SitlIp to the ip address of your virtual machine set the LocalIpAddress to the address of your host machine running the Unreal engine. Remote Controller # There are several options for flying the simulated drone using a remote control or joystick like xbox gamepad. See remote controllers","title":"PX4 in SITL"},{"location":"px4_sitl/#setting-up-px4-software-in-loop","text":"The PX4 software provides a \"software-in-loop\" simulation (SITL) version of their stack that runs in Linux. If you are on Windows then you must use the Cygwin Toolchain as the Bash On Windows toolchain no longer works for SITL. Note that every time you stop the unreal app you have to restart the px4 app. From your bash terminal follow these steps for Linux and follow all the instructions under NuttX based hardware to install prerequisites. We've also included out own copy of the PX4 build instructions which is a bit more concise about what we need exactly. Get the PX4 source code and build the posix SITL version of PX4: mkdir -p PX4 cd PX4 git clone https://github.com/PX4/Firmware.git cd Firmware git checkout v1.10.1 # recommended version Use following command to build and start PX4 firmware in SITL mode: make px4_sitl_default none_iris If you are using older version v1.8.* use this command instead: make posix_sitl_ekf2 none_iris . You should see a message saying the SITL PX4 app is waiting for the simulator (AirSim) to connect. You will also see information about which ports are configured for mavlink connection to the PX4 app. The default ports have changed recently, so check them closely to make sure AirSim settings are correct. INFO [simulator] Waiting for simulator to connect on TCP port 4560 INFO [init] Mixer: etc/mixers/quad_w.main.mix on /dev/pwm_output0 INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 14570 remote port 14550 INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540 Note: this is also an interactive PX4 console, type help to see the list of commands you can enter here. They are mostly low level PX4 commands, but some of them can be useful for debugging. Now edit AirSim settings file to make sure you have matching UDP and TCP port settings: json { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"UseSerial\": false, \"UseTcp\": true, \"TcpPort\": 4560, \"ControlPort\": 14580, \"params\": { \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, \"COM_OBL_ACT\": 1 } } } } Notice the PX4 [simulator] is using TCP, which is why we need to add: \"UseTcp\": true, . Now run your Unreal AirSim environment and it should connect to SITL PX4 via TCP. You should see a bunch of messages from the SITL PX4 window. Specifically, the following messages tell you that AirSim is connected properly and GPS fusion is stable: INFO [simulator] Simulator connected on UDP port 14560 INFO [mavlink] partner IP: 127.0.0.1 INFO [ecl/EKF] EKF GPS checks passed (WGS-84 origin set) INFO [ecl/EKF] EKF commencing GPS fusion If you do not see these messages then check your port settings. You should also be able to use QGroundControl with SITL mode. Make sure there is no Pixhawk hardware plugged in, otherwise QGroundControl will choose to use that instead. Note that as we don't have a physical board, an RC cannot be connected directly to it. So the alternatives are either use XBox 360 Controller or connect your RC using USB (for example, in case of FrSky Taranis X9D Plus) or using trainer USB cable to your PC. This makes your RC look like a joystick. You will need to do extra set up in QGroundControl to use virtual joystick for RC control. You do not need to do this unless you plan to fly a drone manually in AirSim. Autonomous flight using the Python API does not require RC, see No Remote Control below.","title":"Setting up PX4 Software-in-Loop"},{"location":"px4_sitl/#setting-gps-origin","text":"Notice the above settings are provided in the params section of the settings.json file: \"LPE_LAT\": 47.641468, \"LPE_LON\": -122.140165, PX4 SITL mode needs to be configured to get the home location correct. There is a bug in AirSim that makes it such that flight does not work unless the home location is set to the same coordinates defined in AVehiclePawnBase::HomeLatitude and HomeLongitude. You can also run the following in the SITL PX4 console window to check that these values are set correctly. param show LPE_LAT param show LPE_LON","title":"Setting GPS origin"},{"location":"px4_sitl/#smooth-offboard-transitions","text":"Notice the above setting is provided in the params section of the settings.json file: \"COM_OBL_ACT\": 1 This tells the drone automatically hover after each offboard control command finishes (the default setting is to land). Hovering is a smoother transition between multiple offboard commands. You can check this setting by running the following PX4 console command: param show COM_OBL_ACT","title":"Smooth Offboard Transitions"},{"location":"px4_sitl/#check-the-home-position","text":"If you are using DroneShell to execute commands (arm, takeoff, etc) then you should wait until the Home position is set. You will see the PX4 SITL console output this message: INFO [commander] home: 47.6414680, -122.1401672, 119.99 INFO [tone_alarm] home_set Now DroneShell 'pos' command should report this position and the commands should be accepted by PX4. If you attempt to takeoff without a home position you will see the message: WARN [commander] Takeoff denied, disarm and re-try After home position is set check the local position reported by 'pos' command : Local position: x=-0.0326988, y=0.00656854, z=5.48506 If the z coordinate is large like this then takeoff might not work as expected. Resetting the SITL and simulation should fix that problem.","title":"Check the Home Position"},{"location":"px4_sitl/#no-remote-control","text":"Notice the above setting is provided in the params section of the settings.json file: \"NAV_RCL_ACT\": 0, \"NAV_DLL_ACT\": 0, This is required if you plan to fly the SITL mode PX4 with no remote control, just using python scripts, for example. These parameters stop the PX4 from triggering \"failsafe mode on\" every time a move command is finished. You can use the following PX4 command to check these values are set correctly: param show NAV_RCL_ACT param show NAV_DLL_ACT NOTE: Do NOT do this on a real drone as it is too dangerous to fly without these failsafe measures.","title":"No Remote Control"},{"location":"px4_sitl/#manually-set-parameters","text":"You can also run the following in the PX4 console to set all these parameters: param set LPE_LAT 47.641468 param set LPE_LON -122.140165 param set COM_OBL_ACT 1 param set NAV_RCL_ACT 0 param set NAV_DLL_ACT 0","title":"Manually set parameters"},{"location":"px4_sitl/#using-virtualbox-ubuntu","text":"If you want to run the above posix_sitl in a VirtualBox Ubuntu machine then it will have a different ip address from localhost. So in this case you need to edit the settings file and change the UdpIp and SitlIp to the ip address of your virtual machine set the LocalIpAddress to the address of your host machine running the Unreal engine.","title":"Using VirtualBox Ubuntu"},{"location":"px4_sitl/#remote-controller","text":"There are several options for flying the simulated drone using a remote control or joystick like xbox gamepad. See remote controllers","title":"Remote Controller"},{"location":"reinforcement_learning/","text":"Reinforcement Learning in AirSim # We below describe how we can implement DQN in AirSim using CNTK. The easiest way is to first install python only CNTK ( instructions ). CNTK provides several demo examples of deep RL . We will modify the DeepQNeuralNetwork.py to work with AirSim. We can utilize most of the classes and methods corresponding to the DQN algorithm. However, there are certain additions we need to make for AirSim. Disclaimer # This is still in active development. What we share below is a framework that can be extended and tweaked to obtain better performance. RL with Car # Source code This example works with AirSimNeighborhood environment available in releases . First, we need to get the images from simulation and transform them appropriately. Below, we show how a depth image can be obtained from the ego camera and transformed to an 84X84 input to the network. (you can use other sensor modalities, and sensor inputs as well \u2013 of course you\u2019ll have to modify the code accordingly). responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthPerspective, True, False)]) current_state = transform_input(responses) We further define the six actions (breaking, straight with throttle, full-left with throttle, full-right with throttle, half-left with throttle, half-right with throttle) that an agent can execute. This is done via the function interpret_action : def interpret_action(action): car_controls.brake = 0 car_controls.throttle = 1 if action == 0: car_controls.throttle = 0 car_controls.brake = 1 elif action == 1: car_controls.steering = 0 elif action == 2: car_controls.steering = 0.5 elif action == 3: car_controls.steering = -0.5 elif action == 4: car_controls.steering = 0.25 else: car_controls.steering = -0.25 return car_controls We then define the reward function in compute_reward as a convex combination of how fast the vehicle is travelling and how much it deviates from the center line. The agent gets a high reward when its moving fast and staying in the center of the lane. def compute_reward(car_state): MAX_SPEED = 300 MIN_SPEED = 10 thresh_dist = 3.5 beta = 3 z = 0 pts = [np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, -1, z])] pd = car_state.position car_pt = np.array(list(pd.values())) dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((car_pt - pts[i]), (car_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) #print(dist) if dist > thresh_dist: reward = -3 else: reward_dist = (math.exp(-beta*dist) - 0.5) reward_speed = (((car_state.speed - MIN_SPEED)/(MAX_SPEED - MIN_SPEED)) - 0.5) reward = reward_dist + reward_speed return reward The function isDone determines if the episode has terminated (e.g. due to collision). We look at the speed of the vehicle and if it is less than a threshold than the episode is considered to be terminated. def isDone(car_state, car_controls, reward): done = 0 if reward < -1: done = 1 if car_controls.brake == 0: if car_state.speed <= 5: done = 1 return done The main loop then sequences through obtaining the image, computing the action to take according to the current policy, getting a reward and so forth. If the episode terminates then we reset the vehicle to the original state via: client.reset() client.enableApiControl(True) client.armDisarm(True) car_control = interpret_action(1) // Reset position and drive straight for one second client.setCarControls(car_control) time.sleep(1) Note that the simulation needs to be up and running before you execute DQNcar.py. The video below shows first few episodes of DQN training. RL with Quadrotor # Source code This example works with AirSimMountainLandscape environment available in releases . We can similarly apply RL for various autonomous flight scenarios with quadrotors. Below is an example on how RL could be used to train quadrotors to follow high tension power lines (e.g. application for energy infrastructure inspection). There are seven actions here that correspond to different directions in which the quadrotor can move in (six directions + one hovering action). def interpret_action(action): if action == 0: quad_offset = (0, 0, 0) elif action == 1: quad_offset = (1, 0, 0) elif action == 2: quad_offset = (0, 1, 0) elif action == 3: quad_offset = (0, 0, 1) elif action == 4: quad_offset = (-1, 0, 0) elif action == 5: quad_offset = (0, -1, 0) elif action == 6: quad_offset = (0, 0, -1) return quad_offset The reward again is a function how how fast the quad travels in conjunction with how far it gets from the known powerlines. def compute_reward(quad_state, quad_vel, collision_info): thresh_dist = 10 beta = 1 z = -10 pts = [np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, 0, z])] quad_pt = np.array(list((quad_state.x_val, quad_state.y_val, quad_state.z_val))) if collision_info.has_collided: reward = -100 else: dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((quad_pt - pts[i]), (quad_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) if dist > thresh_dist: reward = -10 else: reward = 2*(0.5 - math.exp(-beta*dist)) + np.linalg.norm([quad_vel.x_val, quad_vel.y_val, quad_vel.z_val]) return reward We consider an episode to terminate if it drifts too much away from the known power line coordinates. The reset function here flies the quadrotor to the initial starting point: if done: client.moveToZAsync(clearZ, 2).join() client.moveToPositionAsync(initX, initY, clearZ, 2).join() client.moveToPositionAsync(initZ, initY, initZ, 2).join() current_step +=1 Here is the video of first few episodes during the training. Related # Please also see The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter.","title":"Reinforcement Learning"},{"location":"reinforcement_learning/#reinforcement-learning-in-airsim","text":"We below describe how we can implement DQN in AirSim using CNTK. The easiest way is to first install python only CNTK ( instructions ). CNTK provides several demo examples of deep RL . We will modify the DeepQNeuralNetwork.py to work with AirSim. We can utilize most of the classes and methods corresponding to the DQN algorithm. However, there are certain additions we need to make for AirSim.","title":"Reinforcement Learning in AirSim"},{"location":"reinforcement_learning/#disclaimer","text":"This is still in active development. What we share below is a framework that can be extended and tweaked to obtain better performance.","title":"Disclaimer"},{"location":"reinforcement_learning/#rl-with-car","text":"Source code This example works with AirSimNeighborhood environment available in releases . First, we need to get the images from simulation and transform them appropriately. Below, we show how a depth image can be obtained from the ego camera and transformed to an 84X84 input to the network. (you can use other sensor modalities, and sensor inputs as well \u2013 of course you\u2019ll have to modify the code accordingly). responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthPerspective, True, False)]) current_state = transform_input(responses) We further define the six actions (breaking, straight with throttle, full-left with throttle, full-right with throttle, half-left with throttle, half-right with throttle) that an agent can execute. This is done via the function interpret_action : def interpret_action(action): car_controls.brake = 0 car_controls.throttle = 1 if action == 0: car_controls.throttle = 0 car_controls.brake = 1 elif action == 1: car_controls.steering = 0 elif action == 2: car_controls.steering = 0.5 elif action == 3: car_controls.steering = -0.5 elif action == 4: car_controls.steering = 0.25 else: car_controls.steering = -0.25 return car_controls We then define the reward function in compute_reward as a convex combination of how fast the vehicle is travelling and how much it deviates from the center line. The agent gets a high reward when its moving fast and staying in the center of the lane. def compute_reward(car_state): MAX_SPEED = 300 MIN_SPEED = 10 thresh_dist = 3.5 beta = 3 z = 0 pts = [np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, -1, z]), np.array([130, -1, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, -1, z])] pd = car_state.position car_pt = np.array(list(pd.values())) dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((car_pt - pts[i]), (car_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) #print(dist) if dist > thresh_dist: reward = -3 else: reward_dist = (math.exp(-beta*dist) - 0.5) reward_speed = (((car_state.speed - MIN_SPEED)/(MAX_SPEED - MIN_SPEED)) - 0.5) reward = reward_dist + reward_speed return reward The function isDone determines if the episode has terminated (e.g. due to collision). We look at the speed of the vehicle and if it is less than a threshold than the episode is considered to be terminated. def isDone(car_state, car_controls, reward): done = 0 if reward < -1: done = 1 if car_controls.brake == 0: if car_state.speed <= 5: done = 1 return done The main loop then sequences through obtaining the image, computing the action to take according to the current policy, getting a reward and so forth. If the episode terminates then we reset the vehicle to the original state via: client.reset() client.enableApiControl(True) client.armDisarm(True) car_control = interpret_action(1) // Reset position and drive straight for one second client.setCarControls(car_control) time.sleep(1) Note that the simulation needs to be up and running before you execute DQNcar.py. The video below shows first few episodes of DQN training.","title":"RL with Car"},{"location":"reinforcement_learning/#rl-with-quadrotor","text":"Source code This example works with AirSimMountainLandscape environment available in releases . We can similarly apply RL for various autonomous flight scenarios with quadrotors. Below is an example on how RL could be used to train quadrotors to follow high tension power lines (e.g. application for energy infrastructure inspection). There are seven actions here that correspond to different directions in which the quadrotor can move in (six directions + one hovering action). def interpret_action(action): if action == 0: quad_offset = (0, 0, 0) elif action == 1: quad_offset = (1, 0, 0) elif action == 2: quad_offset = (0, 1, 0) elif action == 3: quad_offset = (0, 0, 1) elif action == 4: quad_offset = (-1, 0, 0) elif action == 5: quad_offset = (0, -1, 0) elif action == 6: quad_offset = (0, 0, -1) return quad_offset The reward again is a function how how fast the quad travels in conjunction with how far it gets from the known powerlines. def compute_reward(quad_state, quad_vel, collision_info): thresh_dist = 10 beta = 1 z = -10 pts = [np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, 125, z]), np.array([0, 125, z]), np.array([0, 0, z]), np.array([130, 0, z]), np.array([130, -128, z]), np.array([0, -128, z]), np.array([0, 0, z])] quad_pt = np.array(list((quad_state.x_val, quad_state.y_val, quad_state.z_val))) if collision_info.has_collided: reward = -100 else: dist = 10000000 for i in range(0, len(pts)-1): dist = min(dist, np.linalg.norm(np.cross((quad_pt - pts[i]), (quad_pt - pts[i+1])))/np.linalg.norm(pts[i]-pts[i+1])) if dist > thresh_dist: reward = -10 else: reward = 2*(0.5 - math.exp(-beta*dist)) + np.linalg.norm([quad_vel.x_val, quad_vel.y_val, quad_vel.z_val]) return reward We consider an episode to terminate if it drifts too much away from the known power line coordinates. The reset function here flies the quadrotor to the initial starting point: if done: client.moveToZAsync(clearZ, 2).join() client.moveToPositionAsync(initX, initY, clearZ, 2).join() client.moveToPositionAsync(initZ, initY, initZ, 2).join() current_step +=1 Here is the video of first few episodes during the training.","title":"RL with Quadrotor"},{"location":"reinforcement_learning/#related","text":"Please also see The Autonomous Driving Cookbook by Microsoft Deep Learning and Robotics Garage Chapter.","title":"Related"},{"location":"remote_control/","text":"Remote Control # To fly manually, you need remote control or RC. If you don't have one then you can use APIs to fly programmatically or use so-called Computer Vision mode to move around using keyboard. RC Setup for Default Config # By default AirSim uses simple_flight as its flight controller which connects to RC via USB port to your computer. You can either use XBox controller or FrSky Taranis X9D Plus . Note that XBox 360 controller is not precise enough and is not recommended if you wanted more real world experience. See FAQ below if things are not working. Other Devices # AirSim can detect large variety of devices however devices other than above might need extra configuration. In future we will add ability to set this config through settings.json. For now, if things are not working then you might want to try workarounds such as x360ce or change code in SimJoystick.cpp file . Note on FrSky Taranis X9D Plus # FrSky Taranis X9D Plus is real UAV remote control with an advantage that it has USB port so it can be directly connected to PC. You can download AirSim config file and follow this tutorial to import it in your RC. You should then see \"sim\" model in RC with all channels configured properly. Note on Linux # Currently default config on Linux is for using Xbox controller. This means other devices might not work properly. In future we will add ability to configure RC in settings.json but for now you might have to change code in SimJoystick.cpp file to use other devices. RC Setup for PX4 # AirSim supports PX4 flight controller however it requires different setup. There are many remote control options that you can use with quadrotors. We have successfully used FrSky Taranis X9D Plus, FlySky FS-TH9X and Futaba 14SG with AirSim. Following are the high level steps to configure your RC: If you are going to use Hardware-in-Loop mode, you need transmitter for your specific brand of RC and bind it. You can find this information in RC's user guide. For Hardware-in-Loop mode, you connect transmitter to Pixhawk. Usually you can find online doc or YouTube video tutorial on how to do that. Calibrate your RC in QGroundControl . See PX4 RC configuration and Please see this guide for more information. Using XBox 360 USB Gamepad # You can also use an xbox controller in SITL mode, it just won't be as precise as a real RC controller. See xbox controller for details on how to set that up. Using Playstation 3 controller # A Playstation 3 controller is confirmed to work as an AirSim controller. On Windows, an emulator to make it look like an Xbox 360 controller, is required however. Many different solutions are available online, for example x360ce Xbox 360 Controller Emulator . DJI Controller # Nils Tijtgat wrote an excellent blog on how to get the DJI controller working with AirSim . FAQ # I'm using default config and AirSim says my RC is not detected on USB. # This typically happens if you have multiple RCs and or XBox/Playstation gamepads etc connected. In Windows, hit Windows+S key and search for \"Set up USB Game controllers\" (in older versions of Windows try \"joystick\"). This will show you all game controllers connected to your PC. If you don't see yours than Windows haven't detected it and so you need to first solve that issue. If you do see yours but not at the top of the list (i.e. index 0) than you need to tell AirSim because AirSim by default tries to use RC at index 0. To do this, navigate to your ~/Documents/AirSim folder, open up settings.json and add/modify following setting. Below tells AirSim to use RC at index = 2. { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"RC\": { \"RemoteControlID\": 2 } } } } Vehicle seems unstable when using XBox/PS3 controller # Regular gamepads are not very precise and have lot of random noise. Most of the times you may see significant offsets as well (i.e. output is not zero when sticks are at zero). So this behavior is expected. Where is RC calibration in AirSim? # We haven't implemented it yet. This means your RC firmware will need to have a capability to do calibration for now. My RC is not working with PX4 setup. # First you want to make sure your RC is working in QGroundControl . If it doesn't then it will sure not work in AirSim. The PX4 mode is suitable for folks who have at least intermediate level of experience to deal with various issues related to PX4 and we would generally refer you to get help from PX4 forums.","title":"Remote Control"},{"location":"remote_control/#remote-control","text":"To fly manually, you need remote control or RC. If you don't have one then you can use APIs to fly programmatically or use so-called Computer Vision mode to move around using keyboard.","title":"Remote Control"},{"location":"remote_control/#rc-setup-for-default-config","text":"By default AirSim uses simple_flight as its flight controller which connects to RC via USB port to your computer. You can either use XBox controller or FrSky Taranis X9D Plus . Note that XBox 360 controller is not precise enough and is not recommended if you wanted more real world experience. See FAQ below if things are not working.","title":"RC Setup for Default Config"},{"location":"remote_control/#other-devices","text":"AirSim can detect large variety of devices however devices other than above might need extra configuration. In future we will add ability to set this config through settings.json. For now, if things are not working then you might want to try workarounds such as x360ce or change code in SimJoystick.cpp file .","title":"Other Devices"},{"location":"remote_control/#note-on-frsky-taranis-x9d-plus","text":"FrSky Taranis X9D Plus is real UAV remote control with an advantage that it has USB port so it can be directly connected to PC. You can download AirSim config file and follow this tutorial to import it in your RC. You should then see \"sim\" model in RC with all channels configured properly.","title":"Note on FrSky Taranis X9D Plus"},{"location":"remote_control/#note-on-linux","text":"Currently default config on Linux is for using Xbox controller. This means other devices might not work properly. In future we will add ability to configure RC in settings.json but for now you might have to change code in SimJoystick.cpp file to use other devices.","title":"Note on Linux"},{"location":"remote_control/#rc-setup-for-px4","text":"AirSim supports PX4 flight controller however it requires different setup. There are many remote control options that you can use with quadrotors. We have successfully used FrSky Taranis X9D Plus, FlySky FS-TH9X and Futaba 14SG with AirSim. Following are the high level steps to configure your RC: If you are going to use Hardware-in-Loop mode, you need transmitter for your specific brand of RC and bind it. You can find this information in RC's user guide. For Hardware-in-Loop mode, you connect transmitter to Pixhawk. Usually you can find online doc or YouTube video tutorial on how to do that. Calibrate your RC in QGroundControl . See PX4 RC configuration and Please see this guide for more information.","title":"RC Setup for PX4"},{"location":"remote_control/#using-xbox-360-usb-gamepad","text":"You can also use an xbox controller in SITL mode, it just won't be as precise as a real RC controller. See xbox controller for details on how to set that up.","title":"Using XBox 360 USB Gamepad"},{"location":"remote_control/#using-playstation-3-controller","text":"A Playstation 3 controller is confirmed to work as an AirSim controller. On Windows, an emulator to make it look like an Xbox 360 controller, is required however. Many different solutions are available online, for example x360ce Xbox 360 Controller Emulator .","title":"Using Playstation 3 controller"},{"location":"remote_control/#dji-controller","text":"Nils Tijtgat wrote an excellent blog on how to get the DJI controller working with AirSim .","title":"DJI Controller"},{"location":"remote_control/#faq","text":"","title":"FAQ"},{"location":"remote_control/#im-using-default-config-and-airsim-says-my-rc-is-not-detected-on-usb","text":"This typically happens if you have multiple RCs and or XBox/Playstation gamepads etc connected. In Windows, hit Windows+S key and search for \"Set up USB Game controllers\" (in older versions of Windows try \"joystick\"). This will show you all game controllers connected to your PC. If you don't see yours than Windows haven't detected it and so you need to first solve that issue. If you do see yours but not at the top of the list (i.e. index 0) than you need to tell AirSim because AirSim by default tries to use RC at index 0. To do this, navigate to your ~/Documents/AirSim folder, open up settings.json and add/modify following setting. Below tells AirSim to use RC at index = 2. { \"SettingsVersion\": 1.2, \"SimMode\": \"Multirotor\", \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"RC\": { \"RemoteControlID\": 2 } } } }","title":"I'm using default config and AirSim says my RC is not detected on USB."},{"location":"remote_control/#vehicle-seems-unstable-when-using-xboxps3-controller","text":"Regular gamepads are not very precise and have lot of random noise. Most of the times you may see significant offsets as well (i.e. output is not zero when sticks are at zero). So this behavior is expected.","title":"Vehicle seems unstable when using XBox/PS3 controller"},{"location":"remote_control/#where-is-rc-calibration-in-airsim","text":"We haven't implemented it yet. This means your RC firmware will need to have a capability to do calibration for now.","title":"Where is RC calibration in AirSim?"},{"location":"remote_control/#my-rc-is-not-working-with-px4-setup","text":"First you want to make sure your RC is working in QGroundControl . If it doesn't then it will sure not work in AirSim. The PX4 mode is suitable for folks who have at least intermediate level of experience to deal with various issues related to PX4 and we would generally refer you to get help from PX4 forums.","title":"My RC is not working with PX4 setup."},{"location":"retexturing/","text":"Runtime Texture Swapping # How to Make An Actor Retexturable # To be made texture-swappable, an actor must derive from the parent class TextureShuffleActor. The parent class can be set via the settings tab in the actor's blueprint. After setting the parent class to TextureShuffActor, the object gains the member DynamicMaterial. DynamicMaterial needs to be set--on all actor instances in the scene--to TextureSwappableMaterial. Warning: Statically setting the Dynamic Material in the blueprint class may cause rendering errors. It seems to work better to set it on all the actor instances in the scene, using the details panel. How to Define the Set(s) of Textures to Choose From # Typically, certain subsets of actors will share a set of texture options with each other. (e.g. walls that are part of the same building) It's easy to set up these groupings by using Unreal Engine's group editing functionality. Select all the instances that should have the same texture selection, and add the textures to all of them simultaneously via the Details panel. Use the same technique to add descriptive tags to groups of actors, which will be used to address them in the API. It's ideal to work from larger groupings to smaller groupings, simply deselecting actors to narrow down the grouping as you go, and applying any individual actor properties last. How to Swap Textures from the API # The following API is available in C++ and python. (C++ shown) std::vector simSwapTextures(const std::string& tags, int tex_id); The string of \",\" or \", \" delimited tags identifies on which actors to perform the swap. The tex_id indexes the array of textures assigned to each actor undergoing a swap. The function will return the list of objects which matched the provided tags and had the texture swap perfomed. If tex_id is out-of-bounds for some object's texture set, it will be taken modulo the number of textures that were available. Demo (Python): import airsim import time c = airsim.client.MultirotorClient() print(c.simSwapTextures(\"furniture\", 0)) time.sleep(2) print(c.simSwapTextures(\"chair\", 1)) time.sleep(2) print(c.simSwapTextures(\"table\", 1)) time.sleep(2) print(c.simSwapTextures(\"chair, right\", 0)) Results: ['RetexturableChair', 'RetexturableChair2', 'RetexturableTable'] ['RetexturableChair', 'RetexturableChair2'] ['RetexturableTable'] ['RetexturableChair2'] Note that in this example, different textures were chosen on each actor for the same index value.","title":"Domain Randomization"},{"location":"retexturing/#runtime-texture-swapping","text":"","title":"Runtime Texture Swapping"},{"location":"retexturing/#how-to-make-an-actor-retexturable","text":"To be made texture-swappable, an actor must derive from the parent class TextureShuffleActor. The parent class can be set via the settings tab in the actor's blueprint. After setting the parent class to TextureShuffActor, the object gains the member DynamicMaterial. DynamicMaterial needs to be set--on all actor instances in the scene--to TextureSwappableMaterial. Warning: Statically setting the Dynamic Material in the blueprint class may cause rendering errors. It seems to work better to set it on all the actor instances in the scene, using the details panel.","title":"How to Make An Actor Retexturable"},{"location":"retexturing/#how-to-define-the-sets-of-textures-to-choose-from","text":"Typically, certain subsets of actors will share a set of texture options with each other. (e.g. walls that are part of the same building) It's easy to set up these groupings by using Unreal Engine's group editing functionality. Select all the instances that should have the same texture selection, and add the textures to all of them simultaneously via the Details panel. Use the same technique to add descriptive tags to groups of actors, which will be used to address them in the API. It's ideal to work from larger groupings to smaller groupings, simply deselecting actors to narrow down the grouping as you go, and applying any individual actor properties last.","title":"How to Define the Set(s) of Textures to Choose From"},{"location":"retexturing/#how-to-swap-textures-from-the-api","text":"The following API is available in C++ and python. (C++ shown) std::vector simSwapTextures(const std::string& tags, int tex_id); The string of \",\" or \", \" delimited tags identifies on which actors to perform the swap. The tex_id indexes the array of textures assigned to each actor undergoing a swap. The function will return the list of objects which matched the provided tags and had the texture swap perfomed. If tex_id is out-of-bounds for some object's texture set, it will be taken modulo the number of textures that were available. Demo (Python): import airsim import time c = airsim.client.MultirotorClient() print(c.simSwapTextures(\"furniture\", 0)) time.sleep(2) print(c.simSwapTextures(\"chair\", 1)) time.sleep(2) print(c.simSwapTextures(\"table\", 1)) time.sleep(2) print(c.simSwapTextures(\"chair, right\", 0)) Results: ['RetexturableChair', 'RetexturableChair2', 'RetexturableTable'] ['RetexturableChair', 'RetexturableChair2'] ['RetexturableTable'] ['RetexturableChair2'] Note that in this example, different textures were chosen on each actor for the same index value.","title":"How to Swap Textures from the API"},{"location":"sensors/","text":"Sensors in AirSim # AirSim currently supports the following sensors. Each sensor is associated with a integer enum specifying its sensor type. Camera Barometer = 1 Imu = 2 Gps = 3 Magnetometer = 4 Distance Sensor = 5 Lidar = 6 Note : Cameras are configured differently than the other sensors and do not have an enum associated with them. Look at general settings and image API for camera config and API. Default sensors # If no sensors are specified in the settings.json , the the following sensors are enabled by default based on the sim mode. Multirotor # Imu Magnetometer Gps Barometer Car # Gps ComputerVision # None Behind the scenes, createDefaultSensorSettings method in AirSimSettings.hpp sets up the above sensors with their default parameters, depending on the sim mode specified in the settings.json file. Configuring the default sensor list # The default sensor list can be configured in settings json: \"DefaultSensors\": { \"Barometer\": { \"SensorType\": 1, \"Enabled\" : true }, \"Imu\": { \"SensorType\": 2, \"Enabled\" : true }, \"Gps\": { \"SensorType\": 3, \"Enabled\" : true }, \"Magnetometer\": { \"SensorType\": 4, \"Enabled\" : true }, \"Distance\": { \"SensorType\": 5, \"Enabled\" : true }, \"Lidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000 } }, Configuring vehicle-specific sensor list # If a vehicle provides its sensor list, it must provide the whole list. Selective add/remove/update of the default sensor list is NOT supported. A vehicle specific sensor list can be specified in the vehicle settings part of the json. e.g., \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"AutoCreate\": true, ... \"Sensors\": { \"MyLidar1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true }, \"MyLidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true } } } } Sensor specific settings # Each sensor-type has its own set of settings as well. Please see lidar for example of Lidar specific settings. Distance Sensor # By default, Distance Sensor points to the front of the vehicle. It can be pointed in any direction by modifying the settings Configurable Parameters - Parameter Description X Y Z Position of the sensor relative to the vehicle (in NED, in meters) (Default (0,0,0)-Multirotor, (0,0,-1)-Car) Yaw Pitch Roll Orientation of the sensor relative to the vehicle (degrees) (Default (0,0,0)) MinDistance Minimum distance measured by distance sensor (metres, only used to fill Mavlink message for PX4) (Default 0.2m) MaxDistance Maximum distance measured by distance sensor (metres) (Default 40.0m) For example, to make the sensor point towards the ground (for altitude measurement similar to barometer), the orientation can be modified as follows - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, \"Yaw\": 0, \"Pitch\": -90, \"Roll\": 0 } Note: For Cars, the sensor is placed 1 meter above the vehicle center by default. This is required since otherwise the sensor gives strange data due it being inside the vehicle. This doesn't affect the sensor values say when measuring the distance between 2 cars. See PythonClient/car/distance_sensor_multi.py for an example usage. Server side visualization for debugging # Be default, the points hit by distance sensor are not drawn on the viewport. To enable the drawing of hit points on the viewport, please enable setting DrawDebugPoints via settings json. E.g. - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, ... \"DrawDebugPoints\": true } Sensor APIs # Jump straight to hello_drone.py or hello_drone.cpp for example usage, or see follow below for the full API. Barometer C++ cpp msr::airlib::BarometerBase::Output getBarometerData(const std::string& barometer_name, const std::string& vehicle_name); Python python barometer_data = client.getBarometerData(barometer_name = \"\", vehicle_name = \"\") IMU C++ cpp msr::airlib::ImuBase::Output getImuData(const std::string& imu_name = \"\", const std::string& vehicle_name = \"\"); Python python imu_data = client.getImuData(imu_name = \"\", vehicle_name = \"\") GPS C++ cpp msr::airlib::GpsBase::Output getGpsData(const std::string& gps_name = \"\", const std::string& vehicle_name = \"\"); Python python gps_data = client.getGpsData(gps_name = \"\", vehicle_name = \"\") Magnetometer C++ cpp msr::airlib::MagnetometerBase::Output getMagnetometerData(const std::string& magnetometer_name = \"\", const std::string& vehicle_name = \"\"); Python python magnetometer_data = client.getMagnetometerData(magnetometer_name = \"\", vehicle_name = \"\") Distance sensor C++ cpp msr::airlib::DistanceSensorData getDistanceSensorData(const std::string& distance_sensor_name = \"\", const std::string& vehicle_name = \"\"); Python python distance_sensor_data = client.getDistanceSensorData(distance_sensor_name = \"\", vehicle_name = \"\") Lidar See lidar for Lidar API.","title":"Sensors"},{"location":"sensors/#sensors-in-airsim","text":"AirSim currently supports the following sensors. Each sensor is associated with a integer enum specifying its sensor type. Camera Barometer = 1 Imu = 2 Gps = 3 Magnetometer = 4 Distance Sensor = 5 Lidar = 6 Note : Cameras are configured differently than the other sensors and do not have an enum associated with them. Look at general settings and image API for camera config and API.","title":"Sensors in AirSim"},{"location":"sensors/#default-sensors","text":"If no sensors are specified in the settings.json , the the following sensors are enabled by default based on the sim mode.","title":"Default sensors"},{"location":"sensors/#multirotor","text":"Imu Magnetometer Gps Barometer","title":"Multirotor"},{"location":"sensors/#car","text":"Gps","title":"Car"},{"location":"sensors/#computervision","text":"None Behind the scenes, createDefaultSensorSettings method in AirSimSettings.hpp sets up the above sensors with their default parameters, depending on the sim mode specified in the settings.json file.","title":"ComputerVision"},{"location":"sensors/#configuring-the-default-sensor-list","text":"The default sensor list can be configured in settings json: \"DefaultSensors\": { \"Barometer\": { \"SensorType\": 1, \"Enabled\" : true }, \"Imu\": { \"SensorType\": 2, \"Enabled\" : true }, \"Gps\": { \"SensorType\": 3, \"Enabled\" : true }, \"Magnetometer\": { \"SensorType\": 4, \"Enabled\" : true }, \"Distance\": { \"SensorType\": 5, \"Enabled\" : true }, \"Lidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000 } },","title":"Configuring the default sensor list"},{"location":"sensors/#configuring-vehicle-specific-sensor-list","text":"If a vehicle provides its sensor list, it must provide the whole list. Selective add/remove/update of the default sensor list is NOT supported. A vehicle specific sensor list can be specified in the vehicle settings part of the json. e.g., \"Vehicles\": { \"Drone1\": { \"VehicleType\": \"SimpleFlight\", \"AutoCreate\": true, ... \"Sensors\": { \"MyLidar1\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 16, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true }, \"MyLidar2\": { \"SensorType\": 6, \"Enabled\" : true, \"NumberOfChannels\": 4, \"PointsPerSecond\": 10000, \"X\": 0, \"Y\": 0, \"Z\": -1, \"DrawDebugPoints\": true } } } }","title":"Configuring vehicle-specific sensor list"},{"location":"sensors/#sensor-specific-settings","text":"Each sensor-type has its own set of settings as well. Please see lidar for example of Lidar specific settings.","title":"Sensor specific settings"},{"location":"sensors/#distance-sensor","text":"By default, Distance Sensor points to the front of the vehicle. It can be pointed in any direction by modifying the settings Configurable Parameters - Parameter Description X Y Z Position of the sensor relative to the vehicle (in NED, in meters) (Default (0,0,0)-Multirotor, (0,0,-1)-Car) Yaw Pitch Roll Orientation of the sensor relative to the vehicle (degrees) (Default (0,0,0)) MinDistance Minimum distance measured by distance sensor (metres, only used to fill Mavlink message for PX4) (Default 0.2m) MaxDistance Maximum distance measured by distance sensor (metres) (Default 40.0m) For example, to make the sensor point towards the ground (for altitude measurement similar to barometer), the orientation can be modified as follows - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, \"Yaw\": 0, \"Pitch\": -90, \"Roll\": 0 } Note: For Cars, the sensor is placed 1 meter above the vehicle center by default. This is required since otherwise the sensor gives strange data due it being inside the vehicle. This doesn't affect the sensor values say when measuring the distance between 2 cars. See PythonClient/car/distance_sensor_multi.py for an example usage.","title":"Distance Sensor"},{"location":"sensors/#server-side-visualization-for-debugging","text":"Be default, the points hit by distance sensor are not drawn on the viewport. To enable the drawing of hit points on the viewport, please enable setting DrawDebugPoints via settings json. E.g. - \"Distance\": { \"SensorType\": 5, \"Enabled\" : true, ... \"DrawDebugPoints\": true }","title":"Server side visualization for debugging"},{"location":"sensors/#sensor-apis","text":"Jump straight to hello_drone.py or hello_drone.cpp for example usage, or see follow below for the full API. Barometer C++ cpp msr::airlib::BarometerBase::Output getBarometerData(const std::string& barometer_name, const std::string& vehicle_name); Python python barometer_data = client.getBarometerData(barometer_name = \"\", vehicle_name = \"\") IMU C++ cpp msr::airlib::ImuBase::Output getImuData(const std::string& imu_name = \"\", const std::string& vehicle_name = \"\"); Python python imu_data = client.getImuData(imu_name = \"\", vehicle_name = \"\") GPS C++ cpp msr::airlib::GpsBase::Output getGpsData(const std::string& gps_name = \"\", const std::string& vehicle_name = \"\"); Python python gps_data = client.getGpsData(gps_name = \"\", vehicle_name = \"\") Magnetometer C++ cpp msr::airlib::MagnetometerBase::Output getMagnetometerData(const std::string& magnetometer_name = \"\", const std::string& vehicle_name = \"\"); Python python magnetometer_data = client.getMagnetometerData(magnetometer_name = \"\", vehicle_name = \"\") Distance sensor C++ cpp msr::airlib::DistanceSensorData getDistanceSensorData(const std::string& distance_sensor_name = \"\", const std::string& vehicle_name = \"\"); Python python distance_sensor_data = client.getDistanceSensorData(distance_sensor_name = \"\", vehicle_name = \"\") Lidar See lidar for Lidar API.","title":"Sensor APIs"},{"location":"settings/","text":"AirSim Settings # Where are Settings Stored? # AirSim is searching for the settings definition in 4 different ways in the following order. The first match will be used: Looking at the (absolute) path specified by the --settings command line argument. For example, in Windows: AirSim.exe --settings 'C:\\path\\to\\settings.json' In Linux ./Blocks.sh --settings '/home/$USER/path/to/settings.json' Looking for a json document passed as a command line argument by the --settings argument. For example, in Windows: AirSim.exe --settings '{\"foo\" : \"bar\"}' In Linux ./Blocks.sh --settings '{\"foo\" : \"bar\"}' Looking in the folder of the executable for a file called settings.json . Looking in the users home folder for a file called settings.json . The AirSim subfolder is located at Documents\\AirSim on Windows and ~/Documents/AirSim on Linux systems. The file is in usual json format . On first startup AirSim would create settings.json file with no settings at the users home folder. To avoid problems, always use ASCII format to save json file. How to Chose Between Car and Multirotor? # The default is to use multirotor. To use car simple set \"SimMode\": \"Car\" like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } To choose multirotor, set \"SimMode\": \"Multirotor\" . If you want to prompt user to select vehicle type then use \"SimMode\": \"\" . Available Settings and Their Defaults # Below are complete list of settings available along with their default values. If any of the settings is missing from json file, then default value is used. Some default values are simply specified as \"\" which means actual value may be chosen based on the vehicle you are using. For example, ViewMode setting has default value \"\" which translates to \"FlyWithMe\" for drones and \"SpringArmChase\" for cars. WARNING: Do not copy paste all of below in your settings.json. We strongly recommend adding only those settings that you don't want default values. Only required element is \"SettingsVersion\" . { \"SimMode\": \"\", \"ClockType\": \"\", \"ClockSpeed\": 1, \"LocalHostIp\": \"127.0.0.1\", \"RecordUIVisible\": true, \"LogMessagesVisible\": true, \"ViewMode\": \"\", \"RpcEnabled\": true, \"EngineSound\": true, \"PhysicsEngineName\": \"\", \"SpeedUnitFactor\": 1.0, \"SpeedUnitLabel\": \"m/s\", \"Recording\": { \"RecordOnMove\": false, \"RecordInterval\": 0.05, \"Cameras\": [ { \"CameraName\": \"0\", \"ImageType\": 0, \"PixelsAsFloat\": false, \"Compress\": true } ] }, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"AutoExposureBias\": 0, \"AutoExposureMaxBrightness\": 0.64, \"AutoExposureMinBrightness\": 0.03, \"MotionBlurAmount\": 0, \"TargetGamma\": 1.0, \"ProjectionMode\": \"\", \"OrthoWidth\": 5.12 } ], \"NoiseSettings\": [ { \"Enabled\": false, \"ImageType\": 0, \"RandContrib\": 0.2, \"RandSpeed\": 100000.0, \"RandSize\": 500.0, \"RandDensity\": 2, \"HorzWaveContrib\":0.03, \"HorzWaveStrength\": 0.08, \"HorzWaveVertSize\": 1.0, \"HorzWaveScreenSize\": 1.0, \"HorzNoiseLinesContrib\": 1.0, \"HorzNoiseLinesDensityY\": 0.01, \"HorzNoiseLinesDensityXY\": 0.5, \"HorzDistortionContrib\": 1.0, \"HorzDistortionStrength\": 0.002 } ], \"Gimbal\": { \"Stabilization\": 0, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"OriginGeopoint\": { \"Latitude\": 47.641468, \"Longitude\": -122.140165, \"Altitude\": 122 }, \"TimeOfDay\": { \"Enabled\": false, \"StartDateTime\": \"\", \"CelestialClockSpeed\": 1, \"StartDateTimeDst\": false, \"UpdateIntervalSecs\": 60 }, \"SubWindows\": [ {\"WindowID\": 0, \"CameraName\": \"0\", \"ImageType\": 3, \"Visible\": false}, {\"WindowID\": 1, \"CameraName\": \"0\", \"ImageType\": 5, \"Visible\": false}, {\"WindowID\": 2, \"CameraName\": \"0\", \"ImageType\": 0, \"Visible\": false} ], \"SegmentationSettings\": { \"InitMethod\": \"\", \"MeshNamingMethod\": \"\", \"OverrideExisting\": false }, \"PawnPaths\": { \"BareboneCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/Vehicle/VehicleAdvPawn.VehicleAdvPawn_C'\"}, \"DefaultCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/SUV/SuvCarPawn.SuvCarPawn_C'\"}, \"DefaultQuadrotor\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_FlyingPawn.BP_FlyingPawn_C'\"}, \"DefaultComputerVision\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_ComputerVisionPawn.BP_ComputerVisionPawn_C'\"} }, \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Armed\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": false }, \"Cameras\": { //same elements as CameraDefaults above, key as name }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"PhysXCar\": { \"VehicleType\": \"PhysXCar\", \"DefaultVehicleState\": \"\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"RC\": { \"RemoteControlID\": -1 }, \"Cameras\": { \"MyCamera1\": { //same elements as elements inside CameraDefaults above }, \"MyCamera2\": { //same elements as elements inside CameraDefaults above }, }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } } } SimMode # SimMode determines which simulation mode will be used. Below are currently supported values: - \"\" : prompt user to select vehicle type multirotor or car - \"Multirotor\" : Use multirotor simulation - \"Car\" : Use car simulation - \"ComputerVision\" : Use only camera, no vehicle or physics ViewMode # The ViewMode determines which camera to use as default and how camera will follow the vehicle. For multirotors, the default ViewMode is \"FlyWithMe\" while for cars the default ViewMode is \"SpringArmChase\" . FlyWithMe : Chase the vehicle from behind with 6 degrees of freedom GroundObserver : Chase the vehicle from 6' above the ground but with full freedom in XY plane. Fpv : View the scene from front camera of vehicle Manual : Don't move camera automatically. Use arrow keys and ASWD keys for move camera manually. SpringArmChase : Chase the vehicle with camera mounted on (invisible) arm that is attached to the vehicle via spring (so it has some latency in movement). NoDisplay : This will freeze rendering for main screen however rendering for subwindows, recording and APIs remain active. This mode is useful to save resources in \"headless\" mode where you are only interested in getting images and don't care about what gets rendered on main screen. This may also improve FPS for recording images. TimeOfDay # This setting controls the position of Sun in the environment. By default Enabled is false which means Sun's position is left at whatever was the default in the environment and it doesn't change over the time. If Enabled is true then Sun position is computed using longitude, latitude and altitude specified in OriginGeopoint section for the date specified in StartDateTime in the string format as %Y-%m-%d %H:%M:%S , for example, 2018-02-12 15:20:00 . If this string is empty then current date and time is used. If StartDateTimeDst is true then we adjust for day light savings time. The Sun's position is then continuously updated at the interval specified in UpdateIntervalSecs . In some cases, it might be desirable to have celestial clock run faster or slower than simulation clock. This can be specified using CelestialClockSpeed , for example, value 100 means for every 1 second of simulation clock, Sun's position is advanced by 100 seconds so Sun will move in sky much faster. Also see Time of Day API . OriginGeopoint # This setting specifies the latitude, longitude and altitude of the Player Start component placed in the Unreal environment. The vehicle's home point is computed using this transformation. Note that all coordinates exposed via APIs are using NED system in SI units which means each vehicle starts at (0, 0, 0) in NED system. Time of Day settings are computed for geographical coordinates specified in OriginGeopoint . SubWindows # This setting determines what is shown in each of 3 subwindows which are visible when you press 0 key. The WindowsID can be 0 to 2, CameraName is any available camera on the vehicle. ImageType integer value determines what kind of image gets shown according to ImageType enum . For example, for car vehicles below shows driver view, front bumper view and rear view as scene, depth and surface normals respectively. \"SubWindows\": [ {\"WindowID\": 0, \"ImageType\": 0, \"CameraName\": \"3\", \"Visible\": true}, {\"WindowID\": 1, \"ImageType\": 3, \"CameraName\": \"0\", \"Visible\": true}, {\"WindowID\": 2, \"ImageType\": 6, \"CameraName\": \"4\", \"Visible\": true} ] Recording # The recording feature allows you to record data such as position, orientation, velocity along with the captured image at specified intervals. You can start recording by pressing red Record button on lower right or the R key. The data is stored in the Documents\\AirSim folder, in a time stamped subfolder for each recording session, as tab separated file. RecordInterval : specifies minimal interval in seconds between capturing two images. RecordOnMove : specifies that do not record frame if there was vehicle's position or orientation hasn't changed. Cameras : this element controls which cameras are used to capture images. By default scene image from camera 0 is recorded as compressed png format. This setting is json array so you can specify multiple cameras to capture images, each with potentially different image types . When PixelsAsFloat is true, image is saved as pfm file instead of png file. ClockSpeed # This setting allows you to set the speed of simulation clock with respect to wall clock. For example, value of 5.0 would mean simulation clock has 5 seconds elapsed when wall clock has 1 second elapsed (i.e. simulation is running faster). The value of 0.1 means that simulation clock is 10X slower than wall clock. The value of 1 means simulation is running in real time. It is important to realize that quality of simulation may decrease as the simulation clock runs faster. You might see artifacts like object moving past obstacles because collision is not detected. However slowing down simulation clock (i.e. values < 1.0) generally improves the quality of simulation. Segmentation Settings # The InitMethod determines how object IDs are initialized at startup to generate segmentation . The value \"\" or \"CommonObjectsRandomIDs\" (default) means assign random IDs to each object at startup. This will generate segmentation view with random colors assign to each object. The value \"None\" means don't initialize object IDs. This will cause segmentation view to have single solid colors. This mode is useful if you plan to set up object IDs using APIs and it can save lot of delay at startup for large environments like CityEnviron. If OverrideExisting is false then initialization does not alter non-zero object IDs already assigned otherwise it does. If MeshNamingMethod is \"\" or \"OwnerName\" then we use mesh's owner name to generate random hash as object IDs. If it is \"StaticMeshName\" then we use static mesh's name to generate random hash as object IDs. Note that it is not possible to tell individual instances of the same static mesh apart this way, but the names are often more intuitive. Camera Settings # The CameraDefaults element at root level specifies defaults used for all cameras. These defaults can be overridden for individual camera in Cameras element inside Vehicles as described later. Note on ImageType element # The ImageType element in JSON array determines which image type that settings applies to. The valid values are described in ImageType section . In addition, we also support special value ImageType: -1 to apply the settings to external camera (i.e. what you are looking at on the screen). For example, CaptureSettings element is json array so you can add settings for multiple image types easily. CaptureSettings # The CaptureSettings determines how different image types such as scene, depth, disparity, surface normals and segmentation views are rendered. The Width, Height and FOV settings should be self explanatory. The AutoExposureSpeed decides how fast eye adaptation works. We set to generally high value such as 100 to avoid artifacts in image capture. Similarly we set MotionBlurAmount to 0 by default to avoid artifacts in ground truth images. The ProjectionMode decides the projection used by the capture camera and can take value \"perspective\" (default) or \"orthographic\". If projection mode is \"orthographic\" then OrthoWidth determines width of projected area captured in meters. For explanation of other settings, please see this article . NoiseSettings # The NoiseSettings allows to add noise to the specified image type with a goal of simulating camera sensor noise, interference and other artifacts. By default no noise is added, i.e., Enabled: false . If you set Enabled: true then following different types of noise and interference artifacts are enabled, each can be further tuned using setting. The noise effects are implemented as shader created as post processing material in Unreal Engine called CameraSensorNoise . Demo of camera noise and interference simulation: Random noise # This adds random noise blobs with following parameters. * RandContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * RandSpeed : This determines how fast noise fluctuates, 1 means no fluctuation and higher values like 1E6 means full fluctuation. * RandSize : This determines how coarse noise is, 1 means every pixel has its own noise while higher value means more than 1 pixels share same noise value. * RandDensity : This determines how many pixels out of total will have noise, 1 means all pixels while higher value means lesser number of pixels (exponentially). Horizontal bump distortion # This adds horizontal bumps / flickering / ghosting effect. * HorzWaveContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzWaveStrength : This determines overall strength of the effect. * HorzWaveVertSize : This determines how many vertical pixels would be effected by the effect. * HorzWaveScreenSize : This determines how much of the screen is effected by the effect. Horizontal noise lines # This adds regions of noise on horizontal lines. * HorzNoiseLinesContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzNoiseLinesDensityY : This determines how many pixels in horizontal line gets affected. * HorzNoiseLinesDensityXY : This determines how many lines on screen gets affected. Horizontal line distortion # This adds fluctuations on horizontal line. * HorzDistortionContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzDistortionStrength : This determines how large is the distortion. Gimbal # The Gimbal element allows to freeze camera orientation for pitch, roll and/or yaw. This setting is ignored unless ImageType is -1. The Stabilization is defaulted to 0 meaning no gimbal i.e. camera orientation changes with body orientation on all axis. The value of 1 means full stabilization. The value between 0 to 1 acts as a weight for fixed angles specified (in degrees, in world-frame) in Pitch , Roll and Yaw elements and orientation of the vehicle body. When any of the angles is omitted from json or set to NaN, that angle is not stabilized (i.e. it moves along with vehicle body). Vehicles Settings # Each simulation mode will go through the list of vehicles specified in this setting and create the ones that has \"AutoCreate\": true . Each vehicle specified in this setting has key which becomes the name of the vehicle. If \"Vehicles\" element is missing then this list is populated with default car named \"PhysXCar\" and default multirotor named \"SimpleFlight\". Common Vehicle Setting # VehicleType : This could be either PhysXCar , SimpleFlight , PX4Multirotor or ComputerVision . There is no default value therefore this element must be specified. PawnPath : This allows to override the pawn blueprint to use for the vehicle. For example, you may create new pawn blueprint derived from ACarPawn for a warehouse robot in your own project outside the AirSim code and then specify its path here. See also PawnPaths . DefaultVehicleState : Possible value for multirotors is Armed or Disarmed . AutoCreate : If true then this vehicle would be spawned (if supported by selected sim mode). RC : This sub-element allows to specify which remote controller to use for vehicle using RemoteControlID . The value of -1 means use keyboard (not supported yet for multirotors). The value >= 0 specifies one of many remote controllers connected to the system. The list of available RCs can be seen in Game Controllers panel in Windows, for example. X, Y, Z, Yaw, Roll, Pitch : These elements allows you to specify the initial position and orientation of the vehicle. Position is in NED coordinates in SI units with origin set to Player Start location in Unreal environment. The orientation is specified in degrees. IsFpvVehicle : This setting allows to specify which vehicle camera will follow and the view that will be shown when ViewMode is set to Fpv. By default, AirSim selects the first vehicle in settings as FPV vehicle. Cameras : This element specifies camera settings for vehicle. The key in this element is name of the available camera and the value is same as CameraDefaults as described above. For example, to change FOV for the front center camera to 120 degrees, you can use this for Vehicles setting: \"Vehicles\": { \"FishEyeDrone\": { \"VehicleType\": \"SimpleFlight\", \"Cameras\": { \"front-center\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"FOV_Degrees\": 120 } ] } } } } Using PX4 # By default we use simple_flight so you don't have to do separate HITL or SITL setups. We also support \"PX4\" for advanced users. To use PX4 with AirSim, you can use the following for Vehicles setting: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", } } Additional PX4 Settings # The defaults for PX4 is to enable hardware-in-loop setup. There are various other settings available for PX4 as follows with their default values: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"ControlIp\": \"127.0.0.1\", \"ControlPort\": 14580, \"LogViewerHostIp\": \"127.0.0.1\", \"LogViewerPort\": 14388, \"OffboardCompID\": 1, \"OffboardSysID\": 134, \"QgcHostIp\": \"127.0.0.1\", \"QgcPort\": 14550, \"SerialBaudRate\": 115200, \"SerialPort\": \"*\", \"SimCompID\": 42, \"SimSysID\": 142, \"TcpPort\": 4560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14560, \"UseSerial\": true, \"UseTcp\": false, \"VehicleCompID\": 1, \"VehicleSysID\": 135, \"Model\": \"Generic\", \"LocalHostIp\": \"127.0.0.1\" } } These settings define the MavLink SystemId and ComponentId for the Simulator (SimSysID, SimCompID), and for the vehicle (VehicleSysID, VehicleCompID) and the node that allows remote control of the drone from another app this is called the offboard node (OffboardSysID, OffboardCompID). If you want the simulator to also talk to your ground control app (like QGroundControl) you can also set the UDP address for that in case you want to run that on a different machine (QgcHostIp, QgcPort). The default is local host so QGroundControl should \"just work\" if it is running on the same machine. You can connect the simulator to the LogViewer app, provided in this repo, by setting the UDP address for that (LogViewerHostIp, LogViewerPort). And for each flying drone added to the simulator there is a named block of additional settings. In the above you see the default name \"PX4\". You can change this name from the Unreal Editor when you add a new BP_FlyingPawn asset. You will see these properties grouped under the category \"MavLink\". The MavLink node for this pawn can be remote over UDP or it can be connected to a local serial port. If serial then set UseSerial to true, otherwise set UseSerial to false. For serial connections you also need to set the appropriate SerialBaudRate. The default of 115200 works with Pixhawk version 2 over USB. When communicating with the PX4 drone over serial port both the HIL_ messages and vehicle control messages share the same serial port. When communicating over UDP or TCP PX4 requires two separate channels. If UseTcp is false, then UdpIp, UdpPort are used to send HIL_ messages, otherwise the TcpPort is used. TCP support in PX4 was added in 1.9.2 with the lockstep feature because the guarantee of message delivery that TCP provides is required for the proper functioning of lockstep. AirSim becomes a TCP server in that case, and waits for a connection from the PX4 app. The second channel for controlling the vehicle is defined by (ControlIp, ControlPort) and is always a UDP channel. Other Settings # EngineSound # To turn off the engine sound use setting \"EngineSound\": false . Currently this setting applies only to car. PawnPaths # This allows you to specify your own vehicle pawn blueprints, for example, you can replace the default car in AirSim with your own car. Your vehicle BP can reside in Content folder of your own Unreal project (i.e. outside of AirSim plugin folder). For example, if you have a car BP located in file Content\\MyCar\\MySedanBP.uasset in your project then you can set \"DefaultCar\": {\"PawnBP\":\"Class'/Game/MyCar/MySedanBP.MySedanBP_C'\"} . The XYZ.XYZ_C is a special notation required to specify class for BP XYZ . Please note that your BP must be derived from CarPawn class. By default this is not the case but you can re-parent the BP using the \"Class Settings\" button in toolbar in UE editor after you open the BP and then choosing \"Car Pawn\" for Parent Class settings in Class Options. It is also a good idea to disable \"Auto Possess Player\" and \"Auto Possess AI\" as well as set AI Controller Class to None in BP details. Please make sure your asset is included for cooking in packaging options if you are creating binary. PhysicsEngineName # For cars, we support only PhysX for now (regardless of value in this setting). For multirotors, we support \"FastPhysicsEngine\" only. LocalHostIp Setting # Now when connecting to remote machines you may need to pick a specific Ethernet adapter to reach those machines, for example, it might be over Ethernet or over Wi-Fi, or some other special virtual adapter or a VPN. Your PC may have multiple networks, and those networks might not be allowed to talk to each other, in which case the UDP messages from one network will not get through to the others. So the LocalHostIp allows you to configure how you are reaching those machines. The default of 127.0.0.1 is not able to reach external machines, this default is only used when everything you are talking to is contained on a single PC. SpeedUnitFactor # Unit conversion factor for speed related to m/s , default is 1. Used in conjunction with SpeedUnitLabel. This may be only used for display purposes for example on-display speed when car is being driven. For example, to get speed in miles/hr use factor 2.23694. SpeedUnitLabel # Unit label for speed, default is m/s . Used in conjunction with SpeedUnitFactor.","title":"Settings"},{"location":"settings/#airsim-settings","text":"","title":"AirSim Settings"},{"location":"settings/#where-are-settings-stored","text":"AirSim is searching for the settings definition in 4 different ways in the following order. The first match will be used: Looking at the (absolute) path specified by the --settings command line argument. For example, in Windows: AirSim.exe --settings 'C:\\path\\to\\settings.json' In Linux ./Blocks.sh --settings '/home/$USER/path/to/settings.json' Looking for a json document passed as a command line argument by the --settings argument. For example, in Windows: AirSim.exe --settings '{\"foo\" : \"bar\"}' In Linux ./Blocks.sh --settings '{\"foo\" : \"bar\"}' Looking in the folder of the executable for a file called settings.json . Looking in the users home folder for a file called settings.json . The AirSim subfolder is located at Documents\\AirSim on Windows and ~/Documents/AirSim on Linux systems. The file is in usual json format . On first startup AirSim would create settings.json file with no settings at the users home folder. To avoid problems, always use ASCII format to save json file.","title":"Where are Settings Stored?"},{"location":"settings/#how-to-chose-between-car-and-multirotor","text":"The default is to use multirotor. To use car simple set \"SimMode\": \"Car\" like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } To choose multirotor, set \"SimMode\": \"Multirotor\" . If you want to prompt user to select vehicle type then use \"SimMode\": \"\" .","title":"How to Chose Between Car and Multirotor?"},{"location":"settings/#available-settings-and-their-defaults","text":"Below are complete list of settings available along with their default values. If any of the settings is missing from json file, then default value is used. Some default values are simply specified as \"\" which means actual value may be chosen based on the vehicle you are using. For example, ViewMode setting has default value \"\" which translates to \"FlyWithMe\" for drones and \"SpringArmChase\" for cars. WARNING: Do not copy paste all of below in your settings.json. We strongly recommend adding only those settings that you don't want default values. Only required element is \"SettingsVersion\" . { \"SimMode\": \"\", \"ClockType\": \"\", \"ClockSpeed\": 1, \"LocalHostIp\": \"127.0.0.1\", \"RecordUIVisible\": true, \"LogMessagesVisible\": true, \"ViewMode\": \"\", \"RpcEnabled\": true, \"EngineSound\": true, \"PhysicsEngineName\": \"\", \"SpeedUnitFactor\": 1.0, \"SpeedUnitLabel\": \"m/s\", \"Recording\": { \"RecordOnMove\": false, \"RecordInterval\": 0.05, \"Cameras\": [ { \"CameraName\": \"0\", \"ImageType\": 0, \"PixelsAsFloat\": false, \"Compress\": true } ] }, \"CameraDefaults\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"Width\": 256, \"Height\": 144, \"FOV_Degrees\": 90, \"AutoExposureSpeed\": 100, \"AutoExposureBias\": 0, \"AutoExposureMaxBrightness\": 0.64, \"AutoExposureMinBrightness\": 0.03, \"MotionBlurAmount\": 0, \"TargetGamma\": 1.0, \"ProjectionMode\": \"\", \"OrthoWidth\": 5.12 } ], \"NoiseSettings\": [ { \"Enabled\": false, \"ImageType\": 0, \"RandContrib\": 0.2, \"RandSpeed\": 100000.0, \"RandSize\": 500.0, \"RandDensity\": 2, \"HorzWaveContrib\":0.03, \"HorzWaveStrength\": 0.08, \"HorzWaveVertSize\": 1.0, \"HorzWaveScreenSize\": 1.0, \"HorzNoiseLinesContrib\": 1.0, \"HorzNoiseLinesDensityY\": 0.01, \"HorzNoiseLinesDensityXY\": 0.5, \"HorzDistortionContrib\": 1.0, \"HorzDistortionStrength\": 0.002 } ], \"Gimbal\": { \"Stabilization\": 0, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"OriginGeopoint\": { \"Latitude\": 47.641468, \"Longitude\": -122.140165, \"Altitude\": 122 }, \"TimeOfDay\": { \"Enabled\": false, \"StartDateTime\": \"\", \"CelestialClockSpeed\": 1, \"StartDateTimeDst\": false, \"UpdateIntervalSecs\": 60 }, \"SubWindows\": [ {\"WindowID\": 0, \"CameraName\": \"0\", \"ImageType\": 3, \"Visible\": false}, {\"WindowID\": 1, \"CameraName\": \"0\", \"ImageType\": 5, \"Visible\": false}, {\"WindowID\": 2, \"CameraName\": \"0\", \"ImageType\": 0, \"Visible\": false} ], \"SegmentationSettings\": { \"InitMethod\": \"\", \"MeshNamingMethod\": \"\", \"OverrideExisting\": false }, \"PawnPaths\": { \"BareboneCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/Vehicle/VehicleAdvPawn.VehicleAdvPawn_C'\"}, \"DefaultCar\": {\"PawnBP\": \"Class'/AirSim/VehicleAdv/SUV/SuvCarPawn.SuvCarPawn_C'\"}, \"DefaultQuadrotor\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_FlyingPawn.BP_FlyingPawn_C'\"}, \"DefaultComputerVision\": {\"PawnBP\": \"Class'/AirSim/Blueprints/BP_ComputerVisionPawn.BP_ComputerVisionPawn_C'\"} }, \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Armed\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": false }, \"Cameras\": { //same elements as CameraDefaults above, key as name }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN }, \"PhysXCar\": { \"VehicleType\": \"PhysXCar\", \"DefaultVehicleState\": \"\", \"AutoCreate\": true, \"PawnPath\": \"\", \"EnableCollisionPassthrogh\": false, \"EnableCollisions\": true, \"RC\": { \"RemoteControlID\": -1 }, \"Cameras\": { \"MyCamera1\": { //same elements as elements inside CameraDefaults above }, \"MyCamera2\": { //same elements as elements inside CameraDefaults above }, }, \"X\": NaN, \"Y\": NaN, \"Z\": NaN, \"Pitch\": NaN, \"Roll\": NaN, \"Yaw\": NaN } } }","title":"Available Settings and Their Defaults"},{"location":"settings/#simmode","text":"SimMode determines which simulation mode will be used. Below are currently supported values: - \"\" : prompt user to select vehicle type multirotor or car - \"Multirotor\" : Use multirotor simulation - \"Car\" : Use car simulation - \"ComputerVision\" : Use only camera, no vehicle or physics","title":"SimMode"},{"location":"settings/#viewmode","text":"The ViewMode determines which camera to use as default and how camera will follow the vehicle. For multirotors, the default ViewMode is \"FlyWithMe\" while for cars the default ViewMode is \"SpringArmChase\" . FlyWithMe : Chase the vehicle from behind with 6 degrees of freedom GroundObserver : Chase the vehicle from 6' above the ground but with full freedom in XY plane. Fpv : View the scene from front camera of vehicle Manual : Don't move camera automatically. Use arrow keys and ASWD keys for move camera manually. SpringArmChase : Chase the vehicle with camera mounted on (invisible) arm that is attached to the vehicle via spring (so it has some latency in movement). NoDisplay : This will freeze rendering for main screen however rendering for subwindows, recording and APIs remain active. This mode is useful to save resources in \"headless\" mode where you are only interested in getting images and don't care about what gets rendered on main screen. This may also improve FPS for recording images.","title":"ViewMode"},{"location":"settings/#timeofday","text":"This setting controls the position of Sun in the environment. By default Enabled is false which means Sun's position is left at whatever was the default in the environment and it doesn't change over the time. If Enabled is true then Sun position is computed using longitude, latitude and altitude specified in OriginGeopoint section for the date specified in StartDateTime in the string format as %Y-%m-%d %H:%M:%S , for example, 2018-02-12 15:20:00 . If this string is empty then current date and time is used. If StartDateTimeDst is true then we adjust for day light savings time. The Sun's position is then continuously updated at the interval specified in UpdateIntervalSecs . In some cases, it might be desirable to have celestial clock run faster or slower than simulation clock. This can be specified using CelestialClockSpeed , for example, value 100 means for every 1 second of simulation clock, Sun's position is advanced by 100 seconds so Sun will move in sky much faster. Also see Time of Day API .","title":"TimeOfDay"},{"location":"settings/#origingeopoint","text":"This setting specifies the latitude, longitude and altitude of the Player Start component placed in the Unreal environment. The vehicle's home point is computed using this transformation. Note that all coordinates exposed via APIs are using NED system in SI units which means each vehicle starts at (0, 0, 0) in NED system. Time of Day settings are computed for geographical coordinates specified in OriginGeopoint .","title":"OriginGeopoint"},{"location":"settings/#subwindows","text":"This setting determines what is shown in each of 3 subwindows which are visible when you press 0 key. The WindowsID can be 0 to 2, CameraName is any available camera on the vehicle. ImageType integer value determines what kind of image gets shown according to ImageType enum . For example, for car vehicles below shows driver view, front bumper view and rear view as scene, depth and surface normals respectively. \"SubWindows\": [ {\"WindowID\": 0, \"ImageType\": 0, \"CameraName\": \"3\", \"Visible\": true}, {\"WindowID\": 1, \"ImageType\": 3, \"CameraName\": \"0\", \"Visible\": true}, {\"WindowID\": 2, \"ImageType\": 6, \"CameraName\": \"4\", \"Visible\": true} ]","title":"SubWindows"},{"location":"settings/#recording","text":"The recording feature allows you to record data such as position, orientation, velocity along with the captured image at specified intervals. You can start recording by pressing red Record button on lower right or the R key. The data is stored in the Documents\\AirSim folder, in a time stamped subfolder for each recording session, as tab separated file. RecordInterval : specifies minimal interval in seconds between capturing two images. RecordOnMove : specifies that do not record frame if there was vehicle's position or orientation hasn't changed. Cameras : this element controls which cameras are used to capture images. By default scene image from camera 0 is recorded as compressed png format. This setting is json array so you can specify multiple cameras to capture images, each with potentially different image types . When PixelsAsFloat is true, image is saved as pfm file instead of png file.","title":"Recording"},{"location":"settings/#clockspeed","text":"This setting allows you to set the speed of simulation clock with respect to wall clock. For example, value of 5.0 would mean simulation clock has 5 seconds elapsed when wall clock has 1 second elapsed (i.e. simulation is running faster). The value of 0.1 means that simulation clock is 10X slower than wall clock. The value of 1 means simulation is running in real time. It is important to realize that quality of simulation may decrease as the simulation clock runs faster. You might see artifacts like object moving past obstacles because collision is not detected. However slowing down simulation clock (i.e. values < 1.0) generally improves the quality of simulation.","title":"ClockSpeed"},{"location":"settings/#segmentation-settings","text":"The InitMethod determines how object IDs are initialized at startup to generate segmentation . The value \"\" or \"CommonObjectsRandomIDs\" (default) means assign random IDs to each object at startup. This will generate segmentation view with random colors assign to each object. The value \"None\" means don't initialize object IDs. This will cause segmentation view to have single solid colors. This mode is useful if you plan to set up object IDs using APIs and it can save lot of delay at startup for large environments like CityEnviron. If OverrideExisting is false then initialization does not alter non-zero object IDs already assigned otherwise it does. If MeshNamingMethod is \"\" or \"OwnerName\" then we use mesh's owner name to generate random hash as object IDs. If it is \"StaticMeshName\" then we use static mesh's name to generate random hash as object IDs. Note that it is not possible to tell individual instances of the same static mesh apart this way, but the names are often more intuitive.","title":"Segmentation Settings"},{"location":"settings/#camera-settings","text":"The CameraDefaults element at root level specifies defaults used for all cameras. These defaults can be overridden for individual camera in Cameras element inside Vehicles as described later.","title":"Camera Settings"},{"location":"settings/#note-on-imagetype-element","text":"The ImageType element in JSON array determines which image type that settings applies to. The valid values are described in ImageType section . In addition, we also support special value ImageType: -1 to apply the settings to external camera (i.e. what you are looking at on the screen). For example, CaptureSettings element is json array so you can add settings for multiple image types easily.","title":"Note on ImageType element"},{"location":"settings/#capturesettings","text":"The CaptureSettings determines how different image types such as scene, depth, disparity, surface normals and segmentation views are rendered. The Width, Height and FOV settings should be self explanatory. The AutoExposureSpeed decides how fast eye adaptation works. We set to generally high value such as 100 to avoid artifacts in image capture. Similarly we set MotionBlurAmount to 0 by default to avoid artifacts in ground truth images. The ProjectionMode decides the projection used by the capture camera and can take value \"perspective\" (default) or \"orthographic\". If projection mode is \"orthographic\" then OrthoWidth determines width of projected area captured in meters. For explanation of other settings, please see this article .","title":"CaptureSettings"},{"location":"settings/#noisesettings","text":"The NoiseSettings allows to add noise to the specified image type with a goal of simulating camera sensor noise, interference and other artifacts. By default no noise is added, i.e., Enabled: false . If you set Enabled: true then following different types of noise and interference artifacts are enabled, each can be further tuned using setting. The noise effects are implemented as shader created as post processing material in Unreal Engine called CameraSensorNoise . Demo of camera noise and interference simulation:","title":"NoiseSettings"},{"location":"settings/#random-noise","text":"This adds random noise blobs with following parameters. * RandContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * RandSpeed : This determines how fast noise fluctuates, 1 means no fluctuation and higher values like 1E6 means full fluctuation. * RandSize : This determines how coarse noise is, 1 means every pixel has its own noise while higher value means more than 1 pixels share same noise value. * RandDensity : This determines how many pixels out of total will have noise, 1 means all pixels while higher value means lesser number of pixels (exponentially).","title":"Random noise"},{"location":"settings/#horizontal-bump-distortion","text":"This adds horizontal bumps / flickering / ghosting effect. * HorzWaveContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzWaveStrength : This determines overall strength of the effect. * HorzWaveVertSize : This determines how many vertical pixels would be effected by the effect. * HorzWaveScreenSize : This determines how much of the screen is effected by the effect.","title":"Horizontal bump distortion"},{"location":"settings/#horizontal-noise-lines","text":"This adds regions of noise on horizontal lines. * HorzNoiseLinesContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzNoiseLinesDensityY : This determines how many pixels in horizontal line gets affected. * HorzNoiseLinesDensityXY : This determines how many lines on screen gets affected.","title":"Horizontal noise lines"},{"location":"settings/#horizontal-line-distortion","text":"This adds fluctuations on horizontal line. * HorzDistortionContrib : This determines blend ratio of noise pixel with image pixel, 0 means no noise and 1 means only noise. * HorzDistortionStrength : This determines how large is the distortion.","title":"Horizontal line distortion"},{"location":"settings/#gimbal","text":"The Gimbal element allows to freeze camera orientation for pitch, roll and/or yaw. This setting is ignored unless ImageType is -1. The Stabilization is defaulted to 0 meaning no gimbal i.e. camera orientation changes with body orientation on all axis. The value of 1 means full stabilization. The value between 0 to 1 acts as a weight for fixed angles specified (in degrees, in world-frame) in Pitch , Roll and Yaw elements and orientation of the vehicle body. When any of the angles is omitted from json or set to NaN, that angle is not stabilized (i.e. it moves along with vehicle body).","title":"Gimbal"},{"location":"settings/#vehicles-settings","text":"Each simulation mode will go through the list of vehicles specified in this setting and create the ones that has \"AutoCreate\": true . Each vehicle specified in this setting has key which becomes the name of the vehicle. If \"Vehicles\" element is missing then this list is populated with default car named \"PhysXCar\" and default multirotor named \"SimpleFlight\".","title":"Vehicles Settings"},{"location":"settings/#common-vehicle-setting","text":"VehicleType : This could be either PhysXCar , SimpleFlight , PX4Multirotor or ComputerVision . There is no default value therefore this element must be specified. PawnPath : This allows to override the pawn blueprint to use for the vehicle. For example, you may create new pawn blueprint derived from ACarPawn for a warehouse robot in your own project outside the AirSim code and then specify its path here. See also PawnPaths . DefaultVehicleState : Possible value for multirotors is Armed or Disarmed . AutoCreate : If true then this vehicle would be spawned (if supported by selected sim mode). RC : This sub-element allows to specify which remote controller to use for vehicle using RemoteControlID . The value of -1 means use keyboard (not supported yet for multirotors). The value >= 0 specifies one of many remote controllers connected to the system. The list of available RCs can be seen in Game Controllers panel in Windows, for example. X, Y, Z, Yaw, Roll, Pitch : These elements allows you to specify the initial position and orientation of the vehicle. Position is in NED coordinates in SI units with origin set to Player Start location in Unreal environment. The orientation is specified in degrees. IsFpvVehicle : This setting allows to specify which vehicle camera will follow and the view that will be shown when ViewMode is set to Fpv. By default, AirSim selects the first vehicle in settings as FPV vehicle. Cameras : This element specifies camera settings for vehicle. The key in this element is name of the available camera and the value is same as CameraDefaults as described above. For example, to change FOV for the front center camera to 120 degrees, you can use this for Vehicles setting: \"Vehicles\": { \"FishEyeDrone\": { \"VehicleType\": \"SimpleFlight\", \"Cameras\": { \"front-center\": { \"CaptureSettings\": [ { \"ImageType\": 0, \"FOV_Degrees\": 120 } ] } } } }","title":"Common Vehicle Setting"},{"location":"settings/#using-px4","text":"By default we use simple_flight so you don't have to do separate HITL or SITL setups. We also support \"PX4\" for advanced users. To use PX4 with AirSim, you can use the following for Vehicles setting: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", } }","title":"Using PX4"},{"location":"settings/#additional-px4-settings","text":"The defaults for PX4 is to enable hardware-in-loop setup. There are various other settings available for PX4 as follows with their default values: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"ControlIp\": \"127.0.0.1\", \"ControlPort\": 14580, \"LogViewerHostIp\": \"127.0.0.1\", \"LogViewerPort\": 14388, \"OffboardCompID\": 1, \"OffboardSysID\": 134, \"QgcHostIp\": \"127.0.0.1\", \"QgcPort\": 14550, \"SerialBaudRate\": 115200, \"SerialPort\": \"*\", \"SimCompID\": 42, \"SimSysID\": 142, \"TcpPort\": 4560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14560, \"UseSerial\": true, \"UseTcp\": false, \"VehicleCompID\": 1, \"VehicleSysID\": 135, \"Model\": \"Generic\", \"LocalHostIp\": \"127.0.0.1\" } } These settings define the MavLink SystemId and ComponentId for the Simulator (SimSysID, SimCompID), and for the vehicle (VehicleSysID, VehicleCompID) and the node that allows remote control of the drone from another app this is called the offboard node (OffboardSysID, OffboardCompID). If you want the simulator to also talk to your ground control app (like QGroundControl) you can also set the UDP address for that in case you want to run that on a different machine (QgcHostIp, QgcPort). The default is local host so QGroundControl should \"just work\" if it is running on the same machine. You can connect the simulator to the LogViewer app, provided in this repo, by setting the UDP address for that (LogViewerHostIp, LogViewerPort). And for each flying drone added to the simulator there is a named block of additional settings. In the above you see the default name \"PX4\". You can change this name from the Unreal Editor when you add a new BP_FlyingPawn asset. You will see these properties grouped under the category \"MavLink\". The MavLink node for this pawn can be remote over UDP or it can be connected to a local serial port. If serial then set UseSerial to true, otherwise set UseSerial to false. For serial connections you also need to set the appropriate SerialBaudRate. The default of 115200 works with Pixhawk version 2 over USB. When communicating with the PX4 drone over serial port both the HIL_ messages and vehicle control messages share the same serial port. When communicating over UDP or TCP PX4 requires two separate channels. If UseTcp is false, then UdpIp, UdpPort are used to send HIL_ messages, otherwise the TcpPort is used. TCP support in PX4 was added in 1.9.2 with the lockstep feature because the guarantee of message delivery that TCP provides is required for the proper functioning of lockstep. AirSim becomes a TCP server in that case, and waits for a connection from the PX4 app. The second channel for controlling the vehicle is defined by (ControlIp, ControlPort) and is always a UDP channel.","title":"Additional PX4 Settings"},{"location":"settings/#other-settings","text":"","title":"Other Settings"},{"location":"settings/#enginesound","text":"To turn off the engine sound use setting \"EngineSound\": false . Currently this setting applies only to car.","title":"EngineSound"},{"location":"settings/#pawnpaths","text":"This allows you to specify your own vehicle pawn blueprints, for example, you can replace the default car in AirSim with your own car. Your vehicle BP can reside in Content folder of your own Unreal project (i.e. outside of AirSim plugin folder). For example, if you have a car BP located in file Content\\MyCar\\MySedanBP.uasset in your project then you can set \"DefaultCar\": {\"PawnBP\":\"Class'/Game/MyCar/MySedanBP.MySedanBP_C'\"} . The XYZ.XYZ_C is a special notation required to specify class for BP XYZ . Please note that your BP must be derived from CarPawn class. By default this is not the case but you can re-parent the BP using the \"Class Settings\" button in toolbar in UE editor after you open the BP and then choosing \"Car Pawn\" for Parent Class settings in Class Options. It is also a good idea to disable \"Auto Possess Player\" and \"Auto Possess AI\" as well as set AI Controller Class to None in BP details. Please make sure your asset is included for cooking in packaging options if you are creating binary.","title":"PawnPaths"},{"location":"settings/#physicsenginename","text":"For cars, we support only PhysX for now (regardless of value in this setting). For multirotors, we support \"FastPhysicsEngine\" only.","title":"PhysicsEngineName"},{"location":"settings/#localhostip-setting","text":"Now when connecting to remote machines you may need to pick a specific Ethernet adapter to reach those machines, for example, it might be over Ethernet or over Wi-Fi, or some other special virtual adapter or a VPN. Your PC may have multiple networks, and those networks might not be allowed to talk to each other, in which case the UDP messages from one network will not get through to the others. So the LocalHostIp allows you to configure how you are reaching those machines. The default of 127.0.0.1 is not able to reach external machines, this default is only used when everything you are talking to is contained on a single PC.","title":"LocalHostIp Setting"},{"location":"settings/#speedunitfactor","text":"Unit conversion factor for speed related to m/s , default is 1. Used in conjunction with SpeedUnitLabel. This may be only used for display purposes for example on-display speed when car is being driven. For example, to get speed in miles/hr use factor 2.23694.","title":"SpeedUnitFactor"},{"location":"settings/#speedunitlabel","text":"Unit label for speed, default is m/s . Used in conjunction with SpeedUnitFactor.","title":"SpeedUnitLabel"},{"location":"simple_flight/","text":"simple_flight # If you don't know what flight controller does than see What is Flight Controller? . AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In future, we also plan to support ROSFlight and Hackflight . Advantages # The advantage of using simple_flight is zero additional setup you need to do and it \"just works\". Also, simple_flight uses steppable clock which means you can pause the simulation and things are not at mercy of high variance low precision clock that operating system provides. Further, simple_flight is simple, cross platform and 100% header-only dependency-free C++ code which means you can literally step through from simulator to inside flight controller code within same code base! Design # Normally flight controllers are designed to run on actual hardware on vehicles and their support for running in simulator varies widely. They are often fairly difficult to configure for non-expert users and typically have complex build usually lacking cross platform support. All these problems have played significant part in design of simple_flight. simple_flight is designed from ground up as library with clean interface that can work onboard the vehicle as well as simulator. The core principle is that flight controller has no way to specify special simulation mode and there for it has no way to know if it is running under simulation or real vehicle. We thus view flight controller simply as collection of algorithms packaged in a library. Another key emphasis is to develop this code as dependency free header-only pure standard C++11 code. This means there is no special build required to compile simple_flight. You just copy its source code to any project you wish and it just works. Control # simple_flight can control vehicle by taking in desired input as angle rate, angle level, velocity or position. Each axis of control can be specified with one of these modes. Internally simple_flight uses cascade of PID controllers to finally generate actuator signals. This means position PID drives velocity PID which drives angle level PID which finally drives angle rate PID. State Estimation # In current release we are using ground truth from simulator for our state estimation. We plan to add complimentary filter based state estimation for angular velocity and orientation using 2 sensors (gyroscope, accelerometer) in near future. In more longer term, we plan to integrate another library to do velocity and position estimation using 4 sensors (gyroscope, accelerometer, magnetometer and barometer) using EKF. If you have experience this area than we encourage you to engage with us and contribute! Supported Boards # Currently we have implemented simple_flight interfaces for simulated board. We plan to implement it for Pixhawk V2 board and possibly Naze32 board. We expect all our code to remain unchanged and the implementation would mainly involve adding drivers for various sensors, handling ISRs and managing other board specific details. If you have experience this area than we encourage you to engage with us and contribute! Configuration # To have AirSim use simple_flight, you can specify it in settings.json as shown below. Note that this is default so you don't have to do it explicitly. \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", } } By default, vehicle using simple_flight is already armed which is why you would see propellers spinning. However if you don't want that than set DefaultVehicleState to Inactive like this: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Inactive\" } } In this case, you will need to either manually arm using RC sticks in down inward position or using APIs. For safety reasons, flight controllers would disallow API control unless human operator has consented using a switch on RC. Also, when RC control is lost, vehicle should disable API control and enter hover mode for safety reasons. To simplify things a bit, simple_flight enables API control without human consent using RC and even when RC is not detected by default however you can change this using following setting: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": true } } } Finally, simple_flight uses steppable clock by default which means clock advances when simulator tells it to advance (unlike wall clock which advances strictly according to passage of time). This means clock can be paused, for example, if code hits the break point and there is zero variance in clock (clock APIs provides by operating system might have significant variance unless its \"real time\" OS). If you want simple_flight to use wall clock instead than use following settings: \"ClockType\": \"ScalableClock\"","title":"Simple Flight"},{"location":"simple_flight/#simple_flight","text":"If you don't know what flight controller does than see What is Flight Controller? . AirSim has built-in flight controller called simple_flight and it is used by default. You don't need to do anything to use or configure it. AirSim also supports PX4 as another flight controller for advanced users. In future, we also plan to support ROSFlight and Hackflight .","title":"simple_flight"},{"location":"simple_flight/#advantages","text":"The advantage of using simple_flight is zero additional setup you need to do and it \"just works\". Also, simple_flight uses steppable clock which means you can pause the simulation and things are not at mercy of high variance low precision clock that operating system provides. Further, simple_flight is simple, cross platform and 100% header-only dependency-free C++ code which means you can literally step through from simulator to inside flight controller code within same code base!","title":"Advantages"},{"location":"simple_flight/#design","text":"Normally flight controllers are designed to run on actual hardware on vehicles and their support for running in simulator varies widely. They are often fairly difficult to configure for non-expert users and typically have complex build usually lacking cross platform support. All these problems have played significant part in design of simple_flight. simple_flight is designed from ground up as library with clean interface that can work onboard the vehicle as well as simulator. The core principle is that flight controller has no way to specify special simulation mode and there for it has no way to know if it is running under simulation or real vehicle. We thus view flight controller simply as collection of algorithms packaged in a library. Another key emphasis is to develop this code as dependency free header-only pure standard C++11 code. This means there is no special build required to compile simple_flight. You just copy its source code to any project you wish and it just works.","title":"Design"},{"location":"simple_flight/#control","text":"simple_flight can control vehicle by taking in desired input as angle rate, angle level, velocity or position. Each axis of control can be specified with one of these modes. Internally simple_flight uses cascade of PID controllers to finally generate actuator signals. This means position PID drives velocity PID which drives angle level PID which finally drives angle rate PID.","title":"Control"},{"location":"simple_flight/#state-estimation","text":"In current release we are using ground truth from simulator for our state estimation. We plan to add complimentary filter based state estimation for angular velocity and orientation using 2 sensors (gyroscope, accelerometer) in near future. In more longer term, we plan to integrate another library to do velocity and position estimation using 4 sensors (gyroscope, accelerometer, magnetometer and barometer) using EKF. If you have experience this area than we encourage you to engage with us and contribute!","title":"State Estimation"},{"location":"simple_flight/#supported-boards","text":"Currently we have implemented simple_flight interfaces for simulated board. We plan to implement it for Pixhawk V2 board and possibly Naze32 board. We expect all our code to remain unchanged and the implementation would mainly involve adding drivers for various sensors, handling ISRs and managing other board specific details. If you have experience this area than we encourage you to engage with us and contribute!","title":"Supported Boards"},{"location":"simple_flight/#configuration","text":"To have AirSim use simple_flight, you can specify it in settings.json as shown below. Note that this is default so you don't have to do it explicitly. \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", } } By default, vehicle using simple_flight is already armed which is why you would see propellers spinning. However if you don't want that than set DefaultVehicleState to Inactive like this: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"DefaultVehicleState\": \"Inactive\" } } In this case, you will need to either manually arm using RC sticks in down inward position or using APIs. For safety reasons, flight controllers would disallow API control unless human operator has consented using a switch on RC. Also, when RC control is lost, vehicle should disable API control and enter hover mode for safety reasons. To simplify things a bit, simple_flight enables API control without human consent using RC and even when RC is not detected by default however you can change this using following setting: \"Vehicles\": { \"SimpleFlight\": { \"VehicleType\": \"SimpleFlight\", \"AllowAPIAlways\": true, \"RC\": { \"RemoteControlID\": 0, \"AllowAPIWhenDisconnected\": true } } } Finally, simple_flight uses steppable clock by default which means clock advances when simulator tells it to advance (unlike wall clock which advances strictly according to passage of time). This means clock can be paused, for example, if code hits the break point and there is zero variance in clock (clock APIs provides by operating system might have significant variance unless its \"real time\" OS). If you want simple_flight to use wall clock instead than use following settings: \"ClockType\": \"ScalableClock\"","title":"Configuration"},{"location":"steering_wheel_installation/","text":"Logitech G920 Steering Wheel Installation # To use Logitech G920 steering wheel with AirSim follow these steps: Connect the steering wheel to the computer and wait until drivers installation complete. Install Logitech Gaming Software from here Before debug, you\u2019ll have to normalize the values in AirSim code. Perform this changes in CarPawn.cpp (according to the current update in the git): In line 382, change \u201cVal\u201d to \u201c1 \u2013 Val\u201d. (the complementary value in the range [0.0,1.0]). In line 388, change \u201cVal\u201d to \u201c5Val - 2.5\u201d (Change the range of the given input from [0.0,1.0] to [-1.0,1.0]). In line 404, change \u201cVal\u201d to \u201c4(1 \u2013 Val)\u201d. (the complementary value in the range [0.0,1.0]). Debug AirSim project (while the steering wheel is connected \u2013 it\u2019s important). On Unreal Editor, go to Edit->plugins->input devices and enable \u201cWindows RawInput\u201d. Go to Edit->Project Settings->Raw Input, and add new device configuration: Vendor ID: 0x046d (In case of Logitech G920, otherwise you might need to check it). Product ID: 0xc261 (In case of Logitech G920, otherwise you might need to check it). Under \u201cAxis Properties\u201d, make sure that \u201cGenericUSBController Axis 2\u201d, \u201cGenericUSBController Axis 4\u201d and \u201cGenericUSBController Axis 5\u201d are all enabled with an offset of 1.0. Explanation: axis 2 is responsible for steering movement, axis 4 is for brake and axis 5 is for gas. If you need to configure the clutch, it\u2019s on axis 3. Go to Edit->Project Settings->Input, Under Bindings in \u201cAxis Mappings\u201d: Remove existing mappings from the groups \u201cMoveRight\u201d and \u201cMoveForward\u201d. Add new axis mapping to the group \u201cMoveRight\u201d, use GenericUSBController axis 2 with a scale of 1.0. Add new axis mapping to the group \u201cMoveForward\u201d, use GenericUSBController axis 5 with a scale of 1.0. Add a new group of axis mappings, name it \u201cFootBrake\u201d and add new axis mapping to this group, use GenericUSBController axis 4 with a scale of 1.0. Play and drive ! Pay Attention # Notice that in the first time we \"play\" after debug, we need to touch the wheel to \u201creset\u201d the values. Tip # In the gaming software, you can configure buttons as keyboard shortcuts, we used it to configure a shortcut to record dataset or to play in full screen.","title":"Steering Wheel"},{"location":"steering_wheel_installation/#logitech-g920-steering-wheel-installation","text":"To use Logitech G920 steering wheel with AirSim follow these steps: Connect the steering wheel to the computer and wait until drivers installation complete. Install Logitech Gaming Software from here Before debug, you\u2019ll have to normalize the values in AirSim code. Perform this changes in CarPawn.cpp (according to the current update in the git): In line 382, change \u201cVal\u201d to \u201c1 \u2013 Val\u201d. (the complementary value in the range [0.0,1.0]). In line 388, change \u201cVal\u201d to \u201c5Val - 2.5\u201d (Change the range of the given input from [0.0,1.0] to [-1.0,1.0]). In line 404, change \u201cVal\u201d to \u201c4(1 \u2013 Val)\u201d. (the complementary value in the range [0.0,1.0]). Debug AirSim project (while the steering wheel is connected \u2013 it\u2019s important). On Unreal Editor, go to Edit->plugins->input devices and enable \u201cWindows RawInput\u201d. Go to Edit->Project Settings->Raw Input, and add new device configuration: Vendor ID: 0x046d (In case of Logitech G920, otherwise you might need to check it). Product ID: 0xc261 (In case of Logitech G920, otherwise you might need to check it). Under \u201cAxis Properties\u201d, make sure that \u201cGenericUSBController Axis 2\u201d, \u201cGenericUSBController Axis 4\u201d and \u201cGenericUSBController Axis 5\u201d are all enabled with an offset of 1.0. Explanation: axis 2 is responsible for steering movement, axis 4 is for brake and axis 5 is for gas. If you need to configure the clutch, it\u2019s on axis 3. Go to Edit->Project Settings->Input, Under Bindings in \u201cAxis Mappings\u201d: Remove existing mappings from the groups \u201cMoveRight\u201d and \u201cMoveForward\u201d. Add new axis mapping to the group \u201cMoveRight\u201d, use GenericUSBController axis 2 with a scale of 1.0. Add new axis mapping to the group \u201cMoveForward\u201d, use GenericUSBController axis 5 with a scale of 1.0. Add a new group of axis mappings, name it \u201cFootBrake\u201d and add new axis mapping to this group, use GenericUSBController axis 4 with a scale of 1.0. Play and drive !","title":"Logitech G920 Steering Wheel Installation"},{"location":"steering_wheel_installation/#pay-attention","text":"Notice that in the first time we \"play\" after debug, we need to touch the wheel to \u201creset\u201d the values.","title":"Pay Attention"},{"location":"steering_wheel_installation/#tip","text":"In the gaming software, you can configure buttons as keyboard shortcuts, we used it to configure a shortcut to record dataset or to play in full screen.","title":"Tip"},{"location":"unity_api_support/","text":"Api Car Multirotor reset Supported Supported enableApiControl Supported Supported armDisarm Supported Supported confirmConnection Supported Supported isApiControlEnabled Supported Supported ping Supported Supported simGetObjectPose Supported Supported simSetObjectPose Supported Supported pause Supported Supported continueForTime Supported Supported simGetCollisionInfo Supported Supported setCarControls Supported NA getCarState Supported NA moveByAngleThrottleAsync NA Supported getMultirotorState NA Supported takeoffAsync NA Supported waitOnLastTask Supported Supported cancelLastTask Supported Supported simGetGroundTruthKinematics Supported Supported getCameraInfo Supported Supported getClientVersion Supported Supported getCollisionInfo Supported Supported getGpsLocation TODO TODO getHomeGeoPoint TODO TODO getLandedState TODO TODO getLidarData TODO TODO getMinRequiredClientVersion Supported Supported getMinRequiredServerVersion Supported Supported getOrientation TODO TODO getPosition TODO TODO getServerVersion Supported Supported getVelocity TODO TODO goHome TODO TODO hover TODO TODO land TODO TODO moveByAngleThrottle TODO TODO moveByAngleZ TODO TODO moveByManual TODO TODO moveByVelocity TODO TODO moveByVelocityZ TODO TODO moveOnPath TODO TODO moveToPosition TODO TODO moveToZ TODO TODO rotateByYawRate TODO TODO rotateToYaw TODO TODO setCameraOrientation TODO TODO setRCData TODO TODO simContinueForTime Supported Supported simGetCameraInfo TODO TODO simGetGroundTruthEnvironment TODO TODO simGetImage TODO TODO simGetImages Supported Supported simGetPose TODO TODO simGetSegmentationObjectID TODO TODO simGetVehiclePose TODO TODO simIsPause TODO TODO simPause TODO TODO simPrintLogMessage TODO TODO simSetCameraOrientation TODO TODO simSetPose TODO TODO simSetSegmentationObjectID TODO TODO simSetVehiclePose TODO TODO takeoff TODO TODO goHomeAsync NA TODO hoverAsync NA TODO landAsync NA TODO moveByAngleZAsync NA TODO moveByManualAsync NA TODO moveByRC NA TODO moveByVelocityAsync NA Supported moveByVelocityZAsync NA TODO moveOnPathAsync NA TODO moveToPositionAsync NA TODO moveToZAsync NA TODO rotateByYawRateAsync NA TODO rotateToYawAsync NA TODO","title":"Unity APIs"},{"location":"unreal_blocks/","text":"Setup Blocks Environment for AirSim # Blocks environment is available in repo in folder Unreal/Environments/Blocks and is designed to be lightweight in size. That means its very basic but fast. Here are quick steps to get Blocks environment up and running: Windows # Make sure you have installed Unreal and built AirSim . Navigate to folder AirSim\\Unreal\\Environments\\Blocks and run update_from_git.bat . Double click on generated .sln file to open in Visual Studio 2019 or newer. Make sure Blocks project is the startup project, build configuration is set to DebugGame_Editor and Win64 . Hit F5 to run. Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim . Changing Code and Rebuilding # For Windows, you can just change the code in Visual Studio, press F5 and re-run. There are few batch files available in folder AirSim\\Unreal\\Environments\\Blocks that lets you sync code, clean etc. Linux # Make sure you have built the Unreal Engine and AirSim . Navigate to your UnrealEngine repo folder and run Engine/Binaries/Linux/UE4Editor which will start Unreal Editor. On first start you might not see any projects in UE4 editor. Click on Projects tab, Browse button and then navigate to AirSim/Unreal/Environments/Blocks/Blocks.uproject . If you get prompted for incompatible version and conversion, select In-place conversion which is usually under \"More\" options. If you get prompted for missing modules, make sure to select No so you don't exit. Finally, when prompted with building AirSim, select Yes. Now it might take a while so go get some coffee :). Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim . Changing Code and Rebuilding # For Linux, make code changes in AirLib or Unreal/Plugins folder and then run ./build.sh to rebuild. This step also copies the build output to Blocks sample project. You can then follow above steps again to re-run. Chosing Your Vehicle: Car or Multirotor # By default AirSim spawns multirotor. You can easily change this to car and use all of AirSim goodies. Please see using car guide. FAQ # I see warnings about like \"_BuitData\" file is missing. # These are intermediate files and you can safely ignore it.","title":"Blocks Environment"},{"location":"unreal_blocks/#setup-blocks-environment-for-airsim","text":"Blocks environment is available in repo in folder Unreal/Environments/Blocks and is designed to be lightweight in size. That means its very basic but fast. Here are quick steps to get Blocks environment up and running:","title":"Setup Blocks Environment for AirSim"},{"location":"unreal_blocks/#windows","text":"Make sure you have installed Unreal and built AirSim . Navigate to folder AirSim\\Unreal\\Environments\\Blocks and run update_from_git.bat . Double click on generated .sln file to open in Visual Studio 2019 or newer. Make sure Blocks project is the startup project, build configuration is set to DebugGame_Editor and Win64 . Hit F5 to run. Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim .","title":"Windows"},{"location":"unreal_blocks/#changing-code-and-rebuilding","text":"For Windows, you can just change the code in Visual Studio, press F5 and re-run. There are few batch files available in folder AirSim\\Unreal\\Environments\\Blocks that lets you sync code, clean etc.","title":"Changing Code and Rebuilding"},{"location":"unreal_blocks/#linux","text":"Make sure you have built the Unreal Engine and AirSim . Navigate to your UnrealEngine repo folder and run Engine/Binaries/Linux/UE4Editor which will start Unreal Editor. On first start you might not see any projects in UE4 editor. Click on Projects tab, Browse button and then navigate to AirSim/Unreal/Environments/Blocks/Blocks.uproject . If you get prompted for incompatible version and conversion, select In-place conversion which is usually under \"More\" options. If you get prompted for missing modules, make sure to select No so you don't exit. Finally, when prompted with building AirSim, select Yes. Now it might take a while so go get some coffee :). Press the Play button in Unreal Editor and you will see something like in below video. Also see how to use AirSim .","title":"Linux"},{"location":"unreal_blocks/#changing-code-and-rebuilding_1","text":"For Linux, make code changes in AirLib or Unreal/Plugins folder and then run ./build.sh to rebuild. This step also copies the build output to Blocks sample project. You can then follow above steps again to re-run.","title":"Changing Code and Rebuilding"},{"location":"unreal_blocks/#chosing-your-vehicle-car-or-multirotor","text":"By default AirSim spawns multirotor. You can easily change this to car and use all of AirSim goodies. Please see using car guide.","title":"Chosing Your Vehicle: Car or Multirotor"},{"location":"unreal_blocks/#faq","text":"","title":"FAQ"},{"location":"unreal_blocks/#i-see-warnings-about-like-_buitdata-file-is-missing","text":"These are intermediate files and you can safely ignore it.","title":"I see warnings about like \"_BuitData\" file is missing."},{"location":"unreal_custenv/","text":"Creating and Setting Up Unreal Environment # This page contains the complete instructions start to finish for setting up Unreal environment with AirSim. The Unreal Marketplace has several environment available that you can start using in just few minutes. It is also possible to use environments available on websites such as turbosquid.com or cgitrader.com with bit more effort (here's tutorial video ). In addition there also several free environments available. Below we will use a freely downloadable environment from Unreal Marketplace called Landscape Mountain but the steps are same for any other environments. You can also view these steps performed in Unreal AirSim Setup Video . Note for Linux Users # There is no Epic Games Launcher for Linux which means that if you need to create custom environment, you will need Windows machine to do that. Once you have Unreal project folder, just copy it over to your Linux machine. Step by Step Instructions # Make sure AirSim is built and Unreal 4.24 is installed as described in build instructions . In Epic Games Launcher click the Learn tab then scroll down and find Landscape Mountains . Click the Create Project and download this content (~2GB download). Open LandscapeMountains.uproject , it should launch the Unreal Editor. From the File menu select New C++ class , leave default None on the type of class, click Next , leave default name MyClass , and click Create Class . We need to do this because Unreal requires at least one source file in project. It should trigger compile and open up Visual Studio solution LandscapeMountains.sln . Go to your folder for AirSim repo and copy Unreal\\Plugins folder in to your LandscapeMountains folder. This way now your own Unreal project has AirSim plugin. Edit the LandscapeMountains.uproject so that it looks like this { \"FileVersion\": 3, \"EngineAssociation\": \"4.24\", \"Category\": \"Samples\", \"Description\": \"\", \"Modules\": [ { \"Name\": \"LandscapeMountains\", \"Type\": \"Runtime\", \"LoadingPhase\": \"Default\", \"AdditionalDependencies\": [ \"AirSim\" ] } ], \"TargetPlatforms\": [ \"MacNoEditor\", \"WindowsNoEditor\" ], \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ] } Close Visual Studio and the Unreal Editor and right click the LandscapeMountains.uproject in Windows Explorer and select Generate Visual Studio Project Files . This step detects all plugins and source files in your Unreal project and generates .sln file for Visual Studio. Tip: If the Generate Visual Studio Project Files option is missing you may need to reboot your machine for the Unreal Shell extensions to take effect. If it is still missing then open the LandscapeMountains.uproject in the Unreal Editor and select Refresh Visual Studio Project from the File menu. Reopen LandscapeMountains.sln in Visual Studio, and make sure \"DebugGame Editor\" and \"Win64\" build configuration is the active build configuration. Press F5 to run . This will start the Unreal Editor. The Unreal Editor allows you to edit the environment, assets and other game related settings. First thing you want to do in your environment is set up PlayerStart object. In Landscape Mountains environment, PlayerStart object already exist and you can find it in the World Outliner . Make sure its location is setup as shown. This is where AirSim plugin will create and place the vehicle. If its too high up then vehicle will fall down as soon as you press play giving potentially random behavior In Window/World Settings as shown below, set the GameMode Override to AirSimGameMode : Go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. If you don't do this then UE will be slowed down dramatically when UE window loses focus. Be sure to Save these edits. Hit the Play button in the Unreal Editor. See how to use AirSim . Congratulations! You are now running AirSim in your own Unreal environment. Choosing Your Vehicle: Car or Multirotor # By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . Please see using car guide. Updating Your Environment to Latest Version of AirSim # Once you have your environment using above instructions, you should frequently update your local AirSim code to latest version from GitHub. Below are the instructions to do this: First put clean.bat (or clean.sh for Linux users) in the root folder of your environment. Run this file to clean up all intermediate files in your Unreal project. Do git pull in your AirSim repo followed by build.cmd (or ./build.sh for Linux users). Replace [your project]/Plugins folder with AirSim/Unreal/Plugins folder. Right click on your .uproject file and chose \"Generate Visual Studio project files\" option. This is not required for Linux. FAQ # What are other cool environments? # Unreal Marketplace has dozens of prebuilt extra-ordinarily detailed environments ranging from Moon to Mars and everything in between. The one we have used for testing is called Modular Neighborhood Pack but you can use any environment. Another free environment is Infinity Blade series . Alternatively, if you look under the Learn tab in Epic Game Launcher, you will find many free samples that you can use. One of our favorites is \"A Boy and His Kite\" which is a 100 square miles of highly detailed environment (caution: you will need very beefy PC to run it!). When I press Play button some kind of video starts instead of my vehicle. # If the environment comes with MatineeActor, delete it to avoid any startup demo sequences. There might be other ways to remove it as well, for example, click on Blueprints button, then Level Blueprint and then look at Begin Play event in Event Graph. You might want to disconnect any connections that may be starting \"matinee\". Is there easy way to sync code in my Unreal project with code in AirSim repo? # Sure, there is! You can find bunch of .bat files (for linux, .sh ) in AirSim\\Unreal\\Environments\\Blocks . Just copy them over to your own Unreal project. Most of these are quite simple and self explanatory. I get some error about map. # You might have to set default map for your project. For example, if you are using Modular Neighborhood Pack, set the Editor Starter Map as well as Game Default Map to Demo_Map in Project Settings > Maps & Modes. I see \"Add to project\" option for environment but not \"Create project\" option. # In this case, create a new blank C++ project with no Starter Content and add your environment in to it. I already have my own Unreal project. How do I use AirSim with it? # Copy the Unreal\\Plugins folder from the build you did in the above section into the root of your Unreal project's folder. In your Unreal project's .uproject file, add the key AdditionalDependencies to the \"Modules\" object as we showed in the LandscapeMountains.uproject above. \"AdditionalDependencies\": [ \"AirSim\" ] and the Plugins section to the top level object: \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ]","title":"Custom Unreal Environment"},{"location":"unreal_custenv/#creating-and-setting-up-unreal-environment","text":"This page contains the complete instructions start to finish for setting up Unreal environment with AirSim. The Unreal Marketplace has several environment available that you can start using in just few minutes. It is also possible to use environments available on websites such as turbosquid.com or cgitrader.com with bit more effort (here's tutorial video ). In addition there also several free environments available. Below we will use a freely downloadable environment from Unreal Marketplace called Landscape Mountain but the steps are same for any other environments. You can also view these steps performed in Unreal AirSim Setup Video .","title":"Creating and Setting Up Unreal Environment"},{"location":"unreal_custenv/#note-for-linux-users","text":"There is no Epic Games Launcher for Linux which means that if you need to create custom environment, you will need Windows machine to do that. Once you have Unreal project folder, just copy it over to your Linux machine.","title":"Note for Linux Users"},{"location":"unreal_custenv/#step-by-step-instructions","text":"Make sure AirSim is built and Unreal 4.24 is installed as described in build instructions . In Epic Games Launcher click the Learn tab then scroll down and find Landscape Mountains . Click the Create Project and download this content (~2GB download). Open LandscapeMountains.uproject , it should launch the Unreal Editor. From the File menu select New C++ class , leave default None on the type of class, click Next , leave default name MyClass , and click Create Class . We need to do this because Unreal requires at least one source file in project. It should trigger compile and open up Visual Studio solution LandscapeMountains.sln . Go to your folder for AirSim repo and copy Unreal\\Plugins folder in to your LandscapeMountains folder. This way now your own Unreal project has AirSim plugin. Edit the LandscapeMountains.uproject so that it looks like this { \"FileVersion\": 3, \"EngineAssociation\": \"4.24\", \"Category\": \"Samples\", \"Description\": \"\", \"Modules\": [ { \"Name\": \"LandscapeMountains\", \"Type\": \"Runtime\", \"LoadingPhase\": \"Default\", \"AdditionalDependencies\": [ \"AirSim\" ] } ], \"TargetPlatforms\": [ \"MacNoEditor\", \"WindowsNoEditor\" ], \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ] } Close Visual Studio and the Unreal Editor and right click the LandscapeMountains.uproject in Windows Explorer and select Generate Visual Studio Project Files . This step detects all plugins and source files in your Unreal project and generates .sln file for Visual Studio. Tip: If the Generate Visual Studio Project Files option is missing you may need to reboot your machine for the Unreal Shell extensions to take effect. If it is still missing then open the LandscapeMountains.uproject in the Unreal Editor and select Refresh Visual Studio Project from the File menu. Reopen LandscapeMountains.sln in Visual Studio, and make sure \"DebugGame Editor\" and \"Win64\" build configuration is the active build configuration. Press F5 to run . This will start the Unreal Editor. The Unreal Editor allows you to edit the environment, assets and other game related settings. First thing you want to do in your environment is set up PlayerStart object. In Landscape Mountains environment, PlayerStart object already exist and you can find it in the World Outliner . Make sure its location is setup as shown. This is where AirSim plugin will create and place the vehicle. If its too high up then vehicle will fall down as soon as you press play giving potentially random behavior In Window/World Settings as shown below, set the GameMode Override to AirSimGameMode : Go to 'Edit->Editor Preferences' in Unreal Editor, in the 'Search' box type 'CPU' and ensure that the 'Use Less CPU when in Background' is unchecked. If you don't do this then UE will be slowed down dramatically when UE window loses focus. Be sure to Save these edits. Hit the Play button in the Unreal Editor. See how to use AirSim . Congratulations! You are now running AirSim in your own Unreal environment.","title":"Step by Step Instructions"},{"location":"unreal_custenv/#choosing-your-vehicle-car-or-multirotor","text":"By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . Please see using car guide.","title":"Choosing Your Vehicle: Car or Multirotor"},{"location":"unreal_custenv/#updating-your-environment-to-latest-version-of-airsim","text":"Once you have your environment using above instructions, you should frequently update your local AirSim code to latest version from GitHub. Below are the instructions to do this: First put clean.bat (or clean.sh for Linux users) in the root folder of your environment. Run this file to clean up all intermediate files in your Unreal project. Do git pull in your AirSim repo followed by build.cmd (or ./build.sh for Linux users). Replace [your project]/Plugins folder with AirSim/Unreal/Plugins folder. Right click on your .uproject file and chose \"Generate Visual Studio project files\" option. This is not required for Linux.","title":"Updating Your Environment to Latest Version of AirSim"},{"location":"unreal_custenv/#faq","text":"","title":"FAQ"},{"location":"unreal_custenv/#what-are-other-cool-environments","text":"Unreal Marketplace has dozens of prebuilt extra-ordinarily detailed environments ranging from Moon to Mars and everything in between. The one we have used for testing is called Modular Neighborhood Pack but you can use any environment. Another free environment is Infinity Blade series . Alternatively, if you look under the Learn tab in Epic Game Launcher, you will find many free samples that you can use. One of our favorites is \"A Boy and His Kite\" which is a 100 square miles of highly detailed environment (caution: you will need very beefy PC to run it!).","title":"What are other cool environments?"},{"location":"unreal_custenv/#when-i-press-play-button-some-kind-of-video-starts-instead-of-my-vehicle","text":"If the environment comes with MatineeActor, delete it to avoid any startup demo sequences. There might be other ways to remove it as well, for example, click on Blueprints button, then Level Blueprint and then look at Begin Play event in Event Graph. You might want to disconnect any connections that may be starting \"matinee\".","title":"When I press Play button some kind of video starts instead of my vehicle."},{"location":"unreal_custenv/#is-there-easy-way-to-sync-code-in-my-unreal-project-with-code-in-airsim-repo","text":"Sure, there is! You can find bunch of .bat files (for linux, .sh ) in AirSim\\Unreal\\Environments\\Blocks . Just copy them over to your own Unreal project. Most of these are quite simple and self explanatory.","title":"Is there easy way to sync code in my Unreal project with code in AirSim repo?"},{"location":"unreal_custenv/#i-get-some-error-about-map","text":"You might have to set default map for your project. For example, if you are using Modular Neighborhood Pack, set the Editor Starter Map as well as Game Default Map to Demo_Map in Project Settings > Maps & Modes.","title":"I get some error about map."},{"location":"unreal_custenv/#i-see-add-to-project-option-for-environment-but-not-create-project-option","text":"In this case, create a new blank C++ project with no Starter Content and add your environment in to it.","title":"I see \"Add to project\" option for environment but not \"Create project\" option."},{"location":"unreal_custenv/#i-already-have-my-own-unreal-project-how-do-i-use-airsim-with-it","text":"Copy the Unreal\\Plugins folder from the build you did in the above section into the root of your Unreal project's folder. In your Unreal project's .uproject file, add the key AdditionalDependencies to the \"Modules\" object as we showed in the LandscapeMountains.uproject above. \"AdditionalDependencies\": [ \"AirSim\" ] and the Plugins section to the top level object: \"Plugins\": [ { \"Name\": \"AirSim\", \"Enabled\": true } ]","title":"I already have my own Unreal project. How do I use AirSim with it?"},{"location":"unreal_proj/","text":"Unreal Environment # Setting Up the Unreal Project # Option 1: Built-in Blocks Environment # To get up and running fast, you can use the Blocks project that already comes with AirSim. This is not very highly detailed environment to keep the repo size reasonable but we use it for various testing all the times and it is the easiest way to get your feet wet in this strange land. Follow these quick steps . Option 2: Create Your Own Unreal Environment # If you want to setup photo-realistic high quality environments, then you will need to create your own Unreal project. This is little bit more involved but worthwhile! Follow this step-by-step guide . Changing Code and Development Workflow # To see how you can change and test AirSim code, please read our recommended development workflow .","title":"Setting up Unreal Environment"},{"location":"unreal_proj/#unreal-environment","text":"","title":"Unreal Environment"},{"location":"unreal_proj/#setting-up-the-unreal-project","text":"","title":"Setting Up the Unreal Project"},{"location":"unreal_proj/#option-1-built-in-blocks-environment","text":"To get up and running fast, you can use the Blocks project that already comes with AirSim. This is not very highly detailed environment to keep the repo size reasonable but we use it for various testing all the times and it is the easiest way to get your feet wet in this strange land. Follow these quick steps .","title":"Option 1: Built-in Blocks Environment"},{"location":"unreal_proj/#option-2-create-your-own-unreal-environment","text":"If you want to setup photo-realistic high quality environments, then you will need to create your own Unreal project. This is little bit more involved but worthwhile! Follow this step-by-step guide .","title":"Option 2: Create Your Own Unreal Environment"},{"location":"unreal_proj/#changing-code-and-development-workflow","text":"To see how you can change and test AirSim code, please read our recommended development workflow .","title":"Changing Code and Development Workflow"},{"location":"unreal_upgrade/","text":"Upgrading to Unreal Engine 4.24 # These instructions apply if you are already using AirSim on Unreal Engine 4.16. If you have never installed AirSim, please see How to get it . Caution: The below steps will delete any of your unsaved work in AirSim or Unreal folder. Do this first # For Windows Users # Install Visual Studio 2019 with VC++, Python and C#. Install UE 4.24 through Epic Games Launcher. Start x64 Native Tools Command Prompt for VS 2019 and navigate to AirSim repo. Run clean_rebuild.bat to remove all unchecked/extra stuff and rebuild everything. See also Build AirSim on Windows for more information. For Linux Users # From your AirSim repo folder, run 'clean_rebuild.sh`. Rename or delete your existing folder for Unreal Engine. Follow step 1 and 2 to install Unreal Engine 4.24 . See also Build AirSim on Linux for more information. Upgrading Your Custom Unreal Project # If you have your own Unreal project created in an older version of Unreal Engine then you need to upgrade your project to Unreal 4.24. To do this, Open .uproject file and look for the line \"EngineAssociation\" and make sure it reads like \"EngineAssociation\": \"4.24\" . Delete Plugins/AirSim folder in your Unreal project's folder. Go to your AirSim repo folder and copy Unreal\\Plugins folder to your Unreal project's folder. Copy .bat (or .sh for Linux) from Unreal\\Environments\\Blocks to your project's folder. Run clean.bat (or clean.sh for Linux) followed by GenerateProjectFiles.bat (only for Windows). FAQ # I have an Unreal project that is older than 4.16. How do I upgrade it? # Option 1: Just Recreate Project # If your project doesn't have any code or assets other than environment you downloaded then you can also simply recreate the project in Unreal 4.24 Editor and then copy Plugins folder from AirSim/Unreal/Plugins . Option 2: Modify Few Files # Unreal versions newer than Unreal 4.15 has breaking changes. So you need to modify your .Build.cs and .Target.cs which you can find in the Source folder of your Unreal project. So what are those changes? Below is the gist of it but you should really refer to Unreal's official 4.16 transition post . In your project's *.Target.cs # Change the contructor from, public MyProjectTarget(TargetInfo Target) to public MyProjectTarget(TargetInfo Target) : base(Target) Remove SetupBinaries method if you have one and instead add following line in contructor above: ExtraModuleNames.AddRange(new string[] { \"MyProject\" }); In your project's *.Build.cs # Change the constructor from public MyProject(TargetInfo Target) to public MyProject(ReadOnlyTargetRules Target) : base(Target) . And finally... # Follow above steps to continue the upgrade. The warning box might show only \"Open Copy\" button. Don't click that. Instead, click on More Options which will reveal more buttons. Choose Convert-In-Place option . Caution: Always keep backup of your project first! If you don't have anything nasty, in place conversion should go through and you are now on the new version of Unreal.","title":"Upgrading Unreal"},{"location":"unreal_upgrade/#upgrading-to-unreal-engine-424","text":"These instructions apply if you are already using AirSim on Unreal Engine 4.16. If you have never installed AirSim, please see How to get it . Caution: The below steps will delete any of your unsaved work in AirSim or Unreal folder.","title":"Upgrading to Unreal Engine 4.24"},{"location":"unreal_upgrade/#do-this-first","text":"","title":"Do this first"},{"location":"unreal_upgrade/#for-windows-users","text":"Install Visual Studio 2019 with VC++, Python and C#. Install UE 4.24 through Epic Games Launcher. Start x64 Native Tools Command Prompt for VS 2019 and navigate to AirSim repo. Run clean_rebuild.bat to remove all unchecked/extra stuff and rebuild everything. See also Build AirSim on Windows for more information.","title":"For Windows Users"},{"location":"unreal_upgrade/#for-linux-users","text":"From your AirSim repo folder, run 'clean_rebuild.sh`. Rename or delete your existing folder for Unreal Engine. Follow step 1 and 2 to install Unreal Engine 4.24 . See also Build AirSim on Linux for more information.","title":"For Linux Users"},{"location":"unreal_upgrade/#upgrading-your-custom-unreal-project","text":"If you have your own Unreal project created in an older version of Unreal Engine then you need to upgrade your project to Unreal 4.24. To do this, Open .uproject file and look for the line \"EngineAssociation\" and make sure it reads like \"EngineAssociation\": \"4.24\" . Delete Plugins/AirSim folder in your Unreal project's folder. Go to your AirSim repo folder and copy Unreal\\Plugins folder to your Unreal project's folder. Copy .bat (or .sh for Linux) from Unreal\\Environments\\Blocks to your project's folder. Run clean.bat (or clean.sh for Linux) followed by GenerateProjectFiles.bat (only for Windows).","title":"Upgrading Your Custom Unreal Project"},{"location":"unreal_upgrade/#faq","text":"","title":"FAQ"},{"location":"unreal_upgrade/#i-have-an-unreal-project-that-is-older-than-416-how-do-i-upgrade-it","text":"","title":"I have an Unreal project that is older than 4.16. How do I upgrade it?"},{"location":"unreal_upgrade/#option-1-just-recreate-project","text":"If your project doesn't have any code or assets other than environment you downloaded then you can also simply recreate the project in Unreal 4.24 Editor and then copy Plugins folder from AirSim/Unreal/Plugins .","title":"Option 1: Just Recreate Project"},{"location":"unreal_upgrade/#option-2-modify-few-files","text":"Unreal versions newer than Unreal 4.15 has breaking changes. So you need to modify your .Build.cs and .Target.cs which you can find in the Source folder of your Unreal project. So what are those changes? Below is the gist of it but you should really refer to Unreal's official 4.16 transition post .","title":"Option 2: Modify Few Files"},{"location":"unreal_upgrade/#in-your-projects-targetcs","text":"Change the contructor from, public MyProjectTarget(TargetInfo Target) to public MyProjectTarget(TargetInfo Target) : base(Target) Remove SetupBinaries method if you have one and instead add following line in contructor above: ExtraModuleNames.AddRange(new string[] { \"MyProject\" });","title":"In your project's *.Target.cs"},{"location":"unreal_upgrade/#in-your-projects-buildcs","text":"Change the constructor from public MyProject(TargetInfo Target) to public MyProject(ReadOnlyTargetRules Target) : base(Target) .","title":"In your project's *.Build.cs"},{"location":"unreal_upgrade/#and-finally","text":"Follow above steps to continue the upgrade. The warning box might show only \"Open Copy\" button. Don't click that. Instead, click on More Options which will reveal more buttons. Choose Convert-In-Place option . Caution: Always keep backup of your project first! If you don't have anything nasty, in place conversion should go through and you are now on the new version of Unreal.","title":"And finally..."},{"location":"upgrade_apis/","text":"Upgrading API Client Code # There have been several API changes in AirSim v1.2 that we hope removes inconsistency, adds future extensibility and presents cleaner interface. Many of these changes are however breaking changes which means you will need to modify your client code that talks to AirSim. Quicker Way # While most changes you need to do in your client code are fairly easy, a quicker way is simply to take a look at the example code such as Hello Drone or Hello Car to get gist of changes. Importing AirSim # Instead of, from AirSimClient import * use this: import airsim Above assumes you have installed AirSim module using, pip install --user airsim If you are running you code from PythonClient folder in repo then you can also do this: import setup_path import airsim Here setup_path.py should exist in your folder and it will set the path of airsim package in PythonClient repo folder. All examples in PythonClient folder uses this method. Using AirSim Classes # As we have everything now in package, you will need to use explicit namespace for AirSim classes like shown below. Instead of, client1 = CarClient() use this: client1 = airsim.CarClient() AirSim Types # We have moved all types in airsim namespace. Instead of, image_type = AirSimImageType.DepthVis d = DrivetrainType.MaxDegreeOfFreedom use this: image_type = airsim.ImageType.DepthVis d = airsim.DrivetrainType.MaxDegreeOfFreedom Getting Images # Nothing new below, it's just combination of above. Note that all APIs that previously took camera_id , now takes camera_name instead. You can take a look at available cameras here. Instead of, responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthVis)]) use this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis)]) Utility Methods # In earlier version, we provided several utility methods as part of AirSimClientBase . These methods are now moved to airsim namespace for more pythonic interface. Instead of, AirSimClientBase.write_png(my_path, img_rgba) AirSimClientBase.wait_key('Press any key') use this: airsim.write_png(my_path, img_rgba) airsim.wait_key('Press any key') Camera Names # AirSim now uses names to reference cameras instead of index numbers. However to retain backward compatibility, these names are aliased with old index numbers as string. Instead of, client.simGetCameraInfo(0) use this: client.simGetCameraInfo(\"0\") # or client.simGetCameraInfo(\"front-center\") Async Methods # For multirotors, AirSim had various methods such as takeoff or moveByVelocityZ that would take long time to complete. All of such methods are now renamed by adding the suffix Async as shown below. Instead of, client.takeoff() client.moveToPosition(-10, 10, -10, 5) use this: client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() Here .join() is a call on Python's Future class to wait for the async call to complete. You can also choose to do some other computation instead while the call is in progress. Simulation-Only Methods # Now we have clear distinction between methods that are only available in simulation from the ones that may be available on actual vehicle. The simulation only methods are prefixed with sim as shown below. getCollisionInfo() is renamed to simGetCollisionInfo() getCameraInfo() is renamed to simGetCameraInfo() setCameraOrientation() is renamed to simSetCameraOrientation() State Information # Previously CarState mixed simulation-only information like kinematics_true . Moving forward, CarState will only contain information that can be obtained in real world. k = car_state.kinematics_true use this: k = car_state.kinematics_estimated # or k = client.simGetGroundTruthKinematics()","title":"Upgrading APIs"},{"location":"upgrade_apis/#upgrading-api-client-code","text":"There have been several API changes in AirSim v1.2 that we hope removes inconsistency, adds future extensibility and presents cleaner interface. Many of these changes are however breaking changes which means you will need to modify your client code that talks to AirSim.","title":"Upgrading API Client Code"},{"location":"upgrade_apis/#quicker-way","text":"While most changes you need to do in your client code are fairly easy, a quicker way is simply to take a look at the example code such as Hello Drone or Hello Car to get gist of changes.","title":"Quicker Way"},{"location":"upgrade_apis/#importing-airsim","text":"Instead of, from AirSimClient import * use this: import airsim Above assumes you have installed AirSim module using, pip install --user airsim If you are running you code from PythonClient folder in repo then you can also do this: import setup_path import airsim Here setup_path.py should exist in your folder and it will set the path of airsim package in PythonClient repo folder. All examples in PythonClient folder uses this method.","title":"Importing AirSim"},{"location":"upgrade_apis/#using-airsim-classes","text":"As we have everything now in package, you will need to use explicit namespace for AirSim classes like shown below. Instead of, client1 = CarClient() use this: client1 = airsim.CarClient()","title":"Using AirSim Classes"},{"location":"upgrade_apis/#airsim-types","text":"We have moved all types in airsim namespace. Instead of, image_type = AirSimImageType.DepthVis d = DrivetrainType.MaxDegreeOfFreedom use this: image_type = airsim.ImageType.DepthVis d = airsim.DrivetrainType.MaxDegreeOfFreedom","title":"AirSim Types"},{"location":"upgrade_apis/#getting-images","text":"Nothing new below, it's just combination of above. Note that all APIs that previously took camera_id , now takes camera_name instead. You can take a look at available cameras here. Instead of, responses = client.simGetImages([ImageRequest(0, AirSimImageType.DepthVis)]) use this: responses = client.simGetImages([airsim.ImageRequest(\"0\", airsim.ImageType.DepthVis)])","title":"Getting Images"},{"location":"upgrade_apis/#utility-methods","text":"In earlier version, we provided several utility methods as part of AirSimClientBase . These methods are now moved to airsim namespace for more pythonic interface. Instead of, AirSimClientBase.write_png(my_path, img_rgba) AirSimClientBase.wait_key('Press any key') use this: airsim.write_png(my_path, img_rgba) airsim.wait_key('Press any key')","title":"Utility Methods"},{"location":"upgrade_apis/#camera-names","text":"AirSim now uses names to reference cameras instead of index numbers. However to retain backward compatibility, these names are aliased with old index numbers as string. Instead of, client.simGetCameraInfo(0) use this: client.simGetCameraInfo(\"0\") # or client.simGetCameraInfo(\"front-center\")","title":"Camera Names"},{"location":"upgrade_apis/#async-methods","text":"For multirotors, AirSim had various methods such as takeoff or moveByVelocityZ that would take long time to complete. All of such methods are now renamed by adding the suffix Async as shown below. Instead of, client.takeoff() client.moveToPosition(-10, 10, -10, 5) use this: client.takeoffAsync().join() client.moveToPositionAsync(-10, 10, -10, 5).join() Here .join() is a call on Python's Future class to wait for the async call to complete. You can also choose to do some other computation instead while the call is in progress.","title":"Async Methods"},{"location":"upgrade_apis/#simulation-only-methods","text":"Now we have clear distinction between methods that are only available in simulation from the ones that may be available on actual vehicle. The simulation only methods are prefixed with sim as shown below. getCollisionInfo() is renamed to simGetCollisionInfo() getCameraInfo() is renamed to simGetCameraInfo() setCameraOrientation() is renamed to simSetCameraOrientation()","title":"Simulation-Only Methods"},{"location":"upgrade_apis/#state-information","text":"Previously CarState mixed simulation-only information like kinematics_true . Moving forward, CarState will only contain information that can be obtained in real world. k = car_state.kinematics_true use this: k = car_state.kinematics_estimated # or k = client.simGetGroundTruthKinematics()","title":"State Information"},{"location":"upgrade_settings/","text":"Upgrading Settings # The settings schema in AirSim 1.2 is changed for more flexibility and cleaner interface. If you have older settings.json file then you can either delete it and restart AirSim or use this guide to make manual upgrade. Quicker Way # We recommend simply deleting the settings.json and add back the settings you need. Please see the doc for complete information on available settings. Changes # UsageScenario # Previously we used UsageScenario to specify the ComputerVision mode. Now we use \"SimMode\": \"ComputerVision\" instead. CameraDefaults and Changing Camera Settings # Previously we had CaptureSettings and NoiseSettings in root. Now these are combined in new CameraDefaults element. The schema for this element is later used to configure cameras on vehicle. Gimbal # The Gimbal element (instead of old Gimble element) is now moved out of CaptureSettings . CameraID to CameraName # All settings now reference cameras by name instead of ID. Using PX4 # The new Vehicles element allows to specify which vehicles to create. To use PX4, please see this section . AdditionalCameras # The old AdditionalCameras setting is now replaced by Cameras element within vehicle setting.","title":"Upgrading Settings"},{"location":"upgrade_settings/#upgrading-settings","text":"The settings schema in AirSim 1.2 is changed for more flexibility and cleaner interface. If you have older settings.json file then you can either delete it and restart AirSim or use this guide to make manual upgrade.","title":"Upgrading Settings"},{"location":"upgrade_settings/#quicker-way","text":"We recommend simply deleting the settings.json and add back the settings you need. Please see the doc for complete information on available settings.","title":"Quicker Way"},{"location":"upgrade_settings/#changes","text":"","title":"Changes"},{"location":"upgrade_settings/#usagescenario","text":"Previously we used UsageScenario to specify the ComputerVision mode. Now we use \"SimMode\": \"ComputerVision\" instead.","title":"UsageScenario"},{"location":"upgrade_settings/#cameradefaults-and-changing-camera-settings","text":"Previously we had CaptureSettings and NoiseSettings in root. Now these are combined in new CameraDefaults element. The schema for this element is later used to configure cameras on vehicle.","title":"CameraDefaults and Changing Camera Settings"},{"location":"upgrade_settings/#gimbal","text":"The Gimbal element (instead of old Gimble element) is now moved out of CaptureSettings .","title":"Gimbal"},{"location":"upgrade_settings/#cameraid-to-cameraname","text":"All settings now reference cameras by name instead of ID.","title":"CameraID to CameraName"},{"location":"upgrade_settings/#using-px4","text":"The new Vehicles element allows to specify which vehicles to create. To use PX4, please see this section .","title":"Using PX4"},{"location":"upgrade_settings/#additionalcameras","text":"The old AdditionalCameras setting is now replaced by Cameras element within vehicle setting.","title":"AdditionalCameras"},{"location":"use_precompiled/","text":"Download Binaries # You can simply download precompiled binaries and run to get started immediately. If you want to set up your own Unreal environment then please see these instructions . Unreal Engine # Windows, Linux : Download the binaries for the environment of your choice from the latest release . macOS : You will need to build it yourself Unity (Experimental) # A free environment called Windridge City is available at Unity Asset Store as an experimental release of AirSim on Unity. Note : This is an old release, and many of the features and APIs might not work. Controlling Vehicles # Most of our users typically use APIs to control the vehicles. However if you can also control vehicles manually. You can drive the car using keyboard, gamepad or steering wheel . To fly drone manually, you will need either XBox controller or a remote control (feel free to contribute keyboard support). Please see remote control setup for more details. Alternatively you can use APIs for programmatic control or use so-called Computer Vision mode to move around in environment using the keyboard. Don't Have Good GPU? # The AirSim binaries, like CityEnviron, requires a beefy GPU to run smoothly. You can run them in low resolution mode by editing the run.bat file on Windows like this: start CityEnviron -ResX=640 -ResY=480 -windowed For Linux binaries, use the Blocks.sh or corresponding shell script as follows - ./Blocks.sh -ResX=640 -ResY=480 -windowed UE 4.24 uses Vulkan drivers by default, but they can consume more GPU memory. If you get memory allocation errors, then you can try switching to OpenGL using -opengl","title":"Download Binaries"},{"location":"use_precompiled/#download-binaries","text":"You can simply download precompiled binaries and run to get started immediately. If you want to set up your own Unreal environment then please see these instructions .","title":"Download Binaries"},{"location":"use_precompiled/#unreal-engine","text":"Windows, Linux : Download the binaries for the environment of your choice from the latest release . macOS : You will need to build it yourself","title":"Unreal Engine"},{"location":"use_precompiled/#unity-experimental","text":"A free environment called Windridge City is available at Unity Asset Store as an experimental release of AirSim on Unity. Note : This is an old release, and many of the features and APIs might not work.","title":"Unity (Experimental)"},{"location":"use_precompiled/#controlling-vehicles","text":"Most of our users typically use APIs to control the vehicles. However if you can also control vehicles manually. You can drive the car using keyboard, gamepad or steering wheel . To fly drone manually, you will need either XBox controller or a remote control (feel free to contribute keyboard support). Please see remote control setup for more details. Alternatively you can use APIs for programmatic control or use so-called Computer Vision mode to move around in environment using the keyboard.","title":"Controlling Vehicles"},{"location":"use_precompiled/#dont-have-good-gpu","text":"The AirSim binaries, like CityEnviron, requires a beefy GPU to run smoothly. You can run them in low resolution mode by editing the run.bat file on Windows like this: start CityEnviron -ResX=640 -ResY=480 -windowed For Linux binaries, use the Blocks.sh or corresponding shell script as follows - ./Blocks.sh -ResX=640 -ResY=480 -windowed UE 4.24 uses Vulkan drivers by default, but they can consume more GPU memory. If you get memory allocation errors, then you can try switching to OpenGL using -opengl","title":"Don't Have Good GPU?"},{"location":"using_car/","text":"How to Use Car in AirSim # By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . For example, if you want to use car instead then just set the SimMode in your settings.json which you can find in your ~/Documents/AirSim folder, like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } Now when you restart AirSim, you should see the car spawned automatically. Manual Driving # Please use the keyboard arrow keys to drive manually. Spacebar for the handbrake. In manual drive mode, gears are set in \"auto\". Using APIs # You can control the car, get state and images by calling APIs in variety of client languages including C++ and Python. Please see APIs doc for more details. Changing Views # By default camera will chase the car from the back. You can get the FPV view by pressing F key and switch back to chasing from back view by pressing / key. More keyboard shortcuts can be seen by pressing F1. Cameras # By default car is installed with 5 cameras: center, left and right, driver and reverse. You can chose the images from these camera by specifying the name .","title":"Car Mode"},{"location":"using_car/#how-to-use-car-in-airsim","text":"By default AirSim prompts user for which vehicle to use. You can easily change this by setting SimMode . For example, if you want to use car instead then just set the SimMode in your settings.json which you can find in your ~/Documents/AirSim folder, like this: { \"SettingsVersion\": 1.2, \"SimMode\": \"Car\" } Now when you restart AirSim, you should see the car spawned automatically.","title":"How to Use Car in AirSim"},{"location":"using_car/#manual-driving","text":"Please use the keyboard arrow keys to drive manually. Spacebar for the handbrake. In manual drive mode, gears are set in \"auto\".","title":"Manual Driving"},{"location":"using_car/#using-apis","text":"You can control the car, get state and images by calling APIs in variety of client languages including C++ and Python. Please see APIs doc for more details.","title":"Using APIs"},{"location":"using_car/#changing-views","text":"By default camera will chase the car from the back. You can get the FPV view by pressing F key and switch back to chasing from back view by pressing / key. More keyboard shortcuts can be seen by pressing F1.","title":"Changing Views"},{"location":"using_car/#cameras","text":"By default car is installed with 5 cameras: center, left and right, driver and reverse. You can chose the images from these camera by specifying the name .","title":"Cameras"},{"location":"who_is_using/","text":"Who is Using AirSim? # Would you like to see your own group or project here? # Just add a GitHub issue with quick details and link to your website or paper or send us email at msrair at microsoft.com. NASA Ames Research Center \u2013 Systems Analysis Office Astrobotic GRASP Lab, Univ of Pennsylvania Department of Aeronautics and Astronautics, Stanford University Formula Technion Ghent University ICARUS UC, Santa Barbara WISE Lab, Univ of Waterloo HAMS project, MSR India Washington and Lee University University of Oklahoma Robotics Institute, Carnegie Mellon University Texas A&M Robotics and Perception Group, University of Zurich National University of Ireland, Galway (NUIG) Soda Mobility Technologies","title":"Who is Using AirSim"},{"location":"who_is_using/#who-is-using-airsim","text":"","title":"Who is Using AirSim?"},{"location":"who_is_using/#would-you-like-to-see-your-own-group-or-project-here","text":"Just add a GitHub issue with quick details and link to your website or paper or send us email at msrair at microsoft.com. NASA Ames Research Center \u2013 Systems Analysis Office Astrobotic GRASP Lab, Univ of Pennsylvania Department of Aeronautics and Astronautics, Stanford University Formula Technion Ghent University ICARUS UC, Santa Barbara WISE Lab, Univ of Waterloo HAMS project, MSR India Washington and Lee University University of Oklahoma Robotics Institute, Carnegie Mellon University Texas A&M Robotics and Perception Group, University of Zurich National University of Ireland, Galway (NUIG) Soda Mobility Technologies","title":"Would you like to see your own group or project here?"},{"location":"working_with_plugin_contents/","text":"How to use plugin contents # Plugin contents are not shown in Unreal projects by default. To view plugin content, you need to click on few semi-hidden buttons: Causion Changes you make in content folder are changes to binary files so be careful.","title":"Working with UE Plugin Contents"},{"location":"working_with_plugin_contents/#how-to-use-plugin-contents","text":"Plugin contents are not shown in Unreal projects by default. To view plugin content, you need to click on few semi-hidden buttons: Causion Changes you make in content folder are changes to binary files so be careful.","title":"How to use plugin contents"},{"location":"xbox_controller/","text":"XBox Controller # To use an XBox controller with AirSim follow these steps: Connect XBox controller so it shows up in your PC Game Controllers: Launch QGroundControl and you should see a new Joystick tab under settings: Now calibrate the radio, and setup some handy button actions. For example, I set mine so that the 'A' button arms the drone, 'B' put it in manual flight mode, 'X' puts it in altitude hold mode and 'Y' puts it in position hold mode. I also prefer the feel of the controller when I check the box labelled \"Use exponential curve on roll,pitch, yaw\" because this gives me more sensitivity for small movements. QGroundControl will find your Pixhawk via the UDP proxy port 14550 setup by MavLinkTest above. AirSim will find your Pixhawk via the other UDP server port 14570 also setup by MavLinkTest above. You can also use all the QGroundControl controls for autonomous flying at this point too. Connect to Pixhawk serial port using MavLinkTest.exe like this: MavLinkTest.exe -serial:*,115200 -proxy:127.0.0.1:14550 -server:127.0.0.1:14570 Run AirSim Unreal simulator with these ~/Documents/AirSim/settings.json settings: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"SitlIp\": \"\", \"SitlPort\": 14560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14570, \"UseSerial\": false } } Advanced # If the Joystick tab doesn't show up in QGroundControl then Click on the purple \"Q\" icon on left in tool bar to reveal the Preferences panel. Go to General tab and check the Virtual Joystick checkbox. Go back to settings screen (gears icon), click on Parameters tab, type COM_RC_IN_MODE in search box and change its value to either Joystick/No RC Checks or Virtual RC by Joystick . Other Options # See remote controller options","title":"XBox Controller"},{"location":"xbox_controller/#xbox-controller","text":"To use an XBox controller with AirSim follow these steps: Connect XBox controller so it shows up in your PC Game Controllers: Launch QGroundControl and you should see a new Joystick tab under settings: Now calibrate the radio, and setup some handy button actions. For example, I set mine so that the 'A' button arms the drone, 'B' put it in manual flight mode, 'X' puts it in altitude hold mode and 'Y' puts it in position hold mode. I also prefer the feel of the controller when I check the box labelled \"Use exponential curve on roll,pitch, yaw\" because this gives me more sensitivity for small movements. QGroundControl will find your Pixhawk via the UDP proxy port 14550 setup by MavLinkTest above. AirSim will find your Pixhawk via the other UDP server port 14570 also setup by MavLinkTest above. You can also use all the QGroundControl controls for autonomous flying at this point too. Connect to Pixhawk serial port using MavLinkTest.exe like this: MavLinkTest.exe -serial:*,115200 -proxy:127.0.0.1:14550 -server:127.0.0.1:14570 Run AirSim Unreal simulator with these ~/Documents/AirSim/settings.json settings: \"Vehicles\": { \"PX4\": { \"VehicleType\": \"PX4Multirotor\", \"SitlIp\": \"\", \"SitlPort\": 14560, \"UdpIp\": \"127.0.0.1\", \"UdpPort\": 14570, \"UseSerial\": false } }","title":"XBox Controller"},{"location":"xbox_controller/#advanced","text":"If the Joystick tab doesn't show up in QGroundControl then Click on the purple \"Q\" icon on left in tool bar to reveal the Preferences panel. Go to General tab and check the Virtual Joystick checkbox. Go back to settings screen (gears icon), click on Parameters tab, type COM_RC_IN_MODE in search box and change its value to either Joystick/No RC Checks or Virtual RC by Joystick .","title":"Advanced"},{"location":"xbox_controller/#other-options","text":"See remote controller options","title":"Other Options"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index dc0b4b553c..3358b56962 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ