Skip to content

Commit

Permalink
Add ATEK documentation pages.
Browse files Browse the repository at this point in the history 
Summary: Adding doc pages to ATEK.

Reviewed By: echo-xiao9

Differential Revision: D63333948

fbshipit-source-id: 6fe3b946d3d598ed5b723b270a338e00eaebdc9e
  • Loading branch information
@YLouWashU @facebook-github-bot
YLouWashU authored and facebook-github-bot committed Sep 24, 2024
1 parent ff1af48 commit 3021aee
Show file tree
Hide file tree
Showing 18 changed files with 995 additions and 260 deletions.
160 changes: 76 additions & 84 deletions docs/ATEK_Data_Store.md
View file
Original file line number Diff line number Diff line change
@@ -1,92 +1,84 @@
# ATEK Data Store

We preprocessed open Aria datasets with selected preprocessing configurations,
where users can directly download and load into PyTorch.
ATEK Data Store is a data platform where preprocessed open Aria datasets in WebDataset (WDS) formats, with selected preprocessing configurations, are available for users to directly download and load into PyTorch.

## List of available ATEK datasets
## ATEK datasets in WDS format

| Aria Open Dataset | Preprocess configuration | Download link |
| ------------------------ | ------------------------ | -------------------------------------------------------- |
| AriaDigitalTwin | cubercnn | [access link](https://www.projectaria.com/datasets/adt/) |
| AriaDigitalTwin | efm | [access link](https://www.projectaria.com/datasets/adt/) |
| AriaSyntheticEnvironment | cubercnn | [access link](https://www.projectaria.com/datasets/ase/) |
| AriaSyntheticEnvironment | cubercnn_eval | [access link](https://www.projectaria.com/datasets/ase/) |
| AriaSyntheticEnvironment | efm | [access link](https://www.projectaria.com/datasets/ase/) |
| AriaSyntheticEnvironment | efm_eval | [access link](https://www.projectaria.com/datasets/ase/) |
Aria Open Dataset | Preprocess configuration | Download link
------------------------ | ------------------------ | --------------------------------------------------------
AriaDigitalTwin | cubercnn | [access link](https://www.projectaria.com/datasets/adt/)
AriaDigitalTwin | efm | [access link](https://www.projectaria.com/datasets/adt/)
AriaSyntheticEnvironment | cubercnn | [access link](https://www.projectaria.com/datasets/ase/)
AriaSyntheticEnvironment | cubercnn_eval | [access link](https://www.projectaria.com/datasets/ase/)
AriaSyntheticEnvironment | efm | [access link](https://www.projectaria.com/datasets/ase/)
AriaSyntheticEnvironment | efm_eval | [access link](https://www.projectaria.com/datasets/ase/)

Details of each preprocessing configuration are listed [here](TODO).
Details of each preprocessing configuration are listed [here](../data/atek_data_store_confs/).

## How to access ATEK data

1. First, make sure that you have followed the [Install.md](./Install.md) guide
to install the **core** version of ATEK library, and cloned ATEK repo
locally, e.g. to `${ATEK_SRC}`.
2. Click **access link** in the above table, you can find the **Access The
Dataset** button on the bottom of the page. Input your email address, you
will be redirected to a page where you will find a button to download **...
in ATEK format (PyTorch ready)** data.

<img src="./images/atek_data_store_download_button.png" alt="Download button" width="700">

3. This will download a json file, e.g.
`AriaDigitalTwin_ATEK_download_urls.json`, that contains the URLs of the
actual preprocessed data. Note that for the same dataset, all preprocessing
configuration's URLs are contained in the same json file.

4. User now have 2 options to access the data, using ATEK-provided downloader
script:

a. **Optiona 1**: user can download the files to local:

```
python3 ${ATEK_SRC}/tools/atek_wds_data_downloader.py \
--config-name ${config_name} \
--input-json-path ${downloaded_url_json} \
--output-folder-path ${output_folder} \
--max-num-sequences 10 \
--train-val-split-ratio 0.8 \
--random-seed 42 \
--download-wds-to-local
```

where :

- `--config-name` specifies which [preprocessing configuration](TODO) you
would like to download.
- `--max-num-sequences` specifies number of sequences to download, and you
can use `--random-seed` to get a determined list of sequences to download.
- `--download-wds-to-local` this flag **needs to be set** to download to
local.

This command will download ATEK preprocessed WebDataset files with the
following folder structure, where `local_*_tars.yaml` file can be loaded by
[this API](TODO: add). Note that if the download breaks in the middle,
simply run it again **with the same random seed (default 42)** to pick up
from the middle.

```
./downloaded_local_wds
├── 0
│   ├── shards-0000.tar
│   ├── shards-0001.tar
│   └── shards-0002.tar
├── 1
│   ├── shards-0000.tar
│   └── shards-0001.tar
├── ...
├── local_all_tars.yaml
├── local_train_tars.yaml
└── local_validation_tars.yaml
```

a. **Optiona 2**: user can also create 3 streamable yaml files, by using the
same command but **without** the `--download-wds-to-local` flag. The created
`streamable_*_tars.yaml` files can be loaded by [this API](TODO: add), where
the preprocessed data can be streamed through web protocols via WebDataset
library. See
[ATEK_Demo_2](../examples/v3_demo_2_data_store_and_inference.ipynb) for an
example.

The streaming option will be impacted by internet connections, hence we
**strongly recommend** users to use Option 1 to download the preprocessed data
in model training.
We provide 2 options to access data from ATEK Data Store:

1. By default, the preprocessed dataset is **downloaded** to user's local computer. This is recommended for model training.
2. Or, the dataset can also be **streamed** via their URLs. However this will be impacted by internet connections, hence only recommended for small-scale testing.

To access the data:

1. Click the **access link** in the above table, you can find the **Access The Dataset** button on the bottom of the page. Input your email address, you will be redirected to a page where you will find a button to download **[dataset] in ATEK format (PyTorch ready)**.

![Download button](./images/atek_data_store_download_button.png)

2. This will download a json file, e.g. `[dataset_name]_ATEK_download_urls.json`, that contains the URLs of the actual preprocessed data. Note that for the same dataset, all preprocessing configuration's URLs are contained in the same json file.

3. First, make sure you have [installed ATEK lib](./Install.md#core-lib-installation). You can then download the data using ATEK-provided downloader script:

```bash
python3 ${ATEK_SRC}/tools/atek_wds_data_downloader.py \
--config-name ${config_name} \
--input-json-path ${downloaded_url_json} \
--output-folder-path ${output_folder} \
--download-wds-to-local
```

where :

- `--config-name` specifies which [preprocessing configuration](./preprocessing_configurations.md) you would like to download.
- `--download-wds-to-local` user can remove this flag to create **streamable** yaml files.

User can also specify other options including maximum number of sequences to download, training validation split ratio, etc. See [src code](../tools/atek_wds_data_downloader.py) for details.

4. **Note that these URLs will EXPIRE AFTER 30 DAYS**, user will need to re-download and re-generate the streamable yaml files.

These steps will download ATEK preprocessed WebDataset files with the following folder structure. Note that if the download breaks in the middle, simply run it again to pick up from the middle.

```bash
./downloaded_local_wds
├── 0
│ ├── shards-0000.tar
│ ├── shards-0001.tar
│ └── shards-0002.tar
├── 1
│ ├── shards-0000.tar
│ └── shards-0001.tar
├── ...
├── local_all_tars.yaml
├── local_train_tars.yaml
└── local_validation_tars.yaml
```

If `--download-wds-to-local` flag is removed, the script will only create 3 streamable yaml files without downloading any data:

```bash
./output
├── streamable_all_tars.yaml
├── streamable_train_tars.yaml
└── streamable_validation_tars.yaml
```

## Load ATEK Data Store yaml file

The generated `streamable_*.yaml` and `local_*.yaml` files contain local or remote URLs of the preprocessed WDS files. The following API can load them into a `List` of URLs that can be input into [ATEK data loader APIs](./data_loading_and_inference.md)

```python
tar_file_urls = load_yaml_and_extract_tar_list(yaml_path="./local_all_tars.yaml")
```
Empty file removed docs/ATEK_Data_Store_cubercnn_conf.md
View file
Empty file.
94 changes: 30 additions & 64 deletions docs/Install.md
View file
Original file line number Diff line number Diff line change
@@ -1,84 +1,50 @@
# Installation

## Install ATEK-core lib
We provided 2 ways to install ATEK:

1. If you just need **the core functionalities of ATEK**, including data pre-processing, data loading, and visualization, you can simply [install ATEK's core lib](#core-lib-installation)
2. If you want to run the CubeRCNN demos and all task-specific evaluation benchmarking, you can follow this guide to [install **full dependencies**](#install-all-dependencies-using-mambaconda).

## Core lib installation

1. First install PyTorch following the [official guide](https://pytorch.org/).
2. Install ATEK core lib. This should allow you to use all core functionalities
of ATEK, including data preprocessing, data loading, and visualization
2. Then install ATEK core lib:

```
pip install projectaria-atek
```

## [Optional] Install dependencies for object detection evaluation lib

If you want to run ATEK's object detection evaluation lib and benchmarking, you
need to install Pytorch3D library by following their
[official guide](https://github.com/facebookresearch/pytorch3d). Or, ATEK
provides a [one-stop way](#bookmark_one_stop) to install full dependencies in
Mamba.

## [Optional] Install dependencies for running CubeRCNN examples

If you want to run ATEK's example inference and training scripts using CubeRCNN,
you need to install both
[Detectron2](https://github.com/facebookresearch/detectron2), and
[(a forked copy of) CubeRCNN](https://github.com/YLouWashU/omni3d). Or, ATEK
provides a [one-stop way](#bookmark_one_stop) to install full dependencies in
Mamba:
## Full dependencies installation using Mamba/Conda

## One stop way to install all dependencies using Mamba {#bookmark_one_stop}
The following steps will install ATEK with full dependencies, where you will be able to run all task-specific evaluations, along with our CubeRCNN examples. We have tested this under CUDA 12.4 + Python 3.9, on platforms including local Linux, AWS, and M1 MacBook. Note that all `mamba` commands below can be replaced by `conda`.

The following steps will install ATEK with full dependencies. We have tested
this under CUDA 12.4 + Python 3.9, AWS, and M1 macbook.
1. Install **`mamba` (recommended)** by following instructions [here](https://github.com/conda-forge/miniforge). `mamba` is a drop-in replacement for `conda` and is fully compatible with `conda` commands. For Mac / Linux, you can just do:

1. Install **`mamba` (recommended)** by following instructions
[here](https://github.com/conda-forge/miniforge?tab=readme-ov-file#install).
`mamba` is a drop-in replacement for `conda` and is fully compatible with
`conda` commands. For Mac / Linux, you can just do:

```
cd ~/Documents/
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh
mamba init; # may need to reopen terminal after this
```
```bash
cd ~/Documents/
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh
mamba init # may need to reopen terminal after this
```

2. Clone ATEK git repo.

```
cd ~/Documents/
git clone https://github.com/facebookresearch/ATEK.git
cd ~/Documents/ATEK
```

3. Install dependencies in Mamba. Depending on use cases, you can choose from
the following flavors: `full`, `mac`, and `aws`:

a. `full`: if you are on linux, run:

```
mamba env create -f envs/env_atek_full.yml
```
```bash
cd ~/Documents/
git clone https://github.com/facebookresearch/ATEK.git
cd ~/Documents/ATEK
```

c. `aws`: if you are running on AWS, you can run the following command, it
will install the `full` version:
3. Install dependencies in Mamba. You can choose from the following flavors: `linux`, `mac`, and `aws`:

```
mamba env create -f envs/env_atek_aws.yml
```

d. `mac`: if you are using mac, you can run the following command:

```
mamba env create -f envs/env_atek_mac.yml
```
```bash
mamba env create -f envs/env_atek_linux[mac/aws].yml
```

4. Install ATEK library from source code:

```
mamba activate atek_full
cd ~/Document/ATEK
pip install -e .
```
```bash
mamba activate atek_full
cd ~/Documents/ATEK
pip install -e .
```
82 changes: 82 additions & 0 deletions docs/ML_task_object_detection.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Machine Learning tasks supported by ATEK: static 3D object detection

## Task introduction

Static 3D object detection is the task of perceiving objects in 3D from image / video inputs. The primary goal is to accurately estimate the 3D location, orientation, and extent of each object within the scene. This is achieved by placing a tight, oriented 3D bounding box around each detected object, effectively translating 2D data into a 3D spatial understanding.

**Input**: For 3D object detection task in Aria, the input would include the images captured by the RGB camera. In some variations, additional inputs may include Slam camera images, camera calibration, and trajecotry + point-cloud data from Aria's [Machine Perception Services](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps).

**Output**: the task expects to output detected 3D objects within the scene. Each detection consists of the pose of the 3D bounding box enclosing the object, and the detected category name of the object.

## ATEK standardized Evaluation library

In ATEK, we provide an standardized evaluation library for this task, and supports the following datasets and models:
- **supported Aria datasets**:
- [Aria Digital Twin Dataset](https://www.projectaria.com/datasets/adt/)
- [Aria Synthetic Environments Dataset](https://www.projectaria.com/datasets/ase/)
- **supported ML models**:
- [Cube R-CNN](https://github.com/facebookresearch/omni3d/tree/main)
- [Egocentric Voxel Lifting](https://arxiv.org/abs/2406.10224)

### Dependencies

User will need to install [`PyTorch3D`](https://github.com/facebookresearch/pytorch3d) following their official guidance in order to run ATEK evaluation for the object detection task. ATEK also provides a guide to install this using mamba / conda in [Install.md](./Install.md#full-dependencies-installation-using-mambaconda)

### ATEK Evaluation dataset

Follow the instructions in [ATEK Data Store](http://docs/atek_data_store.md) section to download **ASE** dataset with **cubercnn-eval** configuration.

### Prediction Files Format

For this task, ATEK defines both the prediction file and groundtruth files to be a `csv` file, with the following fields. For each Aria sequence, user just need to parse the GT data and their model output into this format (see [here](../atek/evaluation/static_object_detection/obb3_csv_io.py) for an example):

- **`​​time_ns`**: Timestamp of the detection in nanoseconds.
- **`t[x,y,z]_world_object`**: coordinate of the object's position in the world coordinate system, in meters.
- **`q[w,x,y,z]_world_object`**: orientation in quaternions of the object in the world coorindate system.
- **`scale_[x,y,z]`**: Size of the object.
- **`name`**: category name of the object.
- **`instance`**: Unique identifier for each instance of an object in a scene. If not available, this value is set to 1.
- **`sem_id`**: category id of the object.
- **`prob`**: Probability that the object is correctly identified, ranging from 0 to 1\. For groundtruth, this value is set to 1.

For multiple sequences, please follow the following folder structure to store the results:

```
results/
├── sequence_01/
│ ├── prediction.csv
│ └── gt.csv
├── sequence_02/
│ ├── prediction.csv
│ └── gt.csv
└── sequence_03/
├── prediction.csv
└── gt.csv
```

### Run ATEK benchmarking

Once the `prediction.csv` and `gt.csv` files are prepared into the following folder structure, user can run ATEK-provided benchmarking script to generate metrics: [`tools/benchmarking_static_object_detection.py`](../tools/benchmarking_static_object_detection.py).

#### `benchmarking_static_object_detection.py` script

This Python script evaluates the performance of static 3D object detection models using oriented bounding boxes (OBBs). It supports both dataset-level and file-level evaluations.

##### Command Line Arguments

- `--input-folder`: If specificied, will perform Dataset-level evaluation. Path to the folder containing ground truth and prediction CSV files.
- `--pred-csv`: Filename of the prediction csv file.
- `--gt-csv`: Filename of the groundtruth csv file..
- `--output-file`: Path where the output metrics JSON file will be saved.
- `--iou-threshold`:[Optional] IOU threshold for determining a match between predicted and actual OBBs, used in computing average precision / recall metrics (default: 0.2).
- `--confidence-lower-threshold`:[Optional] Minimum confidence score for predictions to be considered, used in computing average precision / recall metrics (default: 0.3).
- `--max-num-sequences`:[Optional] Maximum number of sequences to evaluate (default: -1, which means no limit).

##### Reported metrics

- **`map_3D`** : the mean average precision computed across all the categories in the dataset, over a range of 3D IoU thresholds of [0.05, 0.10,..., 0.50].
- **`[email protected],Confidence0.3`** : averaged precision value over all sequences, where the IoU threshold and confidence levels can be set through commandline arguments.
- **`[email protected],Cofidence0.3`**: averaged recall value over all sequences, where the IoU threshold and confidence levels can be set through commandline arguments.
- **`map_per_class@...`**: the mean average precision computed

See the generated `--output-file` for full metrics.
Loading

0 comments on commit 3021aee

Please sign in to comment.