diff --git a/docs/ATEK_Data_Store.md b/docs/ATEK_Data_Store.md
index 7127272..634c462 100644
--- a/docs/ATEK_Data_Store.md
+++ b/docs/ATEK_Data_Store.md
@@ -1,6 +1,6 @@
# ATEK Data Store
-ATEK Data Store is a data platform where preprocessed open Aria datasets in WebDataset (WDS) formats, with selected preprocessing configurations, are available for users to directly download and load into PyTorch.
+ATEK Data Store is a data platform where preprocessed open Aria datasets in [WebDataset](https://github.com/webdataset/webdataset) (WDS) formats, with selected preprocessing configurations, are available for users to directly download and load into PyTorch.
## ATEK datasets in WDS format
@@ -26,7 +26,7 @@ To access the data:
1. Click the **access link** in the above table, you can find the **Access The Dataset** button on the bottom of the page. Input your email address, you will be redirected to a page where you will find a button to download **[dataset] in ATEK format (PyTorch ready)**.
- 
+ 
2. This will download a json file, e.g. `[dataset_name]_ATEK_download_urls.json`, that contains the URLs of the actual preprocessed data. Note that for the same dataset, all preprocessing configuration's URLs are contained in the same json file.
@@ -42,14 +42,15 @@ To access the data:
where :
- - `--config-name` specifies which [preprocessing configuration](./preprocessing_configurations.md) you would like to download.
+ - `--config-name` specifies which [preprocessing configuration](./preprocessing_configurations.md) you would like to download. You should choose one from [this table](#atek-datasets-in-wds-format).
- `--download-wds-to-local` user can remove this flag to create **streamable** yaml files.
User can also specify other options including maximum number of sequences to download, training validation split ratio, etc. See [src code](../tools/atek_wds_data_downloader.py) for details.
4. **Note that these URLs will EXPIRE AFTER 30 DAYS**, user will need to re-download and re-generate the streamable yaml files.
-These steps will download ATEK preprocessed WebDataset files with the following folder structure. Note that if the download breaks in the middle, simply run it again to pick up from the middle.
+## Downloaded WDS files
+Following the above steps will download ATEK preprocessed WebDataset files with the following folder structure. Note that if the download breaks in the middle, simply run it again to pick up from the middle.
```bash
./downloaded_local_wds
diff --git a/docs/Install.md b/docs/Install.md
index 0eb4fb7..97fea26 100644
--- a/docs/Install.md
+++ b/docs/Install.md
@@ -2,7 +2,7 @@
We provided 2 ways to install ATEK:
-1. If you just need **the core functionalities of ATEK**, including data pre-processing, data loading, and visualization, you can simply [install ATEK's core lib](#core-lib-installation)
+1. If you just need the core functionalities of ATEK, including data pre-processing, data loading, and visualization, you can simply [install **ATEK core lib**](#core-lib-installation)
2. If you want to run the CubeRCNN demos and all task-specific evaluation benchmarking, you can follow this guide to [install **full dependencies**](#install-all-dependencies-using-mambaconda).
## Core lib installation
diff --git a/docs/example_cubercnn_customization.md b/docs/example_cubercnn_customization.md
index 2c67a86..f928467 100644
--- a/docs/example_cubercnn_customization.md
+++ b/docs/example_cubercnn_customization.md
@@ -139,22 +139,18 @@ print(f"Loading WDS into CubeRCNN format, each sample contains the following key
## CuberCNN model trainng / inference
-With the created Pytorch DataLoader, user will be able to easily run model training / inference for CubeRCNN model:
+With the created Pytorch DataLoader, user will be able to easily run model training or inference for CubeRCNN model.
+**Training script**
```python
-# Load pre-trained model for training / inference
-model_config, model = create_inference_model(
- model_config_file, model_ckpt_path, use_cpu_only = use_cpu_only
-)
+# Load pre-trained model for training
+model_config, model = create_training_model(model_config_file, model_ckpt_path)
-# training / inference loop
+# Training loop
for cubercnn_input_data in tqdm(
cubercnn_dataloader,
- desc="Training / Inference progress: ",
+ desc="Training progress: ",
):
- # Inference step
- cubercnn_model_output = model(cubercnn_input_data)
-
# Training step
loss = model(cubercnn_input_data)
losses = sum(loss.values())
@@ -163,3 +159,21 @@ for cubercnn_input_data in tqdm(
optimizer.step()
...
```
+
+
+**Inference script**
+
+```python
+# Load pre-trained model for inference
+model_config, model = create_inference_model(model_config_file, model_ckpt_path)
+
+# Inference loop
+for cubercnn_input_data in tqdm(
+ cubercnn_dataloader,
+ desc="Training progress: ",
+):
+ # Inference step
+ cubercnn_model_output = model(cubercnn_input_data)
+ ...
+```
+# Inference step
diff --git a/docs/example_training.md b/docs/example_training.md
index 68eef4e..74983b0 100644
--- a/docs/example_training.md
+++ b/docs/example_training.md
@@ -88,4 +88,4 @@ Here is the tensorboard results for ASE trained results, where the left 2 figure
Once model training is finished, user can proceed to [example_inference.md] to run model inference on the trained weights.
-We also provided 2 sets of CubeRCNN trained weights by us, one on ASE 10K dataset, the other on ADT dataset. The weights can be downloaded [here](https://www.projectaria.com/async/sample/download/?bucket=adt&filename=ATEK_example_model_weights.tar)
+We also provided 2 sets of CubeRCNN trained weights by us, one on ASE 10K dataset, the other on ADT dataset. The weights can be downloaded [here](https://www.projectaria.com/async/sample/download/?bucket=atek&filename=ATEK_example_model_weights.tar). By downloading this file, you acknowledge that you have read, understood, and agree to be bound by the terms of the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en) software license.
diff --git a/docs/images/atek_github_video_small.webm b/docs/images/atek_github_video_small.webm
new file mode 100644
index 0000000..916f125
Binary files /dev/null and b/docs/images/atek_github_video_small.webm differ
diff --git a/docs/preprocessing.md b/docs/preprocessing.md
index f20b5b1..4dabfe3 100644
--- a/docs/preprocessing.md
+++ b/docs/preprocessing.md
@@ -17,9 +17,9 @@ Before ATEK, users will need to hand-craft all these code on their own, which is
## Simple customization through preprocessing config
-ATEK allows user to **customize the preprocessing workflow by simply modifying the preprocessing configuration yaml file** (see [preprocessing_configurations.md](./preprocessing_configurations.md) for details).
+ATEK allows user to **customize the preprocessing workflow by simply modifying the preprocessing configuration yaml file** (see [Preprocessing configurations page](./preprocessing_configurations.md) for details).
-The following is the core code to load an open Aria data sequence, preprocess according to a given configuration file, and write the preprocessed results to disk as WebDataset ([full example](../examples/Demo_1_data_preprocessing.ipynb)). We also use a visualization library based on `ReRun` to visualize the preprocessed results. The results are stored as `Dict` in memory containing tensors, strings, and sub-dicts, and also saved to local disk in WebDataset (WDS) format for further use.
+The following is the core code to load an open Aria data sequence, preprocess according to a given configuration file, and write the preprocessed results to disk as WebDataset. We also use a visualization library based on `ReRun` to visualize the preprocessed results. The results are stored as `Dict` in memory containing tensors, strings, and sub-dicts, and also saved to local disk in WebDataset (WDS) format for further use.
```python
from omegaconf import OmegaConf
@@ -35,11 +35,11 @@ num_samples = preprocessor.process_all_samples(write_to_wds_flag = True, viz_fla
### `create_general_atek_preprocessor_from_conf`
-This is a factory method that initializes a `GeneralAtekPreprocessor` based on a configuration object. It selects the appropriate preprocessor configuration for ATEK using the `atek_config_name` field in the provided Omega configuration. See [here](./preprocessing_configurations.md) for currently supported configs.
+This is a factory method that initializes a `GeneralAtekPreprocessor` based on a configuration object. It selects the appropriate preprocessor configuration for ATEK using the `atek_config_name` field in the provided Omega configuration.
#### Parameters
-- **conf** (`DictConfig`): Configuration object with preprocessing settings. The `atek_config_name` key specifies the preprocessor type,
+- **conf** (`DictConfig`): Configuration object with preprocessing settings.
- **raw_data_folder** (`str`): Path to the folder with raw data files.
- **sequence_name** (`str`): Name of the data sequence to process.
- **output_wds_folder** (`Optional[str]`): Path for saving preprocessed data in WebDataset (WDS) format. If `None`, data is not saved in WDS format.
diff --git a/examples/ATEK_CoLab_Notebook.ipynb b/examples/ATEK_CoLab_Notebook.ipynb
index fe0c1fe..a5d47e0 100644
--- a/examples/ATEK_CoLab_Notebook.ipynb
+++ b/examples/ATEK_CoLab_Notebook.ipynb
@@ -1,664 +1,701 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "924Eqh3ErHWn"
- },
- "source": [
- "# ATEK CoLab Notebook\n",
- "\n",
- "Welcome to ATEK CoLab Notebook. This notebook\n",
- " will walk through the steps of preparing an Aria data sequence with annotations ([AriaDigitalTwin (ADT)](https://www.projectaria.com/datasets/adt/)), for use in a 3D object detection ML task.\n",
- "We will go through the following steps:\n",
- "1. downloading ADT sample data\n",
- "2. preprocess ADT sample data\n",
- "3. visualize the preprocessed data\n",
- "4. run model inference with ATEK preprocessed data\n",
- "5. evaluate model performance.\n",
- "\n",
- "## Environment set up"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "YygI9xSxQJen"
- },
- "outputs": [],
- "source": [
- "!pip install 'git+https://github.com/YLouWashU/omni3d.git'\n",
- "!pip install projectaria-atek==1.0.0\n",
- "!pip install 'git+https://github.com/facebookresearch/detectron2.git'\n",
- "!pip install iopath\n",
- "import sys\n",
- "import torch\n",
- "pyt_version_str=torch.__version__.split(\"+\")[0].replace(\".\", \"\")\n",
- "version_str=\"\".join([\n",
- " f\"py3{sys.version_info.minor}_cu\",\n",
- " torch.version.cuda.replace(\".\",\"\"),\n",
- " f\"_pyt{pyt_version_str}\"\n",
- "])\n",
- "!pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
- "!pip install rerun-sdk[notebook]"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "import requests\n",
- "\n",
- "url = \"https://www.projectaria.com/async/sample/download/?bucket=adt&filename=ATEK_example_model_weights.tar\"\n",
- "response = requests.get(url)\n",
- "\n",
- "# Specify the path where you want to save the file\n",
- "file_path = \"/content/ATEK_example_model_weights.tar\"\n",
- "\n",
- "# Open the file in binary write mode and write the contents of the response\n",
- "with open(file_path, \"wb\") as file:\n",
- " file.write(response.content)"
- ],
- "metadata": {
- "id": "mejA1LJYoqny"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "s2-IjOuiBsim"
- },
- "outputs": [],
- "source": [
- "!mkdir -p /content/data\n",
- "!tar -xvf /content/ATEK_example_model_weights.tar -C /content/data"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "!cd /content/data\n",
- "!git clone https://github.com/facebookresearch/ATEK.git\n",
- "!unzip /content/atek.zip -d /content/data"
- ],
- "metadata": {
- "id": "EUZA4zzGfrdJ",
- "collapsed": true
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "print(\"----debug: manually upload zip file\")\n",
- "!unzip /content/ATEK-main.zip -d /content/data"
- ],
- "metadata": {
- "id": "3j8s4r7VM1y_"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "tTp3xS-Ggpe4"
- },
- "source": [
- "## Part1: Data proprocessing\n",
- "\n",
- "### Data preprocessing requirements\n",
- "ADT sequence has:\n",
- "1. Aria recording (VRS).\n",
- "2. MPS trajectory file (CSV).\n",
- "3. Object detection annotation files (3 csv files + 1 json file).\n",
- "\n",
- "CubeRCNN model needs synchronized data frame containing:\n",
- "1. Upright RGB camera image.\n",
- "2. Linear camera calibration matrix.\n",
- "3. Object bounding box annotations in 2D + 3D.\n",
- "4. Camera-to-object poses.\n",
- "\n",
- "**Before ATEK**, users need to implement all the followings to prepare ADT sequence into CubeRCNN model:\n",
- "1. Parse in ADT sequence data using `projectaria_tools` lib. \n",
- "2. Properly synchronize sensor + annotation data into training samples.\n",
- "3. Perform additional image & data processing:\n",
- " 1. Undistort image + camera calibration.\n",
- " 2. Rescale camera resolution.\n",
- " 3. Rotate image + camera calibration.\n",
- " 4. Undistort + rescale + rotate object 2D bounding boxes accordingly."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "Yy79EFR8UZMS"
- },
- "outputs": [],
- "source": [
- "import faulthandler\n",
- "import logging\n",
- "import os\n",
- "from logging import StreamHandler\n",
- "import numpy as np\n",
- "from typing import Dict, List, Optional\n",
- "import torch\n",
- "import sys\n",
- "from itertools import islice\n",
- "from tqdm import tqdm\n",
- "\n",
- "from atek.data_preprocess.genera_atek_preprocessor_factory import (\n",
- " create_general_atek_preprocessor_from_conf,\n",
- ")\n",
- "from atek.viz.atek_visualizer import NativeAtekSampleVisualizer\n",
- "from atek.data_preprocess.general_atek_preprocessor import GeneralAtekPreprocessor\n",
- "from atek.data_loaders.atek_wds_dataloader import (\n",
- " create_native_atek_dataloader\n",
- ")\n",
- "\n",
- "from atek.data_loaders.cubercnn_model_adaptor import (\n",
- " cubercnn_collation_fn,\n",
- " create_atek_dataloader_as_cubercnn\n",
- ")\n",
- "from atek.evaluation.static_object_detection.obb3_csv_io import AtekObb3CsvWriter\n",
- "\n",
- "from atek.data_preprocess.atek_data_sample import (\n",
- " create_atek_data_sample_from_flatten_dict,\n",
- ")\n",
- "from cubercnn.config import get_cfg_defaults\n",
- "from cubercnn.modeling.backbone import build_dla_from_vision_fpn_backbone\n",
- "from cubercnn.modeling.meta_arch import build_model\n",
- "\n",
- "from detectron2.checkpoint import DetectionCheckpointer\n",
- "from detectron2.config import get_cfg\n",
- "from omegaconf import OmegaConf\n",
- "import subprocess"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "YJNAP3W2g_CL"
- },
- "source": [
- "## Download ADT sequences from projectaria.com"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "Fy-pj5ejrMn8"
- },
- "outputs": [],
- "source": [
- "adt_sample_path = \"./adt_sample_data\"\n",
- "data_sequence_url = \"https://www.projectaria.com/async/sample/download/?bucket=adt&filename=aria_digital_twin_test_data_v2.zip\"\n",
- "command_list = [\n",
- " f\"mkdir -p {adt_sample_path}\",\n",
- " # Download sample data\n",
- " f'curl -o {adt_sample_path}/adt_sample_data.zip -C - -O -L \"{data_sequence_url}\"',\n",
- " # Unzip the sample data\n",
- " f\"unzip -o {adt_sample_path}/adt_sample_data.zip -d {adt_sample_path}\"\n",
- "]\n",
- "sequence_path = f\"{adt_sample_path}/Apartment_release_golden_skeleton_seq100_10s_sample_M1292\"\n",
- "\n",
- "# Execute the commands for downloading dataset\n",
- "for command in command_list:\n",
- " subprocess.run(command, shell=True, check=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "mGJc-2B_hF03"
- },
- "source": [
- "### Set up data and code paths"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "KGaKsE5IEol8"
- },
- "outputs": [],
- "source": [
- "example_adt_data_dir = f\"{adt_sample_path}/Apartment_release_golden_skeleton_seq100_10s_sample_M1292\"\n",
- "sequence_name = \"Apartment_release_golden_skeleton_seq100_10s_sample_M1292\"\n",
- "atek_src_path = \"/content/data/ATEK-main\"\n",
- "category_mapping_file = f\"{atek_src_path}/data/adt_prototype_to_atek.csv\"\n",
- "atek_preprocess_config_path = f\"{atek_src_path}/examples/data/adt_cubercnn_preprocess_config.yaml\"\n",
- "preprocess_conf = OmegaConf.load(atek_preprocess_config_path)\n",
- "# Take viz conf out of preprocess conf\n",
- "viz_conf = preprocess_conf.visualizer\n",
- "del preprocess_conf.visualizer\n",
- "\n",
- "# Create viz conf for inference viewer\n",
- "infer_viz_config_path = f\"{atek_src_path}/examples/data/infer_viz_conf.yaml\"\n",
- "infer_viz_conf = OmegaConf.load(infer_viz_config_path)\n",
- "\n",
- "output_wds_path = f\"{atek_src_path}/examples/data/wds_output\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "nBcS5GTahgx5"
- },
- "source": [
- "### Helper functions"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "uw-Yzp1pjyJd"
- },
- "outputs": [],
- "source": [
- "faulthandler.enable()\n",
- "\n",
- "# Configure logging to display the log messages in the notebook\n",
- "logging.basicConfig(\n",
- " level=logging.INFO,\n",
- " format='%(asctime)s - %(levelname)s - %(message)s',\n",
- " handlers=[\n",
- " logging.StreamHandler(sys.stdout)\n",
- " ]\n",
- ")\n",
- "\n",
- "logger = logging.getLogger()\n",
- "\n",
- "\n",
- "# -------------------- Helper functions --------------------#\n",
- "def print_data_sample_dict_content(data_sample, if_pretty: bool = False):\n",
- " \"\"\"\n",
- " A helper function to print the content of data sample dict\n",
- " \"\"\"\n",
- " logger.info(\"Printing the content in a ATEK data sample dict: \")\n",
- " for key, val in data_sample.items():\n",
- " if if_pretty and \"#\" in key:\n",
- " key = key.split(\"#\", 1)[1]\n",
- "\n",
- " msg = f\"\\t {key}: is a {type(val)}, \"\n",
- " if isinstance(val, torch.Tensor):\n",
- " msg += f\"\\n \\t\\t\\t\\t with tensor dtype of {val.dtype}, and shape of : {val.shape}\"\n",
- " elif isinstance(val, list):\n",
- " msg += f\"with len of : {len(val)}\"\n",
- " elif isinstance(val, str):\n",
- " msg += f\"value is {val}\"\n",
- " else:\n",
- " pass\n",
- " logger.info(msg)\n",
- "\n",
- "def run_command_and_display_output(command):\n",
- " # Start the process\n",
- " process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n",
- "\n",
- " # Poll process.stdout to show stdout live\n",
- " while True:\n",
- " output = process.stdout.readline()\n",
- " if output == '' and process.poll() is not None:\n",
- " break\n",
- " if output:\n",
- " print(output.strip())\n",
- " rc = process.poll()\n",
- " return rc\n",
- "\n",
- "def create_inference_model(config_file, ckpt_dir, use_cpu_only=False):\n",
- " \"\"\"\n",
- " Create the model for inference pipeline, with the model config.\n",
- " \"\"\"\n",
- " # Create default model configuration\n",
- " model_config = get_cfg()\n",
- " model_config.set_new_allowed(True)\n",
- " get_cfg_defaults(model_config)\n",
- "\n",
- " # add extra configs for data\n",
- " model_config.MAX_TRAINING_ATTEMPTS = 3\n",
- " model_config.TRAIN_LIST = \"\"\n",
- " model_config.TEST_LIST = \"\"\n",
- " model_config.TRAIN_WDS_DIR = \"\"\n",
- " model_config.TEST_WDS_DIR = \"\"\n",
- " model_config.ID_MAP_JSON = \"\"\n",
- " model_config.OBJ_PROP_JSON = \"\"\n",
- " model_config.CATEGORY_JSON = \"\"\n",
- " model_config.DATASETS.OBJECT_DETECTION_MODE = \"\"\n",
- " model_config.SOLVER.VAL_MAX_ITER = 0\n",
- " model_config.SOLVER.MAX_EPOCH = 0\n",
- "\n",
- " model_config.merge_from_file(config_file)\n",
- " if use_cpu_only:\n",
- " model_config.MODEL.DEVICE = \"cpu\"\n",
- " model_config.freeze()\n",
- "\n",
- " model = build_model(model_config, priors=None)\n",
- "\n",
- " _ = DetectionCheckpointer(model, save_dir=ckpt_dir).resume_or_load(\n",
- " model_config.MODEL.WEIGHTS, resume=True\n",
- " )\n",
- " model.eval()\n",
- "\n",
- " return model_config, model"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "rvdmyAIrhngs"
- },
- "source": [
- "### Set up and run ATEK data preprocessor"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "Ua3oo2TSsYm0"
- },
- "outputs": [],
- "source": [
- "# Create ATEK preprocessor from conf. It will automatically choose which type of sample to build.\n",
- "atek_preprocessor = create_general_atek_preprocessor_from_conf(\n",
- " # [required]\n",
- " conf=preprocess_conf,\n",
- " raw_data_folder = example_adt_data_dir,\n",
- " sequence_name = sequence_name,\n",
- " # [optional]\n",
- " output_wds_folder=output_wds_path,\n",
- " output_viz_file=os.path.join(example_adt_data_dir, \"atek_preprocess_viz.rrd\"),\n",
- " category_mapping_file=category_mapping_file,\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "cxgiEJAFiDDH"
- },
- "source": [
- "## Preprocessed ATEK data sample content\n",
- "* Preprocessing input: VRS + csv + jsons\n",
- "* Preprocessing output (in memory): ATEK data samples: `Dict[torch.Tensor, str, or Dict]`\n",
- "* Preprocessing output (on local disk): WebDataset (WDS) tar files."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "BS-OuwW9scUP"
- },
- "outputs": [],
- "source": [
- "atek_data_sample = atek_preprocessor[0]\n",
- "atek_data_sample_dict = atek_data_sample.to_flatten_dict()\n",
- "print_data_sample_dict_content(atek_data_sample_dict)\n",
- "\n",
- "# Loop over all samples, and write valid ones to local tar files.\n",
- "atek_preprocessor.process_all_samples(write_to_wds_flag=True, viz_flag=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "qDUCTF0gs48Y"
- },
- "source": [
- "## Visualize preprocessed ATEK data sample"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "cVCo0il9abVi"
- },
- "outputs": [],
- "source": [
- "tar_file_urls = [os.path.join(output_wds_path, f\"shards-000{i}.tar\") for i in range(2)]\n",
- "atek_dataloader = create_native_atek_dataloader(urls = tar_file_urls, batch_size = None, num_workers = 1)\n",
- "atek_viz = NativeAtekSampleVisualizer(viz_prefix = \"notebook atek viz\", show_on_notebook = True, conf = viz_conf)\n",
- "for atek_sample in tqdm(islice(atek_dataloader, 10)):\n",
- " atek_viz.plot_atek_sample_as_dict(atek_sample)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "LVbIgzI6jciw"
- },
- "source": [
- "# Part 2: Run Object detection inference using pre-trained CubeRCNN model\n",
- "In this example, we demonstrate how to run model inference with preprocessed ATEK data streamed from Data Store."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "J2tOwBZ8k4cp"
- },
- "source": [
- "### Create PyTorch DataLoader, converted to CubeRCNN format\n",
- "User can add a data transform function from ATEK format -> CubeRCNN format:\n",
- "1. Dict key remapping.\n",
- "2. Tensor reshaping & reordering.\n",
- "3. Other data transformations.\n",
- "\n",
- "Example data transform function for CubeRCNN model: [src code](https://www.internalfb.com/code/fbsource/[a5c3831c045bc718862d1c512e84d4ed6f79d722]/fbcode/surreal/data_services/atek/atek/data_loaders/cubercnn_model_adaptor.py?lines=44-74)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "id": "i7GgyyaWZvFB"
- },
- "outputs": [],
- "source": [
- "import logging\n",
- "logger = logging.getLogger()\n",
- "logger.info(\n",
- " \"-------------------- ATEK WDS data can loaded into Model-specific format --------------- \"\n",
- ")\n",
- "# The CubeRCNN ModelAdaptor class is wrapped in this function\n",
- "cubercnn_dataloader = create_atek_dataloader_as_cubercnn(urls = tar_file_urls, batch_size = 1, num_workers = 1)\n",
- "first_cubercnn_sample = next(iter(cubercnn_dataloader))\n",
- "logger.info(f\"Loading WDS into CubeRCNN format, each sample contains the following keys: {first_cubercnn_sample[0].keys()}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "pWyaUUQPk6Te"
- },
- "source": [
- "### Run model inference"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "-0RWOhlrjaKE"
- },
- "outputs": [],
- "source": [
- "from tqdm import tqdm\n",
- "model_ckpt_path = \"/content/data/model_weights/ATEK_example_cubercnn_weights_trained_on_ADT\"\n",
- "# load pre-trained CubeRCNN model\n",
- "model_config_file = os.path.join(model_ckpt_path, \"config.yaml\")\n",
- "conf = OmegaConf.load(model_config_file)\n",
- "\n",
- "# setup config and model\n",
- "model_config, model = create_inference_model(\n",
- " model_config_file, model_ckpt_path, use_cpu_only = True\n",
- ")\n",
- "\n",
- "# Cache inference results for visualization\n",
- "input_output_data_pairs = []\n",
- "\n",
- "# Loop over created Pytorch Dataloader, only 5 batches for demonstration\n",
- "with torch.no_grad():\n",
- " for cubercnn_input_data in tqdm(\n",
- " islice(cubercnn_dataloader, 5),\n",
- " desc=\"Inference progress: \",\n",
- " ):\n",
- " cubercnn_model_output = model(cubercnn_input_data)\n",
- "\n",
- " # cache inference results for visualization\n",
- " input_output_data_pairs.append((cubercnn_input_data, cubercnn_model_output))\n",
- "\n",
- "logger.info(\"Inference completed.\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "uKegL5oFtEK-"
- },
- "source": [
- "## Visualize inference result\n",
- "\n",
- "If you did not see the visualization, please run the following code block again. Sometimes rerun visualization on colab notebook is not stable."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "SHTdX-AxkZR_",
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "from atek.viz.cubercnn_visualizer import CubercnnVisualizer\n",
- "\n",
- "# Visualize cached inference results\n",
- "logger.info(\"Visualizing inference results.\")\n",
- "cubercnn_visualizer = CubercnnVisualizer(viz_prefix = \"inference_visualizer\", show_on_notebook = True, conf = viz_conf)\n",
- "# cubercnn_visualizer = CubercnnVisualizer(viz_prefix = \"inference_visualizer\", conf = infer_viz_conf)\n",
- "for input_data_as_list, output_data_as_list in input_output_data_pairs:\n",
- " for single_cubercnn_input, single_cubercnn_output in zip(input_data_as_list, output_data_as_list):\n",
- " timestamp_ns = single_cubercnn_input[\"timestamp_ns\"]\n",
- " # Plot RGB image\n",
- " cubercnn_visualizer.plot_cubercnn_img(single_cubercnn_input[\"image\"], timestamp_ns = timestamp_ns)\n",
- "\n",
- " # Plot GT and prediction in different colors\n",
- " single_cubercnn_output[\"T_world_camera\"] = single_cubercnn_input[\"T_world_camera\"] # This patch is needed for visualization\n",
- " cubercnn_visualizer.plot_cubercnn_dict(cubercnn_dict = single_cubercnn_input, timestamp_ns = timestamp_ns, plot_color = cubercnn_visualizer.COLOR_GREEN, suffix = \"_model_input\")\n",
- " cubercnn_visualizer.plot_cubercnn_dict(cubercnn_dict = single_cubercnn_output, timestamp_ns = timestamp_ns, plot_color = cubercnn_visualizer.COLOR_RED, suffix = \"_model_output\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Ttd-XYROmW1E"
- },
- "source": [
- "## Part 3: Evaluate model performance\n",
- "ATEK provides **per-task**:\n",
- "1. Standardized prediction file formats.\n",
- "2. Lib for common eval metrics.\n",
- "3. Benchmarking scripts.\n",
- "\n",
- "Example prediction file format for 3D object detection:\n",
- "\n",
- "| time_ns | tx_world_object | ty_world_object | tz_world_object | qw_world_object | qx_world_object | qy_world_object | qz_world_object | scale_x | scale_y | scale_z | name | instance | sem_id | prob |\n",
- "|---------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|---------|---------|---------|---------|----------|--------|--------|\n",
- "| 14588033546600| -4.119894 | 0.986124 | 2.796770 | 0.008052 | -0.022706 | -0.010150 | 0.999658 | 0.190 | 2.146 | 0.900 | door | -1 | 32 | 0.994462|\n",
- "| 14588033546600| -3.875954 | 0.837941 | 4.056602 | 0.009215 | -0.015670 | 0.999661 | -0.018645 | 0.325 | 1.697 | 0.964 | display | -1 | 37 | 0.994381|"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ISxm_8RqmdIt"
- },
- "source": [
- "### Write inference results into ATEK-format csv files\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "AU9azyidmVUG"
- },
- "outputs": [],
- "source": [
- "from atek.evaluation.static_object_detection.obb3_csv_io import AtekObb3CsvWriter\n",
- "data_dir = \"/content/data\"\n",
- "os.makedirs(data_dir, exist_ok = True)\n",
- "\n",
- "gt_writer = AtekObb3CsvWriter(output_filename = os.path.join(data_dir, \"gt_obbs.csv\"))\n",
- "prediction_writer = AtekObb3CsvWriter(output_filename = os.path.join(data_dir, \"prediction_obbs.csv\"))\n",
- "\n",
- "for input_data_as_list, output_data_as_list in input_output_data_pairs:\n",
- " for single_cubercnn_input, single_cubercnn_output in zip(input_data_as_list, output_data_as_list):\n",
- " timestamp_ns = single_cubercnn_input[\"timestamp_ns\"]\n",
- " single_cubercnn_output[\"T_world_camera\"] = single_cubercnn_input[\"T_world_camera\"]\n",
- "\n",
- " gt_writer.write_from_cubercnn_dict(cubercnn_dict = single_cubercnn_input, timestamp_ns = timestamp_ns)\n",
- " prediction_writer.write_from_cubercnn_dict(cubercnn_dict = single_cubercnn_output, timestamp_ns = timestamp_ns)\n",
- "logger.info(\"Finished writing obb csv files\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "oUmOT3t8mq2n"
- },
- "source": [
- "### Call ATEK's benchmarking script to evaluate the results"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "6snS7A2NmVEW"
- },
- "outputs": [],
- "source": [
- "benchmarking_command = [\n",
- " \"python3\", f\"{atek_src_path}/tools/benchmarking_static_object_detection.py\",\n",
- " \"--pred-csv\", f\"{data_dir}/prediction_obbs.csv\",\n",
- " \"--gt-csv\", f\"{data_dir}/gt_obbs.csv\",\n",
- " \"--output-file\", f\"{data_dir}/atek_metrics.json\"\n",
- "]\n",
- "return_code = run_command_and_display_output(benchmarking_command)"
- ]
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "924Eqh3ErHWn"
+ },
+ "source": [
+ "# ATEK CoLab Notebook\n",
+ "\n",
+ "Welcome to ATEK CoLab Notebook. This notebook\n",
+ " will walk through the steps of preparing an Aria data sequence with annotations ([AriaDigitalTwin (ADT)](https://www.projectaria.com/datasets/adt/)), for use in a 3D object detection ML task.\n",
+ "We will go through the following steps:\n",
+ "1. downloading ADT sample data\n",
+ "2. preprocess ADT sample data\n",
+ "3. visualize the preprocessed data\n",
+ "4. run model inference with ATEK preprocessed data\n",
+ "5. evaluate model performance.\n",
+ "\n",
+ "## Environment set up"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "YygI9xSxQJen",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "!pip install 'git+https://github.com/YLouWashU/omni3d.git'\n",
+ "!pip install projectaria-atek==1.0.0\n",
+ "!pip install 'git+https://github.com/facebookresearch/detectron2.git'\n",
+ "!pip install iopath\n",
+ "import sys\n",
+ "import torch\n",
+ "pyt_version_str=torch.__version__.split(\"+\")[0].replace(\".\", \"\")\n",
+ "version_str=\"\".join([\n",
+ " f\"py3{sys.version_info.minor}_cu\",\n",
+ " torch.version.cuda.replace(\".\",\"\"),\n",
+ " f\"_pyt{pyt_version_str}\"\n",
+ "])\n",
+ "!pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
+ "!pip install rerun-sdk[notebook]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "mejA1LJYoqny"
+ },
+ "outputs": [],
+ "source": [
+ "import requests\n",
+ "\n",
+ "url = \"https://www.projectaria.com/async/sample/download/?bucket=atek&filename=ATEK_example_model_weights.tar\"\n",
+ "response = requests.get(url)\n",
+ "\n",
+ "# Specify the path where you want to save the file\n",
+ "file_path = \"/content/ATEK_example_model_weights.tar\"\n",
+ "\n",
+ "# Open the file in binary write mode and write the contents of the response\n",
+ "with open(file_path, \"wb\") as file:\n",
+ " file.write(response.content)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "s2-IjOuiBsim",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "!mkdir -p /content/data\n",
+ "!tar -xvf /content/ATEK_example_model_weights.tar -C /content/data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "EUZA4zzGfrdJ",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "!cd /content/data\n",
+ "!git clone https://github.com/facebookresearch/ATEK.git\n",
+ "!unzip /content/atek.zip -d /content/data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "3j8s4r7VM1y_"
+ },
+ "outputs": [],
+ "source": [
+ "print(\"----debug: manually upload zip file\")\n",
+ "!unzip /content/ATEK-main.zip -d /content/data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "tTp3xS-Ggpe4"
+ },
+ "source": [
+ "## Part1: Data proprocessing\n",
+ "\n",
+ "### Data preprocessing requirements\n",
+ "ADT sequence has:\n",
+ "1. Aria recording (VRS).\n",
+ "2. MPS trajectory file (CSV).\n",
+ "3. Object detection annotation files (3 csv files + 1 json file).\n",
+ "\n",
+ "CubeRCNN model needs synchronized data frame containing:\n",
+ "1. Upright RGB camera image.\n",
+ "2. Linear camera calibration matrix.\n",
+ "3. Object bounding box annotations in 2D + 3D.\n",
+ "4. Camera-to-object poses.\n",
+ "\n",
+ "**Before ATEK**, users need to implement all the followings to prepare ADT sequence into CubeRCNN model:\n",
+ "1. Parse in ADT sequence data using `projectaria_tools` lib. \n",
+ "2. Properly synchronize sensor + annotation data into training samples.\n",
+ "3. Perform additional image & data processing:\n",
+ " 1. Undistort image + camera calibration.\n",
+ " 2. Rescale camera resolution.\n",
+ " 3. Rotate image + camera calibration.\n",
+ " 4. Undistort + rescale + rotate object 2D bounding boxes accordingly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "Yy79EFR8UZMS",
+ "jupyter": {
+ "outputs_hidden": true
}
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "name": "python"
+ },
+ "outputs": [],
+ "source": [
+ "import faulthandler\n",
+ "import logging\n",
+ "import os\n",
+ "from logging import StreamHandler\n",
+ "import numpy as np\n",
+ "from typing import Dict, List, Optional\n",
+ "import torch\n",
+ "import sys\n",
+ "from itertools import islice\n",
+ "from tqdm import tqdm\n",
+ "\n",
+ "from atek.data_preprocess.genera_atek_preprocessor_factory import (\n",
+ " create_general_atek_preprocessor_from_conf,\n",
+ ")\n",
+ "from atek.viz.atek_visualizer import NativeAtekSampleVisualizer\n",
+ "from atek.data_preprocess.general_atek_preprocessor import GeneralAtekPreprocessor\n",
+ "from atek.data_loaders.atek_wds_dataloader import (\n",
+ " create_native_atek_dataloader\n",
+ ")\n",
+ "\n",
+ "from atek.data_loaders.cubercnn_model_adaptor import (\n",
+ " cubercnn_collation_fn,\n",
+ " create_atek_dataloader_as_cubercnn\n",
+ ")\n",
+ "from atek.evaluation.static_object_detection.obb3_csv_io import AtekObb3CsvWriter\n",
+ "\n",
+ "from atek.data_preprocess.atek_data_sample import (\n",
+ " create_atek_data_sample_from_flatten_dict,\n",
+ ")\n",
+ "from cubercnn.config import get_cfg_defaults\n",
+ "from cubercnn.modeling.backbone import build_dla_from_vision_fpn_backbone\n",
+ "from cubercnn.modeling.meta_arch import build_model\n",
+ "\n",
+ "from detectron2.checkpoint import DetectionCheckpointer\n",
+ "from detectron2.config import get_cfg\n",
+ "from omegaconf import OmegaConf\n",
+ "import subprocess"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "YJNAP3W2g_CL"
+ },
+ "source": [
+ "## Download ADT sequences from projectaria.com"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "Fy-pj5ejrMn8",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "adt_sample_path = \"./adt_sample_data\"\n",
+ "data_sequence_url = \"https://www.projectaria.com/async/sample/download/?bucket=adt&filename=aria_digital_twin_test_data_v2.zip\"\n",
+ "command_list = [\n",
+ " f\"mkdir -p {adt_sample_path}\",\n",
+ " # Download sample data\n",
+ " f'curl -o {adt_sample_path}/adt_sample_data.zip -C - -O -L \"{data_sequence_url}\"',\n",
+ " # Unzip the sample data\n",
+ " f\"unzip -o {adt_sample_path}/adt_sample_data.zip -d {adt_sample_path}\"\n",
+ "]\n",
+ "sequence_path = f\"{adt_sample_path}/Apartment_release_golden_skeleton_seq100_10s_sample_M1292\"\n",
+ "\n",
+ "# Execute the commands for downloading dataset\n",
+ "for command in command_list:\n",
+ " subprocess.run(command, shell=True, check=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "mGJc-2B_hF03"
+ },
+ "source": [
+ "### Set up data and code paths"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "KGaKsE5IEol8"
+ },
+ "outputs": [],
+ "source": [
+ "example_adt_data_dir = f\"{adt_sample_path}/Apartment_release_golden_skeleton_seq100_10s_sample_M1292\"\n",
+ "sequence_name = \"Apartment_release_golden_skeleton_seq100_10s_sample_M1292\"\n",
+ "atek_src_path = \"/content/data/ATEK-main\"\n",
+ "category_mapping_file = f\"{atek_src_path}/data/adt_prototype_to_atek.csv\"\n",
+ "atek_preprocess_config_path = f\"{atek_src_path}/examples/data/adt_cubercnn_preprocess_config.yaml\"\n",
+ "preprocess_conf = OmegaConf.load(atek_preprocess_config_path)\n",
+ "# Take viz conf out of preprocess conf\n",
+ "viz_conf = preprocess_conf.visualizer\n",
+ "del preprocess_conf.visualizer\n",
+ "\n",
+ "# Create viz conf for inference viewer\n",
+ "infer_viz_config_path = f\"{atek_src_path}/examples/data/infer_viz_conf.yaml\"\n",
+ "infer_viz_conf = OmegaConf.load(infer_viz_config_path)\n",
+ "\n",
+ "output_wds_path = f\"{atek_src_path}/examples/data/wds_output\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "nBcS5GTahgx5"
+ },
+ "source": [
+ "### Helper functions"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "uw-Yzp1pjyJd"
+ },
+ "outputs": [],
+ "source": [
+ "faulthandler.enable()\n",
+ "\n",
+ "# Configure logging to display the log messages in the notebook\n",
+ "logging.basicConfig(\n",
+ " level=logging.INFO,\n",
+ " format='%(asctime)s - %(levelname)s - %(message)s',\n",
+ " handlers=[\n",
+ " logging.StreamHandler(sys.stdout)\n",
+ " ]\n",
+ ")\n",
+ "\n",
+ "logger = logging.getLogger()\n",
+ "\n",
+ "\n",
+ "# -------------------- Helper functions --------------------#\n",
+ "def print_data_sample_dict_content(data_sample, if_pretty: bool = False):\n",
+ " \"\"\"\n",
+ " A helper function to print the content of data sample dict\n",
+ " \"\"\"\n",
+ " logger.info(\"Printing the content in a ATEK data sample dict: \")\n",
+ " for key, val in data_sample.items():\n",
+ " if if_pretty and \"#\" in key:\n",
+ " key = key.split(\"#\", 1)[1]\n",
+ "\n",
+ " msg = f\"\\t {key}: is a {type(val)}, \"\n",
+ " if isinstance(val, torch.Tensor):\n",
+ " msg += f\"\\n \\t\\t\\t\\t with tensor dtype of {val.dtype}, and shape of : {val.shape}\"\n",
+ " elif isinstance(val, list):\n",
+ " msg += f\"with len of : {len(val)}\"\n",
+ " elif isinstance(val, str):\n",
+ " msg += f\"value is {val}\"\n",
+ " else:\n",
+ " pass\n",
+ " logger.info(msg)\n",
+ "\n",
+ "def run_command_and_display_output(command):\n",
+ " # Start the process\n",
+ " process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n",
+ "\n",
+ " # Poll process.stdout to show stdout live\n",
+ " while True:\n",
+ " output = process.stdout.readline()\n",
+ " if output == '' and process.poll() is not None:\n",
+ " break\n",
+ " if output:\n",
+ " print(output.strip())\n",
+ " rc = process.poll()\n",
+ " return rc\n",
+ "\n",
+ "def create_inference_model(config_file, ckpt_dir, use_cpu_only=False):\n",
+ " \"\"\"\n",
+ " Create the model for inference pipeline, with the model config.\n",
+ " \"\"\"\n",
+ " # Create default model configuration\n",
+ " model_config = get_cfg()\n",
+ " model_config.set_new_allowed(True)\n",
+ " get_cfg_defaults(model_config)\n",
+ "\n",
+ " # add extra configs for data\n",
+ " model_config.MAX_TRAINING_ATTEMPTS = 3\n",
+ " model_config.TRAIN_LIST = \"\"\n",
+ " model_config.TEST_LIST = \"\"\n",
+ " model_config.TRAIN_WDS_DIR = \"\"\n",
+ " model_config.TEST_WDS_DIR = \"\"\n",
+ " model_config.ID_MAP_JSON = \"\"\n",
+ " model_config.OBJ_PROP_JSON = \"\"\n",
+ " model_config.CATEGORY_JSON = \"\"\n",
+ " model_config.DATASETS.OBJECT_DETECTION_MODE = \"\"\n",
+ " model_config.SOLVER.VAL_MAX_ITER = 0\n",
+ " model_config.SOLVER.MAX_EPOCH = 0\n",
+ "\n",
+ " model_config.merge_from_file(config_file)\n",
+ " if use_cpu_only:\n",
+ " model_config.MODEL.DEVICE = \"cpu\"\n",
+ " model_config.freeze()\n",
+ "\n",
+ " model = build_model(model_config, priors=None)\n",
+ "\n",
+ " _ = DetectionCheckpointer(model, save_dir=ckpt_dir).resume_or_load(\n",
+ " model_config.MODEL.WEIGHTS, resume=True\n",
+ " )\n",
+ " model.eval()\n",
+ "\n",
+ " return model_config, model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rvdmyAIrhngs"
+ },
+ "source": [
+ "### Set up and run ATEK data preprocessor"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "Ua3oo2TSsYm0",
+ "jupyter": {
+ "outputs_hidden": true
}
+ },
+ "outputs": [],
+ "source": [
+ "# Create ATEK preprocessor from conf. It will automatically choose which type of sample to build.\n",
+ "atek_preprocessor = create_general_atek_preprocessor_from_conf(\n",
+ " # [required]\n",
+ " conf=preprocess_conf,\n",
+ " raw_data_folder = example_adt_data_dir,\n",
+ " sequence_name = sequence_name,\n",
+ " # [optional]\n",
+ " output_wds_folder=output_wds_path,\n",
+ " output_viz_file=os.path.join(example_adt_data_dir, \"atek_preprocess_viz.rrd\"),\n",
+ " category_mapping_file=category_mapping_file,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cxgiEJAFiDDH"
+ },
+ "source": [
+ "## Preprocessed ATEK data sample content\n",
+ "* Preprocessing input: VRS + csv + jsons\n",
+ "* Preprocessing output (in memory): ATEK data samples: `Dict[torch.Tensor, str, or Dict]`\n",
+ "* Preprocessing output (on local disk): WebDataset (WDS) tar files."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "BS-OuwW9scUP",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "atek_data_sample = atek_preprocessor[0]\n",
+ "atek_data_sample_dict = atek_data_sample.to_flatten_dict()\n",
+ "print_data_sample_dict_content(atek_data_sample_dict)\n",
+ "\n",
+ "# Loop over all samples, and write valid ones to local tar files.\n",
+ "atek_preprocessor.process_all_samples(write_to_wds_flag=True, viz_flag=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qDUCTF0gs48Y"
+ },
+ "source": [
+ "## Visualize preprocessed ATEK data sample"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cVCo0il9abVi"
+ },
+ "outputs": [],
+ "source": [
+ "tar_file_urls = [os.path.join(output_wds_path, f\"shards-000{i}.tar\") for i in range(2)]\n",
+ "atek_dataloader = create_native_atek_dataloader(urls = tar_file_urls, batch_size = None, num_workers = 1)\n",
+ "atek_viz = NativeAtekSampleVisualizer(viz_prefix = \"notebook atek viz\", show_on_notebook = True, conf = viz_conf)\n",
+ "for atek_sample in tqdm(islice(atek_dataloader, 10)):\n",
+ " atek_viz.plot_atek_sample_as_dict(atek_sample)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LVbIgzI6jciw"
+ },
+ "source": [
+ "# Part 2: Run Object detection inference using pre-trained CubeRCNN model\n",
+ "In this example, we demonstrate how to run model inference with preprocessed ATEK data streamed from Data Store."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "J2tOwBZ8k4cp"
+ },
+ "source": [
+ "### Create PyTorch DataLoader, converted to CubeRCNN format\n",
+ "User can add a data transform function from ATEK format -> CubeRCNN format:\n",
+ "1. Dict key remapping.\n",
+ "2. Tensor reshaping & reordering.\n",
+ "3. Other data transformations.\n",
+ "\n",
+ "Example data transform function for CubeRCNN model: [src code](https://www.internalfb.com/code/fbsource/[a5c3831c045bc718862d1c512e84d4ed6f79d722]/fbcode/surreal/data_services/atek/atek/data_loaders/cubercnn_model_adaptor.py?lines=44-74)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "i7GgyyaWZvFB",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "logger = logging.getLogger()\n",
+ "logger.info(\n",
+ " \"-------------------- ATEK WDS data can loaded into Model-specific format --------------- \"\n",
+ ")\n",
+ "# The CubeRCNN ModelAdaptor class is wrapped in this function\n",
+ "cubercnn_dataloader = create_atek_dataloader_as_cubercnn(urls = tar_file_urls, batch_size = 1, num_workers = 1)\n",
+ "first_cubercnn_sample = next(iter(cubercnn_dataloader))\n",
+ "logger.info(f\"Loading WDS into CubeRCNN format, each sample contains the following keys: {first_cubercnn_sample[0].keys()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pWyaUUQPk6Te"
+ },
+ "source": [
+ "### Run model inference"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "-0RWOhlrjaKE"
+ },
+ "outputs": [],
+ "source": [
+ "from tqdm import tqdm\n",
+ "model_ckpt_path = \"/content/data/model_weights/ATEK_example_cubercnn_weights_trained_on_ADT\"\n",
+ "# load pre-trained CubeRCNN model\n",
+ "model_config_file = os.path.join(model_ckpt_path, \"config.yaml\")\n",
+ "conf = OmegaConf.load(model_config_file)\n",
+ "\n",
+ "# setup config and model\n",
+ "model_config, model = create_inference_model(\n",
+ " model_config_file, model_ckpt_path, use_cpu_only = True\n",
+ ")\n",
+ "\n",
+ "# Cache inference results for visualization\n",
+ "input_output_data_pairs = []\n",
+ "\n",
+ "# Loop over created Pytorch Dataloader, only 5 batches for demonstration\n",
+ "with torch.no_grad():\n",
+ " for cubercnn_input_data in tqdm(\n",
+ " islice(cubercnn_dataloader, 5),\n",
+ " desc=\"Inference progress: \",\n",
+ " ):\n",
+ " cubercnn_model_output = model(cubercnn_input_data)\n",
+ "\n",
+ " # cache inference results for visualization\n",
+ " input_output_data_pairs.append((cubercnn_input_data, cubercnn_model_output))\n",
+ "\n",
+ "logger.info(\"Inference completed.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "uKegL5oFtEK-"
+ },
+ "source": [
+ "## Visualize inference result\n",
+ "\n",
+ "If you did not see the visualization, please run the following code block again. Sometimes rerun visualization on colab notebook is not stable."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true,
+ "id": "SHTdX-AxkZR_",
+ "jupyter": {
+ "outputs_hidden": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "from atek.viz.cubercnn_visualizer import CubercnnVisualizer\n",
+ "\n",
+ "# Visualize cached inference results\n",
+ "logger.info(\"Visualizing inference results.\")\n",
+ "cubercnn_visualizer = CubercnnVisualizer(viz_prefix = \"inference_visualizer\", show_on_notebook = True, conf = viz_conf)\n",
+ "# cubercnn_visualizer = CubercnnVisualizer(viz_prefix = \"inference_visualizer\", conf = infer_viz_conf)\n",
+ "for input_data_as_list, output_data_as_list in input_output_data_pairs:\n",
+ " for single_cubercnn_input, single_cubercnn_output in zip(input_data_as_list, output_data_as_list):\n",
+ " timestamp_ns = single_cubercnn_input[\"timestamp_ns\"]\n",
+ " # Plot RGB image\n",
+ " cubercnn_visualizer.plot_cubercnn_img(single_cubercnn_input[\"image\"], timestamp_ns = timestamp_ns)\n",
+ "\n",
+ " # Plot GT and prediction in different colors\n",
+ " single_cubercnn_output[\"T_world_camera\"] = single_cubercnn_input[\"T_world_camera\"] # This patch is needed for visualization\n",
+ " cubercnn_visualizer.plot_cubercnn_dict(cubercnn_dict = single_cubercnn_input, timestamp_ns = timestamp_ns, plot_color = cubercnn_visualizer.COLOR_GREEN, suffix = \"_model_input\")\n",
+ " cubercnn_visualizer.plot_cubercnn_dict(cubercnn_dict = single_cubercnn_output, timestamp_ns = timestamp_ns, plot_color = cubercnn_visualizer.COLOR_RED, suffix = \"_model_output\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ttd-XYROmW1E"
+ },
+ "source": [
+ "## Part 3: Evaluate model performance\n",
+ "ATEK provides **per-task**:\n",
+ "1. Standardized prediction file formats.\n",
+ "2. Lib for common eval metrics.\n",
+ "3. Benchmarking scripts.\n",
+ "\n",
+ "Example prediction file format for 3D object detection:\n",
+ "\n",
+ "| time_ns | tx_world_object | ty_world_object | tz_world_object | qw_world_object | qx_world_object | qy_world_object | qz_world_object | scale_x | scale_y | scale_z | name | instance | sem_id | prob |\n",
+ "|---------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|---------|---------|---------|---------|----------|--------|--------|\n",
+ "| 14588033546600| -4.119894 | 0.986124 | 2.796770 | 0.008052 | -0.022706 | -0.010150 | 0.999658 | 0.190 | 2.146 | 0.900 | door | -1 | 32 | 0.994462|\n",
+ "| 14588033546600| -3.875954 | 0.837941 | 4.056602 | 0.009215 | -0.015670 | 0.999661 | -0.018645 | 0.325 | 1.697 | 0.964 | display | -1 | 37 | 0.994381|"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ISxm_8RqmdIt"
+ },
+ "source": [
+ "### Write inference results into ATEK-format csv files\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "AU9azyidmVUG"
+ },
+ "outputs": [],
+ "source": [
+ "from atek.evaluation.static_object_detection.obb3_csv_io import AtekObb3CsvWriter\n",
+ "data_dir = \"/content/data\"\n",
+ "os.makedirs(data_dir, exist_ok = True)\n",
+ "\n",
+ "gt_writer = AtekObb3CsvWriter(output_filename = os.path.join(data_dir, \"gt_obbs.csv\"))\n",
+ "prediction_writer = AtekObb3CsvWriter(output_filename = os.path.join(data_dir, \"prediction_obbs.csv\"))\n",
+ "\n",
+ "for input_data_as_list, output_data_as_list in input_output_data_pairs:\n",
+ " for single_cubercnn_input, single_cubercnn_output in zip(input_data_as_list, output_data_as_list):\n",
+ " timestamp_ns = single_cubercnn_input[\"timestamp_ns\"]\n",
+ " single_cubercnn_output[\"T_world_camera\"] = single_cubercnn_input[\"T_world_camera\"]\n",
+ "\n",
+ " gt_writer.write_from_cubercnn_dict(cubercnn_dict = single_cubercnn_input, timestamp_ns = timestamp_ns)\n",
+ " prediction_writer.write_from_cubercnn_dict(cubercnn_dict = single_cubercnn_output, timestamp_ns = timestamp_ns)\n",
+ "logger.info(\"Finished writing obb csv files\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "oUmOT3t8mq2n"
+ },
+ "source": [
+ "### Call ATEK's benchmarking script to evaluate the results"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "6snS7A2NmVEW"
+ },
+ "outputs": [],
+ "source": [
+ "benchmarking_command = [\n",
+ " \"python3\", f\"{atek_src_path}/tools/benchmarking_static_object_detection.py\",\n",
+ " \"--pred-csv\", f\"{data_dir}/prediction_obbs.csv\",\n",
+ " \"--gt-csv\", f\"{data_dir}/gt_obbs.csv\",\n",
+ " \"--output-file\", f\"{data_dir}/atek_metrics.json\"\n",
+ "]\n",
+ "return_code = run_command_and_display_output(benchmarking_command)"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
},
- "nbformat": 4,
- "nbformat_minor": 0
-}
\ No newline at end of file
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.19"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/examples/Demo_2_data_store_and_inference.ipynb b/examples/Demo_2_data_store_and_inference.ipynb
index a866248..74c8f84 100644
--- a/examples/Demo_2_data_store_and_inference.ipynb
+++ b/examples/Demo_2_data_store_and_inference.ipynb
@@ -2,22 +2,16 @@
"cells": [
{
"cell_type": "markdown",
- "id": "034c6e39-f092-4232-8bf8-67b057ca36b9",
+ "id": "17062c55-e989-4102-80c9-36af4fa9dd7e",
"metadata": {},
"source": [
- "# Demo 2: ATEK Data Store and Model Inference\n",
- "\n",
- "Note: if you are seeing contaminated warning messages about `webdataset/autodecode.py: Future Warning`, this can be resolved by installing the main-branch of `webdataset` library: \n",
- "```\n",
- "mamba uninstall webdataset\n",
- "pip install git+https://github.com/webdataset/webdataset.git@69673059ab680195f6f6d2727077f69693c60af9\n",
- "```"
+ "# Demo 2: ATEK Data Store and Model Inference"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "cc9edbdc-95e5-4e57-82af-306b152ee5de",
+ "id": "c92d6196-e905-4812-897d-152c968a4f3f",
"metadata": {},
"outputs": [],
"source": [
@@ -122,18 +116,23 @@
},
{
"cell_type": "markdown",
- "id": "4bc7ebd6-f61a-4639-8e1e-64f57f7cc2e7",
+ "id": "0bc1e381-e1a5-4edb-a389-04407a62898b",
"metadata": {},
"source": [
"### Setup data and code path\n",
- "- Download trained model weights from [here](https://www.projectaria.com/async/sample/download/?bucket=adt&filename=ATEK_example_model_weights.tar), and by using the trained model weights, you need to agree to the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en). \n",
- "- Download ATEK Data Store json file from [here](https://www.projectaria.com/datasets/adt/). \n"
+ "- Download trained model weights from [here](https://www.projectaria.com/async/sample/download/?bucket=atek&filename=ATEK_example_model_weights.tar). By downloading this file, you acknowledge that you have read, understood, and agree to be bound by the terms of the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en) software license. \n",
+ "- Download ATEK Data Store json file from [here](https://www.projectaria.com/datasets/adt/). \n",
+ "- Note: if you are seeing contaminated warning messages about `webdataset/autodecode.py: Future Warning`, this can be resolved by installing the `beta` version of `webdataset` library: \n",
+ "```\n",
+ "pip uninstall webdataset\n",
+ "pip install git+https://github.com/webdataset/webdataset.git@69673059ab680195f6f6d2727077f69693c60af9\n",
+ "```\n"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "a5702ba5-28b9-444f-9f82-535850af6d1f",
+ "id": "3d00265e-93c2-43be-93b4-c6b50d0042a2",
"metadata": {},
"outputs": [],
"source": [
@@ -150,7 +149,7 @@
},
{
"cell_type": "markdown",
- "id": "d5fcfab0-c9ea-473d-9227-270e21536cae",
+ "id": "33947132-d97f-4edc-9be3-bfc9e764ae34",
"metadata": {},
"source": [
"## Part 1: ATEK Data Store\n",
@@ -166,7 +165,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "ea37f2f5-d2bd-4df6-adfb-6c3c6238bc1f",
+ "id": "e9148953-6090-44ca-a0cb-88f6c21ea6e0",
"metadata": {},
"outputs": [],
"source": [
@@ -191,7 +190,7 @@
},
{
"cell_type": "markdown",
- "id": "d2a0254c-5be1-43fc-a233-ed1f582c09c8",
+ "id": "ccac525a-6738-44ba-a858-b38ad7e09708",
"metadata": {},
"source": [
"# Part 2: Run Object detection inference using pre-trained CubeRCNN model\n",
@@ -200,7 +199,7 @@
},
{
"cell_type": "markdown",
- "id": "809894fe-da3b-45f7-a01b-cb8454803dfe",
+ "id": "16a2cf3d-f838-44ae-9700-923933664abc",
"metadata": {},
"source": [
"### Create a PyTorch DataLoader from ATEK WDS files\n",
@@ -210,7 +209,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "2a2a8aac-739b-4869-9633-c9869b1f7554",
+ "id": "799babf7-0685-4190-bbca-9a09eb11f40b",
"metadata": {},
"outputs": [],
"source": [
@@ -225,7 +224,7 @@
},
{
"cell_type": "markdown",
- "id": "cc64898a-4630-4f46-bd9e-25d38b6f5951",
+ "id": "1cc8b72f-c60f-4dc6-b962-9760a0b7cff9",
"metadata": {},
"source": [
"### Create PyTorch DataLoader, converted to CubeRCNN format\n",
@@ -240,7 +239,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "5b4f6edf-fe8f-498d-8813-de8d415340e8",
+ "id": "3aa1415b-90a2-47aa-a0fb-b794e880fd7b",
"metadata": {},
"outputs": [],
"source": [
@@ -251,7 +250,7 @@
},
{
"cell_type": "markdown",
- "id": "e94fe5a5-6b65-40ef-9c37-34ebed8166d3",
+ "id": "f848d0ea-a201-4206-ab51-edc8c7699240",
"metadata": {},
"source": [
"### Run model inference"
@@ -260,7 +259,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "7de44e4a-5617-481e-9376-61d57f0c804d",
+ "id": "7276080b-dab7-422f-abf0-0d9d55369445",
"metadata": {},
"outputs": [],
"source": [
@@ -296,7 +295,7 @@
},
{
"cell_type": "markdown",
- "id": "66b0a0ba-d0c7-4e94-9db4-ab31c1f6c4fa",
+ "id": "0d0fabca-b8d8-433e-8ece-840ea7294378",
"metadata": {},
"source": [
"### Visualize inference results"
@@ -305,7 +304,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "1699c668-03ca-4237-88b1-f31269f485df",
+ "id": "209bf47a-558d-45b3-b7e3-652fd653651d",
"metadata": {},
"outputs": [],
"source": [
@@ -328,7 +327,7 @@
},
{
"cell_type": "markdown",
- "id": "0e2b41bd-8fc5-4517-9e3b-a5659889db0b",
+ "id": "fce6fc93-d10a-4b6f-9df8-99b182661f81",
"metadata": {},
"source": [
"## Part 3: Evaluate model performance\n",
@@ -347,7 +346,7 @@
},
{
"cell_type": "markdown",
- "id": "9cb1b335-2109-42e2-bd12-1addc5bb19fc",
+ "id": "f9e01cd6-e82e-4c5e-99a2-b9a7a50229ac",
"metadata": {},
"source": [
"### Write inference results into ATEK-format csv files\n"
@@ -356,7 +355,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "fead1dd1-2318-4d59-af84-f62932eb4325",
+ "id": "8898790f-6606-40e3-ad98-9d2fa5b0646d",
"metadata": {},
"outputs": [],
"source": [
@@ -376,7 +375,7 @@
},
{
"cell_type": "markdown",
- "id": "4426e44a-89ed-47b4-839e-7c7f14926133",
+ "id": "3feecc51-1a7b-46b5-a4f7-62adcaf98953",
"metadata": {},
"source": [
"### Call ATEK's benchmarking script to evaluate the results"
@@ -385,7 +384,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "a62dbd85-a367-40cf-a30e-dbef0bbcbf46",
+ "id": "5d3045ac-b67e-4bc6-ba8f-99bb1efe7e0a",
"metadata": {},
"outputs": [],
"source": [
diff --git a/examples/Demo_3_model_training.ipynb b/examples/Demo_3_model_training.ipynb
index 5d37cff..2952eb4 100644
--- a/examples/Demo_3_model_training.ipynb
+++ b/examples/Demo_3_model_training.ipynb
@@ -2,22 +2,16 @@
"cells": [
{
"cell_type": "markdown",
- "id": "702636c5-600d-4fb1-a0f0-4ae108330d88",
+ "id": "6043f900-8982-4805-9936-b69b8e117952",
"metadata": {},
"source": [
- "# Demo 3: Model training in ATEK\n",
- "\n",
- "Note: if you are seeing contaminated warning messages about `webdataset/autodecode.py: Future Warning`, this can be resolved by installing the main-branch of `webdataset` library: \n",
- "```\n",
- "mamba uninstall webdataset\n",
- "pip install git+https://github.com/webdataset/webdataset.git@69673059ab680195f6f6d2727077f69693c60af9\n",
- "```"
+ "# Demo 3: Model training in ATEK"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "8e83e890-0945-4f50-8ada-fa427d08d7ba",
+ "id": "10474d37-f0be-4f2d-ab0d-0de1ea8391b7",
"metadata": {},
"outputs": [],
"source": [
@@ -73,17 +67,22 @@
},
{
"cell_type": "markdown",
- "id": "87b61ab1-661e-450f-961d-ed6fb0fa082f",
+ "id": "12a14cfc-f9de-4d7b-ab2f-2b3b8f9f58ab",
"metadata": {},
"source": [
"### Set up data and code paths\n",
- "- Download ATEK Data Store json file from [here](https://www.projectaria.com/datasets/adt/). "
+ "- Download ATEK Data Store json file from [here](https://www.projectaria.com/datasets/adt/). \n",
+ "- Note: if you are seeing contaminated warning messages about `webdataset/autodecode.py: Future Warning`, this can be resolved by installing the main-branch of `webdataset` library: \n",
+ "```\n",
+ "mamba uninstall webdataset\n",
+ "pip install git+https://github.com/webdataset/webdataset.git@69673059ab680195f6f6d2727077f69693c60af9\n",
+ "```"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "cd0fed8b-301d-4df5-9510-699471de3962",
+ "id": "d8e4064a-b9cd-4612-a7a0-2beac423e154",
"metadata": {},
"outputs": [],
"source": [
@@ -101,7 +100,7 @@
},
{
"cell_type": "markdown",
- "id": "054a7c68-a63b-4c1a-b720-9b1d28c3533a",
+ "id": "55da180e-340f-4888-a0d0-0a20498c1c1c",
"metadata": {},
"source": [
"### Step 1: Download preprocessed data from ATEK Data Store"
@@ -110,7 +109,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "3fe01ea3-422e-4411-a555-66df9dcf89b6",
+ "id": "d42b9dd4-feb1-416b-8dea-38391327b592",
"metadata": {},
"outputs": [],
"source": [
@@ -131,7 +130,7 @@
},
{
"cell_type": "markdown",
- "id": "2e269682-b1fe-4148-9e41-d3aa0b4951d5",
+ "id": "59d7e9cf-9e4f-409b-83d7-a6dfea145d26",
"metadata": {},
"source": [
"## ATEK Training example with CubeRCNN\n",
@@ -158,7 +157,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "196b9219-7a94-4ac2-9e10-20ba40eacf3d",
+ "id": "2d6dc845-ae2e-42b4-a4e0-30323e14504a",
"metadata": {},
"outputs": [],
"source": [
@@ -182,7 +181,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "7dd72942-5454-4f8e-8544-32d594fdcb7f",
+ "id": "00833d14-c3fa-4d8d-87f2-2f1bdbfb853d",
"metadata": {},
"outputs": [],
"source": [
diff --git a/readme.md b/readme.md
index 61e9364..b48b87a 100644
--- a/readme.md
+++ b/readme.md
@@ -1,36 +1,34 @@
# Aria Training and Evaluation toolkit (ATEK)
-ATEK is a toolbox from the `projectaria` team specially built for the Machine Learning (ML) community. It seeks to lower the frictions for users using Aria data in ML research, and accelerate their development cycle, by addressing the following pain points:
+
-- `VRS` format in open Aria datasets are **not PyTorch compatible**.
-- Users need to write a lot of **hand-crafted boiler plate code** to preprocess Aria data, which requires expertise on Aria specs.
-- It is time and resource consuming to preprocess large Aria datasets.
-- There is no fair competing ground to compare model performacnes for Aria-specific tasks.
+The Aria Training and Evaluation Kit (ATEK) is a toolbox for accelerating the development of Machine Learning tasks using Aria datasets. Specifically, it provides
-To address these issues, ATEK provides the followings to the community: 
-
-- An easy-to-use data preprocessing library for Aria datasets.
-- Data Store with downloadable preprocessed Aria datasets.
+- An easy-to-use data preprocessing library for Aria datasets for converting VRS-format Aria datasets in to datasets in PyTorch-compatible format
+- Datastore of preprocessed data sets in WebDataset format
- Standardized evaluation libraries that supports the following ML perception tasks for Aria:
-
- static 3D object detection
- 3D surface reconstruction
+- Notebooks and script examples including model training, inference, and visualization
-- Rich notebook and script examples including model training, inference, and visualization.
-
-And users can engage ATEK in their projects with 3 different starting points: 
+
-- Just want to run some evaluation, even on non-Aria data? Check out [ATEK evaluation libraries](./docs/evaluation.md)!
+ATEK users can engage ATEK in their projects with 3 different starting points:
-- Want to try your trained-model on ego-centric Aria data? Just download processed data from our [Data Store](./docs/ATEK_Data_Store.md), and check out how to run [model inference](./docs/data_loading_and_inference.md)!
+- Just want to run some evaluation, even on non-Aria data? Check out [ATEK evaluation libraries](http://./docs/evaluation.md)\!
+- Want to try your trained-model on ego-centric Aria data? Just download processed data from our [Data Store](http://./docs/ATEK_Data_Store.md), and check out how to run [model inference](http://./docs/data_loading_and_inference.md)\!
+- Now ready for the full ML adventure from raw Aria data? Check out our full [table of contents](#table-of-content)\!
-- Now ready for the full ML adventure from raw Aria data? Check out our full [table of contents](#table-of-content)!
+
## Interactive Python notebook playground (Google Colab)
-[](https://colab.research.google.com/github/facebookresearch/ATEK/tree/main/examples/ATEK_CoLab_Notebook.ipynb)
+[](https://colab.research.google.com/github/facebookresearch/ATEK/blob/main/examples/ATEK_CoLab_Notebook.ipynb)
-User can start with our Google Colab notebook, which shows an example of running and evaluating a 3D object detection model called `CubeRCNN`, on an Aria Digital Twin data sequence, which involes data-preprocessing, model inference, and evaluation.
+User can start with our Google Colab notebook, which shows an example of running and evaluating a 3D object detection model called `CubeRCNN`, on an Aria Digital Twin data sequence, which involves data-preprocessing, model inference, and evaluation.
## Table of content
@@ -40,11 +38,12 @@ User can start with our Google Colab notebook, which shows an example of running
- [Data Preprocessing](./docs/preprocessing.md)
- [Data Loader for inference and training](./docs/data_loading_and_inference.md)
+ - [Model Adaptors](./docs/ModelAdaptors.md)
- [Evaluation](./docs/evaluation.md)
-- [Machine Learning tasks supported by ATEK](docs/evaluation.md)
- - [static 3D object detection](./ML_task_object_detection.md)
- - [3D surface reconstruction](./ML_task_surface_recon.md)
+- [Machine Learning tasks supported by ATEK]
+ - [static 3D object detection](./docs/ML_task_object_detection.md)
+ - [3D surface reconstruction](./docs/ML_task_surface_recon.md)
- Examples
- [Example: demo notebooks](./docs/example_demos.md)