Summary: tsia

Reviewed By: YLouWashU

Differential Revision: D53844466

fbshipit-source-id: 913a7d76be4b47174f752e73761f1553a18f6103

Summary: Fixing typo. Plus a deprecated approach: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary.

Reviewed By: chengshy

Differential Revision: D54097958

fbshipit-source-id: effb53a054c35626e2f16578dccc5a57fc9a7403

Summary:
CubeRCNN modeling needs to be imported first. `atek.model.cubercnn` has the imports.

Pull Request resolved: #9

Reviewed By: YLouWashU

Differential Revision: D54264215

Pulled By: zhengkang86

fbshipit-source-id: 1c0bd30ca9504d51768899d46644f3735a6a7a8f

Summary:
1. Enable category-based vs instance based training, via adding a `DATASETS.OBJECT_DETECTION_MODE` variable in the config yaml file.
2. Enable basic training in ASE datasets. The training seems to be (at least )up and running with following tensorboard results:
3.
![4](https://github.com/fairinternal/ATEK/assets/7026362/bbef2de6-78db-463d-9dfd-0c22d911f191)

Pull Request resolved: #10

Reviewed By: zhengkang86

Differential Revision: D54282791

Pulled By: YLouWashU

fbshipit-source-id: 3f085cdd9d1eab93085f7ed7f0534bcceef1932a

Summary: tsia

Reviewed By: YLouWashU

Differential Revision: D54318431

fbshipit-source-id: 24cdd6d854ed64226593df183371cd3248ce3303

Summary:
This diff adds a unique file name for ADT-converted WDS datasets. Dev can specify a `dataset_prefix` string, which will further append `sequence_name` and `subtour_name` after it. The generated new filename will look like:
`AsePublicRelease2023_100000_euston_simulation_100000_device0_fsg_000008`

Reviewed By: zhengkang86

Differential Revision: D54293719

fbshipit-source-id: 65ccf38aeaa0a0947c326d1eecfeb8e0511768a6

Summary:
Formats the covered files with pyfmt.

paintitblack

Reviewed By: aleivag

Differential Revision: D54447727

fbshipit-source-id: 8844b1caa08de94d04ac4df3c768dbf8c865fd2f

Summary: As title.

Reviewed By: zhengkang86

Differential Revision: D54399027

fbshipit-source-id: 8ccae7d1b452fcd4a5a04c990a6ae1c5b6e6f9db

Summary: It seems that ASE uses category name for mapping, while ADT uses Prototype name. This diff will enable either in `AdtGtDataProcessor`, and will determine which one to use by checking the header name in the taxonomy mapping file.

Reviewed By: zhengkang86

Differential Revision: D54438232

fbshipit-source-id: 1fe42e70edc284f2e17e9837f064df130116164d

Summary: "Iteration" means something different in optimization than ML. So changing it to "batch", which means the current data "batch", to avoid confusion.

Reviewed By: YLouWashU

Differential Revision: D54543343

fbshipit-source-id: 925eca476db0a72f4cc158227797fc896be4964b

Summary: Create a Obb3 class to represent 3D bounding boxes

Reviewed By: YLouWashU

Differential Revision: D50849167

fbshipit-source-id: 78c0c7e446bef561f008ae428fcce20145d6b492

Reviewed By: YLouWashU

Differential Revision: D50891167

fbshipit-source-id: a70814c141a637c9c3b67809e6fb9bdf277439e5

Summary: tsia

Reviewed By: YLouWashU

Differential Revision: D54640143

fbshipit-source-id: 3bad2672082b76edc7203503b7da6a49f71184cb

Summary:
The 3D bounding box evaluator consists of two steps:
1) per-batch update: compute pairwise metrics (IoU, GIoU, etc) between prediction and GT, match pred to GT based on the match criterion (e.g., IoU > 0.25)
2) final eval: aggregate all true positives, false positives and false negatives to compute mean average precision (mAP)

`eval_utils.py` is copied from https://www.internalfb.com/code/fbsource/fbcode/surreal/index/evaluation/metrics_compute.py to compute precision-recall curve, and mean average precision

Reviewed By: YLouWashU

Differential Revision: D54739283

fbshipit-source-id: 9fefcc21e829929dee97dd96140d9493fc9db3f1

Summary: Add evaluation callbacks in the inference pipeline to print and save evaluation results

Reviewed By: YLouWashU

Differential Revision:
D54875155

Privacy Context Container: L11581

fbshipit-source-id: 39470850bea159c36a9072d94e16622212a5c1b6

Summary: tsia

Reviewed By: YLouWashU

Differential Revision:
D54880351

Privacy Context Container: L11581

fbshipit-source-id: e79e043dc42874534305436c780b7c8bcafae153

Summary: Added Chamfer distance and Hungarian distance as metric to match predicted and GT 3D boxes

Reviewed By: YLouWashU

Differential Revision: D54913585

fbshipit-source-id: bffb508a26d4243a47a8bf88eb1b2205f62d72be

Summary: This diff changes the name of the file and class, because we plan to extend it to support MPS output.

Reviewed By: zhengkang86

Differential Revision: D55042388

fbshipit-source-id: 8c31a866f01b90fc0d7d30ecf38766b0f8817ee9

Summary:
Added three error metrics between predicted and GT 3D boxes: Euclidean distance, Geodesic angular error, diagonal error

Euclidean distance can be used to associate/match predicted and GT 3D boxes

Reviewed By: YLouWashU

Differential Revision:
D54913594

Privacy Context Container: L11581

fbshipit-source-id: a0b4a2e7306d0ed85c702d365c242192fa86e611

Summary: In the IOU/GIOU unit test, two convex hulls' intersection volume is verified by comparing to the results of `intersect_meshes` using trimesh API.

Reviewed By: YLouWashU

Differential Revision: D54399897

fbshipit-source-id: 3aa90d5085a4a7459a8c9651eb0c1f796c121361

Summary: Add unit tests for Chamfer distance, Hungarian distance, pairwise Chamfer distance, and pairwise Hungarian distance for a few simple cases

Reviewed By: chengshy

Differential Revision: D50867309

fbshipit-source-id: 7d51f98007680191f3ff1bd9f96560c00a65bd3f

Summary: Add unit tests for metrics between predicted and GT 3D boxes: Euclidean distance, Geodesic angular error, Diagonal error

Reviewed By: YLouWashU

Differential Revision: D51137891

fbshipit-source-id: 98a92d7e442bdcd6d00d359631ab939d6ee28874

Summary:
Update omni3d_adapter to filter GT categories;
Update evaluator to handle cases where no GT and no predictions are available;
Update callback to skip saving prediction by default

Reviewed By: YLouWashU

Differential Revision: D55145188

fbshipit-source-id: 770027e091bb6268590a446243f5d04c70a5ecf8

Summary: As title.

Reviewed By: zhengkang86

Differential Revision: D54978775

fbshipit-source-id: 93368db50dbdbc5c35ba9d81a46877c955f25388

Summary: This diff adds semidense point cloud to MpsProcessor. I will later break these into separate processors.

Reviewed By: zhengkang86

Differential Revision: D54978776

fbshipit-source-id: e11e918195d1f692fc3c834ea66d124adbad0591

Summary: This diff adds a unit test for semidense point API in `MPSProcessor`.

Reviewed By: zhengkang86

Differential Revision:
D55048214

Privacy Context Container: L11581

fbshipit-source-id: 1c11c4698ca5f5f4a4155314b85c1552164105aa

Summary: Change semidense output format to DF, which is consistent with other data in current DataSchema.

Reviewed By: zhengkang86

Differential Revision: D55883722

fbshipit-source-id: 61ed49ff2b632ccd9c0fdb3faac55ae982b6799d

Summary: As title.

Reviewed By: zhengkang86

Differential Revision: D55883721

fbshipit-source-id: c04ae24b65bbb659abcb1c3301c415cd831c0a74

Summary: It seems that EFM needs inverse distance instead.

Reviewed By: zhengkang86

Differential Revision:
D55883723

Privacy Context Container: L11581

fbshipit-source-id: 54e0bfb45148b3c05096b3f817f6c90a69a7b0fe

Summary: as title

Reviewed By: YLouWashU

Differential Revision:
D56552004

Privacy Context Container: L11581

fbshipit-source-id: 86d6a2169f6ea630d999cf33c7fb66c21c51ad8a

Summary:
This diff moves the current `atek/atek` content to `atek/atek_v1`, and start adding refactored content into `atek/atek` folder.

Add `atek_data_sample`class. It is a structured data class to represent a sample in ATEK.

Reviewed By: chengshy

Differential Revision: D56802361

fbshipit-source-id: fc09b21c7b33c4c990071018eb3a591fd5648a28

Summary:
This diff adds a `AriaCameraProcessor` class, that processes camera data.

We also uses a OmegaConf system in the new design.

Reviewed By: chengshy

Differential Revision: D56802362

fbshipit-source-id: 287a58559a9d188c1a8bcaef9fc959e1f3ee5fe7

Summary: Adding unit test for camera processor class.

Reviewed By: chengshy

Differential Revision: D56802354

fbshipit-source-id: 49e9cbdecb310258de5b012b13332ce033faeb9d

Summary: Adding `mps_traj_processor` class to process MPS traj poses.

Reviewed By: chengshy

Differential Revision: D56802357

fbshipit-source-id: 9c4fa146bdf10c28a197e8154614c8f2ee3c9397

Summary: Unit test for MPS traj processor class.

Reviewed By: zhengkang86

Differential Revision: D56802360

fbshipit-source-id: 2e8a0b59b161f9db66a147d2cdf7316d26992d14

Summary: Add GT processor for OBB3 class.

Reviewed By: chengshy

Differential Revision: D56802363

fbshipit-source-id: 26631dbd2cd54995ac1d1dacd5353fe6b7afa6e2

Summary: Unit test for obb3 gt processor class.

Reviewed By: chengshy

Differential Revision: D56802355

fbshipit-source-id: ef498812702cedc083137dde20b1dde650eceb2c

Summary: Add a `obb3_sample_builder` class to build ATEK samples. Currently only adding 1 frameset per sample, different cameras are aligned through a given timestamp.

Reviewed By: chengshy

Differential Revision: D56802359

fbshipit-source-id: a736c39a9c5575e4f03dfd0b40f4d0661e39f2c4

Summary: This diff adds unit test for the `obb3_sample_builder` class.

Reviewed By: chengshy

Differential Revision: D56802358

fbshipit-source-id: 5a6ae5385975a5ba11efd7bfc1e9d5bc702596fe

Summary: This diff adds a TemporalSubsampler class to get timestamps for samples when subsampling is turned on. Currently it only returns a single timestamp for the main camera. I might later add multi-frame features.

Reviewed By: chengshy

Differential Revision: D56893766

fbshipit-source-id: 2278510da3902919bf6d1737d2c501db4c6ec582

Summary: This diff adds `to_flatten_dict()` functions to each data class in `atek_data_sample.py`. The dict key also contains a file extension to indicate what is the output format when written to WDS.

Reviewed By: chengshy

Differential Revision: D56893769

fbshipit-source-id: d08c16c2d520abd92cab63db8b68f4e6133a5b6c

Summary: Adding WDS writer class (similiar to previous WDS writer). Now images are written as tensors directly.

Reviewed By: chengshy

Differential Revision: D56893767

fbshipit-source-id: a28dddf634bd8b265bed4aa10a204ef211570b03

Summary: changes `setup.py` to only pick up python files. I will remove `atek_v1` folder completely when the refactoring is done.

Reviewed By: chengshy

Differential Revision: D56893765

fbshipit-source-id: e400de9532e27e1e4e7b64a96c765e6aceb9c12a

Summary: This diff updates the `atek_data_preprocessing` example jupyter notebook, to show a basic example of how to run data-preprocessing.

Reviewed By: chengshy

Differential Revision: D56909697

fbshipit-source-id: 4c19a0e188473ccb2178f190fee07fc12d20217d

Summary:
This diff creates a DataLoader class that loads ATEK WDS dataset for `obb3_sample_builder` samples.

I also updated the notebook example to load a new WDS dataset.

Reviewed By: chengshy

Differential Revision: D56893768

fbshipit-source-id: 327aa1f96b75b7690956dbcd7ea5e1e7d89a11b9

Summary: This diff changes how `category_mapping` is handled. It is now parsed into `obb3_gt_processor` as an optional `gt_file`.

Reviewed By: chengshy

Differential Revision:
D57023377

Privacy Context Container: L1257120

fbshipit-source-id: 39f6837446921ba2828c607af5df19d43d3d29e3

Summary: This diff adds some APIs in `TemporalSubsampler` so we can write samples for a full sequence.

Reviewed By: chengshy

Differential Revision: D57023379

fbshipit-source-id: 1176b676e74cb9050fddb49f26e7c412625a1212

Summary: This diff modifies the `adt_to_wds` fblearner workflow to the new code design.

Reviewed By: chengshy

Differential Revision:
D56922611

Privacy Context Container: L1257120

fbshipit-source-id: 00ac66e44ea0af79947df06bc590eb78286fb2e4

Summary: This diff adds a `mps_semidense_processor` class, along with unit test. It is essentially same as my previous implementation.

Reviewed By: chengshy

Differential Revision: D57083864

fbshipit-source-id: bfc69059cd751fb858070978d52688034231599d

Summary: Fixing a bug in `DistortByCalibrationTVWrapper`.

Reviewed By: chengshy

Differential Revision:
D57083863

Privacy Context Container: L1257120

fbshipit-source-id: 9c276d295b98f9759fd7416a2ffeb5ba1f7a16ce

Summary: Since some (including EFM) model take multiple frames in a sample (snippet), this diff adds this functionality for AriaCameraProcessor class, and changes the unit test accordingly.

Reviewed By: chengshy

Differential Revision: D57146644

fbshipit-source-id: a4f234fa06684cfdb6241edfe0846f8935907825

Summary: As title.

Reviewed By: chengshy

Differential Revision: D57146646

fbshipit-source-id: 9e26ac47733d0f36fdee68359ea70e2ccaf6669b

Summary: This diff also modifies the `CameraTemporalSubsampler` class to return a "packet" of main camera's timestamps in one sample, if the sample contains multiple frames.

Reviewed By: chengshy

Differential Revision: D57146641

fbshipit-source-id: 386910ba752978cc40d1680dcd9bbf68fecc06b6

Summary: This diff adds a EfmGtProcessor class, which returns OBB3 Dict with multiple frames. Therefore I made it a child class.

Reviewed By: chengshy

Differential Revision: D57146643

fbshipit-source-id: 0574e30502be28759543065fab8d253aeff22dd0

Summary: As title.

Reviewed By: chengshy

Differential Revision: D57146645

fbshipit-source-id: 96f3d90c1e53ce539f4cc1d7bde1bc7ee081fcb1

Summary: Since EFM needs Depth images, this diff adds DepthImageProcessor class. In the mean time, Depth images will need to perform the same transformations as the raw images, except that the interpolation method needs to be NEAREST. Hence I exposed the transformation as a list of `torchvision.transforms` out of `AriaCameraProcess` class to reuse across processors.

Reviewed By: hlzz

Differential Revision: D57146859

fbshipit-source-id: d90eca438d79cd71cd9e9a510a00db6e15260815

Summary: As title. Semidense data are represented by a list of tensors, hence they need to be handled slightly differently where the list of tensor are stacked when writing to WDS.

Reviewed By: chengshy

Differential Revision: D57470031

fbshipit-source-id: 20eb5e754ade594f5393a7f1e78fc8b42ab4da28

Summary: As title. Just a few small clean-ups. And added time profiling when loading semidense data, which is a little slow, we might improve it later.

Reviewed By: chengshy

Differential Revision: D57470033

fbshipit-source-id: 49d545e658f90cb2a6658be4d1285095bc6953b7

Summary: This diff separates the python notebooks for obb3 and efm into different notebooks, for convenient testing.

Reviewed By: chengshy

Differential Revision: D57470035

fbshipit-source-id: 8cab6a0cc70313658599ac1c1b4f0fb749d7aa8b

Summary: This diff moves the function `load_category_mapping_from_csv` out of `Obb3GTProcessor` class, into `file_io_utils` lib, because it will be reused by other classes.

Reviewed By: chengshy

Differential Revision: D57470026

fbshipit-source-id: 20317f2f963ea39b07cc90902e021e01aad6c1ea

Summary: This diff unpack the stacked semi-dense tensors back to a list of tensor of shape [K, 3] during WDS data loading. The EFM adaptor will further pad them.

Reviewed By: chengshy

Differential Revision: D57470027

fbshipit-source-id: a2c85489feae3235d23ec56585c2bee4413a3173

Summary: This diff adds key-remapping and data transform APIs for ATEK WDS data loading, which will be used by EFM Model adaptor.

Reviewed By: chengshy

Differential Revision: D57470034

fbshipit-source-id: e34e48c6eba4c41ad640fe8998a24f7bcb90a249

Summary: Added a `fill_or_trim_tensor` util function, which will be used by EFM's model adaptor class. It allows both filling fixed value, or repeating last element, by setting the `fill_value` parameter.

Reviewed By: hlzz, chengshy

Differential Revision: D57470032

fbshipit-source-id: 837862753b59957c356a0717dc2752129de283be

Summary:
This diff does a few things regarding `aria_camera_processor`:
1. It changes the return type of `get_image_transform` to directly return a Callable (instead of a list of transforms).
2. It exposes a new API, `get_pixel_transform`, which will perform the same transformation as `get_image_transform()`, but for batched pixel coordinates.

These are necessary when distorting 2D bboxes in CubeRCNN GT.

Reviewed By: chengshy

Differential Revision: D57470028

fbshipit-source-id: a4ae37f6e22944baa232bb8c2632347b60535ecf

Summary: This diff adds a Obb3GT processor class with unit test. This is needed by CubeRCNN because it also requires 2d OBB data.

Reviewed By: chengshy

Differential Revision: D57470029

fbshipit-source-id: 3dd49d1e95df656a5ff8e000c05b457c247edb5c

Summary:
This diff adds a few more fields in `aria_camera_processor`'s data field.

It also changes the processors to directly return data type defined in `atek_data_sample.py`.

Reviewed By: chengshy

Differential Revision:
D57804186

Privacy Context Container: L1257120

fbshipit-source-id: 2cb99e82d532acde880cb0c157b8fd0fce8ca8bd

Summary: This diff adds `dist_std` and `capture_timestamps_ns` to mps_semidense processor class.

Reviewed By: chengshy

Differential Revision:
D57791547

Privacy Context Container: L1257120

fbshipit-source-id: 406ea3184070c297e1402718ef60f133c307c836

Summary: This diff adds comment docstrings to gt processor classes to explain the content.

Reviewed By: chengshy

Differential Revision: D58094944

fbshipit-source-id: ebaeb7a0fc03a302a84ac47d19b9fc7f1460ede3

Summary: This diff creates the SampleBuilder for CubeRCNN. The main difference between this and OBB3 is that it also has OBB2 GT information in the GT dict.

Reviewed By: chengshy

Differential Revision: D58094948

fbshipit-source-id: fc11f62c590d70a07d0c85c9fa87a0edd6aebc6f

Summary: A few more bug fixes.

Reviewed By: chengshy

Differential Revision: D58094945

fbshipit-source-id: cbfd0fd63ff106311ced139603c7fdcdc05a3efa

Summary: Implementing CubeRCNN model adaptor class.

Reviewed By: chengshy

Differential Revision: D58094943

fbshipit-source-id: c319af926cbd0338c82111be346e210943436753

Summary: This diff changes the OBB3 loading notebook to load CubeRCNN WDS files. In the mean time, it also adds visualizations to visualize ATEK WDS data.

Differential Revision: D57804188

fbshipit-source-id: 242dd64581ae4882a09f7f6a129d68fd86d658bf

Summary: As title. Simply some variable renaming etc.

Differential Revision: D58772742

fbshipit-source-id: bee19048a2fd11797d0c1db0f6e105ab907043eb

Summary: DuringWDS data loading, the image are loaded in order of 0000, 0001, 0010, 0011, etc, which is apparently wrong. This diff explicitly enforced the order of image loading.

Differential Revision: D58772747

fbshipit-source-id: 93d02a064104d80eb09b5d453fca26b2e0794245

… to behave weird.

Summary: This diff fixes an issue where torch's clone does not seem to work very well with the Image pointer manipulations in `Distort` function. Therefore used np.copy() instead.

Differential Revision: D58772751

fbshipit-source-id: 43f02ccb71adce68c340996fe6209b806cbb24f0

…der classes.

Summary:
As title. These are all important features.

TODO: will add a default collation function to ATEK data samples, once the GT flattening task is done T192965615.

Differential Revision: D58772741

fbshipit-source-id: 63ad29f6674387edaa4aa2e57a48b2b212d72df7

Summary:
In OBB3 GT class, we want to store only visible instances.

TODO: consider store only visible instance content instead of storing visible instance IDs?

Differential Revision: D58772743

fbshipit-source-id: 0f67e4b836ee2c975d4f157bd4ca03b2ae3a6c8d

Summary: As title.

Differential Revision: D58772748

fbshipit-source-id: b8a304c429d54559935c3ccb4e14c2d35856776e

Summary: change projectaria_tools version to 1.5.2, which is the most updated version.

Reviewed By: YLouWashU

Differential Revision: D59178707

fbshipit-source-id: 7270309bbbebcd0e67578e4bfef96c0aeb37704b

Summary: As title.

Reviewed By: echo-xiao9

Differential Revision: D58772749

fbshipit-source-id: 1eda1d6fbd28e35f359a2b3db6bfd387964276d5

Summary: As title.

Reviewed By: echo-xiao9

Differential Revision: D58772745

fbshipit-source-id: 744669a5eab13f577363922700eebe170029a418

Summary:
As title. CV2 does NOT have antialiasing, while PILLOW always have this. Therefore enable this flag during pre-processing.

See this post: https://fb.workplace.com/groups/374285425240042/permalink/405477545454163/

Reviewed By: hlzz, echo-xiao9

Differential Revision: D58772746

fbshipit-source-id: aad08bab9c7b8b72e345f416f5e4d71a1f7c40e3

…n notebook with pure python scripts.

Summary:
This diff does 2 things:
1. It merges `cubercnn_sample_builder` with `obb3_sample_builder`, since the former contains the latter, and rename it as `obb_sample_builder`.
2. It also puts jupyter notebooks in `legacy` folder, and added native python scripts as examples. We will revive the jupyter notebook at a later stage when examples are more consolidated.

Viz results:  {F1729997394}

Reviewed By: echo-xiao9

Differential Revision: D59180576

fbshipit-source-id: dfd739027fd3d41838326aa0abfd4600e067d27a

Summary: Fixed some typo found in the codebase

Reviewed By: YLouWashU

Differential Revision: D59338595

fbshipit-source-id: c9028423679a50c2be77324f70a0455347c2d2eb

Summary: This diff adds loading CubeRCNN into the obb3 data loading script.

Reviewed By: echo-xiao9

Differential Revision: D59347343

fbshipit-source-id: a45a3e2b0ef4d3dbfa0c1c47a082b7586350c7b4

Summary: A simple renaming to be consistent with `atek_wds_dataloader` file.

Reviewed By: echo-xiao9

Differential Revision: D59254652

fbshipit-source-id: 707cdd1e767152a920d0054ec97c3b40591c5bc2

Summary:
I always noticed that our WDS files contain upper case letters, while the dict loaded by WDS always convert them to lower case. This is due to WDS's [default behavior]( webdataset/webdataset#371)

 To comply with this, I changed ATEK to store lower-cased filenames in WDS, and fixed the corresponding unit tests.

Reviewed By: echo-xiao9

Differential Revision: D59306287

fbshipit-source-id: f4bab47a0af5de537a9f4c455b6c7f8d44bc43cf

Summary: This diff adds an example code for obtaining depth images with ATEK lib.

Reviewed By: echo-xiao9

Differential Revision: D59235789

fbshipit-source-id: c65faa54888bf674ba8137174d7532b3c898a1cd

Summary: This diff slightly changes the GT dict in obb3 processor class. Before we had pretty nested structure, and GT content is directly saved as a json blob. With this diff, GT dict actually contains slightly more flattened content, where tensors in GT will be later directly saved as .pth file for efficiency (we are seeing some 10% save on CubeRCNN example data).

Reviewed By: echo-xiao9

Differential Revision: D59313728

fbshipit-source-id: 5a6fef7703f17a446ca6cd1854e2df07211c4fdf

Summary: As title. I found it is more convenient to save obb3 as per-camera-observed instances, because using `visible_instances` to re-construct observed instances has computation complexity of O(MN), which is expensive.

Reviewed By: echo-xiao9

Differential Revision: D59508436

fbshipit-source-id: 83f42e51d324c06f7d207ff0ca046d90e48cfad5

Summary: Similar to previous diff, but on obb2 processor class.

Reviewed By: echo-xiao9

Differential Revision: D59313730

fbshipit-source-id: 79b74cc11a2d6f0173df1ae290de75e91dcdaab7

…aty GT dict.

Summary:
These 2 util functions are important, they can separate tensors from a `gt_dict` to form a new dict, or merge tensors back in `gt_dict_wo_tensors`.

Currently we are already converting tensors to list during WDS writing, but we never implemented how this is done during loading. With these util functions, I plan to separate tensors from gt during WDS writing, and merge them back in when loading WDS, which seems a cleaner solution.

Reviewed By: echo-xiao9

Differential Revision: D59313732

fbshipit-source-id: e56a7a18fa3da0383752ed0f304af3785c3d531e

Summary: As title. Adding unit test.

Reviewed By: echo-xiao9

Differential Revision: D59313726

fbshipit-source-id: 2b3c6644955bcedc1bc179c8d12c143ef3201f86