GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval
This repository is the official PyTorch implementation of our AAAI 2024 paper GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval.
1. Clone this repository:
git clone https://github.com/huangmozhi9527/GMMFormer.git
cd GMMFormer
2. Create a conda environment and install the dependencies:
conda create -n prvr python=3.9
conda activate prvr
conda install pytorch==1.9.0 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install -r requirements.txt
3. Download Datasets: All features of TVR, ActivityNet Captions and Charades-STA are kindly provided by the authors of MS-SL.
4. Set root and data_root in config files (e.g., ./Configs/tvr.py).
To train GMMFormer on TVR:
cd src
python main.py -d tvr --gpu 0
To train GMMFormer on ActivityNet Captions:
cd src
python main.py -d act --gpu 0
To train GMMFormer on Charades-STA:
cd src
python main.py -d cha --gpu 0
We provide trained GMMFormer checkpoints. You can download them from Baiduyun disk.
Dataset | ckpt |
---|---|
TVR | Baidu disk |
ActivityNet Captions | Baidu disk |
Charades-STA | Baidu disk |
For this repository, the expected performance is:
Dataset | R@1 | R@5 | R@10 | R@100 | SumR |
---|---|---|---|---|---|
TVR | 13.9 | 33.3 | 44.5 | 84.9 | 176.6 |
ActivityNet Captions | 8.3 | 24.9 | 36.7 | 76.1 | 146.0 |
Charades-STA | 2.1 | 7.8 | 12.5 | 50.6 | 72.9 |
If you find this repository useful, please consider citing our work:
@inproceedings{wang2023gmmformer,
title={GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval},
author={Wang, Yuting and Wang, Jinpeng and Chen, Bin and Zeng, Ziyun and Xia, Shu-Tao},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2024}
}