StreamMamba: Self-Predictive Frame Skipping for Real-Time Prompt-Based Peak Frame Detection

This repository contains the official PyTorch implementation for the paper: "StreamMamba: Self-Predictive Frame Skipping for Real-Time Prompt-Based Peak Frame Detection".

StreamMamba introduces a novel framework for efficient video understanding that dramatically reduces computational cost without significant loss in performance. It leverages a Mamba state-space model trained to predict future video content, allowing it to dynamically skip processing redundant frames. This makes it ideal for real-time applications on resource-constrained devices like smartphones.

Note: The mamba folder in this repository is copied from the original implementation at https://github.com/state-spaces/mamba.

Training and Inference

For detailed instructions on how to prepare datasets and models, as well as how to run training and inference, please refer to:

• Dataset and Model Preparation
• Installation and Usage