This project aims to finetune the descript-audio-codec for speech enhancement and noise removal.
- Data Preparation: Custom dataloaders and datasets manage voice and noise data, and creating noisy speech data.
- Model Architecture: Utilizes
dac(Descript Audio Codec), a high-fidelity general neural audio codec, with both generator and discriminator networks for adversarial training. - HifiPlusPlus: The discriminators from HPP can be used to improve the quality of the generated audio.
- Loss Functions: A diverse set of losses, including L1, Mel-Spectrogram, SI-SDR, and GAN losses, are used to capture various aspects of audio reconstruction quality.
- Training: A training loop with loss weighting, noise addition, and gradient clipping to create stable learning.
- Evaluation: The model's output is periodically saved for quality inspection, and performance metrics are tracked during training to Weights and Biases.
Training is is done using the train script with config for folders, parameters etc. in the config file. To run training there must be both a clean and noisy dataset path provided in the config.
- train.py:
To start the training process with default config, run the desired script with:
python3 train.pyTo train on DTU HPC with appropriate dataset run:
train.py -c ./configs/config_HPC.jsonTo run inference of the trained model weights use the notebook called "test_model_weights.ipynb". Download weights from Drive and place in models folder.
Adjust hyperparameters such as learning rates, batch sizes, folder setup and epochs within the config file.
Models, logs, and audio samples are saved in the ./output/ directory, allowing for incremental evaluation of the audio processing quality.
See Wandb for training logs: 1 and 2
Please refer to the individual scripts for detailed parameter settings and training customizations.