Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiCut to store multi-channel recordings with shared supervision #822

Merged
merged 72 commits into from
Oct 5, 2022

Conversation

desh2608
Copy link
Collaborator

@desh2608 desh2608 commented Sep 26, 2022

This PR adds a MultiCut which is a new type of Cut that represents multi-channel data. We also define a DataCut as a Cut which contains actual data (as opposed to a PaddingCut or MixedCut which are empty or operations on other cuts). This DataCut is the abstract base class from which MonoCut and MultiCut inherit.

 >>> 2-channel telephone recording with 2 supervisions, one for each channel (e.g., Switchboard):

                  ╔══════════════════════════════  MultiCut  ═════════════════╗
                  ║ ┌──────────────────────────┐                              ║
     Channel 1  ──╬─│   Hello this is John.    │──────────────────────────────╬────────
                  ║ └──────────────────────────┘                              ║
                  ║                               ┌──────────────────────────┐║
     Channel 2  ──╬───────────────────────────────│ Hey, John. How are you?  │╠────────
                  ║                               └──────────────────────────┘║
                  ╚═══════════════════════════════════════════════════════════╝

   
>>> Multi-array multi-microphone recording with shared supervisions (e.g., CHiME-6),
    along with close-talk microphones (A and B are distant arrays, C is close-talk):

               ╔═══════════════════════════════════════════════════════════════════════════╗
               ║ ┌───────────────────┐                         ┌───────────────────┐       ║
       A-1   ──╬─┤                   ├─────────────────────────┤                   ├───────╬─
               ║ │ What did you do?  │                         │I cleaned my room. │       ║
       A-2   ──╬─┤                   ├─────────────────────────┤                   ├───────╬─
               ║ └───────────────────┘  ┌───────────────────┐  └───────────────────┘       ║
       B-1   ──╬────────────────────────┤Yeah, we were going├──────────────────────────────╬─
               ║                        │   to the mall.    │                              ║
       B-2   ──╬────────────────────────┤                   ├──────────────────────────────╬─
               ║                        └───────────────────┘        ┌───────────────────┐ ║
        C    ──╬─────────────────────────────────────────────────────┤      Right.       ├─╬─
               ║                                                     └───────────────────┘ ║
               ╚════════════════════════════════  MultiCut  ═══════════════════════════════╝

lhotse/cut/multi.py Outdated Show resolved Hide resolved
@desh2608 desh2608 marked this pull request as ready for review October 1, 2022 00:21
@desh2608 desh2608 changed the title WIP: MultiCut to store multi-channel recordings with shared supervision MultiCut to store multi-channel recordings with shared supervision Oct 1, 2022
@desh2608
Copy link
Collaborator Author

desh2608 commented Oct 1, 2022

I think this one's ready for review once the tests pass.

@csukuangfj
Copy link
Contributor

 >>> 2-channel telephone recording with 2 supervisions, one for each channel (e.g., Switchboard):

                  ╔══════════════════════════════  MultiCut  ═════════════════╗
                  ║ ┌──────────────────────────┐                              ║
     Channel 1  ──╬─│   Hello this is John.    │──────────────────────────────╬────────
                  ║ └──────────────────────────┘                              ║
                  ║                               ┌──────────────────────────┐║
     Channel 2  ──╬───────────────────────────────│ Hey, John. How are you?  │╠────────
                  ║                               └──────────────────────────┘║
                  ╚═══════════════════════════════════════════════════════════╝

The visualization looks great!

Copy link
Collaborator

@pzelasko pzelasko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, very impressive work!! I left just a few comments, mostly minor ones + asking for a bit more test coverage, otherwise it looks great!

lhotse/audio.py Show resolved Hide resolved
lhotse/audio.py Outdated Show resolved Hide resolved
lhotse/cut/__init__.py Show resolved Hide resolved
lhotse/cut/mixed.py Show resolved Hide resolved
lhotse/cut/mixed.py Outdated Show resolved Hide resolved
lhotse/cut/multi.py Show resolved Hide resolved
lhotse/cut/set.py Show resolved Hide resolved
lhotse/audio.py Outdated Show resolved Hide resolved
lhotse/cut/data.py Show resolved Hide resolved
lhotse/cut/multi.py Show resolved Hide resolved
@desh2608
Copy link
Collaborator Author

desh2608 commented Oct 4, 2022

@pzelasko Addressed all comments.

@pzelasko pzelasko added this to the v1.9 milestone Oct 5, 2022
@pzelasko
Copy link
Collaborator

pzelasko commented Oct 5, 2022

LGTM! Again, great work.

@pzelasko pzelasko merged commit 1ce212e into lhotse-speech:master Oct 5, 2022
@desh2608 desh2608 deleted the multi_cut branch November 2, 2023 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants