Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding temperature scaling on Joiner logits: #789

Merged
merged 2 commits into from
Apr 26, 2024

Commits on Apr 18, 2024

  1. Adding temperature scaling on Joiner logits:

    - T hard-coded to 2.0
    - so far best result NCE 0.122 (still not so high)
        - the BPE scores were rescaled with 0.2 (but then also incorrect words
          get high confidence, visually reasonable histograms are for 0.5 scale)
        - BPE->WORD score merging done by min(.) function
          (tried also prob-product, and also arithmetic, geometric, harmonic mean)
    
    - without temperature scaling (i.e. scale 1.0), the best NCE was 0.032 (here product merging was best)
    
    Results seem consistent with: https://arxiv.org/abs/2110.15222
    
    Everything tuned on a very-small set of 100 sentences with 813 words and 10.2% WER, a Czech model.
    
    I also experimented with blank posteriors mixed into the BPE confidences,
    but no NCE improvement found, so not pushing that.
    
    Temperature scling added also to the Greedy search confidences.
    KarelVesely84 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    6ec96cd View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. Configuration menu
    Copy the full SHA
    8b57f73 View commit details
    Browse the repository at this point in the history