Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decoder API rework (Fixes #2672) #2681

Closed
wants to merge 17 commits into from
Closed

Decoder API rework (Fixes #2672) #2681

wants to merge 17 commits into from

Conversation

reuben
Copy link
Contributor

@reuben reuben commented Jan 20, 2020

No description provided.

@reuben reuben force-pushed the decoder-api-changes branch 3 times, most recently from 211a18d to 1eac7e4 Compare January 21, 2020 10:39
@reuben reuben changed the title WIP test decoder API changes Decoder API rework Jan 21, 2020
@reuben reuben force-pushed the decoder-api-changes branch from 1eac7e4 to b904beb Compare January 21, 2020 10:54
@reuben
Copy link
Contributor Author

reuben commented Jan 22, 2020

This is a big one, so I highly recommend reviewing on a per commit basis. GitHub now lets you see changes from a commit range, which helps. The main interesting parts are:

  • The new data/lm/generate_package.py script
  • The changes to native_client/ctcdecode/scorer.{h,cpp} for the new format
  • API changes in b07e426

@reuben
Copy link
Contributor Author

reuben commented Jan 22, 2020

And the changes to the examples are here: mozilla/DeepSpeech-examples@f3dee79

@mozilla mozilla deleted a comment from reuben Jan 22, 2020
@reuben reuben changed the title Decoder API rework Decoder API rework (Fixes #2672) Jan 22, 2020
lissyx
lissyx reviewed Jan 22, 2020
View reviewed changes
lissyx
lissyx reviewed Jan 22, 2020
View reviewed changes
lissyx
lissyx reviewed Jan 22, 2020
View reviewed changes
lissyx
lissyx approved these changes Jan 22, 2020
View reviewed changes
Copy link
Collaborator

@lissyx lissyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we might want to ensure doc is uptodate as well :)

@reuben
Copy link
Contributor Author

reuben commented Jan 22, 2020

Thanks! I was going to update the docs but then I got flashbacks of previous API breaks where users read master docs and the options don't work with the stable release and we get several issues. Maybe it's unavoidable, but perhaps we can land the docs update closer to the v0.7 release when this goes out? What do you think?

@reuben reuben force-pushed the decoder-api-changes branch from 7c64029 to 4b7359d Compare February 3, 2020 11:58
kdavis-mozilla
kdavis-mozilla approved these changes Feb 4, 2020
View reviewed changes
Copy link
Contributor

@kdavis-mozilla kdavis-mozilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow that was big! Just a few questions, see the comments.

@reuben reuben force-pushed the decoder-api-changes branch 4 times, most recently from 65ec5a3 to 3603ad3 Compare February 11, 2020 09:30
reuben added 16 commits February 11, 2020 14:12
@reuben
Update KenLM to b9f35777d112ce2fc10bd3986302517a16dc3883
8a06cef
@reuben
Stop including vocabulary data in LM.binary.
dbfd294
@reuben
Refactor Scorer so model/trie package can be created by an external tool
bd6c20a
@reuben
Add generate_package tool to create combined scorer package
5e9bcb8
@reuben
Load combined format from Scorer
4db606e
@reuben
Write default values for alpha and beta into trie header
00e4c2b
@reuben
Change decoder API
3a686d8
@reuben
Switch to new scorer format
c792b92
@reuben
Switch smoke test scorer to new format
af15edb
@reuben
Add tool to extract vocabulary from the old LM binary format
494a1f7
@reuben
Update all API consumers
a9163eb
@reuben
Fix linter errors
5dc4235 
X-DeepSpeech: NOBUILD
@reuben
Improve error handling around Scorer loading
1c7d9cd
@reuben
Address review comments and update docs
14aefdf
@reuben
Address review comments
f4e8986
@reuben
Update examples model asset
0d84923
@reuben reuben force-pushed the decoder-api-changes branch from 3603ad3 to d9a324d Compare February 11, 2020 13:13
@reuben reuben force-pushed the decoder-api-changes branch from d9a324d to 1d5931d Compare February 11, 2020 13:53
reuben added a commit that referenced this pull request Feb 11, 2020
@reuben
Merge branch 'decoder-api-changes' (PR #2681)
5366f90
@reuben reuben closed this Feb 11, 2020
@reuben reuben mentioned this pull request Feb 13, 2020
Closed
10 tasks
@lock
Copy link

lock bot commented Mar 17, 2020

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 17, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@lissyx lissyx lissyx approved these changes

@kdavis-mozilla kdavis-mozilla kdavis-mozilla approved these changes

@carlfm01 carlfm01 carlfm01 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@reuben @lissyx @kdavis-mozilla @carlfm01