Skip to content

mini_librispeech: fixing some bugs and limiting the repeated downloads #2861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danpovey merged 1 commit into from
Nov 27, 2018
Merged

mini_librispeech: fixing some bugs and limiting the repeated downloads #2861

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions egs/mini_librispeech/s5/local/download_and_untar.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,11 @@ if [ ! -d "$data" ]; then
exit 1;
fi

data=$(readlink -f $data)

part_ok=false
list="dev-clean-2 train-clean-5"
for x in $list; do
for x in $list; do
if [ "$part" == $x ]; then part_ok=true; fi
done
if ! $part_ok; then
Expand All @@ -49,7 +51,8 @@ if [ -f $data/LibriSpeech/$part/.complete ]; then
fi


sizes="126046265 332747356"
#sizes="126046265 332747356"
sizes="126046265 332954390"

if [ -f $data/$part.tar.gz ]; then
size=$(/bin/ls -l $data/$part.tar.gz | awk '{print $5}')
Expand Down
18 changes: 10 additions & 8 deletions egs/mini_librispeech/s5/local/download_lm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@
# 2017 Daniel Povey
# Apache 2.0

if [ $# -ne "2" ]; then
echo "Usage: $0 <base-url> <download_dir>"
echo "e.g.: $0 http://www.openslr.org/resources/11 data/local/lm"
if [ $# -ne "3" ]; then
echo "Usage: $0 <base-url> <download_dir> <local?"
echo "e.g.: $0 http://www.openslr.org/resources/11 ./corpus/ data/local/lm"
exit 1
fi

base_url=$1
dst_dir=$2
local_dir=$3

# given a filename returns the corresponding file size in bytes
# The switch cases below can be autogenerated by entering the data directory and running:
Expand Down Expand Up @@ -64,9 +65,10 @@ for f in 3-gram.arpa.gz 3-gram.pruned.1e-7.arpa.gz 3-gram.pruned.3e-7.arpa.gz \
check_and_download $f || exit 1
done

cd $dst_dir
ln -sf 3-gram.pruned.1e-7.arpa.gz lm_tgmed.arpa.gz
ln -sf 3-gram.pruned.3e-7.arpa.gz lm_tgsmall.arpa.gz
ln -sf 3-gram.arpa.gz lm_tglarge.arpa.gz

dst_dir=$(readlink -f $dst_dir)
ln -sf $dst_dir/3-gram.pruned.1e-7.arpa.gz $local_dir/lm_tgmed.arpa.gz
ln -sf $dst_dir/3-gram.pruned.3e-7.arpa.gz $local_dir/lm_tgsmall.arpa.gz
ln -sf $dst_dir/3-gram.arpa.gz $local_dir/lm_tglarge.arpa.gz
ln -sf $dst_dir/librispeech-lexicon.txt $local_dir/librispeech-lexicon.txt
ln -sf $dst_dir/librispeech-vocab.txt $local_dir/librispeech-vocab.txt
exit 0
4 changes: 2 additions & 2 deletions egs/mini_librispeech/s5/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

# Change this location to somewhere where you want to put the data.
data=/export/a05/dgalvez/
data=./corpus/

data_url=www.openslr.org/resources/31
lm_url=www.openslr.org/resources/11
Expand All @@ -21,7 +21,7 @@ for part in dev-clean-2 train-clean-5; do
done

if [ $stage -le 0 ]; then
local/download_lm.sh $lm_url data/local/lm
local/download_lm.sh $lm_url $data data/local/lm
fi

if [ $stage -le 1 ]; then
Expand Down