Skip to content

Sed compatibility #2981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danpovey merged 11 commits into from
Mar 8, 2019
Merged

Sed compatibility #2981

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
941cc12
sed compatibility
desh2608 Jan 9, 2019
96e7d29
more fixes for sed
desh2608 Jan 12, 2019
d40fed7
remove trailing whitespace at the end of line
desh2608 Jan 12, 2019
1a28ad1
resolved merge conflict
desh2608 Mar 5, 2019
cfccf4b
single-pass segment modification with perl
desh2608 Mar 6, 2019
e34d23e
perl for string modifications in files
desh2608 Mar 6, 2019
2f6fed7
perl for string modification in file
desh2608 Mar 6, 2019
a24149f
perl modifications for robustness
desh2608 Mar 7, 2019
48c09a5
minor change
desh2608 Mar 7, 2019
73ec088
retain diagnostic output
desh2608 Mar 7, 2019
d8d225c
minor fix
desh2608 Mar 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion egs/ami/s5/local/ami_ihm_scoring_data_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ if [ `cat $dir/segments_to_fix | wc -l` -gt 0 ]; then
while read line; do
p1=`echo $line | awk -F'>' '{print $1}'`
p2=`echo $line | awk -F'>' '{print $2}'`
sed -ir "s!$p1!$p2!" $dir/segments
perl -i -pe "s!$p1!$p2!" $dir/segments
done < $dir/segments_to_fix
fi

Expand Down
2 changes: 1 addition & 1 deletion egs/ami/s5/local/ami_mdm_scoring_data_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ if [ `cat $tmpdir/segments_to_fix | wc -l` -gt 0 ]; then
while read line; do
p1=`echo $line | awk -F'>' '{print $1}'`
p2=`echo $line | awk -F'>' '{print $2}'`
sed -ir "s:$p1:$p2:" $tmpdir/segments
perl -i -pe "s:$p1:$p2:" $tmpdir/segments
done < $tmpdir/segments_to_fix
fi

Expand Down
2 changes: 1 addition & 1 deletion egs/ami/s5/local/ami_sdm_scoring_data_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ if [ `cat $tmpdir/segments_to_fix | wc -l` -gt 0 ]; then
while read line; do
p1=`echo $line | awk -F'>' '{print $1}'`
p2=`echo $line | awk -F'>' '{print $2}'`
sed -ir "s:$p1:$p2:" $tmpdir/segments
perl -i -pe "s:$p1:$p2:" $tmpdir/segments
done < $tmpdir/segments_to_fix
fi

Expand Down
2 changes: 1 addition & 1 deletion egs/ami/s5b/local/ami_ihm_scoring_data_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ if [ `cat $dir/segments_to_fix | wc -l` -gt 0 ]; then
while read line; do
p1=`echo $line | awk -F'>' '{print $1}'`
p2=`echo $line | awk -F'>' '{print $2}'`
sed -ir "s!$p1!$p2!" $dir/segments
perl -i -pe "s!$p1!$p2!" $dir/segments
done < $dir/segments_to_fix
fi

Expand Down
2 changes: 1 addition & 1 deletion egs/ami/s5b/local/ami_mdm_scoring_data_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ if [ `cat $tmpdir/segments_to_fix | wc -l` -gt 0 ]; then
while read line; do
p1=`echo $line | awk -F'>' '{print $1}'`
p2=`echo $line | awk -F'>' '{print $2}'`
sed -ir "s:$p1:$p2:" $tmpdir/segments
perl -i -pe "s:$p1:$p2:" $tmpdir/segments
done < $tmpdir/segments_to_fix
fi

Expand Down
2 changes: 1 addition & 1 deletion egs/ami/s5b/local/ami_sdm_scoring_data_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ if [ `cat $tmpdir/segments_to_fix | wc -l` -gt 0 ]; then
while read line; do
p1=`echo $line | awk -F'>' '{print $1}'`
p2=`echo $line | awk -F'>' '{print $2}'`
sed -ir "s:$p1:$p2:" $tmpdir/segments
perl -i -pe "s:$p1:$p2:" $tmpdir/segments
Comment thread
desh2608 marked this conversation as resolved.
Outdated
done < $tmpdir/segments_to_fix
fi

Expand Down
2 changes: 1 addition & 1 deletion egs/aspire/s5/local/multi_condition/prepare_impulses_noises.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ cp ${output_dir}_non_normalized/info/* $output_dir/info

# rename file location in the noise-rir pairing files
for file in `ls $output_dir/info/noise_impulse*`; do
sed -i "s/_non_normalized//g" $file
perl -i -pe "s/_non_normalized//g" $file
done

# generating the rir-list with probabilities alloted for each rir
Expand Down
4 changes: 2 additions & 2 deletions egs/babel/s5c/local/syllab/generate_syllable_lang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,8 @@ ln -s lex.syllabs2phones.disambig.fst $out/L_disambig.fst
echo "Validating the output lang dir"
utils/validate_lang.pl $out || exit 1

sed -i'' 's/#1$//g' $lout/lexicon.txt
sed -i'' 's/#1$//g' $lout/lexiconp.txt
perl -i'' -pe 's/#1$//g' $lout/lexicon.txt
perl -i'' -pe 's/#1$//g' $lout/lexiconp.txt
Comment thread
desh2608 marked this conversation as resolved.
Outdated

echo "Done OK."
exit 0
4 changes: 2 additions & 2 deletions egs/babel/s5d/local/syllab/generate_phone_lang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,8 @@ ln -s lex.syllabs2phones.disambig.fst $out/L_disambig.fst
echo "Validating the output lang dir"
utils/validate_lang.pl $out || exit 1

sed -i'' 's/#1$//g' $lout/lexicon.txt
sed -i'' 's/#1$//g' $lout/lexiconp.txt
perl -i'' -pe 's/#1$//g' $lout/lexicon.txt
perl -i'' -pe 's/#1$//g' $lout/lexiconp.txt

echo "Done OK."
exit 0
4 changes: 2 additions & 2 deletions egs/babel/s5d/local/syllab/generate_syllable_lang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,8 @@ ln -s lex.syllabs2phones.disambig.fst $out/L_disambig.fst
echo "Validating the output lang dir"
utils/validate_lang.pl $out || exit 1

sed -i'' 's/#1$//g' $lout/lexicon.txt
sed -i'' 's/#1$//g' $lout/lexiconp.txt
perl -i'' -pe 's/#1$//g' $lout/lexicon.txt
perl -i'' -pe 's/#1$//g' $lout/lexiconp.txt

echo "Done OK."
exit 0
6 changes: 3 additions & 3 deletions egs/bentham/v1/local/create_splits.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ function split {
echo $name $spkid >> $split_dir/utt2spk
done < "$line_file"

sed -i '/^\s*$/d' $split_dir/images.scp
sed -i '/^\s*$/d' $split_dir/text
sed -i '/^\s*$/d' $split_dir/utt2spk
perl -i -pe '/^\s*$/d' $split_dir/images.scp
perl -i -pe '/^\s*$/d' $split_dir/text
perl -i -pe '/^\s*$/d' $split_dir/utt2spk
utils/utt2spk_to_spk2utt.pl $split_dir/utt2spk > $split_dir/spk2utt
}

Expand Down
2 changes: 1 addition & 1 deletion egs/callhome_egyptian/s5/local/callhome_prepare_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ $dir/extra_questions.txt || exit 1;

# Add prons for laughter, noise, oov
for w in `grep -v sil $dir/silence_phones.txt`; do
sed -i "/\[$w\]/d" $tmpdir/lexicon.3
perl -i -pe "/\[$w\]/d" $tmpdir/lexicon.3
done

for w in `grep -v sil $dir/silence_phones.txt`; do
Expand Down
6 changes: 3 additions & 3 deletions egs/callhome_egyptian/s5/local/ctm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ fi
steps/get_ctm.sh $data_dir $lang_dir $decode_dir

# Make sure that channel markers match
#sed -i "s:\s.*_fsp-([AB]): \1:g" data/dev/stm
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} sed -i -r 's:fsp\s1\s:fsp A :g' {}
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} sed -i -r 's:fsp\s2\s:fsp B :g' {}
#perl -i -pe "s:\s.*_fsp-([AB]): \1:g" data/dev/stm
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} perl -i -pe 's:fsp\s1\s:fsp A :g' {}
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} perl -i -pe 's:fsp\s2\s:fsp B :g' {}

# Get the environment variables
. /export/babel/data/software/env.sh
Expand Down
6 changes: 3 additions & 3 deletions egs/fisher_callhome_spanish/s5/local/ctm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ fi
steps/get_ctm.sh $data_dir $lang_dir $decode_dir

# Make sure that channel markers match
#sed -i "s:\s.*_fsp-([AB]): \1:g" data/dev/stm
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} sed -i -r 's:fsp\s1\s:fsp A :g' {}
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} sed -i -r 's:fsp\s2\s:fsp B :g' {}
#perl -i -pe "s:\s.*_fsp-([AB]): \1:g" data/dev/stm
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} perl -i -pe 's:fsp\s1\s:fsp A :g' {}
#ls exp/tri5a/decode_dev/score_*/dev.ctm | xargs -I {} perl -i -pe 's:fsp\s2\s:fsp B :g' {}

# Get the environment variables
. /export/babel/data/software/env.sh
Expand Down
2 changes: 1 addition & 1 deletion egs/fisher_callhome_spanish/s5/local/fsp_prepare_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ if [ $stage -le 4 ]; then

# Add prons for laughter, noise, oov
for w in `grep -v sil $dir/silence_phones.txt`; do
sed -i "/\[$w\]/d" $tmpdir/lexicon.2
perl -i -pe "/\[$w\]/d" $tmpdir/lexicon.2
done

for w in `grep -v sil $dir/silence_phones.txt`; do
Expand Down
2 changes: 1 addition & 1 deletion egs/gale_arabic/s5/local/gale_prep_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ cat $dir/lexicon.txt | cut -d ' ' -f2- | tr -s ' ' '\n' |\
sort -u > $dir/nonsilence_phones.txt || exit 1;


sed -i '1i<UNK> SIL' $dir/lexicon.txt
perl -i -pe '1i<UNK> SIL' $dir/lexicon.txt

echo Dictionary preparation succeeded

Expand Down
2 changes: 1 addition & 1 deletion egs/gale_arabic/s5b/local/gale_prep_grapheme_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ echo SIL > $dir/optional_silence.txt
# really to the same base phone.
cat tmp2.$$ | tr -s ' ' '\n' | grep -v ^$ | sort -u > $dir/nonsilence_phones.txt || exit 1;

sed -i '1i<UNK> SIL' $dir/lexicon.txt # insert word <UNK> with phone sil at the begining of the dictionary
perl -i -pe '1i<UNK> SIL' $dir/lexicon.txt # insert word <UNK> with phone sil at the begining of the dictionary

rm -fr ar-ar_lexicon_2014-03-17.txt.bz2 ar-ar_grapheme_lexicon_2016-02-09.bz2 tmp$$ tmp1.$$ tmp2.$$
echo Dictionary preparation succeeded
Expand Down
2 changes: 1 addition & 1 deletion egs/iam/v1/local/prepare_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ while(<>){
}' | sort -u > $dir/lexicon.txt


sed -i "s/#/<HASH>/" $dir/nonsilence_phones.txt
perl -i -pe "s/#/<HASH>/" $dir/nonsilence_phones.txt

echo '<sil> SIL' >> $dir/lexicon.txt
echo '<unk> SIL' >> $dir/lexicon.txt
Expand Down
2 changes: 1 addition & 1 deletion egs/iam/v2/local/prepare_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ while(<>){
}' | sort -u > $dir/lexicon.txt


sed -i "s/#/<HASH>/" $dir/nonsilence_phones.txt
perl -i -pe "s/#/<HASH>/" $dir/nonsilence_phones.txt

echo '<sil> SIL' >> $dir/lexicon.txt

Expand Down
14 changes: 7 additions & 7 deletions egs/reverb/s5/local/download_se_eval_tool.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ unzip REVERB_scores.zip -d local/REVERB_scores_source
rm REVERB_scores.zip

pushd local/REVERB_scores_source/REVERB-SPEENHA.Release04Oct/evaltools
sed -i 's/wavread/audioread/g' prog/score_sim.m
perl -i -pe 's/wavread/audioread/g' prog/score_sim.m
git clone https://github.com/MuSAELab/SRMRToolbox.git
sed -i 's/wavread/audioread/g' SRMRToolbox/libs/preprocess.m
sed -i 's/SRMR_main/SRMR/g' prog/score_real.m
sed -i 's/SRMR_main/SRMR/g' prog/score_sim.m
sed -i 's/+wb\ //g' prog/calcpesq.m
sed -i 's/pesq_/_pesq_/g' prog/calcpesq.m
sed -ie '30d;31d' prog/calcpesq.m
perl -i -pe 's/wavread/audioread/g' SRMRToolbox/libs/preprocess.m
perl -i -pe 's/SRMR_main/SRMR/g' prog/score_real.m
perl -i -pe 's/SRMR_main/SRMR/g' prog/score_sim.m
perl -i -pe 's/+wb\ //g' prog/calcpesq.m
Comment thread
desh2608 marked this conversation as resolved.
Outdated
perl -i -pe 's/pesq_/_pesq_/g' prog/calcpesq.m
perl -i -pe '30d;31d' prog/calcpesq.m
patch score_RealData.m -i ../../../score_RealData.patch -o score_RealData_new.m
mv score_RealData_new.m score_RealData.m
patch score_SimData.m -i ../../../score_SimData.patch -o score_SimData_new.m
Expand Down
2 changes: 1 addition & 1 deletion egs/yomdle_fa/v1/local/prepare_dict.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ mkdir -p $dir

local/prepare_lexicon.py --data-dir $data_dir $dir

sed -i '/^\s*$/d' $dir/lexicon.txt
perl -i -pe '/^\s*$/d' $dir/lexicon.txt
cut -d' ' -f2- $dir/lexicon.txt | sed 's/SIL//g' | tr ' ' '\n' | sort -u | sed '/^$/d' >$dir/nonsilence_phones.txt || exit 1;

echo '<sil> SIL' >> $dir/lexicon.txt
Expand Down
2 changes: 1 addition & 1 deletion egs/yomdle_zh/v1/local/create_download.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,4 @@ local/create_line_image_from_page_image.py \

echo "Downloading table for CangJie."
wget -P $download_dir/ $cangjie_url || exit 1;
sed -ie '1,8d' $download_dir/cj5-cc.txt
Comment thread
desh2608 marked this conversation as resolved.
perl -i -pe '1,8d' $download_dir/cj5-cc.txt
12 changes: 6 additions & 6 deletions misc/maintenance/fix_cpplint_whitespace.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@

# Let's run a set of in-place modifications by sed-commands,
for file in $@; do
sed -i 's/; \/\//; \/\//' $file # '; //' -> '; //'
sed -i 's/{ \/\//{ \/\//' $file # '{ //' -> '{ //'
sed -i 's/} \/\//} \/\//' $file # '} //' -> '} //'
sed -i 's/for(/for (/' $file # 'for(' -> 'for ('
sed -i 's/if(/if (/' $file # 'if(' -> 'if ('
sed -i 's/\s\s*$//' $file # 'remove white-space at the end of lines'
perl -i -pe 's/; \/\//; \/\//' $file # '; //' -> '; //'
perl -i -pe 's/{ \/\//{ \/\//' $file # '{ //' -> '{ //'
perl -i -pe 's/} \/\//} \/\//' $file # '} //' -> '} //'
perl -i -pe 's/for(/for (/' $file # 'for(' -> 'for ('
perl -i -pe 's/if(/if (/' $file # 'if(' -> 'if ('
perl -i -pe 's/\s\s*$//' $file # 'remove white-space at the end of lines'

@danpovey danpovey Jan 11, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this and it seemed to remove the final newline.

@kkm000 kkm000 Mar 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danpovey, as a project to use clang-tidy is under way, maybe we should not touch this file at all? It's a maintainer's file, so presumably sed is fine here for the time being, only a handful of people run it if ever.

done

2 changes: 1 addition & 1 deletion tools/extras/install_portaudio.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ if [ -z "$MACOS" ]; then
fi

./configure --prefix=`pwd`/install --with-pic
sed -i.bk '40s:src/common/pa_ringbuffer.o::g; 40s:$: src/common/pa_ringbuffer.o:' Makefile
perl -i.bk -pe '40s:src/common/pa_ringbuffer.o::g; 40s:$: src/common/pa_ringbuffer.o:' Makefile
Comment thread
desh2608 marked this conversation as resolved.
Outdated

if [ "$MACOS" != "" ]; then
echo "detected MacOS operating system ... trying to fix Makefile"
Expand Down