Add scripts and update readmes

Work for #140

README.md

@@ -55,15 +55,16 @@ This project is very old. We've released the full history for the sake of mainta
 Unfortunately this means all code before our open source release is not open sourced.
 
 In practice this should never be a problem. We never use our old code, except for historical purposes, and you never
-should need to either.
+should need to either. If you do need to use any of this code, just raise an issue and we'll make an exeption
+for your use case.
 
-## Newer code - All commits after [a275d0bc5744ba43096b43de2ef2aee32dc14c18](https://github.com/QutEcoacoustics/audio-analysis/commit/a275d0bc5744ba43096b43de2ef2aee32dc14c18)
+## Newer code - All commits after 2018-01-30
 
-All code after a275d0bc5744ba43096b43de2ef2aee32dc14c18 (<time>2018-01-30T06:38:46Z</time>) are licensed under the 
+All code after [a275d0bc5744ba43096b43de2ef2aee32dc14c18](https://github.com/QutEcoacoustics/audio-analysis/commit/a275d0bc5744ba43096b43de2ef2aee32dc14c18) (<time>2018-01-30T06:38:46Z</time>) are licensed under the 
 [Apache License 2.0](https://choosealicense.com/licenses/apache-2.0/)
 
-## Older code
+## Older code - All commits before 2018-01-30
 
-All commits before a275d0bc5744ba43096b43de2ef2aee32dc14c18 (<time>2018-01-30T06:38:46Z</time>) are not licensed under
-an open source license. All rights and copyright are retained, however, the public has permission to view, link to, cite
+All commits before [a275d0bc5744ba43096b43de2ef2aee32dc14c18](https://github.com/QutEcoacoustics/audio-analysis/commit/a275d0bc5744ba43096b43de2ef2aee32dc14c18) (<time>2018-01-30T06:38:46Z</time>) are not licensed under
+an open source license. All rights and copyright are retained, however, the public has permission to view, link to, and cite
 the code on GitHub.

scripts/README.md

@@ -0,0 +1,62 @@
+# Scripts
+
+_AnalysisPrograms.exe_ works best when processing single audio files.
+This has a few advantages:
+
+**It simplifies the code**. We don't need to know about the particular 
+way you store your data, or the way you want your data processed.
+
+**It allows for greater customiszation**. By building a set of composable
+tools, it allows you to choose what analysis done, and when. You wan't it
+all? You got it. You only want the first bit done once, and the second done
+100 times with parameter sweeps? Go for it.
+
+**It is easier to parallelize**. You want to use your university HPC?
+Write your own distributed analysis? Or just run it in sequence? That's all
+your choice.
+
+Despite these advantages, we recognize it is useful to have example of
+how large analyses might be completed. Thus the scripts you find in this
+folder contain various examples (in various languages) of composing 
+workflows with _AnalysisPrograms.exe_.
+
+## Contributions appreciated!
+
+If you right a script that gets the job done for you, we'd be happy to
+include it here as an example for others to use.
+
+## PowerShell
+
+You'll see a lot of scripts in this folder that are written in PowerShell.
+If you're not familar with it, you can consider it as the Windows equivalent
+of the Bash shell. @atruskie like's PowerShell because in their personal
+opinion, the syntax is more reasonable than Bash, and the enhanced support
+for dates, times, and regular expressions are well worth the investment.
+
+As of [PowerShell 6.0](https://github.com/PowerShell/PowerShell#-powershell)
+the shell is cross platform and well worth investigating. If you're not
+convinced, the scripts should be easy enough to reimplment in your favourite
+language (like Bash)--and we would of course appreciate any translated
+scripts sent back to us as contributed examples.
+
+## Example headers
+
+We'd like to see each example script prefaced with a documentation header.
+A suggested format:
+
+```
+# A simple loop to XXX for a folder of files.
+# Jolene Blogs 2017
+#
+# For each audio file found,
+# - It runs does XXX
+# - It then also does YYY
+# - And sometimes does ZZZ
+# - ...
+#
+# Assumptions:
+# - AnalysisPrograms.exe is in current directory
+# - ...
+
+... script starts here ...
+```

scripts/generate_zooming.ps1

@@ -0,0 +1,109 @@
+# A loop to generate indices and zooming images, with support for remote file downloading
+# and retry semantics.
+# Anthony Truskinger 2018
+#
+# For each audio file found,
+# - It copies the file off a remote server
+# - it generates high reoslution indices
+# - It generates zooming tiles for each file.
+#
+# Assumptions:
+# - AnalysisPrograms.exe is on PATH (or in current directory)
+# - You're working in Windows (though only trivial changes are required to work in Unix)
+# - Users will customize the below variables before running the script
+
+$working_dir = "D:/Temp/zooming"
+$csv = "$working_dir\all_oxley_creek_recordings.csv"
+$output = $working_dir
+# The config files we copied from the ConfigFiles directory and customised for our purposes.
+$indices_config_file = "$working_dir\Towsey.Acoustic.Zooming.yml"
+$zooming_config_file = "$working_dir\SpectrogramZoomingConfig.yml"
+
+$remote_user = "ubuntnu"
+$remote_server = "server.example.com"
+$remote_path = "/home/ubuntu/data/original_audio"
+
+# helper functions
+
+# find the path to analysis programs
+$ap_path = (gcm AnalysisPrograms.exe).Path
+
+# just checks whether a previous run was successful 
+function HasAlreadyRun($results_dir) {
+    $log = Join-Path $results_dir "log.txt"
+    if (Test-Path $log) {
+        $match = Get-Content -Tail 1 $log | Select-String "ERRORLEVEL: (\d+)"
+        return $match.Matches.Groups[1].Value -eq 0
+    }
+
+    return $false
+}
+
+# a trivial utility to indent output from analysis programs
+# (makes it easier to separate script logging from AP.exe logging)
+function IndentOutput {
+    Process {
+        "`t" + $_
+    }
+}
+
+# import the csv data
+$recordings = Import-Csv $csv
+
+# for each file
+$results = @()
+foreach ($recording in $recordings) {
+    Write-Output "Starting new recording $($recording.uuid)"
+
+    # create a results object to store results
+    $result =  New-Object "pscustomobject" | Select-Object Download,Indices,Images
+
+    # extract all needed meta data to create a path to the remote file
+    # constructs a path that looks like: ".../data/b2/b24460cf-e25e-44c9-9034-af9b0a1ddcbe_20121019-140000Z.wav
+    $uuid = $recording.uuid
+    $prefix = $uuid.Substring(0,2)
+    $date = (Get-Date $recording.recorded_date).ToString("yyyyMMdd-HHmmssZ")
+    $name = "$uuid`_$date.wav"
+    $remote_path = "$remote_path/$prefix/$name"
+    $local_dir = "$output/$prefix"
+    $local_path = "$local_dir/$name"
+
+    mkdir.exe -p $local_dir
+    Set-Location $local_dir
+
+    # download the file via ssh
+    sftp "$remote_user@$remote_server`:$remote_path" $name
+    $result.Download = $LASTEXITCODE
+    if ($result.Download -ne 0) {
+        $results += $result
+        continue;
+    }
+
+    $instance_output = $local_path + "_results"
+    # generate indices
+
+    if (HasAlreadyRun $instance_output) {    
+        $result.Indices = "0*"
+        Write-Output "Skipping indices generation for $uuid - already completed"
+    }
+    else {
+        AnalysisPrograms.exe audio2csv $local_path $indices_config_file $instance_output -p --when-exit-copy-log -n | IndentOutput
+        $result.Indices = $LASTEXITCODE
+        if ($result.Indices -ne 0) {
+            $results += $result
+            continue;
+        }
+    }
+
+    # generate zooming tiles
+    $indices_dir = Join-Path $instance_output "Towsey.Acoustic"
+    AnalysisPrograms.exe DrawZoomingSpectrograms $indices_dir $zooming_config_file $instance_output -o "sqlite3" -z "Tile" -n  | IndentOutput
+    $result.Images = $LASTEXITCODE
+    $results += $result
+
+    # copy log
+    Copy-Item (Join-Path $ap_path "../Logs/log.txt") (Join-Path $instance_output "zooming_log.txt")
+}
+
+Write-Output "Analysis complete"
+$results | Export-Csv $output/results.csv -NoTypeInformation

scripts/indices_loop.ps1

@@ -0,0 +1,38 @@
+# A simple loop to generate indices for a folder of files.
+# Karlina Indraswari 2017
+#
+# For each audio file found,
+# - It runs audio2csv to generate indices
+#
+# Assumptions:
+# - AnalysisPrograms.exe is in current directory
+# - You're working in Windows (though only trivial changes are required to work in Unix)
+# - Users will customize the below variables before running the script
+
+
+# Select the directory containing the files
+$directory = "C:\temp\Emerald River Audio Snippets\20131227\"
+# The directory to store the results
+$base_output_directory = "C:\temp\indices_output"
+
+# Get a list of audio files inside the directory
+# (Get-ChildItem is just like ls, or dir)
+$files = Get-ChildItem "$directory\*" -Include "*.mp3", "*.wav"
+
+# iterate through each file
+foreach($file in $files) {
+    Write-Output ("Processing " + $file.FullName)
+
+    # get just the name of the file
+    $file_name = $file.Name
+
+    # make a folder for results
+    $output_directory = "$base_output_directory\$file_name"
+    mkdir $output_directory
+
+    # prepare command
+    $command = ".\AnalysisPrograms.exe audio2csv -source `"$file`" -config `".\configFiles\Towsey.Acoustic.30.yml`" -output `"$output_directory`" -n"
+
+    # finally, execute the command
+    Invoke-Expression $command
+}