diff --git a/README.md b/README.md index e68061a4..d81e5b5a 100644 --- a/README.md +++ b/README.md @@ -11,16 +11,12 @@ table of contents 2. [installing mediamicroservices](https://github.com/mediamicroservices/mm#installing-mediamicroservices) 3. [configuring mediamicroservices](https://github.com/mediamicroservices/mm#configuring-mediamicroservices) 1. [variable explanations](https://github.com/mediamicroservices/mm#variable-explanations) -3. [Database Instructions](#database-instructions) - 1. [Database Configuration](#database-configuration) - 2. [Database Backup](#database-backup) -4. [mediamicroservices functions and instructions for use](https://github.com/mediamicroservices/mm#mediamicroservices-functions-and-instructions-for-use) +3. [mediamicroservices functions and instructions for use](https://github.com/mediamicroservices/mm#mediamicroservices-functions-and-instructions-for-use) * [aipupgrade](https://github.com/mediamicroservices/mm#aipupgrade) * [barcodeinterpret](https://github.com/mediamicroservices/mm#barcodeinterpret) * [blackatends](https://github.com/mediamicroservices/mm#blackatends) * [checksum2filemaker](https://github.com/mediamicroservices/mm#checksum2filemaker) * [checksumpackage](https://github.com/mediamicroservices/mm#checksumpackage) - * [createpremisdb](#createpremisdb) * [finishpackage](https://github.com/mediamicroservices/mm#finishpackage) * [fix_left2stereo](https://github.com/mediamicroservices/mm#fix_left2stereo) * [fix_rewrap](https://github.com/mediamicroservices/mm#fix_rewrap) @@ -198,49 +194,11 @@ this variable stores the name of the FileMaker database that is used in checksum **21. VOLADJUST** This variable must be set to yes (Y) or no (N). If set to yes, volume will be run through a volume adjustment filter and adjusted accordingly during transcoding. -**22. PREMIS_DB** -This variable must be set to yes (Y) or no (N). If set to yes, database reporting for mm will be enabled. - -**23. SQL_ERROR_EMAIL_TO** -This can be set to include an email address to receive error reports for database insertion. - -**24. PREMIS_PROFILE** -Enter the login path information for the database here. (Use the output supplied by the createpremisdb script, or create your own using mysql_config_editor). -** PREMIS_NAME** -Enter the name of the database used for mm reporting. -**26 PERCEPTUAL_HASH** -Set this to 1 (on) to enable the creation of perceptual hashes during the ingestfile process -**27. Quit** +**22. Quit** if editing in the terminal, use this option to leave the configuration file editor. *** -## Database Instructions - -Media Microservices can be integrated with a database for the capture of a variety of preservation metadata generated through the mm workflow. Currently the MM Database supports PREMIS event information, fixity information, perceptual hashes for video files, mediainfo output and ingest logs. - -### Database specific scripts -* createpremisdb -* dbbackup -* makefingerprint -* searchfingerprint -* updatingplist - -### Database Configuration - -To configure the database, run the command `createpremisdb -c` on your __host__ computer and follow the prompts. This will set up the database as well as facilitate user creation. To create users without creating a new database, use the command `createpremisdb -u`. At the end of user creation, the script will supply a command to create a log-in profile for the database. It should look something like this: `mysql_config_editor set --login-path=your_user_config --host=xx.xx.xxx.xxx --user=your_user --password`. Run this command on your __user__ computer and enter the password for the user you created when prompted. This will create the SQL log-in profile that you will use when configuring the microservices. NOTE: When supplying the suggested command, the script does its best to auto-fill the host IP address. You may wish to verify that this is the correct IP. - -To finalize the database setup, run `mmconfig` (GUI mode) or `mmconfig -t` (CLI) and select 'Y' in the PREMIS Database logging section, (or set to Y if using CLI). Enter the database name and log-in profile you have created to finalize DB connectivity. Additional database options (such as fingerprint generation) can be set at this time. - -![Database GUI Example](https://github.com/mediamicroservices/mm/blob/master/Resources/mmgui_dbsetup.png) - -### Database Backup -MM includes a script for database backup. This can be either run manually or set up as an automated process. After configuring the backup script, running `dbbackup` will create a zipped backup in the chosen location. To automate this process use Brew Services. Typing `brew services start mm` will cause the db to be backed up automatically using the included plist file. By default, backups will occur daily at 2:00 AM. - -To configure database backup use the command `dbbackup -e`. This will open the config file in a terminal editor. Set the necessary variables to your desired values. - -*** - ## mediamicroservices functions and instructions for use For all microservices, the structure of the command looks like this: @@ -275,18 +233,6 @@ To view the specific ffmpeg encoding options for each file, view the sourcecode * Another option is to use -c in conjunction with -u, which will create new checksums and version the previous ones if the check is unsuccessful, meaning your checksums have changed. Type `checksumpackage -cu [input]` for this option. * Finally, use -v as an option if you want to fully verify checksums. If no checksums exist, the script will create the initial ones. Verification will version existing checksums by adding the date they were created to the filename and create new ones, and log the difference to a checksumprocess log, which will be placed in the metadata directory of the package. To use -v, type `checksumpackage -v [input]`. -#### createmmdb -* createmmdb configures a database, users and login profiles. Your command will look like this 'Usage: -c (create database) -u (create user) -h (help)'. - -#### createpremisdb -* createpremisdb will create a database for the logging of microservices information as well as facilitate user creation. For more information on use, see the [database configuration](#configuring-premisfixity-logging-database) section of this readme. -* commands are `createpremisdb -c` for database creation and `createpremisdb -u` for user creation. - -#### dbbackup -* Creates backups of selected database. Allows a user specified number of backups to be saved. Script can be run manually or set up as a recurring task using the `brew services start mm` command. - -* Usage: -e edit configurations, -c create login credentials for backup -h help - #### finishpackage * finishpackage is a combination of the microservices makelossless, makebroadcast, makeyoutube, makemetadata, and checksumpackage. The purpose is to losslessly transcode, create access copies, and create metadata and directory structure information for a file or package input. To use finishpackage, type finishpackage and drag your input into the command line: `finishpackage [input]`. finish package is typically used in conjunction with restructureForCompliance. diff --git a/Resources/dbinfo.md b/Resources/dbinfo.md deleted file mode 100644 index 535d1db1..00000000 --- a/Resources/dbinfo.md +++ /dev/null @@ -1,100 +0,0 @@ -# mediamicroservices/LTOpers Database Features - -## Introduction -The mediamicroservices database supports the storing and querying of a variety of information created by both [mediamicroservices](https://github.com/mediamicroservices/mm) and [LTOpers](https://github.com/amiaopensource/ltopers). This document provides database setup information, a breakdown of the structure of the database and examples of the scripts currently used to interact with it. - -## Database specific scripts -* createmmdb -* dbbackup -* makefingerprint -* searchfingerprint -* updatingplist - -## Database Configuration - -To configure the database, run the command `createmmdb -c` on your __host__ computer and follow the prompts. This will set up the database as well as facilitate user creation. To create users without creating a new database, use the command `createmmdb -u`. At the end of user creation, the script will supply a command to create a log-in profile for the database. It should look something like this: `mysql_config_editor set --login-path=your_user_config --host=xx.xx.xxx.xxx --user=your_user --password`. Run this command on your __user__ computer and enter the password for the user you created when prompted. This will create the SQL log-in profile that you will use when configuring the microservices. NOTE: When supplying the suggested command, the script does its best to auto-fill the host IP address. You may wish to verify that this is the correct IP. - -To finalize the database setup, run `mmconfig` (GUI mode) or `mmconfig -t` (CLI) and select 'Y' in the PREMIS Database logging section, (or set to Y if using CLI). Enter the database name and log-in profile you have created to finalize DB connectivity. Additional database options (such as fingerprint generation) can be set at this time. - -![Database GUI Example](https://github.com/mediamicroservices/mm/blob/master/Resources/mmgui_dbsetup.png) - -## Database Backup -Mediamicroservices includes a script for database backup. This can be either run manually or set up as an automated process. After configuring the backup script, running `dbbackup` will create a zipped backup in the chosen location. To automate this process use Brew Services. Typing `brew services start mm` will cause the db to be backed up automatically using the included plist file. By default, backups will occur daily at 2:00 AM. - -To configure database backup use the command `dbbackup -e`. This will open the config file in a terminal editor. Set the neccessary variables to your desired values. - -## Database structure -The Database is divided into six tables with the following structure: - -*** - -#### object: -__Contains identifying information for objects in the database.__ - -|objectIdentifierValueID|objectIdentifierValue|objectDB_Insertion|object_LastTouched| -|---|---|---|---| -|Auto-incrementing number for object records|Name of object|Date/Time Object added to DB|Date/Time Object was last updated| - -*** - -#### event: -__Tracks occurences of events affecting Objects contained in database.__ - -|eventIdentifierValueID|objectIdentifierValue|eventType|eventDateTime|eventDetail|eventOutcome|eventDetailOPT|eventDetailCOMPNAME|linkingAgentIdentifierValue| -|---|---|---|---|---|---|---|---|---| -|Auto-incrementing number for event records|Name of target object for event|PREMIS type for event|Date/Time of event|Name of script|Outcome of event|Options used for script|Computer used for event|User who initiated event| - -*** - -#### fixity -__Captures fixity information generated by fixity events.__ - -|fixityIdentifierValueID|eventIdentifierValue|objectIdentifierValue|eventDateTime|eventDetail|messageDigestAlgorithm|messageDigestSOURCE|messageDigestPATH|messageDigestFILENAME|messageDigestHASH| -|---|---|---|---|---|---|---|---|---|---| -|Auto-incrementing number for fixity records|ID of corresponding event|Name of corresponding object|Date/Time of fixity event|Script name|Hashing algorithm|Source of hash|Location of hash|Name of file hashed|Hash| - -*** - -#### objectCharacteristics -__Captures information from sidecar files generated during microservices ingest.__ - -|objectCharacteristicValueID|mediaInfo|captureLog| -|---|---|---| -|Auto-incrementing number for object characteristic records|Contents of associated media info xml file|Contents of associated capture log| - -*** - -#### ltoSchema -__Captures contents of LTO schema files generated through LTOpers.__ - -|ltoSchemaValueID|ltoID|fileName|filePath|fileSize|modifyTime| -|---|---|---|---|---|---| -|Auto-incrementing number for object characteristic records|ID for corresponding tape|file name|File path|File size|Date/Time of last file modification| - -*** - -#### fingerprints -__Stores fingerprints (perceptual hash information).__ - -|hashNumber|objectIdentifierValue|startframe|endframe|hash1|hash2|hash3|hash4|hash5| -|---|---|---|---|---|---|---|---|---| -|Auto-incrementing number for hashes|Name of corresponding item|Start frame for hash|End frame for hash|First component of hash|Second component of hash|Third component of hash|Fourth component of hash|Fifth component of hash| - -## Database Search Scripts - -#### searchlto - -Searches the LTO schema information stored in the database. Input for standard search can consist either of a Tape ID or a Media ID. If the `-c` flag is used, input is a file to be compared against information stored in the DB. Input file will be compared for modification time, file size and file name. - -Usage: searchlto [-h] (help) [-c] (compares an input file to file information stored in DB) [-f] (shows full schema) [Input] - -#### searchfingerprint - -Generates fingerprints (perceptual hashes) from whole or specified portion of an input video file and compares them against fingerprints stored in the database. Outputs any detected matches in 500 frame segments both in the terminal and in an optional preview window. Preview window will attempt to locate the portions of input video for which matches were found. - -Usage: searchfingerprint [ -h ] (help) [ -i ] (set in time) [ -o ] (set out time) [ -t ] (text only-don't display video preview) [Input] - -## Database Support Scripts -#### createmmdb -#### dbbackup -#### updatingplist diff --git a/checksumpackage b/checksumpackage index 0cf78c14..ca34b83d 100755 --- a/checksumpackage +++ b/checksumpackage @@ -125,7 +125,6 @@ while [ "${*}" != "" ] ; do fi #perform checking and verification if [ "${CHECK}" = "Y" ] ; then - _report_to_db if [ -s "${PACKAGE}/${OUTDIR}/${DFXMLNAME}" ] ; then _check_dfxml "${PACKAGE}" if [ "${CHECKERR}" != "0" ] ; then @@ -182,11 +181,8 @@ while [ "${*}" != "" ] ; do else xmlstarlet sel -T -t -m "/dfxml/fileobject" -v "hashdigest" -o " " -v "filename" -n "${PACKAGE}/${OUTDIR}/${DFXMLNAME}" > "${PACKAGE}/${OUTDIR}/${CHECKSUMNAME}" fi - db_source="${PACKAGE}/${OUTDIR}/${CHECKSUMNAME}" - db_fixity=$(cat "${PACKAGE}/${OUTDIR}/${CHECKSUMNAME}") - _report_to_db + echo "$(_get_iso8601): $(basename "${0}"):${VERSION}: ${CHECKSUMNAME} is generated" >> "${PACKAGE}/${OUTDIR}/${CHECKSUMPROCESSLOG}" - event_outcome="Fixity Creation" if [ "${UPDATE}" = "Y" ] ; then exit @@ -233,9 +229,6 @@ while [ "${*}" != "" ] ; do else _report -dt "ERROR: ${0} requires directories as input and ${PACKAGE} is not a directory." fi - _eventoutcome_update - _report_fixity_db - _db_email_delivery "${SQL_ERROR_EMAIL_TO}" _log -e shift done diff --git a/ingestfile b/ingestfile index 60885b27..31193613 100755 --- a/ingestfile +++ b/ingestfile @@ -731,4 +731,3 @@ fi # clean up queue [ "${PRIORITY}" == "put in queue" ] && _remove_mediaid_from_queue -_report_to_db diff --git a/makederiv b/makederiv index 1a56ae63..88c5887b 100755 --- a/makederiv +++ b/makederiv @@ -217,7 +217,6 @@ while [[ "${@}" != "" ]] ; do INGESTLOG="${INPUT}/metadata/submissionDocumentation/logs/capture.log" _report -wt "The capture.log is found in the ./metadata/submissionDocumentation/logs/ directory. Using that for now but consider moving it to ./metadata/logs/" fi - _report_to_db # get information on the input _get_height "${SOURCEFILE}" diff --git a/makeframes b/makeframes index e5dabddf..7eed220e 100755 --- a/makeframes +++ b/makeframes @@ -50,7 +50,6 @@ while [ "${*}" != "" ] ; do _find_input "${INPUT}" MEDIAID=$(basename "${INPUT}" | cut -d. -f1) - _report_to_db # encoding options _get_codectagstring "${SOURCEFILE}" diff --git a/makegifsummary b/makegifsummary index 2c7e98d3..432a995b 100755 --- a/makegifsummary +++ b/makegifsummary @@ -71,8 +71,6 @@ while [ "${*}" != "" ] ; do fi fi - _report_to_db - # encoding options _get_codectagstring "${SOURCEFILE}" _get_dar "${SOURCEFILE}" diff --git a/mmconfig b/mmconfig index b40c86f8..684fe9d1 100755 --- a/mmconfig +++ b/mmconfig @@ -107,42 +107,10 @@ export MAIN_DIALOG=$(cat << MAINFORM $(funcEntry DVDLABELPREFIX "Please enter the prefix you would like added to the filename in makedvd. You may leave this blank:" "80") - - - - - - - if true enable:PREMIS_PROFILE - if true enable:PREMIS_NAME - - PREMIS_DB - - - if true disable:PREMIS_PROFILE - if true disable:PREMIS_NAME - - PREMIS_DBN - - - $(funcEntry PREMIS_PROFILE "Enter the name of the database profile:" "60") - $(funcEntry PREMIS_NAME "Enter the name of the database:" "60") - - - - - PREMIS_PLIST - - - - PERCEPTUAL_HASH - - $(funcEntry EMAIL_FROM "Please enter the email account that notifications will be sent from:" "80") - $(funcEntry SQL_ERROR_EMAIL_TO "Please the email account for the delivery of SQL Error Reports:" "80") $(funcEntry MAKEBROADCAST_DELIVERY_EMAIL_TO "Please enter the email account (or multiple accounts separated by a comma) makebroadcast notifications will be delivered to:" "80") @@ -181,21 +149,6 @@ elif [[ "${EXIT}" = "OK" ]] ; then else VOLADJUST="N" fi - if [[ "${PREMIS_DB}" = "true" ]] ; then - PREMIS_DB="Y" - else - PREMIS_DB="N" - fi - if [[ "${PREMIS_PLIST}" = "true" ]] ; then - PREMIS_PLIST="1" - else - PREMIS_PLIST="0" - fi - if [[ "${PERCEPTUAL_HASH}" = "true" ]] ; then - PERCEPTUAL_HASH="1" - else - PERCEPTUAL_HASH="0" - fi #echo "${VOLADJUST}" | sed 's:true:Y:g;s:false:N:g' echo " OUTDIR_INGESTFILE = ${OUTDIR_INGESTFILE}" echo " OUTDIR_INGESTXDCAM = ${OUTDIR_INGESTXDCAM}" @@ -217,13 +170,7 @@ elif [[ "${EXIT}" = "OK" ]] ; then echo " MAKEYOUTUBE_DELIVERY_EMAIL_TO = ${MAKEYOUTUBE_DELIVERY_EMAIL_TO}" echo " MAKEBROADCAST_DELIVERY_EMAIL_TO = ${MAKEBROADCAST_DELIVERY_EMAIL_TO}" echo " FILE_MAKER_DB = ${FILE_MAKER_DB}" - echo " SQL_ERROR_EMAIL_TO = ${SQL_ERROR_EMAIL_TO}" echo " VOLADJUST = ${VOLADJUST}" - echo " PREMIS_DB = ${PREMIS_DB}" - echo " PREMIS_PROFILE = ${PREMIS_PROFILE}" - echo " PREMIS_NAME = ${PREMIS_NAME}" - echo " PREMIS_PLIST = ${PREMIS_PLIST}" - echo " PERCEPTUAL_HASH = ${PERCEPTUAL_HASH}" { echo "OUTDIR_INGESTFILE=\"${OUTDIR_INGESTFILE}\"" echo "OUTDIR_INGESTXDCAM=\"${OUTDIR_INGESTXDCAM}\"" @@ -245,12 +192,6 @@ elif [[ "${EXIT}" = "OK" ]] ; then echo "MAKEYOUTUBE_DELIVERY_EMAIL_TO=\"${MAKEYOUTUBE_DELIVERY_EMAIL_TO}\"" echo "MAKEBROADCAST_DELIVERY_EMAIL_TO=\"${MAKEBROADCAST_DELIVERY_EMAIL_TO}\"" echo "FILE_MAKER_DB=\"${FILE_MAKER_DB}\"" - echo "SQL_ERROR_EMAIL_TO=\"${SQL_ERROR_EMAIL_TO}\"" echo "VOLADJUST=\"${VOLADJUST}\"" - echo "PREMIS_DB=\"${PREMIS_DB}\"" - echo "PREMIS_PROFILE=\"${PREMIS_PROFILE}\"" - echo "PREMIS_NAME=\"${PREMIS_NAME}\"" - echo "PREMIS_PLIST=\"${PREMIS_PLIST}\"" - echo "PERCEPTUAL_HASH=\"${PERCEPTUAL_HASH}\"" } > "${MM_CONFIG_FILE}" fi diff --git a/mmfunctions b/mmfunctions index cd3207af..a836630d 100755 --- a/mmfunctions +++ b/mmfunctions @@ -47,286 +47,6 @@ _check_for_colons(){ fi } -_escape_for_db(){ - echo "${1}" | tr "’" "'"| sed "s/'/\''/g" -} - -_report_to_db(){ - if [ "${PREMIS_DB}" = "Y" ] ; then - eventDetail=$(basename "${0}") - if [ -z "${MEDIA_ID}" ] ; then - MEDIA_ID=$(basename "${INPUT}") - fi - LOGDIR="${OUTDIR_INGESTFILE}/${MEDIAID}/metadata/logs" - INGESTLOG="${LOGDIR}/capture.log" - if [ -f "${INGESTLOG}" ] ; then - OP=$(grep "operator" "${INGESTLOG}" | cut -c 11-) - else - OP="${OP}" - fi - if [ -z "${OP}" ] ; then - OP="${USER}" - fi - _premis_event_list - table_name="object" - echo "INSERT IGNORE INTO object (objectIdentifierValue,object_LastTouched) VALUES ('${MEDIA_ID}',NOW()) ON DUPLICATE KEY UPDATE object_LastTouched = NOW()" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" 2> /dev/null - _db_error_check - table_name="event" - echo "INSERT INTO event (objectIdentifierValue,eventType,eventDetail,eventDetailOPT,eventDetailCOMPNAME,linkingAgentIdentifierValue) VALUES ('${MEDIA_ID}','${eventType}','${eventDetail}','${user_input}','${HOSTNAME}', '${OP}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" 2> /dev/null - _db_error_check - if [ -n "${MEDIAINFO}" ] || [ -n "${CAPTURELOG}" ] ; then - table_name="objectCharacteristics" - echo "INSERT INTO objectCharacteristics (objectIdentifierValue,mediaInfo,captureLog) VALUES ('${MEDIA_ID}','${MEDIAINFO}','${CAPTURELOG}') ON DUPLICATE KEY UPDATE mediaInfo=COALESCE(NULLIF(mediaInfo,''),'${MEDIAINFO}'),captureLog=COALESCE(NULLIF(captureLog,''),'${CAPTURELOG}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" 2> /dev/null - _db_error_check - fi - fi -} - -_report_schema_db(){ - if [ "${PREMIS_DB}" = "Y" ] ; then - _premis_event_list - table_name="ltoSchema" - if [ -z "${xmlschema}" ] ; then - xmlschema="$LTO_LOGS/${TAPE_SERIAL}.schema" - fi - schema_sorted=$(xmlstarlet sel -t -m ".//file" -v "concat(name,'|',length, '|', modifytime)" -o "|" -m "ancestor-or-self::directory" -v "name" -o "/" -b -n "${xmlschema}") - schema_tape=$(basename "${xmlschema}" | cut -d'.' -f1) - IFS=$(echo -en "\n\b") - (for i in ${schema_sorted}; do LINE=$i; - schema_name=$(_escape_for_db $(echo "${LINE}" | cut -d'|' -f1)) - schema_path=$(_escape_for_db $(echo "${LINE}" | cut -d'|' -f4)) - schema_length=$(echo "${LINE}" | cut -d'|' -f2) - schema_date=$(echo "${LINE}" | cut -d'|' -f3) - echo "INSERT IGNORE INTO ltoSchema (ltoID,fileName,fileSize,modifyTime,filePath) VALUES ('${schema_tape}','${schema_name}','${schema_length}','$schema_date','${schema_path}${schema_name}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" 2> /dev/null - if [ "${?}" != "0" ]; then - sleep 1 - echo "INSERT IGNORE INTO ltoSchema (ltoID,fileName,fileSize,modifyTime,filePath) VALUES ('${schema_tape}','${schema_name}','${schema_length}','$schema_date','${schema_path}${schema_name}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" || exit 1 - fi - done) - unset IFS - _db_error_check - fi -} - -_report_fingerprint_db(){ - table_name="fingerprints" - (IFS=$'\n' - for i in ${VIDEOFINGERPRINT} ; do - hash1=$(echo "$i" | cut -d':' -f3) - hash2=$(echo "$i" | cut -d':' -f4) - hash3=$(echo "$i" | cut -d':' -f5) - hash4=$(echo "$i" | cut -d':' -f6) - hash5=$(echo "$i" | cut -d':' -f7) - startframe=$(echo "$i" | cut -d':' -f1) - endframe=$(echo "$i" | cut -d':' -f2) - echo "INSERT INTO fingerprints (objectIdentifierValue,startframe,endframe,hash1,hash2,hash3,hash4,hash5) VALUES ('${MEDIA_ID}','${startframe}','${endframe}','${hash1}','${hash2}','${hash3}','${hash4}','${hash5}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" - if [ "${?}" != "0" ]; then - echo "INSERT INTO fingerprints (objectIdentifierValue,startframe,endframe,hash1,hash2,hash3,hash4,hash5) VALUES ('${MEDIA_ID}','${startframe}','${endframe}','${hash1}','${hash2}','${hash3}','${hash4}','${hash5}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" || exit 1 - fi - done) - _db_error_check -} - -_report_fixity_db(){ - if [ "${PREMIS_DB}" = "Y" ] ; then - _premis_event_list - table_name="fixity" - if [ -z "${db_fixity}" ] ; then - echo -e "\033[1;31mNo fixity data found! Fixity information not reported to the database.\033[0m" - event_outcome="No fixity information" - fi - if [ -z "${lastrecord}" ] ; then - lastrecord=$(echo "SELECT eventIdentifierValue FROM event WHERE objectIdentifierValue = '${MEDIA_ID}' AND eventDetail = '${eventDetail}' ORDER BY eventIdentifierValue DESC LIMIT 1" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}") - fi - EVENT_ID_TRIM=$(echo "$lastrecord" | cut -d " " -f2) - IFS=$(echo -en "\n\b") - (for i in ${db_fixity}; do LINE=$i; - messageDigestHASH=$(echo "$LINE" | cut -c -32 ) - messageDigestPATH=$(_escape_for_db $(echo "$LINE" | cut -c 35- )) - messageDigestFILENAME=$(basename "${messageDigestPATH}") - echo "INSERT INTO fixity (eventIdentifierValue,objectIdentifierValue,eventDetail,messageDigestAlgorithm,messageDigestSOURCE,messageDigestPATH,messageDigestFILENAME,messageDigestHASH) VALUES ('${EVENT_ID_TRIM}','${MEDIA_ID}','${eventDetail}','md5','${db_source}','${messageDigestPATH}','${messageDigestFILENAME}','${messageDigestHASH}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" - if [ "${?}" != "0" ]; then - sleep 1 - echo "INSERT INTO fixity (eventIdentifierValue,objectIdentifierValue,eventDetail,messageDigestAlgorithm,messageDigestSOURCE,messageDigestPATH,messageDigestFILENAME,messageDigestHASH) VALUES ('${EVENT_ID_TRIM}','${MEDIA_ID}','${eventDetail}','md5','${db_source}','${messageDigestPATH}','${messageDigestFILENAME}','${messageDigestHASH}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" || exit 1 - fi - done) - _db_error_check - unset IFS - fi -} - -_db_email_delivery(){ - if [ "${PREMIS_DB}" = "Y" ] ; then - EMAILTO="${1}" - if [ "${EMAILTO}" ] ; then - temp_message=$(_maketemp) - if [[ -f "${REPORT_DUMP}" ]] ; then - echo -e "\033[1;103;95mZipping and emailing a copy of "${REPORT_DUMP} to to ${EMAILTO}"\033[0m" - gzip -f "${REPORT_DUMP}" - echo -e "Subject:Database insertion report for ${MEDIA_ID}\nThere was a problem with database reporting for the Media ID ${MEDIA_ID}.\n A database insertion report has been created at ${REPORT_DUMP}.gz on ${HOSTNAME}. A copy is attached to this email. \n\n" > "$temp_message" - uuencode "${REPORT_DUMP}".gz $(basename "${REPORT_DUMP}".gz) >> "$temp_message" - sendmail -f "${EMAIL_FROM}" -F "${EMAILTO}" "${EMAILTO}" < "$temp_message" - if [ -f "$temp_message" ] ; then - rm "$temp_message" - fi - fi - fi - fi -} - -_eventoutcome_update(){ - if [ "${PREMIS_DB}" = "Y" ] ; then - if [ -z "$event_outcome" ] ; then - event_outcome="Success" - fi - if [ -n "${RUN_ERR}" ] ; then - if [[ "${RUN_ERR}" != 0 ]] ; then - event_outcome="Fail" - fi - fi - lastrecord=$(echo "SELECT eventIdentifierValue FROM event WHERE objectIdentifierValue = '${MEDIA_ID}' AND eventDetail = '${eventDetail}' ORDER BY eventIdentifierValue DESC LIMIT 1" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}") - lastrecord_trim=$(echo $lastrecord | cut -d " " -f2) - echo "update event set eventOutcome='${event_outcome}' WHERE eventIdentifierValue='${lastrecord_trim}'" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" 2> /dev/null - fi -} - -_db_error_check(){ - if [ "$?" != "0" ] ; then - ERROR_DATE=$(date +%Y%m%d) - REPORT_DUMP=~/Desktop/"${MEDIA_ID}"_"${ERROR_DATE}"_dbreport.sh - echo -e "\033[1;5;95mERROR:\033[0m \033[1;103;95mCould not access database "${PREMIS_NAME}". Information was not added to the "${table_name}" table. SQL command history has been written to "${REPORT_DUMP}"\033[0m" - if [ "$table_name" = "object" ] ; then - echo "echo \"INSERT IGNORE INTO object (objectIdentifierValue,object_LastTouched) VALUES ('${MEDIA_ID}',NOW()) ON DUPLICATE KEY UPDATE object_LastTouched = NOW()\" | mysql --login-path=\"${PREMIS_PROFILE}\" \"${PREMIS_NAME}\"" >> "${REPORT_DUMP}" - fi - - if [ "${table_name}" = "fixity" ] ; then - IFS=$(echo -en "\n\b") - for i in ${db_fixity} ; do - LINE=$i; - messageDigestHASH=$(echo "$LINE" | cut -c -32 ) - messageDigestPATH=$(_escape_for_db $(echo "$LINE" | cut -c 35- )) - messageDigestFILENAME=$(basename "${messageDigestPATH}") - echo "echo \"INSERT INTO fixity (eventIdentifierValue,objectIdentifierValue,eventDetail,messageDigestAlgorithm,messageDigestSOURCE,messageDigestPATH,messageDigestFILENAME,messageDigestHASH) VALUES ('${EVENT_ID_TRIM}','${MEDIA_ID}','${eventDetail}','md5','${db_source}','${messageDigestPATH}','${messageDigestFILENAME}','${messageDigestHASH}')\" | mysql --login-path=\"${PREMIS_PROFILE}\" \"${PREMIS_NAME}\"" >> "${REPORT_DUMP}" - done - unset IFS - fi - - if [ "${table_name}" = "event" ] ; then - echo "echo \"INSERT INTO event (objectIdentifierValue,eventType,eventDetail,eventDetailOPT,eventDetailCOMPNAME,linkingAgentIdentifierValue) VALUES ('${MEDIAID}','${eventType}','${eventDetail}','${user_input}','${HOSTNAME}', '${OP}')\" | mysql --login-path=\"${PREMIS_PROFILE}\" \"${PREMIS_NAME}\"" >> "${REPORT_DUMP}" - fi - if [ "${table_name}" = "objectCharacteristics" ] ; then - echo "echo \"INSERT INTO objectCharacteristics (objectIdentifierValue,mediaInfo,captureLog,videoFingerprint,videoFingerprintSorted) VALUES ('${MEDIA_ID}','${MEDIAINFO}','${CAPTURELOG}','${VIDEOFINGERPRINT}','${VIDEOFINGERPRINT_SORTED}') ON DUPLICATE KEY UPDATE mediaInfo=COALESCE(NULLIF(mediaInfo,''),'${MEDIAINFO}'),captureLog=COALESCE(NULLIF(captureLog,''),'${CAPTURELOG}'),videoFingerprint=COALESCE(NULLIF(videoFingerprint,''),'${VIDEOFINGERPRINT}'),videoFingerprintSorted=COALESCE(NULLIF(videoFingerprintSorted,''),'${VIDEOFINGERPRINT_SORTED}')\" | mysql --login-path=\"${PREMIS_PROFILE}\" \"$PREMIS_NAME\"" >> "${REPORT_DUMP}" - fi - if [ "${table_name}" = "ltoSchema" ] ; then - echo "echo \"INSERT INTO ltoSchema (ltoID,fileName,fileSize,modifyTime,filePath) VALUES ('${schema_tape}','${schema_name}','${schema_length}','$schema_date','${schema_path}${schema_name}')\" | mysql --login-path=\"${PREMIS_PROFILE}\" \"${PREMIS_NAME}\"" >> "${REPORT_DUMP}" - fi - if [ "${table_name}" = "fingerprints" ] ; then - (IFS=$'\n' - for i in ${VIDEOFINGERPRINT} ; do - hash1=$(echo "$i" | cut -d':' -f3) - hash2=$(echo "$i" | cut -d':' -f4) - hash3=$(echo "$i" | cut -d':' -f5) - hash4=$(echo "$i" | cut -d':' -f6) - hash5=$(echo "$i" | cut -d':' -f7) - startframe=$(echo "$i" | cut -d':' -f1) - endframe=$(echo "$i" | cut -d':' -f2) - echo "echo \"INSERT INTO fingerprints (objectIdentifierValue,startframe,endframe,hash1,hash2,hash3,hash4,hash5) VALUES ('${MEDIA_ID}','${startframe}','${endframe}','${hash1}','${hash2}','${hash3}','${hash4}','${hash5}')\" | mysql --login-path=\"${PREMIS_PROFILE}\" \"${PREMIS_NAME}\"" >> "${REPORT_DUMP}" - done) - fi - else - echo -e "\033[1;103;95mSuccessfully reported infomation to the "${table_name}" table in the database "${PREMIS_NAME}"\033[0m" - fi -} - -_fingerprint_to_db(){ -VIDEOFINGERPRINT=$(xmlstarlet sel -N "m=urn:mpeg:mpeg7:schema:2001" -t -m "m:Mpeg7/m:DescriptionUnit/m:Descriptor/m:VideoSignatureRegion/m:VSVideoSegment" -v m:StartFrameOfSegment -o ':' -v m:EndFrameOfSegment -o ':' -m m:BagOfWords -v "translate(.,' ','')" -o ':' -b -n "${FINGERPRINT_XML}") -} - -_premis_event_list(){ - if [[ -n "${OUTPUT_TYPE}" ]] ;then - eventDetail="${OUTPUT_TYPE}" - else - eventDetail=$(basename "${0}") - fi - eventType="${eventDetail}" - - if [ "$eventDetail" = "ingestfile" ] ; then - eventType="creation" - METADIR="${AIP_STORAGE}/${MEDIAID}/metadata" - MEDIAINFO=$(_escape_for_db "$(cat "${METADIR}/fileMeta/objects/${MEDIA_ID}_mediainfo.xml")") - CAPTURELOG=$(_escape_for_db "$(cat "${METADIR}/logs/capture.log")") - MEDIA_ID="${MEDIAID}" - elif [ "$eventDetail" = "makebroadcast" ] ; then - eventType="creation" - elif [ "$eventDetail" = "makeyoutube" ] ; then - eventType="creation" - elif [ "$eventDetail" = "makemp3" ] ; then - eventType="creation" - elif [ "$eventDetail" = "makedvd" ] ; then - eventType="creation" - elif [ "$eventDetail" = "makeframes" ] ; then - eventType="creation" - elif [ "$eventDetail" = "makewaveform" ] ; then - eventType="creation" - elif [ "$eventDetail" = "makefingerprint" ] ; then - eventType="creation" - MEDIA_ID=$(basename "${SOURCEFILE}") - elif [ "$eventDetail" = "migratefiles" ] ; then - eventType="replication" - elif [ "$eventDetail" = "checksumpackage" ] ; then - eventType="fixity" - LOGDIR="${ABS_PATH}/${MEDIAID}/metadata/logs" - INGESTLOG="${LOGDIR}/capture.log" - if [ -f "${INGESTLOG}" ] ; then - OP=$(grep "operator" "${INGESTLOG}" | cut -c 11-) - fi - if [ -z "${OP}" ] ; then - OP="${USER}" - fi - elif [ "$eventDetail" = "collectionchecksum" ] ; then - eventType="fixity" - db_fixity=$(cat "${TAPECHECKSUMFILE}") - db_source="${TAPECHECKSUMFILE}" - elif [ "$eventDetail" = "writelto" ] ; then - eventType="ingest" - MEDIA_ID="${TAPE_SERIAL}" - if [ "${VERIFY_CHECK}" = "Y" ] ; then - eventDetail="writelto(VERIFY)" - fi - elif [ "$eventDetail" = "verifylto" ] ; then - eventType="fixity" - MEDIA_ID="${TAPE_SERIAL}" - db_fixity=$(cat "${READBACKDIR}/${TAPE_SERIAL}_ReadBack_checksum_${VERIFYTIME}.md5") - db_source="${READBACKDIR}/${TAPE_SERIAL}_ReadBack_checksum_${VERIFYTIME}.md5" - elif [ "$eventDetail" = "ingestcollectionchecksum" ] ; then - eventType="ingest" - fi - #Escape ID for SQL - if [ -z ${ESCAPE_CHECK1} ] ; then - MEDIA_ID=$(_escape_for_db ${MEDIA_ID}) - ESCAPE_CHECK1="Y" - fi - if [ -z ${ESCAPE_CHECK2} ] ; then - db_source=$(_escape_for_db ${db_source}) - ESCAPE_CHECK2="Y" - fi - if [ -z ${ESCAPE_CHECK3} ] ; then - user_input=$(_escape_for_db ${user_input}) - ESCAPE_CHECK3="Y" - fi -} - -_open_mysql_plist(){ - plist_path=$(find /usr/local/Cellar/mysql -iname homebrew.mxcl.mysql.plist | sort | tail -n1) - if [ -n "$plist_path" ] ; then - ip_address="0.0.0.0" - cat "$plist_path" | grep "$ip_address" > /dev/null - if [ "$?" != "0" ] ; then - sed -i '' "12s/.*/--bind-address=$ip_address<\/string>/" "$plist_path" - fi - fi -} - _version_schema(){ SCHEMA_FILE="$LTO_LOGS/${TAPE_SERIAL}.schema" if [ -f "${SCHEMA_FILE}" ] ; then diff --git a/searchfingerprint b/searchfingerprint deleted file mode 100755 index 3ebe167d..00000000 --- a/searchfingerprint +++ /dev/null @@ -1,178 +0,0 @@ -#!/usr/bin/env bash -SCRIPTNAME=$(basename "${0}") -SCRIPTDIR=$(dirname "${0}") -MM_CONFIG_FILE="${SCRIPTDIR}/mm.conf" - -. "${SCRIPTDIR}/mmfunctions" || { echo "Missing '${SCRIPTDIR}/mmfunctions'. Exiting." ; exit 1 ;}; -. "${MM_CONFIG_FILE}" || { echo "Missing '${MM_CONFIG_FILE}/mmfunctions'. Exiting." ; exit 1 ;}; - -_usage(){ -cat << EOF; -Takes an input file, calculates a video fingerprint for specified portion, and compares results to fingerprint database -Usage: "${SCRIPTNAME}" [ -h ] [ -i ] [ -o ] [ -t ] inputfile1 | inputfile2 --h Display this help --i Set input time in seconds for fingerprint comparison --o Set output time in seconds for fingerprint comparison --t Text only results (disables preview window) -EOF -exit -} - -OPTIND=1 -while getopts "hi:o:t" OPT ; do - case "${OPT}" in - i) INTIME="${OPTARG}" ;; - o) OUTTIME="${OPTARG}" ;; - h) _usage ;; - t) MODE="text" ;; - *) echo "bad option -${OPTARG}" ; _usage ;; - esac -done -shift $(( ${OPTIND} - 1 )) - -if ! [ "${1}" ] ; then - _usage -fi - -while [ "${*}" != "" ] ; do - #Set up input and temp files - INPUT="${1}" - shift - - #Confirm input is a video file - if [[ -z $(file -Ib "${INPUT}" | grep video) ]] ; then - echo "Input is not a video file" && continue - fi - - IO=$(mktemp) - TEMPFINGERPRINT=$(mktemp) - TEMPFINGERPRINT_SORTED=$(mktemp) - RESULTS=$(mktemp) - VISUALRESULTS=$(mktemp) - DRAWTEXT=$(mktemp) - #Set up concat input for fingerprint filter - echo "file '${INPUT}'" > "${IO}" - - if [ -n "${INTIME}" ] ; then - if ! [[ "${INTIME}" =~ ^-?[0-9]+$ ]] ; then - echo 'Please use an integer value for input time' && exit 0 - else - echo "inpoint ${INTIME}" >> "${IO}" - fi - fi - - if [ -n "${OUTTIME}" ]; then - if ! [[ "${OUTTIME}" =~ ^-?[0-9]+$ ]] ; then - echo 'Please use an integer value for output time' && exit 0 - else - if [ -n "${INTIME}" ] && ! [ "${OUTTIME}" -gt "${INTIME}" ] ; then - echo "Error! Output time must be greater than input time!" && exit 1 - else - echo "outpoint ${OUTTIME}" >> "${IO}" - fi - fi - fi - - #Create Fingerprint - ffmpeg -f concat -safe 0 -i "${IO}" -vf signature=format=xml:filename="${TEMPFINGERPRINT}" -map 0:v -f null - - xmlstarlet sel -N "m=urn:mpeg:mpeg7:schema:2001" -t -m "m:Mpeg7/m:DescriptionUnit/m:Descriptor/m:VideoSignatureRegion/m:VSVideoSegment" -v m:StartFrameOfSegment -o ':' -v m:EndFrameOfSegment -o ':' -m m:BagOfWords -v "translate(.,' ','')" -o ':' -b -n "${TEMPFINGERPRINT}" > "${TEMPFINGERPRINT_SORTED}" - - #Sort extract relevant values from fingerprint and sort for parsing - (IFS=$'\n' - for i in $(cat "${TEMPFINGERPRINT_SORTED}") ; do - hash1=$(echo "${i}" | cut -d':' -f3) - hash2=$(echo "${i}" | cut -d':' -f4) - hash3=$(echo "${i}" | cut -d':' -f5) - hash4=$(echo "${i}" | cut -d':' -f6) - echo "SELECT objectIdentifierValue,startframe,endframe,hash1,hash2 FROM fingerprints WHERE hash1='${hash1}' AND hash2='${hash2}' AND (hash3='${hash3}' OR hash4='${hash4}')" | mysql --login-path="${PREMIS_PROFILE}" "${PREMIS_NAME}" | tr '\t' ' ' | grep -v "objectIdentifierValue" >> "${RESULTS}" - done) - echo - - #Exit if no results found - if [[ -z $(cat "${RESULTS}") ]] ; then - echo "No potential matches were found." && exit 0 - fi - echo "Potential matches were found in the following frame ranges:" - echo - - #Loop for sorting possible matches divided by files and chunks of approximately 500 frames (Finds an input and then first output after 500) - while read -r RESULT ; do - RESULTNAME=$(echo "$RESULT" | cut -d' ' -f1) - INFRAME=$(echo "$RESULT" | cut -d' ' -f2) - OUTFRAME=$(echo "$RESULT" | cut -d' ' -f3) - HASHMATCH=$(echo "$RESULT" | cut -d' ' -f4- | tr ' ' ':') - - if [ -z ${INTIME} ] ; then - INTIME=0 - fi - - #Checks if the result is a new file. Marks last out time for previous file and new file name. - if ! [[ "${RESULTNAME}" = "${LASTRESULTNAME}" ]] ; then - if [ -n "${FINALOUT}" ] ; then - if ! [ "${FINALOUT}" = "${LASTOUTFRAME}" ] || ! [ "${LARGEMATCH}" = 1 ]; then - echo "Out ${FINALOUT}" | tee -a "${DRAWTEXT}" - echo "outpoint $((( $(grep "${FINALHASH}" "${TEMPFINGERPRINT_SORTED}" | cut -d':' -f2 | head -n 1) / 30 ) + ${INTIME}))" >> "${VISUALRESULTS}" - fi - FINALOUT='' - FINALHASH='' - LARGEMATCH='' - fi - LASTRESULTNAME=$(echo "$RESULT" | cut -d' ' -f1) - LASTOUTFRAME='' - LASTINFRAME='' - echo | tee -a "${DRAWTEXT}" - echo "${LASTRESULTNAME}" | tee -a "${DRAWTEXT}" - echo "------" | tee -a "${DRAWTEXT}" - fi - - #Checks if new frame match preceeds previous matches and resets variables - if [[ ${INFRAME} -lt ${LASTINFRAME} ]] ; then - if ! [ "${NEWIN}" = "1" ] ; then - echo "Out ${FINALOUT}" | tee -a "${DRAWTEXT}" - echo "outpoint $((($(grep "${FINALHASH}" "${TEMPFINGERPRINT_SORTED}" | cut -d':' -f2 | head -n 1) / 30 ) + ${INTIME}))" >> "${VISUALRESULTS}" - fi - LASTOUTFRAME='' - LASTINFRAME='' - SMALLFINALMATCH='1' - fi - - # Checks if match is first matching chunk for that file (or after reset) and sets variable - if [ -z "${LASTINFRAME}" ] ; then - LASTINFRAME="${INFRAME}" - NEWIN=1 - fi - # Sets variable if empty (first match after reset) - if [ -z "${LASTOUTFRAME}" ] ; then - LASTOUTFRAME="${OUTFRAME}" - fi - # Marks input if first match for file or after reset - if [ "${NEWIN}" = "1" ] ; then - echo "In ${INFRAME}" | tee -a "${DRAWTEXT}" - LASTINFRAME="${INFRAME}" - echo "file '${INPUT}'" >> "${VISUALRESULTS}" - echo "inpoint $((($(grep "${HASHMATCH}" "${TEMPFINGERPRINT_SORTED}" | cut -d':' -f1 | head -n 1) / 30 ) + ${INTIME}))" >> "${VISUALRESULTS}" - NEWIN='' - fi - # Marks output after 500 frame threshold is exceded - if [[ "${OUTFRAME}" -gt $(( ${LASTINFRAME} + 500 )) ]] ; then - LASTOUTFRAME="${OUTFRAME}" - NEWIN='1' - LARGEMATCH='1' - SMALLFINALMATCH='' - echo "Out ${LASTOUTFRAME}" | tee -a "${DRAWTEXT}" - echo "outpoint $((($(grep "${HASHMATCH}" "${TEMPFINGERPRINT_SORTED}" | cut -d':' -f2 | head -n 1) / 30 ) + ${INTIME}))" >> "${VISUALRESULTS}" - fi - FINALOUT="${OUTFRAME}" - FINALHASH="${HASHMATCH}" - - done < <(cat "${RESULTS}" | sort -u) - #Marks the last outpoint of last file - if ! [ "${FINALOUT}" = "${LASTOUTFRAME}" ] || ! [ "${LARGEMATCH}" = 1 ] || [ "${SMALLFINALMATCH}" = 1 ] ; then - echo "Out ${FINALOUT}" | tee -a "${DRAWTEXT}" - echo "outpoint $((($(grep "${FINALHASH}" "${TEMPFINGERPRINT_SORTED}" | cut -d':' -f2 | head -n 1) / 30 ) + ${INTIME}))" >> "${VISUALRESULTS}" - fi -done -#Play footage from input for which possible matches were found -if ! [ "${MODE}" = 'text' ] ; then - ffplay -hide_banner -loglevel panic -f concat -safe 0 -i "${VISUALRESULTS}" -vf drawtext=fontcolor=white:box=1:boxcolor=black@.4:fontsize=20:fontfile="/Library/Fonts/Andale Mono.ttf":textfile="${DRAWTEXT}" -fi diff --git a/updatingplist b/updatingplist deleted file mode 100755 index 1709e4b1..00000000 --- a/updatingplist +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env bash -SCRIPTDIR=$(dirname "${0}") -MM_CONFIG_FILE="${SCRIPTDIR}/mm.conf" -if [ -f "${MM_CONFIG_FILE}" ] ; then - . "${MM_CONFIG_FILE}" - . "${SCRIPTDIR}/mmfunctions" -else - exit -fi - -if [ "$PREMIS_PLIST" = "1" ] ; then - _open_mysql_plist -fi