Enable export to PDF and copying of attachments (#11)

Use wkhtmltopdf for PDF support because it has the easiest emoji solution (via twemoji).

heic2jpg.go is untested as it is largely copied from upstream, and I have a PR open to create an interface upstream: adrium/goheif#2

goheif also has issues vendoring because there is no Go code in some of the required C++ subdirectories: golang/go#26366. This should also be fixed upstream, but as a stopgap, we need to use the cp and chmod commands (vendor makefile target).

* Enable output to PDF

* Add support or jpegs in PDFs

* Switch to wkhtmltopdf for emoji support

* Implement copying attachments

* Convert HEIC attachments to JPEG for PDF output

* Update example to match new defaults

* Add "attached" text to attachment references in outfiles

* Update dependencies

* Install wkhtmltopdf in Travis CI

* Use xvfb in Travis CI

Author: tagatac

.travis.yml

@@ -1,9 +1,11 @@
 language: go
 go:
   - 1.x
+before_install:
+  - sudo apt-get -y install wkhtmltopdf
 install:
   - go mod download
 script:
-  - make test
+  - xvfb-run make test
 after_success:
   - make codecov

Makefile

@@ -1,13 +1,20 @@
 COVERAGE_FILE=coverage.out
 ZIPFILE="bagoup-$(shell uname -s)-$(shell uname -m).zip"
+GOHEIF_VENDOR_DIR=vendor/github.com/adrium/goheif
+LIBDE265_VENDOR_DIR=$(GOHEIF_VENDOR_DIR)/libde265
 
 build: bagoup
 
-bagoup: main.go opsys/opsys.go chatdb/chatdb.go vendor
+bagoup: main.go opsys/opsys.go opsys/outfile.go opsys/templates/* chatdb/chatdb.go pathtools/pathtools.go vendor
 	go build -o $@ $<
 
 vendor: go.mod go.sum
 	go mod vendor -v
+	rm -vrf $(LIBDE265_VENDOR_DIR)
+	@echo "Copy files pruned by `go mod vendor` (see https://github.com/golang/go/issues/26366). Sudo permissions will be required"
+	cp -vR $(shell go env GOPATH)/pkg/mod/github.com/adrium/[email protected]/libde265 $(GOHEIF_VENDOR_DIR)
+	sudo chmod -vR u+rw $(LIBDE265_VENDOR_DIR)
+	chmod -v u+x $(LIBDE265_VENDOR_DIR)
 
 .PHONY: deps generate test zip clean codecov
 

README.md

@@ -3,11 +3,11 @@
 bagoup *(pronounced BAAGoop)* is an export utility for Mac OS Messages,
 implemented in Go, inspired by
 [Baskup](http://peterkaminski09.github.io/baskup/). It exports all of the
-conversations saved in Messages to readable, searchable text files.
+conversations saved in Messages to readable, searchable text or PDF files.
 
 ## Example Export
 ```
-$ cat "backup/Novak Djokovic/iMessage;-;+3815555555555.txt"
+$ cat "messages-export/Novak Djokovic/iMessage;-;+3815555555555.txt"
 [2020-03-01 15:34:05] Me: Want to play tennis?
 [2020-03-01 15:34:41] Novak: I can't today. I'm still at the Dubai Open
 [2020-03-01 15:34:53] Me: Ah, okay. When are you back in SF?
@@ -22,11 +22,28 @@ brew tap tagatac/bagoup
 brew install bagoup
 ```
 
-## chat.db Access
+## Protected File Access
 The Messages database is a protected file in Mac OS. See
 [this article](https://appletoolbox.com/seeing-error-operation-not-permitted-in-macos-mojave/)
-for more details. To to backup your messages, you have two options:
-### Option 1 (recommended): Copy chat.db
+for more details. Additionally, attachments can be located in various protected
+places on your filesystem.
+
+To to backup your messages, you have two options. If you wish to export to PDFs
+with images (`--pdf`), or to copy attachments (`--copy-attachments`), you must use
+the first option.
+### Option 1 (required for attachments): Give your terminal emulator full disk access
+From [osxdaily.com](https://osxdaily.com/2018/10/09/fix-operation-not-permitted-terminal-error-macos/):
+1. Pull down the Apple menu and choose "System Preferences"
+1. Choose "Security & Privacy" control panel
+1. Now select the "Privacy" tab, then from the left-side menu select "Full Disk Access"
+1. Click the lock icon in the lower left corner of the preference panel and authenticate with an admin level login
+1. Now click the [+] plus button to add an application with full disk access
+1. Navigate to the /Applications/Utilities/ folder and choose "Terminal" (or your terminal emulator of choice) to grant your terminal with Full Disk Access privileges
+1. Relaunch your terminal emulator, and the “Operation not permitted” error messages will be gone
+
+If you choose this option, bagoup will be able to open **chat.db** in its
+default location, and the `--db-path` flag is not needed.
+### Option 2 (more secure if attachments are not desired): Copy chat.db
 Copy the Messages database to an unprotected folder in Finder, and provide the
 path to the copy via the `--db-path` flag.
 
@@ -37,19 +54,6 @@ path to the copy via the `--db-path` flag.
 1. Right-click in the unprotected folder, and click **Paste Item** in the
 context menu.
 
-### Option 2 (less secure): Give your terminal full disk access
-From [osxdaily.com](https://osxdaily.com/2018/10/09/fix-operation-not-permitted-terminal-error-macos/):
-1. Pull down the Apple menu and choose ‘System Preferences’
-1. Choose “Security & Privacy” control panel
-1. Now select the “Privacy” tab, then from the left-side menu select “Full Disk Access”
-1. Click the lock icon in the lower left corner of the preference panel and authenticate with an admin level login
-1. Now click the [+] plus button to add an application with full disk access
-1. Navigate to the /Applications/Utilities/ folder and choose “Terminal” to grant Terminal with Full Disk Access privileges
-1. Relaunch Terminal, the “Operation not permitted” error messages will be gone
-
-If you choose this option, bagoup will be able to open **chat.db** in its
-default location, and the `--db-path` flag is not needed.
-
 ## Contact Information (optional)
 If you provide your contacts via the `--contacts-path` flag, bagoup will attempt
 to match the handles from the Messages database with full names from your
@@ -65,15 +69,18 @@ Usage:
   bagoup [OPTIONS]
 
 Application Options:
-  -i, --db-path=        Path to the Messages chat database file (default: ~/Library/Messages/chat.db)
-  -o, --export-path=    Path to which the Messages will be exported (default: backup)
-  -m, --mac-os-version= Version of Mac OS, e.g. '10.15', from which the Messages chat database file was copied (not needed if bagoup is running on the same Mac)
-  -c, --contacts-path=  Path to the contacts vCard file
-  -s, --self-handle=    Prefix to use for for messages sent by you (default: Me)
-      --separate-chats  Do not merge chats with the same contact into a single file, e.g. iMessage and SMS
+  -i, --db-path=          Path to the Messages chat database file (default: ~/Library/Messages/chat.db)
+  -o, --export-path=      Path to which the Messages will be exported (default: messages-export)
+  -m, --mac-os-version=   Version of Mac OS, e.g. '10.15', from which the Messages chat database file was copied (not needed if bagoup is running on the same Mac)
+  -c, --contacts-path=    Path to the contacts vCard file
+  -s, --self-handle=      Prefix to use for for messages sent by you (default: Me)
+      --separate-chats    Do not merge chats with the same contact (e.g. iMessage and SMS) into a single file
+  -p, --pdf               Export text and images to PDF files (requires full disk access)
+      --include-ppa       Include plugin payload attachments (e.g. link previews) in generated PDFs
+  -a, --copy-attachments  Copy attachments to the same folder as the chat which included them (requires full disk access)
 
 Help Options:
-  -h, --help            Show this help message
+  -h, --help              Show this help message
 ```
 All conversations will be exported as text files to the specified export path.
 See https://github.com/tagatac/bagoup/tree/master/example-export for an example

chatdb/chatdb.go

@@ -12,12 +12,15 @@ package chatdb
 import (
 	"database/sql"
 	"fmt"
+	"path/filepath"
 	"sort"
 	"strconv"
+	"strings"
 
 	"github.com/Masterminds/semver"
 	"github.com/emersion/go-vcard"
 	"github.com/pkg/errors"
+	"github.com/tagatac/bagoup/pathtools"
 )
 
 const _githubIssueMsg = "open an issue at https://github.com/tagatac/bagoup/issues"
@@ -63,6 +66,9 @@ type (
 		// GetMessage returns a message retrieved from the database formatted for
 		// writing to a chat file.
 		GetMessage(messageID int, handleMap map[int]string, macOSVersion *semver.Version) (string, error)
+		// GetImagePaths returns a list of attachment filepaths associated with
+		// each message ID.
+		GetAttachmentPaths() (map[int][]string, error)
 	}
 
 	// DatedMessageID pairs a message ID and its date, in the legacy date format.
@@ -233,3 +239,49 @@ func (d *chatDB) getDatetimeFormula(macOSVersion *semver.Version) string {
 	}
 	return _datetimeFormula
 }
+
+func (d *chatDB) GetAttachmentPaths() (map[int][]string, error) {
+	attachmentJoins, err := d.DB.Query("SELECT message_id, attachment_id FROM message_attachment_join")
+	if err != nil {
+		return nil, errors.Wrapf(err, "scan message_attachment_join table")
+	}
+	defer attachmentJoins.Close()
+
+	attPaths := map[int][]string{}
+	for attachmentJoins.Next() {
+		var msgID, attID int
+		if err := attachmentJoins.Scan(&msgID, &attID); err != nil {
+			return attPaths, errors.Wrap(err, "read data from message_attachment_join table")
+		}
+		filename, _, err := d.getAttachmentPath(attID)
+		if err != nil {
+			return attPaths, errors.Wrapf(err, "get path for attachment %d to message %d", attID, msgID)
+		}
+		attPaths[msgID] = append(attPaths[msgID], filename)
+	}
+	return attPaths, nil
+}
+
+func (d *chatDB) getAttachmentPath(attachmentID int) (string, string, error) {
+	attachments, err := d.DB.Query(fmt.Sprintf("SELECT filename, COALESCE(mime_type, '') FROM attachment WHERE ROWID=%d", attachmentID))
+	if err != nil {
+		return "", "", errors.Wrapf(err, "query attachment table for ID %d", attachmentID)
+	}
+	defer attachments.Close()
+	attachments.Next()
+	var filename, mimeType string
+	if err := attachments.Scan(&filename, &mimeType); err != nil {
+		return "", "", errors.Wrapf(err, "read data for attachment ID %d", attachmentID)
+	}
+	if attachments.Next() {
+		return "", "", fmt.Errorf("multiple attachments with the same ID: %d - attachment ID uniqueness assumption violated - %s", attachmentID, _githubIssueMsg)
+	}
+	filename, err = pathtools.ReplaceTilde(filename)
+	if err != nil {
+		return "", "", errors.Wrap(err, "replace tilde")
+	}
+	if strings.HasPrefix(filename, "/var") {
+		filename = filepath.Join(filepath.Dir(filename), "0", filepath.Base(filename))
+	}
+	return filename, mimeType, nil
+}

chatdb/chatdb_test.go

@@ -8,6 +8,7 @@ import (
 	"testing"
 
 	"github.com/Masterminds/semver"
+	"github.com/tagatac/bagoup/pathtools"
 
 	"github.com/DATA-DOG/go-sqlmock"
 	"github.com/emersion/go-vcard"
@@ -95,7 +96,7 @@ func TestGetHandleMap(t *testing.T) {
 			cdb := NewChatDB(db, "Me")
 			handleMap, err := cdb.GetHandleMap(tt.contactMap)
 			if tt.wantErr != "" {
-				assert.ErrorContains(t, err, tt.wantErr)
+				assert.Error(t, err, tt.wantErr)
 				return
 			}
 			assert.NilError(t, err)
@@ -217,7 +218,7 @@ func TestGetChats(t *testing.T) {
 
 			chats, err := cdb.GetChats(tt.contactMap)
 			if tt.wantErr != "" {
-				assert.ErrorContains(t, err, tt.wantErr)
+				assert.Error(t, err, tt.wantErr)
 				return
 			}
 			assert.NilError(t, err)
@@ -276,7 +277,7 @@ func TestGetMessageIDs(t *testing.T) {
 
 			ids, err := cdb.GetMessageIDs(42)
 			if tt.wantErr != "" {
-				assert.ErrorContains(t, err, tt.wantErr)
+				assert.Error(t, err, tt.wantErr)
 				return
 			}
 			assert.NilError(t, err)
@@ -328,7 +329,7 @@ func TestGetMessage(t *testing.T) {
 					AddRow(0, nil, "message text", "2019-10-04 18:26:31")
 				query.WillReturnRows(rows)
 			},
-			wantErr: "read data for message ID 42: sql: Scan error on column index 1, name \"handle_id\": converting NULL to int is unsupported",
+			wantErr: `read data for message ID 42: sql: Scan error on column index 1, name "handle_id": converting NULL to int is unsupported`,
 		},
 		{
 			msg: "duplicate message ID",
@@ -353,7 +354,7 @@ func TestGetMessage(t *testing.T) {
 
 			message, err := cdb.GetMessage(42, handleMap, nil)
 			if tt.wantErr != "" {
-				assert.ErrorContains(t, err, tt.wantErr)
+				assert.Error(t, err, tt.wantErr)
 				return
 			}
 			assert.NilError(t, err)
@@ -402,3 +403,120 @@ func TestGetDatetimeFormula(t *testing.T) {
 		})
 	}
 }
+
+func TestGetAttachmentPaths(t *testing.T) {
+	tests := []struct {
+		msg          string
+		setupMock    func(sqlmock.Sqlmock)
+		wantAttPaths map[int][]string
+		wantErr      string
+	}{
+		{
+			msg: "two attachments to one message, one to another",
+			setupMock: func(sMock sqlmock.Sqlmock) {
+				rows := sqlmock.NewRows([]string{"message_id", "attachment_id"}).
+					AddRow(1, 1).
+					AddRow(1, 2).
+					AddRow(2, 3)
+				sMock.ExpectQuery("SELECT message_id, attachment_id FROM message_attachment_join").WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("attachment1.jpeg", "image/jpeg")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=1`).WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("~/attachment2.heic", "image/heic")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=2`).WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("/var/folder/attachment3.mp4", "video/mp4")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=3`).WillReturnRows(rows)
+			},
+			wantAttPaths: map[int][]string{
+				1: {"attachment1.jpeg", pathtools.MustReplaceTilde("~/attachment2.heic")},
+				2: {"/var/folder/0/attachment3.mp4"},
+			},
+		},
+		{
+			msg: "join table query error",
+			setupMock: func(sMock sqlmock.Sqlmock) {
+				sMock.ExpectQuery("SELECT message_id, attachment_id FROM message_attachment_join").WillReturnError(errors.New("this is a DB error"))
+			},
+			wantErr: "scan message_attachment_join table: this is a DB error",
+		},
+		{
+			msg: "join table row scan error",
+			setupMock: func(sMock sqlmock.Sqlmock) {
+				rows := sqlmock.NewRows([]string{"message_id", "attachment_id"}).
+					AddRow(1, nil)
+				sMock.ExpectQuery("SELECT message_id, attachment_id FROM message_attachment_join").WillReturnRows(rows)
+			},
+			wantErr: `read data from message_attachment_join table: sql: Scan error on column index 1, name "attachment_id": converting NULL to int is unsupported`,
+		},
+		{
+			msg: "attachments table query error",
+			setupMock: func(sMock sqlmock.Sqlmock) {
+				rows := sqlmock.NewRows([]string{"message_id", "attachment_id"}).
+					AddRow(1, 1).
+					AddRow(1, 2).
+					AddRow(2, 3)
+				sMock.ExpectQuery("SELECT message_id, attachment_id FROM message_attachment_join").WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("attachment1.jpeg", "image/jpeg")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=1`).WillReturnRows(rows)
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=2`).WillReturnError(errors.New("this is a DB error"))
+			},
+			wantErr: "get path for attachment 2 to message 1: query attachment table for ID 2: this is a DB error",
+		},
+		{
+			msg: "attachments table row scan error",
+			setupMock: func(sMock sqlmock.Sqlmock) {
+				rows := sqlmock.NewRows([]string{"message_id", "attachment_id"}).
+					AddRow(1, 1).
+					AddRow(1, 2).
+					AddRow(2, 3)
+				sMock.ExpectQuery("SELECT message_id, attachment_id FROM message_attachment_join").WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("attachment1.jpeg", "image/jpeg")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=1`).WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("~/attachment2.heic", nil)
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=2`).WillReturnRows(rows)
+			},
+			wantErr: `get path for attachment 2 to message 1: read data for attachment ID 2: sql: Scan error on column index 1, name "mime_type": converting NULL to string is unsupported`,
+		},
+		{
+			msg: "duplicate attachment ID",
+			setupMock: func(sMock sqlmock.Sqlmock) {
+				rows := sqlmock.NewRows([]string{"message_id", "attachment_id"}).
+					AddRow(1, 1).
+					AddRow(1, 2).
+					AddRow(2, 3)
+				sMock.ExpectQuery("SELECT message_id, attachment_id FROM message_attachment_join").WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("attachment1.jpeg", "image/jpeg")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=1`).WillReturnRows(rows)
+				rows = sqlmock.NewRows([]string{"filename", "mime_type"}).
+					AddRow("~/attachment2.heic", "image/heic").
+					AddRow("attachment2.mov", "video/mov")
+				sMock.ExpectQuery(`SELECT filename, COALESCE\(mime_type, ''\) FROM attachment WHERE ROWID\=2`).WillReturnRows(rows)
+			},
+			wantErr: "get path for attachment 2 to message 1: multiple attachments with the same ID: 2 - attachment ID uniqueness assumption violated - open an issue at https://github.com/tagatac/bagoup/issues",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.msg, func(t *testing.T) {
+			db, sMock, err := sqlmock.New()
+			assert.NilError(t, err)
+			defer db.Close()
+			tt.setupMock(sMock)
+			cdb := &chatDB{DB: db}
+
+			attPaths, err := cdb.GetAttachmentPaths()
+			if tt.wantErr != "" {
+				assert.Error(t, err, tt.wantErr)
+				return
+			}
+			assert.NilError(t, err)
+			assert.DeepEqual(t, tt.wantAttPaths, attPaths)
+		})
+	}
+}