Database Panel Implementation

[IAmVigneswaran]

@milanvarady Here is the rough mock-up images from my document.

I believe you can copy and adapt the code from the Configuration Panel?

1. Basically when the users click the +, Marker Data would prompt the user with a dialogue box.
2. The user would be presented with the option to select Notion or Airtable.
3. Example - If the user selects Notion, the user would be able to store/enter their Notion Database URL, Token Key, and Database Profile Name.
4. The those settings would be stored in /Users/UserID/Library/Application Support/Marker Data/Database.
5. And users can store as many Database Profile as possible. But At any given time, only 1 Database Profile can be Active.
6. When uploading, Marker Data would retrieve the stored values of the particular Active Database Profile and use the pre-compiled binary in the background to execute the upload task.
7. It would the the name for Airtable.

Reference:
#4
#2

[IAmVigneswaran]

Notion

For Notion - We will use the latest macOS Binary from CSV2Notion Neo.

https://github.com/TheAcharya/csv2notion-neo/releases

Sample .sh Script

#!/bin/sh

TOOL_PATH="REPLACE"
NOTION_TOKEN="REPLACE"
NOTION_URL="REPLACE"
UPLOAD_CSV="REPLACE"
UPLOAD_LOG="REPLACE"

$TOOL_PATH --token $NOTION_TOKEN --url $NOTION_URL --image-column "Image Filename" --image-column-keep --mandatory-column "Marker ID" --icon-column "Icon Image" --max-threads 5 --merge --log $UPLOAD_LOG "$UPLOAD_CSV"

We can place CSV2Notion Neo binary into this fixed path as

/Applications/Marker Data.app/Contents/Resources/csv2notion_neo

For the Log location -

We can fix the path as

/Users/UserID/Library/Application Support/Marker Data/Logs/notion-log.txt

For the .csv file and path, Marker Data can read from done file generated after successful extraction.

It is crucial we show / print the terminal output into our main window.
That way the user would know the upload process and status.

[milanvarady]

@IAmVigneswaran I am more or less done with the database UI and modeling, and I started to implement the csv to notion export. I need the path to the csv, and you mention this done file. Can you tell me more about it? Because I'm not sure what you mean by that.

[orchetect]

MarkersExtractor outputs a file called done.json if you supply the --create-done-file flag (createDoneFile: Bool property on MarkersExtractor.Settings). It has a single dictionary, and currently only contains the "csvPath" key with its value being the path (not URL) to the csv file.

You can read the file's contents as Data and parse it with JSONDecoder.

[milanvarady]

@orchetect And where is this done file saved? Because I need a path that is relatively easy to locate. Like is it saved in the exported folder or a custom path?

Also in the future if it's not a big deal, it would be great if the .extract() method would return this information. This way I could directly use it in swift code.

[orchetect]

Right now everything MarkersExtractor creates on disk will be in the same output folder, including the done file. Essentially the done file is intended to be the source of truth for any shell scripts or other tools to know where output resources are - which currently is just a CSV file for profiles that use it.

And on reflection, it's actually not that useful right now. There is a library/CLI option to supply the done filename - but not path. Which kind of defeats the purpose of the done file.

• I'm going to change --done-filename (MarkersExtractor .doneFilename) to --done-file which can accept a full path. That way shell tools can know where to find the done file.
• I will add another JSON key called "outputFolder" that is a path to the actual output folder containing all the output files

it would be great if the .extract() method would return this information.

Yeah that's a good point. I'll add that soon. No need for consumers of the library (and not the CLI) to have to deal with the done file.

[milanvarady]

Great! Thanks for the info.

[orchetect]

Yes, everything is in flux and evolving so please let us know if anything about MarkersExtractor is confusing, buggy or needs adding.

[orchetect]

Ok I've made all the changes to the result json file (formerly called 'done' file), and also extract() returns the same content as the result file but as a struct so it's easier to use.

Will be in MarkersExtractor 0.2.6. See TheAcharya/MarkersExtractor#68 (comment)

[IAmVigneswaran]

Right now, our Marker Data Application's workflow assumes (only singular operation) that :

1. User would Export their Timeline in FCP via our Custom Share Destination
2. Marker Data would perform all the extraction, once it is successfully done, our Marker Data app would automatically perform the upload base on the user's Active Configuration and Database profile.

What if the user decides that they would want to perform all their uploads at a later time? They would only want to export a series of timelines currently? For example - They have 10-20 timelines.

I wonder if the result json file could also include date parameter and it would not clear and overwrite the previous json entry?

Basically, the result json file would also act as a history or pending file? Or We have the current approach of result json file and we have another result history json file which is a cumulative of it. I am not sure which approach is best programmatically.

We can have a Upload History Side Tab.

In the History Tab, Marker Data would simply read the result json file or result json file. Users can sort by Date, Profile or Export Folder. We can have a checkbox next to it. When the user wants to upload at a later time, they can select the checkboxes and Press, "Upload Marker Data".

Any thoughts?

[orchetect]

The result json file is for a single MarkersExtractor extraction run. That wouldn't change unless a fundamental part of MarkersExtractor changes (ie: processing multiple projects or multiple FCPXML files at once).

What if the user decides that they would want to perform all their uploads at a later time?

I think you just break the process into two steps. One is the marker extraction. Two is the post-processing (uploading/converting/whatever). Let the user just turn off the post-processing step.

If you want to build some sort of history or batch processing of timelines, then that would be a separate concept/feature of Marker Data and probably not MarkersExtractor. I would suggest you manage any sort of history and its meta data. It could involve the result json files but I wouldn't rely on that per se, since that format could change in future and you may want some data model more stable for Marker Data. But you can output them for the user so that the file exists on disk for each batch anyhow.

[IAmVigneswaran]

If you want to build some sort of history or batch processing of timelines, then that would be a separate concept/feature. I would suggest you manage any sort of history and its meta data. It could involve the result json files but I wouldn't rely on that per se. But you can output them for the user so that the file exists on disk for each batch anyhow.

Maybe Marker Data, could copy the values of result json file and build its own history json file?

Something we can looking once this Database panel is implemented.

[orchetect]

Marker Data, could copy the values of result json file and build its own history

Yes, but it doesn't have to JSON. That's just a way to serialize data. You can use whatever you want depending on how you're storing the history data, and where you're storing it.

[IAmVigneswaran]

@milanvarady

For Notion Setup Panel

We can have a ? button (the yellow circle) that links to

https://markerdata.theacharya.co/databases/notion-prerequisite/

Likewise when switching to Airtable tab, you can set the ? button to

https://markerdata.theacharya.co/databases/airtable-prerequisite/

---

Can we also have "Clear button" (the blue box) at end of both Notion Token and URL field? You can use the same "Trash" icon.

---

For ALL the fields in database profiles (In both Notion and Airtable) except for Profile Name, we need to set the fields as hidden?

You can have a show / hide button.

Example -

This would be for privacy reasons during any screen capture or sharing.

---

We need to two buttons for templates. You can either have it as buttons or as Text URLs under "Open Database Folder in Finder".

"Notion Template"
"Airtable Template"

For both of these buttons you can set blank urls for now.

[milanvarady]

@IAmVigneswaran What are the templates for? We already have this platform selector when creating a new profile.

[IAmVigneswaran]

What are the templates for? We already have this platform selector when creating a new profile.

Nope. It is different. The user can duplicate a pre-made template into their notion account.

For example like the template that I have shared with you. I have already added all the database views, custom formulas and etc.

Without the templates, users will not have staring point.

The platform profile is not equal to templates.

Make sense?

[milanvarady]

Then we should name it something better. Maybe "Create template on notion.so", this makes it clear that things will happen outside the app.

[IAmVigneswaran]

Then we should name it something better. Maybe "Create template on notion.so", this makes it clear that things will happen outside the app.

How about this?

"Duplicate Marker Data Template on your Notion account"
"Duplicate Marker Data Template on your Airtable account"

[milanvarady]

We can go with that.

[milanvarady]

I added secure text fields and a clear button. Also, the ? help link.

As for the template links, the name was too long, so I added a little icon instead that indicates that this does something outside Marker Data.

[milanvarady]

Also, I used this package for the secure the input field, maybe you can add it to credits or somewhere.

https://github.com/MrAsterisco/PasswordField

[IAmVigneswaran]

I added secure text fields and a clear button. Also, the ? help link.

As for the template links, the name was too long, so I added a little icon instead that indicates that this does something outside Marker Data.

Perfect!

Also, I used this package for the secure the input field, maybe you can add it to credits or somewhere.

Noted! I will updated the credits.

[IAmVigneswaran]

@milanvarady

For the Database App menu, we can have the same layout as Configuration menu.

✓ Upload Later
------------------------------------
Open Database Panel
Open Database Folder in Finder
------------------------------------
Select Database Profile
  Profile 01
✓ Profile 02
  Profile 03
  Profile 04
...

[milanvarady]

@IAmVigneswaran Since we no longer have an active database profile nor the upload later option (replaced by recent extractions) I believe this menu becomes redundant.

[IAmVigneswaran]

Noted! I totally overlooked it. In this case we can remove this "Database" App menu.

[milanvarady]

The menu bar item has been removed.

[IAmVigneswaran]

@milanvarady Similar to #15 (comment)

Let's add -

"Would you like to delete this Database Profile? You cannot undo this action." Cancel Delete

[milanvarady]

I added a confirmation dialog.