Merge pull request #29 from mfarragher/dev

v0.10

Author: mfarragher

.github/workflows/ci.yml

@@ -1,5 +1,11 @@
 name: codecov
-on: [push, pull_request]
+on:
+  pull_request:
+    branches-ignore:
+      - main
+  push:
+    branches:
+      - main
 jobs:
   run:
     runs-on: ${{ matrix.os }}
@@ -16,6 +22,7 @@ jobs:
         uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
+          node-version: 16
       - name: Install dependencies
         run: |
           pip install -r requirements.txt --use-pep517

README.md

@@ -4,50 +4,52 @@
 # obsidiantools 🪨⚒️
 **obsidiantools** is a Python package for getting structured metadata about your [Obsidian.md notes](https://obsidian.md/) and analysing your vault.  Complement your Obsidian workflows by getting metrics and detail about all your notes in one place through the widely-used Python data stack.
 
-It's incredibly easy to explore structured data on your vault through this fluent interface.  This is all the code you need to generate a `vault` object that stores the key data:
+It's incredibly easy to explore structured data on your vault through this fluent interface.  This is all the code you need to generate a `vault` object that stores all the data:
 
 ```python
 import obsidiantools.api as otools
 
 vault = otools.Vault(<VAULT_DIRECTORY>).connect().gather()
 ```
 
-These are the basics of the function calls:
-- `connect()`: connect your notes together in a graph structure and get metadata on links (e.g. wikilinks, backlinks, etc.)
+These are the basics of the method calls:
+- `connect()`: connect your notes together in a graph structure and get metadata on links (e.g. wikilinks, backlinks, etc.)  There ais the option to support the inclusion of 'attachment' files in the graph.
 - `gather()`: gather the plaintext content from your notes in one place.  This includes the 'source text' that represent how your notes are written.  There are arguments to support what text you want to remove, e.g. remove code.
 
 See some of the **key features** below - all accessible from the `vault` object either through a method or an attribute.
 
-As this package relies upon note (file)names, it is only recommended for use on vaults where wikilinks are not formatted as paths and where note names are unique.  This should cover the vast majority of vaults that people create.
+The package is built to support the 'shortest path when possible' option for links.  This should cover the vast majority of vaults that people create.  See the [wiki](https://github.com/mfarragher/obsidiantools/wiki) for more info on what sort of wikilink syntax is not well-supported and how the graph may be slightly different to what you see in the Obsidian app.
 
 ## 💡 Key features
 This is how **`obsidiantools`** can complement your workflows for note-taking:
 - **Access a `networkx` graph of your vault** (`vault.graph`)
     - NetworkX is the main Python library for network analysis, enabling sophisticated analyses of your vault.
     - NetworkX also supports the ability to export your graph to other data formats.
     - When instantiating a `vault`, the analysis can also be filtered on specific subdirectories.
-- **Get summary stats about your notes, e.g. number of backlinks and wikilinks, in a Pandas dataframe**
-    - Get the dataframe via `vault.get_note_metadata()`
+- **Get summary stats about your notes & files, e.g. number of backlinks and wikilinks, in a Pandas dataframe**
+    - Get the dataframe via `vault.get_note_metadata()` (notes / md files), `vault.get_media_file_metadata()` (media files that can be embedded in notes) and `vault.get_canvas_file_metadata()` (canvas files).
 - **Retrieve detail about your notes' links and metadata as built-in Python types**
+    - The main indices of files are `md_file_index`, `media_file_index` and `canvas_file_index` (canvas files).
+    - Check whether files included as links in the vault actually exist, via `vault` attributes like `nonexistent_notes`, `nonexistent_media_files` and `nonexistent_canvas_files`.
+    - Check whether actual files are isolated in the graph ('orphans'), via `vault` attributes like `isolated_notes`, `isolated_media_files` and `isolated_canvas_files`.
+    - You can access all the note & file links in one place, or you can load them for an individual note:
+        - e.g. `vault.backlinks_index` for all backlinks in the vault
+        - e.g. `vault.get_backlinks(<NOTE>)` for the backlinks of an individual note
     - **md note info:**
         - The various types of links:
             - Wikilinks (incl. header links, links with alt text)
             - Embedded files
             - Backlinks
             - Markdown links
-        - You can access all the links in one place, or you can load them for an individual note:
-            - e.g. `vault.backlinks_index` for all backlinks in the vault
-            - e.g. `vault.get_backlinks(<NOTE>)` for the backlinks of an individual note
         - Front matter via `vault.get_front_matter(<NOTE>)` or `vault.front_matter_index`
         - Tags via `vault.get_tags(<NOTE>)` or `vault.tags_index`.  Nested tags are supported.
         - LaTeX math via `vault.get_math(<NOTE>)` or `vault.math_index`
-        - Check which notes are isolated (`vault.isolated_notes`)
-        - Check which notes do not exist as files yet (`vault.nonexistent_notes`)
         - As long as `gather()` is called:
             - Get source text of note (via `vault.get_source_text(<NOTE>)`).  This tries to represent how a note's text appears in Obsidian's 'source mode'.
             - Get readable text of note (via `vault.get_readable_text(<NOTE>)`).  This tries to reduce note text to minimal markdown formatting, e.g. preserving paragraphs, headers and punctuation.  Only slight processing is needed for various forms of NLP analysis.
     - **canvas file info:**
         - The JSON content of each canvas file is stored as a Python dict in `vault.canvas_content_index`
+        - Data to recreate the layout of content in a canvas file via the `vault.canvas_graph_detail_index` dict
 
 Check out the functionality in the demo repo.  Launch the '15 minutes' demo in a virtual machine via Binder:
 
@@ -58,7 +60,7 @@ There are other API features that try to mirror the Obsidian.md app, for your co
 The text from vault notes goes through this process: markdown → split out front matter from text → HTML → ASCII plaintext.
 
 ## ⏲️ Installation
-``pip install obsidiantools``
+`pip install obsidiantools`
 
 Requires Python 3.9 or higher.
 

obsidiantools/__init__.py

@@ -2,3 +2,4 @@
 from . import md_utils
 from . import html_processing
 from . import canvas_utils
+from . import media_utils

obsidiantools/_constants.py

@@ -13,3 +13,24 @@
 # helpers:
 WIKILINK_AS_STRING_REGEX = r'\[[^\]]+\]\([^)]+\)'
 EMBEDDED_FILE_LINK_AS_STRING_REGEX = r'!?\[{2}([^\]\]]+)\]{2}'
+
+# Sets of extensions via https://help.obsidian.md/How+to/Embed+files :
+# NB: file.ext and file.EXT can exist in same folder
+IMG_EXT_SET = {'.png', '.jpg', '.jpeg', '.gif', '.bmp', '.svg',
+               '.PNG', '.JPG', '.JPEG', '.GIF', '.BMP', '.SVG'}
+AUDIO_EXT_SET = {'.mp3', '.webm', '.wav', '.m4a', '.ogg', '.3gp', '.flac',
+                 '.MP3', '.WEBM', '.WAV', '.M4A', '.OGG', '.3GP', '.FLAC'}
+VIDEO_EXT_SET = {'.mp4', '.webm', '.ogv', '.mov', '.mkv',
+                 '.MP4', '.WEBM', '.OGV', '.MOV', '.MKV'}
+PDF_EXT_SET = {'.pdf',
+               '.PDF'}
+# canvas files:
+CANVAS_EXT_SET = {'.canvas',
+                  '.CANVAS'}
+
+# metadata df cols order:
+METADATA_DF_COLS_GENERIC_TYPE = [
+    'rel_filepath', 'abs_filepath',
+    'file_exists',
+    'n_backlinks', 'n_wikilinks', 'n_tags', 'n_embedded_files',
+    'modified_time']

obsidiantools/_io.py

@@ -1,5 +1,6 @@
 from pathlib import Path
 from glob import glob
+import numpy as np
 
 
 def get_relpaths_from_dir(dir_path: Path, *, extension: str) -> list[Path]:
@@ -81,3 +82,31 @@ def get_relpaths_matching_subdirs(dir_path: Path, *,
                                                      extension=extension)
                     if str(i.parent.as_posix())
                     in include_subdirs_final]
+
+
+def _get_valid_filepaths_by_ext_set(dirpath: Path, *,
+                                    exts: set[str]):
+    all_files = [p.relative_to(dirpath)
+                 for p in Path(dirpath).glob("**/*")
+                 if p.suffix in exts]
+    return all_files
+
+
+def _get_shortest_path_by_filename(relpaths_list: list[Path]) -> dict[str, Path]:
+    # get filename w/ ext only:
+    all_file_names_list = [f.name for f in relpaths_list]
+
+    # get indices of dupe 'filename w/ ext':
+    _, inverse_ix, counts = np.unique(
+        np.array(all_file_names_list),
+        return_inverse=True,
+        return_counts=True,
+        axis=0)
+    dupe_names_ix = np.where(counts[inverse_ix] > 1)[0]
+
+    # get shortest paths via mask:
+    shortest_paths_arr = np.array(all_file_names_list, dtype=object)
+    shortest_paths_arr[dupe_names_ix] = np.array(
+        [str(fpath)
+         for fpath in relpaths_list])[dupe_names_ix]
+    return {fn: path for fn, path in zip(shortest_paths_arr, relpaths_list)}