added init functions to docs and minor formatting updates

Author: Vedant1

docs/backends.rst

@@ -25,21 +25,25 @@ SQLite
 
 .. automodule:: dsi.backends.sqlite
    :members:
+   :special-members: __init__
 
 SQLAlchemy
 ------
 
 .. automodule:: dsi.backends.sqlalchemy
    :members:
+   :special-members: __init__
 
 GUFI
 ------
 
 .. automodule:: dsi.backends.gufi
    :members:
+   :special-members: __init__
 
 Parquet
 ------
 
 .. automodule:: dsi.backends.parquet
    :members:
+   :special-members: __init__

docs/contributing_readers.rst

@@ -50,7 +50,7 @@ Example ``add_rows``: ::
 
     self.add_to_output(my_data)
 
-*Alternate* Add Rows: ``add_rows(self) -> None``
+*Newer* Add Rows: ``add_rows(self) -> None``
 -------------------------------------
 If you are confident that the the data you read in ``add_rows`` is in the form of an OrderedDict (the data structure used to store all ingested data), you can bypass the use of ``pack_header`` and ``add_to_output`` with an alternate ``set_schema`` function.
 

docs/core.rst

@@ -1,9 +1,9 @@
 Core
 ====
 
-The DSI Core middleware defines the Terminal concept. An instantiated Terminal is the human/machine DSI interface. 
-The person setting up a Core Terminal only needs to know how they want to ask questions, and what metadata they want to ask questions about. 
-If they don’t see an option to ask questions the way they like, or they don’t see the metadata they want to ask questions about, then they should ask a Backend Contributor or a Plugin Contributor, respectively.
+The DSI Core middleware defines the Terminal and Sync concept. 
+An instantiated Terminal is the human/machine DSI interface to connect Reader/Writer plugins and DSI backends.
+An instantiated Sync supports data movement capabilities between local and remote locations and captures metadata documentation
 
 Core: Terminal
 --------------
@@ -24,43 +24,68 @@ Notes for users:
       - Terminal handles errors from any loaded DSI/user-written modules (plugins/backends). If writing an external plugin/backend, return a caught error as a tuple (error, error_message_string). Do not print in a new class
 .. autoclass:: dsi.core.Terminal
       :members:
+      :special-members: __init__
 
 Core: Sync
 ----------
 
-The DSI Core middleware also defines data management functionality in ``Sync``. The purpose of ``Sync`` is to provide file metadata documentation and data movement capabilities when moving data to/from local and remote locations. The purpose of data documentation is to capture and archive metadata (i.e. location of local file structure, their access permissions, file sizes, and creation/access/modification dates) and track their movement to the remote location for future access. The primary functions, ``Copy``, ``Move``, and ``Get`` serve as mechanisms to copy data, move data, or retrieve data from remote locations by creating a DSI database in the process, or retrieving an existing DSI database that contains the location(s) of the target data.
+The DSI Core middleware also defines data management functionality in ``Sync``. 
+The purpose of ``Sync`` is to provide file metadata documentation and data movement capabilities when moving data to/from local and remote locations. 
+The purpose of data documentation is to capture and archive metadata (i.e. location of local file structure, their access permissions, file sizes, and creation/access/modification dates) and track their movement to the remote location for future access. 
+The primary functions, ``Copy``, ``Move``, and ``Get`` serve as mechanisms to copy data, move data, or retrieve data from remote locations by creating a DSI database in the process, or retrieving an existing DSI database that contains the location(s) of the target data.
 
 .. autoclass:: dsi.core.Sync
       :members:
+      :special-members: __init__
 
 Examples
 --------
-Before interacting with the plugins and backends, they must each be loaded. Examples below display various ways that users can incorporate DSI into their data science workflows.
+Before interacting with the plugins and backends, they must each be loaded. 
+Examples below display various ways users can incorporate DSI into their data science workflows.
+They can be found and run in ``examples/core/``
 
-Example 1: Baseline use of DSI to list Modules
+Example 1: Intro use case
+~~~~~~~~~~
+Baseline use of DSI to list Modules
 
 .. literalinclude:: ../examples/core/baseline.py
 
-Example 2: Ingesting data from a Reader to a backend
+Example 2: Ingest data
+~~~~~~~~~~
+Ingesting data from a Reader to a backend
 
 .. literalinclude:: ../examples/core/ingest.py
 
-Example 3: Querying data from a backend
+Example 3: Query data
+~~~~~~~~~~
+Querying data from a backend
 
 .. literalinclude:: ../examples/core/query.py
 
-Example 4: Processing data from a backend to generate an Entity Relationship diagram using a Writer
+Example 4: Process data
+~~~~~~~~~~
+Processing data from a backend to generate an Entity Relationship diagram using a Writer
 
 .. literalinclude:: ../examples/core/process.py
 
-Example 5: Generating a python notebook file (mostly Jupyter notebook) from a backend to view data interactively
+Example 5: Generate notebook
+~~~~~~~~~~
+Generating a python notebook file (mostly Jupyter notebook) from a backend to view data interactively
 
 .. literalinclude:: ../examples/core/notebook.py
 
-Example 6: Finding data from a backend - tables, columns, cells, or all matches
+Example 6: Find data
+~~~~~~~~~~
+Finding data from a backend - tables, columns, cells, or all matches
 
 .. literalinclude:: ../examples/core/find.py
 
-Example 7: Loading an external plugin from another python file
+Example 7: External plugin
+~~~~~~~~~~
+Loading an external python plugin reader from a separate file:
 
-.. literalinclude:: ../examples/core/external_plugin.py
+.. literalinclude:: ../examples/core/external_plugin.py
+
+``text_file_reader``:
+
+.. literalinclude:: ../examples/core/text_file_reader.py

docs/examples.rst

@@ -19,16 +19,12 @@ In the first step, a python script is used to parse the slurm output files and c
 
    ./parse_slurm_output.py --testname leblanc
 
-
-.. .. literalinclude:: ../examples/pennant/parse_slurm_output.py
-
 In the second step, another python script,
 
 .. code-block:: unixconfig
 
    ./create_and_query_dsi_db.py --testname leblanc
 
-
 reads in the CSV file and creates a database:
 
 .. literalinclude:: ../examples/pennant/create_and_query_dsi_db.py

docs/introduction.rst

@@ -48,7 +48,7 @@ Currently, DSI has the following readers:
 ..  figure:: PluginClassHierarchy.png
     :alt: Figure depicting the current plugin class hierarchy.
     :class: with-shadow
-    :scale: 100%
+    :scale: 70%
 
     Figure depicting the current DSI plugin class hierarchy.
 

docs/plugins.rst

@@ -9,17 +9,22 @@ Note that any contributed plugins or extension should include unit tests in  ``p
 ..  figure:: PluginClassHierarchy.png
     :alt: Figure depicting the current plugin class hierarchy.
     :class: with-shadow
-    :scale: 100%
+    :scale: 70%
 
     Figure depicts prominent portion of the current DSI plugin class hierarchy.
 
 .. automodule:: dsi.plugins.plugin
    :members:
+   :special-members: __init__
+
+Metadata Processing
+-------------------
 
 **Note for users:** StructuredMetadata class is used to assign data from a file_reader to the DSI abstraction in core. If data in a user-written reader is restructured as an OrderedDict, only need to call set_schema_2() at bottom of add_rows().
 
 .. automodule:: dsi.plugins.metadata
    :members:
+   :special-members: __init__
 
 File Readers
 ------------
@@ -32,6 +37,7 @@ Note for users:
       - if ingesting multiple tables at a time, users must pad tables with null data (YAML1 uses this and has example code at bottom of add_row() to implement this)
 .. automodule:: dsi.plugins.file_reader
    :members:
+   :special-members: __init__
 
 File Writers
 ------------
@@ -40,14 +46,16 @@ Note for users:
    - If runTable flag is True in Terminal instantiation, the run table is only included in ER Diagram writer if data is processed from a backend. View Example 4 in Core Examples
 .. automodule:: dsi.plugins.file_writer
    :members:
+   :special-members: __init__
 
 Environment Plugins
 -------------------
 .. automodule:: dsi.plugins.env
    :members:
+   :special-members: __init__
 
 Optional Plugin Type Enforcement
-==================================
+--------------------------------
 
 Plugins take data in an arbitrary format, and transform it into metadata which is queriable in DSI. Plugins may enforce types, but they are not required to enforce types. Plugin type enforcement can be static, like the Hostname default plugin. Plugin type enforcement can also be dynamic, like the Bueno default plugin.
 

docs/plugins/generate_plugin_class_hierarchy.py

@@ -36,7 +36,7 @@ def export_png(self, name: str) -> None:
         def process_children(r):
             print(r.clas.__name__)
             for ch in r.subclasses:
-                if ch.clas.__name__ == "Environment" or (r.clas.__name__ == "FileReader" and ch.clas.__name__ in ["Wildfire", "Bueno", "TextFile", "MetadataReader1"]):
+                if ch.clas.__name__ == "Environment" or (r.clas.__name__ == "FileReader" and ch.clas.__name__ in ["Wildfire", "Bueno", "MetadataReader1"]):
                     continue
                 dot.node(ch.clas.__name__)
                 dot.edge(r.clas.__name__, ch.clas.__name__)