minor updates for inline and docs/ text

Vedant1 · Vedant1 · commit e219a089e776 · 2025-03-06T13:53:50.000-07:00
diff --git a/docs/backends.rst b/docs/backends.rst
@@ -1,7 +1,12 @@
 Backends
 ========
 
-Backends connect users to DSI Core middleware and backends allow DSI middleware data structures to read and write to persistent external storage. Backends are modular to support user contribution.  Backend contributors are encouraged to offer custom backend abstract classes and backend implementations.  A contributed backend abstract class may extend another backend to inherit the properties of the parent. In order to be compatible with DSI core middleware, backends should create an interface to Python built-in data structures or data structures from the Python ``collections`` library. Backend extensions will be accepted conditional to the extention of ``backends/tests`` to demonstrate new Backend capability. We can not accept pull requests that are not tested.
+Backends connect users to DSI Core middleware and backends allow DSI middleware data structures to read and write to persistent external storage. 
+Backends are modular to support user contribution.  Backend contributors are encouraged to offer custom backend abstract classes and backend implementations.  
+A contributed backend abstract class may extend another backend to inherit the properties of the parent. 
+In order to be compatible with DSI core middleware, backends should create an interface to Python built-in data structures or data structures from the Python ``collections`` library. 
+Backend extensions will be accepted conditional to the extention of ``backends/tests`` to demonstrate new Backend capability. 
+We can not accept pull requests that are not tested.
 
 Note that any contributed backends or extensions should include unit tests in  ``backends/tests`` to demonstrate the new Backend capability.
 
diff --git a/docs/contributing_readers.rst b/docs/contributing_readers.rst
@@ -2,7 +2,8 @@
 Making a Reader for Your Application
 ====================================
 
-DSI readers are the primary way to transform outside data to metadata that DSI can ingest. Readers are Python classes that must include a few methods, namely ``__init__``, ``pack_header``, and ``add_rows``.
+DSI readers are the primary way to transform outside data to metadata that DSI can ingest. 
+Readers are Python classes that must include a few methods, namely ``__init__``, ``pack_header``, and ``add_rows``.
 
 Initializer: ``__init__(self) -> None:``
 -------------------------------------------
@@ -65,13 +66,13 @@ Example alternate ``add_rows``: ::
     my_data["joey"] = 20
     my_data["amy"] = 30
 
-    self.set_schema2(my_data)
+    self.set_schema_2(my_data)
 
 Implemented Examples
 --------------------------------
 If you want to see some full reader examples in-code, some can be found in 
-`dsi/plugins/env.py <https://github.com/lanl/dsi/blob/main/dsi/plugins/env.py>`_.
-``Hostname`` is an especially simple example to go off of. 
+`dsi/plugins/file_reader.py <https://github.com/lanl/dsi/blob/main/dsi/plugins/file_reader.py>`_.
+``Csv`` is an especially simple example to go off of. 
 
 Loading Your Reader
 -------------------------
diff --git a/docs/core.rst b/docs/core.rst
@@ -1,12 +1,17 @@
 Core
 ====
 
-The DSI Core middleware defines the Terminal concept. An instantiated Terminal is the human/machine DSI interface. The person setting up a Core Terminal only needs to know how they want to ask questions, and what metadata they want to ask questions about. If they don’t see an option to ask questions the way they like, or they don’t see the metadata they want to ask questions about, then they should ask a Backend Contributor or a Plugin Contributor, respectively.
+The DSI Core middleware defines the Terminal concept. An instantiated Terminal is the human/machine DSI interface. 
+The person setting up a Core Terminal only needs to know how they want to ask questions, and what metadata they want to ask questions about. 
+If they don’t see an option to ask questions the way they like, or they don’t see the metadata they want to ask questions about, then they should ask a Backend Contributor or a Plugin Contributor, respectively.
 
 Core: Terminal
 --------------
 
-The Terminal class is a structure through which users can interact with Plugins (Readers/Writers) and Backends as "module" objects. Each reader/writer/backend can be "loaded" to make ready for use and users can further interact with backends by ingesting, querying, or processing data as well as generating an interactive notebook with data. All relevant functions have been listed below for further clarity.
+The Terminal class is a structure through which users can interact with Plugins (Readers/Writers) and Backends as "module" objects. 
+Each reader/writer/backend can be "loaded" to make ready for use and users can further interact with backends by ingesting, querying, processing, or finding data as well as generating an interactive notebook with data. 
+
+All relevant functions have been listed below for further clarity. Examples section displays various workflows using this Terminal class.
 
 Notes for users:
       - All plugin writers that are loaded must be followed by calling transload() after to execute them. Readers are automatically executed upon loading.
@@ -44,7 +49,7 @@ Example 3: Querying data from a backend
 
 .. literalinclude:: ../examples/core/query.py
 
-Example 4: Processing data from a backend to generate an Entity Relationship diagram
+Example 4: Processing data from a backend to generate an Entity Relationship diagram using a Writer
 
 .. literalinclude:: ../examples/core/process.py
 
diff --git a/docs/examples.rst b/docs/examples.rst
@@ -20,9 +20,9 @@ In the first step, a python script is used to parse the slurm output files and c
    ./parse_slurm_output.py --testname leblanc
 
 
-.. literalinclude:: ../examples/pennant/parse_slurm_output.py
+.. .. literalinclude:: ../examples/pennant/parse_slurm_output.py
 
-A second python script,
+In the second step, another python script,
 
 .. code-block:: unixconfig
 
@@ -31,69 +31,17 @@ A second python script,
 
 reads in the CSV file and creates a database:
 
-.. code-block:: python
-
-   """
-   Creates the DSI db from the csv file
-   """
-   """
-   This script reads in the csv file created from parse_slurm_output.py.
-   Then it creates a DSI db from the csv file and performs a query.
-   """
-
-   import argparse
-   import sys
-   from dsi.backends.sqlite import Sqlite, DataType
-   import os
-   from dsi.core import Terminal
-
-   isVerbose = True
-
-   if __name__ == "__main__":
-      """ The testname argument is required """
-      parser = argparse.ArgumentParser()
-      parser.add_argument('--testname', help='the test name')
-      args = parser.parse_args()
-      test_name = args.testname
-      if test_name is None:
-         parser.print_help()
-         sys.exit(0)
-      
-      table_name = "rundata"
-      csvpath = 'pennant_' + test_name + '.csv'
-      dbpath = 'pennant_' + test_name + '.db'
-      output_csv = "pennant_read_query.csv"
-
-      #read in csv
-      core = Terminal(run_table_flag=False)
-      core.load_module('plugin', "Csv", "reader", filenames = csvpath, table_name = table_name)
-
-      if os.path.exists(dbpath):
-         os.remove(dbpath)
-
-      #load data into sqlite db
-      core.load_module('backend','Sqlite','back-write', filename=dbpath)
-      core.artifact_handler(interaction_type='put')
-
-      # update dsi abstraction using a query to the sqlite db
-      query_data = core.artifact_handler(interaction_type='get', query = f"SELECT * FROM {table_name} WHERE hydro_cycle_run_time > 0.006;", dict_return = True)
-      core.update_abstraction(table_name, query_data)
-
-      #export to csv
-      core.load_module('plugin', "Csv_Writer", "writer", filename = output_csv, table_name = table_name)
-      core.transload()
+.. literalinclude:: ../examples/pennant/create_and_query_dsi_db.py
 
 Resulting in the output of the query:
 
 ..  figure:: example-pennant-output.png
     :alt: Screenshot of computer program output.
     :class: with-shadow
 
-
     The output of the PENNANT example.
 
 
-
 Wildfire Dataset
 ----------------
 
@@ -109,12 +57,7 @@ To run this example, load dsi and run:
 
    python3 examples/wildfire/wildfire.py
 
-Within ``wildfire.py``, Sqlite is imported from the available DSI backends and DataType is the derived class for the defined (regular) schema.
-
-.. code-block:: unixconfig
-
-   from dsi.backends.sqlite import Sqlite, DataType
-
+.. literalinclude:: ../examples/wildfire/wildfire.py
 
 This will generate a wildfire.cdb folder with downloaded images from the server and a data.csv file of numerical properties of interest. This cdb folder is called a `Cinema`_ database (CDB). Cinema is an ecosystem for management and analysis of high dimensional data artifacts that promotes flexible and interactive data exploration and analysis.  A Cinema database is comprised of a CSV file where each row of the table is a data element (a run or ensemble member of a simulation or experiment, for example) and each column is a property of the data element. Any column name that starts with 'FILE' is a path to a file associated with the data element.  This could be an image, a plot, a simulation mesh or other data artifact.
 
diff --git a/docs/introduction.rst b/docs/introduction.rst
@@ -1,25 +1,29 @@
+Introduction
+============
 
+The goal of the Data Science Infrastructure Project (DSI) is to manage data through metadata capture and curation.  
+DSI capabilities can be used to develop workflows to support management of simulation data, AI/ML approaches, ensemble data, and other sources of data typically found in scientific computing. 
 
-
-The goal of the Data Science Infrastructure Project (DSI) is to manage data through metadata capture and curation.  DSI capabilities can be used to develop workflows to support management of simulation data, AI/ML approaches, ensemble data, and other sources of data typically found in scientific computing.  DSI infrastructure is designed to be flexible and with these considerations in mind:
-
-- Data management is subject to strict, POSIX-enforced, file security.
-- DSI capabilities support a wide range of common metadata queries.
-- DSI interfaces with multiple database technologies and archival storage options.
-- Query-driven data movement is supported and is transparent to the user.
-- The DSI API can be used to develop user-specific workflows.
+DSI infrastructure is designed to be flexible and with these considerations in mind:
+    - Data management is subject to strict, POSIX-enforced, file security.
+    - DSI capabilities support a wide range of common metadata queries.
+    - DSI interfaces with multiple database technologies and archival storage options.
+    - Query-driven data movement is supported and is transparent to the user.
+    - The DSI API can be used to develop user-specific workflows.
 
 ..  figure:: data_lifecycle.png
     :alt: Figure depicting the data life cycle
     :class: with-shadow
     :scale: 50%
 
-    A depiction of data life cycle can be seen here. The Data Science Infrastructure API supports the user to manage the life cycle aspects of their data.
+    A depiction of data life cycle can be seen here. The DSI API supports the user to manage the life cycle aspects of their data.
 
-DSI system design has been driven by specific use cases, both AI/ML and more generic usage.  These use cases can often be generalized to user stories and needs that can be addressed by specific features, e.g., flexible, human-readable query capabilities.  DSI uses Object Oriented design principles to encourage modularity and to support contributions by the user community.  The DSI API is Python-based.
+DSI system design has been driven by specific use cases, both AI/ML and more generic usage.  
+These use cases can often be generalized to user stories and needs that can be addressed by specific features, e.g., flexible, human-readable query capabilities. 
+DSI uses Object Oriented design principles to encourage modularity and to support contributions by the user community.  The DSI API is Python-based.
 
 Implementation Overview
-=======================
+-----------------------
 
 The DSI API is broken into three main categories:
 
@@ -28,19 +32,19 @@ The DSI API is broken into three main categories:
 - DSI Core: the *middleware* that contains the basic functionality to use the DSI API.
 
 Plugin Abstract Classes
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
 
-Plugins transform an arbitrary data source into a format that is compatible with the DSI core. The parsed and queryable attributes of the data are called *metadata* -- data about the data. Metadata share the same security profile as the source data.
+Plugins transform an arbitrary data source into a format that is compatible with the DSI core. 
+The parsed and queryable attributes of the data are called *metadata* -- data about the data. Metadata share the same security profile as the source data.
 
-Plugins can operate as data readers or data writers. A simple data reader might parse an application's output file and place it into a core-compatible data structure such as Python built-ins and members of the popular Python ``collection`` module. A simple data writer might execute an application to supplement existing data and queryable metadata, e.g., adding locations of outputs data or plots after running an analysis workflow.
+Plugins can operate as data readers or data writers. 
+A simple data reader might parse an application's output file and place it into a core-compatible data structure such as Python built-ins and members of the popular Python ``collection`` module. 
+A simple data writer might execute an application to supplement existing data and queryable metadata, e.g., adding locations of outputs data or plots after running an analysis workflow.
 
 Plugins are defined by a base abstract class, and support child abstract classes which inherit the properties of their ancestors.
 
 Currently, DSI has the following readers:
 
-   - CSV file reader: reads in comma separated value (CSV) files.
-   - Bueno reader: can be used to capture performance data from `Bueno <https://github.com/lanl/bueno>`_.
-
 ..  figure:: PluginClassHierarchy.png
     :alt: Figure depicting the current plugin class hierarchy.
     :class: with-shadow
@@ -49,27 +53,32 @@ Currently, DSI has the following readers:
     Figure depicting the current DSI plugin class hierarchy.
 
 Backend Abstract Classes
-------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
 
 Backends are an interface between the core and a storage medium.
-Backends are designed to support a user-needed functionality.  Given a set of user metadata captured by a DSI frontend, a typical functionality needed by DSI users is to query that metadata by SQL query. Because the files associated with the queryable metadata may be spread across filesystems and security domains, a supporting backend is required to assemble query results and present them to the DSI core for transformation and return.
+Backends are designed to support a user-needed functionality.  
+Given a set of user metadata captured by a DSI frontend, a typical functionality needed by DSI users is to query that metadata by SQL query. 
+Because the files associated with the queryable metadata may be spread across filesystems and security domains, 
+a supporting backend is required to assemble query results and present them to the DSI core for transformation and return.
 
 .. figure:: user_story.png
    :alt: This figure depicts a user asking a typical query on the user's metadata
    :class: with-shadow
    :scale: 50%
 
-   In this typical **user story**, the user has metadata about their data stored in DSI storage of some type.  The user needs to extract all files with the variable **foo** above a specific threshold.  DSI backends query the DSI metadata store to locate and return all such files.
+   In this typical **user story**, the user has metadata about their data stored in DSI storage of some type.  
+   The user needs to extract all files with the variable **foo** above a specific threshold.  
+   DSI backends query the DSI metadata store to locate and return all such files.
 
 Current DSI backends include:
 
-- Sqlite: Python based SQL database and backend; the default DSI API backend.
-- GUFI: the Grand Unified File Index system `Grand Unified File-Index <https://github.com/mar-file-system/GUFI>`_ ; developed at LANL, GUFI is a fast, secure metadata search across a filesystem accessible to both privileged and unprivileged users.
+- SQLite: Python based SQL database and backend; the default DSI API backend.
+- GUFI: the `Grand Unified File Index system <https://github.com/mar-file-system/GUFI>`_ ; developed at LANL. GUFI is a fast, secure metadata search across a filesystem accessible to both privileged and unprivileged users.
 - Parquet: a columnar storage format for `Apache Hadoop <https://hadoop.apache.org>`_.
 
 DSI Core
---------
-
-DSI basic functionality is contained within the middleware known as the *core*.  The DSI core is focused on delivering user-queries on unified metadata which can be distributed across many files and security domains. DSI currently supports Linux, and is tested on RedHat- and Debian-based distributions. The DSI core is a home for DSI Plugins and an interface for DSI Backends.
+~~~~~~~~
 
-Core Documentation
+DSI basic functionality is contained within the middleware known as the *core*.  
+The DSI core is focused on delivering user-queries on unified metadata which can be distributed across many files and security domains. 
+DSI currently supports Linux, and is tested on RedHat- and Debian-based distributions. The DSI core is a home for DSI Plugins and an interface for DSI Backends.
diff --git a/docs/plugins.rst b/docs/plugins.rst
@@ -11,7 +11,7 @@ Note that any contributed plugins or extension should include unit tests in  ``p
     :class: with-shadow
     :scale: 100%
 
-    Figure depicts the current DSI plugin class hierarchy.
+    Figure depicts prominent portion of the current DSI plugin class hierarchy.
 
 .. automodule:: dsi.plugins.plugin
    :members:
@@ -29,15 +29,15 @@ Note for users:
    - Plugin readers in DSI repo can/should handle data files with mismatched number of columns. Ex: file1: table1 has columns a, b, c. file2: table1 has columns a, b, d
    
       - if only reading in one table, users can utilize python pandas to stack mulutiple dataframes vertically (CSV reader)
-      - if ingesting multiple tables at a time, users must pad tables with null data (YAML1 and TOML1 use this. YAML1 has example code at bottom of add_row())
+      - if ingesting multiple tables at a time, users must pad tables with null data (YAML1 uses this and has example code at bottom of add_row() to implement this)
 .. automodule:: dsi.plugins.file_reader
    :members:
 
 File Writers
 ------------
 
 Note for users:
-   - If runTable flag is True in Terminal instantiaton, the run table is only included in ER Diagram writer if data is processed from a backend. View Example 4 in Core Examples
+   - If runTable flag is True in Terminal instantiation, the run table is only included in ER Diagram writer if data is processed from a backend. View Example 4 in Core Examples
 .. automodule:: dsi.plugins.file_writer
    :members:
 
diff --git a/dsi/backends/sqlite.py b/dsi/backends/sqlite.py
diff --git a/dsi/plugins/file_writer.py b/dsi/plugins/file_writer.py
diff --git a/examples/core/process.py b/examples/core/process.py
