Docs

Table of Contents

1. Introduction
2. Project Structure
3. Understanding the Data Model
   3.1 Data Types
   3.2 Views
4. Data Storage/Persistence
5. Metaprogramming
6. Mobile

1. Introduction

It's just a simple note-taking app! So why are there 45,000 lines of code???

There are at least 50 shortcuts that are available to the user for editing and navigating. Each of these shortcuts has a corresponding action-creator, reducer, and unit test. So that, along with the large number of util functions to make common operations easier, creates a large amount of code right off the bat.

bindContext	indentOnSpace	pinOpen
bumpThoughtDown	index	pinSubthoughts
clearThought	join	proseView
collapseContext	moveCursorBackward	redo
cursorBack	moveCursorForward	search
cursorDown	moveThoughtDown	splitSentences
cursorForward	moveThoughtUp	subcategorizeAll
cursorNext	newGrandChild	subcategorizeOne
cursorPrev	newSubthought	toggleContextView
cursorUp	newSubthoughtTop	toggleHiddenThoughts
delete	newThoughtAbove	toggleSidebar
deleteEmptyThoughtOrOutdent	newThoughtOrOutdent	toggleSort
exportContext	newUncle	toggleSplitView
extractThought	note	toggleTableView
home	openShortcutPopup	undo
indent	outdent

Then there are a few features of em that make it more complex than a simple note-taking app:

• em is offline first and supports syncing across multiple devices, including IndexedDB and Firebase. For better or worse (mostly worse), the syncing mechanism is hand-rolled with no 3rd party dependencies. There is some complex logic for handling push, pull, and reconciliation.
• em works as a PWA with a gesture system and detailed control of the caret during editing, which requires some browser-specific code to get right. em is one of the only graph-style note-taking apps that's designed specifically for the mobile experience.

2. Project Structure

Most common source files and folders are shown below. Tests are located in */__tests__/ in their respective folders.

• /src/App.css - All the styles. New components should use styled components (particularly @emotion/styled), but legacy styles are still kept in one giant stylesheet.
• /src/constants.js - Constant values. For constants that are only used in a single module, start by defining them in the module itself. They can be moved to constants.js if they need to be used in multiple modules or if there is a strong case to define them separately, e.g. an app-wide configuration that may need to be changed or tweaked.
• /src/action-creators - Redux action creators. Prefer reducers; only define an action creator if it requires a side effect. Use strings directly rather than defining an action type; I haven't found any benefit in practice for separate action type definitions as typos are immediately exposed. Use action creators rather than dispatching actions directly in order to gain type safety.
• /src/components - React functional components.
• /src/hooks - Custom React hooks. The project started before hooks were released, so we don't leverage them a lot, but we would like to use them more.
• /src/reducers - Redux reducers. Use util/reducerFlow to compose reducers.
• /src/redux-enhancers - Redux enhancers
• /src/redux-middleware - Redux middleware
• /src/selectors - Select, compute, and possibly memoize slices from state.
• /src/shortcuts - Keyboard and gesture shortcuts
• /src/util - Miscellaneous

3. Understanding the Data Model

3.1. Data Types

3.1.0. Thought

We describe em to users in terms of creating, editing, and organizing their thoughts. However a "thought" is not an actual data type in em. It is represented by a few different data types, described below, depending on how it is used or stored.

3.1.1. Context

The word context refers to the ancestor path of a thought. e.g. c is in the context a/b:

- a
  - b
    - c

The data type Context is deprecated, but the common usage of context is still useful for explanatory purposes.

3.1.2. Parent

COMING SOON: Parent will be renamed to Thought.

3.1.2.1. Rank

A rank is a number used to determine a thought's sort order among its sibling thoughts.

Ranks are unique within a single context. There is no relationship between ranks across contexts.

Ranks are relative; the absolute value does not matter. What matters is only if a rank is greater than or less than other ranks in the same context.

A new thought will be assigned a rank depending on where it is inserted:

• at the end of a context → rank of last thought + 1
• at the beginning of a context → rank of first thought - 1 (may be negative!)
• in the middle of a context → rank halfway between surrounding siblings (may be fractional!)

Negative ranks allow new thoughts to be efficiently inserted at the beginning of a context without having to modify the ranks of all other siblings. e.g. If a thought is placed before a thought with ranks 0, it will be assigned a rank of -1.

Fractional ranks allow new thoughts to efficiently be inserted between any two siblings without having to modify the ranks of other siblings. e.g. If a thought is placed between thoughts with ranks 5 and 6, it will be assigned a rank of 5.5.

importJSON autoincrements the ranks of imported thoughts across contexts for efficiency and may result in different ranks that would be produced by manually adding the thoughts, but the sibling-relative ordering will be the same.

3.1.3. Path

/** A sequence of thoughts from contiguous contexts. */
type Path = ThoughtId[]

e.g. ['kv9a-vzva-ac4n', '2mv0-atk3-tjlw', 'vkwt-ftz1-094z']

The most important Path in em is the thought that is being edited: state.cursor. When the user clicks on thought A, state.cursor will be set to [idOfA]. Navigating to a subthought will append the child's id to the cursor. So hitting ArrowDown on A will set the cursor to [idOfA, idOfB], etc.

Circular Paths are allowed. This is possible because of the Context View, described below, which allows jumping across the hierarchy.

3.1.4. SimplePath

/** A contiguous Path with no cycles. */
export type SimplePath = Path & Brand<'SimplePath'>

A SimplePath is a Path that has not crossed any Context Views, and thus has no cycles. Typescript is not expressive enough to capture this property in a type, but we can use brand types to require explicit casting, thus minimizing the chance of using a Path with cycles when a SimplePath is required. A Brand type is a nominal type that disallows implicit conversion. See: https://spin.atomicobject.com/2018/01/15/typescript-flexible-nominal-typing/.

3.1.5. Lexeme

/** An object that contains a list of contexts where a lexeme appears in 
    different word forms (plural, different cases, emojis, etc). */
export interface Lexeme {
  id?: string,
  value: string,
  contexts: ThoughtId[],
  created: Timestamp,
  lastUpdated: Timestamp,
}

A Lexeme stores all the contexts where a thought appears in identical and near-identical word forms (ignoring case, plurality, emojis, etc). Think of these as the inbound links to a thought. e.g. The Lexeme for cat contains two contexts: Animals and Socrates.

- Animals
  - Cats
  - Dogs
- My Pets
  - Socrates
    - cat

Usage Tip: Use getLexeme to get the Lexeme for a thought value.

3.2. Views

3.2.1. Normal View

In normal view (default), a thought's children are rendered in a collapsible tree.

- a
  - m [cursor] 
    - x
    - y
- b
  - m
    - y
    - z

3.2.1.1. Normal View Recursion [DEPRECATED]

3.2.2. Context View

Warning: Context View functionality is currently disabled and under redesign.

Enter Ctrl + Shift + S, or click the button in the toolbar to activate the Context View on a given thought. This will show all the contexts that a thought appears in.

e.g. Given a normal view...

- a
  - m
    - x
    - y
- b
  - m
    - y
    - z

Activating the context view on m (indicated by ~) renders:

- a
  - m~ [cursor]
    - a
    - b
      - y
      - z
- b
  - m
    - y
    - z

a and b are listed under a/m~ because they are the contexts that m appears in. They are the inbound links to m, as opposed to the outbound links that are rendered from a context to a child.

Note: The ranks of the contexts are autogenerated and do not correspond with the rank of the thought within its context, but rather the sorted order of the contexts in the context view.

Usage Tip: Use getContexts() or getThought(...).contexts to get the contexts for a thought value.

3.2.2.1. Context View Recursion

Descendants of contexts within a context view are rendered recursively. The Child thoughts that are generated from the list of contexts mentioned above can render their own Child thoughts (defaulting to Normal View). But what Context to use? When the parent is in Normal View, a Path is converted to a Context. When the parent, is in Context View, e.g. ['a', 'm'], we have direct access to the Context not from a Path but from getContexts('m'): [{ context: ['a'], rank: 0 }, { context: ['b'], rank: 1 }]. We then combine the desired Context with the head thought to render the expected Child thoughts. See the following example.

Note: The cursor here is circular. The underlying data structure allows for cycles. This is possible because only a fixed number of levels of depth are shown at a time.

(~ indicates context view)

- a
  - m~
    - a [cursor]
      - x
      - y
    - b
- b
  - m
    - y
    - z

The ThoughtContexts for m are [{ context: ['a'], rank: 0 }, { context: ['b'], rank: 1 }]. Where do x and y come from? They are the children of ['a', 'm']. When the Context View of m is activated, and the context ['a'] is selected, it renders the children of ['a', 'm'].

3.2.2.2. contextChain

When working with Context Views, it is necessary to switch between the full Path that crosses multiple Context Views, and the contiguous SimplePath segments that make it up. This is the only way to get from a Path that crosses multiple Context Views to a single Context, which does not allow cycles.

This more verbose and explicit representation of a transhierarchical Path and its different Context View boundaries is called a contextChain. contextChain is not stored in state, but derived from the cursor via splitChain(state, cursor). Consider the following:

• a
  • m
    • x
• b
  • m
    • y

When cursor is a/m~/b/y, then contextChain is (ranks omitted for readability):

[
  ['a', 'm'],
  ['b', 'y']
]

That is, the cursor consists of the initial segment a/m, then we enter the Context View of m, then b/y.

This allows the cursor to move across multiple Context Views. A more complicated example (copy and paste into em to test):

  - Books
    - Read
      - C. S. Peirce
        - Philosophical Writings
          - Three Categories
          - Philosophy of Math
          - Philosophy Logic
          - Semiotics
  - Personal
    - Influences
      - Gregory Bateson
      - Michael Polanyi
      - C. S. Peirce
  - Philosophy
    - Philosophy of Math
      - Statistical Inference
      - Probability as Potential
    - Philosophy of Science
    - Metaphysics
      - Sri Aurobindo
      - Forrest Landry
        - Potentiality
          - Probability as Potential
            - Potentiality vs Actuality

The cursor /Books/Read/C.S. Peirce/Philosophical Writings/Philosophy of Math~/Philosophy/Probability as Potential~/Potentiality/Potentiality vs Actuality spans two Context Views (Philosophy of Math and Probability as Potential), thus there are three segments in the contextChain:

[
  ['Books', 'Read', 'C. S. Peirce', 'Philosophical Writings', 'Philosophy of Math'],
  ['Philosophy', 'Probability as Potential'],
  ['Potentiality', 'Potentiality vs Actuality'],
]

Other functions related to contextChain are:

• chain
• contextChainToPath
• lastThoughtsFromContextChain
• splitChain
• thoughtsEditingFromChain

4. Data Storage/Persistence

Thoughts are stored in the underlying objects thoughtIndex and contextIndex, which map hashed values to Lexemes and hashed contexts to Parents, respectively. Only visible thoughts are loaded into state. The pullQueue is responsible for loading additional thoughts into state that need to be rendered.

• state (Redux)
• local (IndexedDB via Dexie)
• remote (Firebase) [optional; if user is logged in]

The syncing and reconciliation logic is done by pull, push, reconcile, and updateThoughts.

5. Metaprogramming

Metaprogramming provides the ability to alter em's behavior from within em itself through hidden subthoughts called metaprogramming attributes. Metaprogramming attributes begin with = and are hidden unless showHiddenThoughts is toggled on from the toolbar. Generally an attribute will affect only its parent context.

Note: User settings are stored as metaprogramming thoughts within [EM, 'Settings']. See INITIAL_SETTINGS for defaults.

List of possible metaprogramming attributes:

• =bullets Hide the bullets of a context. Options: Bullets, None.
• =children Apply attributes to all children. Currently only works with =style and =bullets. e.g. This would make b and c the color tomato:

  - a
    - =children
      - =style
        - color
          - tomato
    - b
    - c
  

• =focus When the cursor is on this thought, hide parent and siblings for additional focus. Options: Normal, Zoom.
• =hidden The thought is only displayed when showHiddenThoughts === true.
• =immovable The thought cannot be moved.
• =label Display alternative text, but continue using the real text when linking contexts. Hide the real text unless editing.
• =note Display a note in smaller text underneath the thought.
• =options Specify a list of allowable subthoughts.
• =pin Keep a thought expanded. Options: true, false.
• =pinChildren Keep all thoughts within a context expanded. Options: true, false.
• =readonly The thought cannot be edited, moved, or extended.
• =style Set CSS styles on the thought. May also use =children/=style or =grandchildren/=style.
• =uneditable The thought cannot be edited.
• =unextendable New subthoughts may not be added to the thought.
• =view Controls how the thought and its subthoughts are displayed. Options: List, Table, Prose.

6. Mobile

6.1 Editing Mode

On mobile, state.editing is set to true if there is an active browser selection. When the user closes their mobile keyboard, editing is set to false. This allows the user to navigate from thought to thought without opening the keyboard. It is not used on desktop.

• To enter editing mode, the user taps on the cursor thought or activates a shortcut that modifies a visible thought, such as newThought, clearText, subcategorizeOne, etc.
• To close editing mode, the user closes the mobile keyboard or navigates to the HOME context.