7x docs and stability improvements (#365)

* doc updates

* more updates

* add prepublish step, document use of workers, how to match tests, publishing workflow

* break out accuracy limitations doc

* deprecate fgbFetchAll for loadFgb, add toolbox doc, drop deprecated overlapRaster

* Break out tutorials and more

* update preprocessing and geoprocessing docs

* genPreprocessor: add minSize and enforceMinSize options, more doc additions

* update reportclient doc

* add projectclient doc, fix links, other updates

* add vite babel plugins for storybook, migrate away from toNullSketch in gp function to childProperties, add toSketchPropertiesArray helper

* remove null sketch from template smoke test output

* add toChildProperties helper, major cleanup 6.1 tutorial

* update 7x tutorials, couple backports to 6.1, add devcontainer guide

* drop global datasources from basic-project template.  add codespaces advanced guide

* update smoke test output

* refactor genRandomFeature script to support sketches too, drop genRandomSketch

* update genRandomFeature, add third part data doc, break out link data tutorial, start newproject revamp

* further newproject, add precalc, third party and data guides

* rework system setup sections, fix broken links

* update windows section

* front page and tutorial updates

* windows clarity

* newproject start refine

* drop global datasources other than boundaries, including from simpleFunction

* Deprecate polygonPreprocessorSmokeTest, create polygonSmokeTest function that allows examples to be loaded by caller, whether they want to pass in features or sketches

* newproject updates

* genRandomFeature: fix outdir to vary whether sketch or feature type output

* drop bbox and planning init questions and basic properties remove template-ocean-eez from testing

* drop template question from init, update tutorial method for getting bbox of datasource

* rename genRandomFeature script to genRandomPolygon

* remove old genRandom scripts on upgrade

* strip down simpleFunction and blankFunction, add to newproject

* tune simpleFunction and blankFunction, requireProperties not required as Handler option

* Fix genRandomPolygon name bug, storybook no open browser, remove default metric group from base-project, add create report section to newproject tutorial, refine Precalc and Testing

* migrate to loadFgb internally

* drop deprecated loadCogWindow and bboxToPixelEdge

* include dataproviders as top-level export

* switch newProject to sampleProject, and add start of sampleProject, drop down base README, highlight concepts

* update newproject, sampleproject, regen api docs with dataproviders changes

* fix broken links, refine sampleproject layout

* dataImport: conditional ask additional properties question

* tune dataImport

* tutorial bump gp version, sort init language alphabetical

* unregister blankFunction in blank project default config, add SimpleReport section.

* breaking: deprecate genClipLoader, remove genPreprocessor and clipToPolygonFeatures,  replace with clipToPolygonFeatures, genClipToPolygonFeatures, clipToPolygonDatasources and genClipToPolygonDatasources

* code comments

* nit

* update example data link

* add validatePolygon preprocessor, with ensureValidPolygon helper, typedoc update

* clean up polygonSmokeTest doc

* sampleproject updates

* move preprocessor test features to base, remove test output, so that we don't hit cloudfront VectorDataSource for every smoke test run, which can error and be brittle

* add clipToOceanEez to basic project, add unit tests for preprocessors with mocks, add numberFormat helper, decrease default memory on preprocessors

* preprocessing docs. other doc updates on rebuild

* drop use of dataproviders module, switch remaining fgbFetchAll to loadFgb, clean up preprocessing doc

* update newproject

* switch back to separate dataproviders module to isolate node-fetch use, causing VectorDataSource metadata fetch issue.

* rollback flatgeobuf to 3.33.0 for now

* migrate last dataprovider use

* upgrade flatgeobuf, remove unused webstreams polyfill, drop node-fetch polyfill

* add vitest-fetch-mock dep to base project

* add back dataproviders module to top-level export and migrate use

* Fix SimpleCard error and exports, add roundDecimalFormat helper function, debug false default for smoke tests

* Fix i18nAsync bug using base english strings instead of letting component render its own, drop report client export from SimpleCard, allow ts-ignore for mock issue

* update base english terms

* update sampleproject, rebuild api docs

* simplify sketchOrCollection variable in SimpleCard and add translation to partials

* update newproject

* bump next version

* newproject nit

* simplify SimpleCard, cleanup BlankCard, update translation

* bump gp

* move upgrade doc to basic tutorials, add gp-workspace upgrade instructions to setup tutorial

* remove .story-cache folders from dist build, update template-ocean-eez smoke test output

* move upgrade doc to basic tutorials, add gp-workspace upgrade instructions to setup tutorial

* remove .story-cache folders from dist build, update template-ocean-eez smoke test output

* move blank functions over to templace-ocean-eez, out of temlate-blank-project

* add back simplefunction smoke test output

* update story-cache to copy from eez template

* template functions cleanup

* switch vectorFunction to use higher level getFeatures instead of loadFgb, in order to make use of datasource filters

* genReportStories - replace spaces in sketch name with underscore

* better link concepts

* improve transform of sketch name to valid variable name

* add sketch prefix to report story variable name

* fix sketch variable name in genReportStories

* Add Card to createReport component name plus other output polish, drop extra prepublish step, it's already run on publish

* Add user attributes to generated sample sketches, nit createReport output

* add user attributes to genRandomPolygons

* update genRandomPolygon notes, add simple-card-view image

* genRandomPolygon: start ids at 1

* Add simple report modifications

* add getFgbPath method to projectClientBase, update createFunction output, add to sampleproject

* Fix getFgbPath

* Add getFeaturesForSketchBboxes helper, clarify MetricGroup type property, doc rebuild, version bump and enhancements

* fix export

* Make MetricGroup type property optional, rebuild docs

* doc updates

* update getFeaturesForBboxes with split feature, add splitBBoxAntimeridian and cleanBBox helpers, rename split.js to antimeridian.js.  Needs more tests.  Remove antimeridian tests from getFeatures

* antimeridian code example

* doc updates

* writeResultOutput, switch to ensureDirSync, flatgeobuf test cleanup

* createFunction - default to async, createClient - look in ocean eez template dir for assets

* undeprecate clip function, createClient - fix story paths, sampleProject - finish coral reef report, lots of code comment updates

* export intersectInChunks, intersectInChunksArea, intersectSum.  Move them over to clip toolbox, build up docs

* bump gp in docs

* update toolbox

* shape up toolbox doc, shift subdividing into data import doc

* shuffle couple functions, toolbox doc

* projectClient - add getClassDatasource helper function, drop requirement to define type in metricGroup schema, refine benthic report

* getFeaturesForSketchBBoxes - create options argument

* benthic report

* createReport: switch vector function to use getFeaturesForSketchBBoxes and getClassDatasource, fix compName to have Card in all

* projectClient: add getClassKey helper

* bump doc gp version

* bug - don't use datasource classKey in vectorFunction

* vectorFunction fix bug in last update, rasterFunction migrate to getClassDatasource

* MetricGroup schema - ensure at least one class item present

* create:report clarify raster stat options, start octocoral report

* metricGroupSchema: require datasourceId and classKey at metric group level or data class level, improve create:report raster metric descriptions.

* rename getClassDatasouce to getMetricGroupDatasource, rename getClassKey to getMetricGroupClassKey, switch them to use options parameter, refactor underlying metricGroup helpers to return class level or top level objectives but not both

* remove default objective

* sampleproject update

* improve precalc logging, use loadCog directly,

* Consolidate report pages into TabReport, drop ViabilityPage

* update sampleProject

* doc rebuild

* projectClientBase - fix getMetricGroupDatasource and getMetricGroupClass key error when classId not defined

* merge advanced concepts into concepts page, update CLI doc, update architecture, other comment nits

* update sampleProject, simplify getting started docs

* update sampleproject, add preprocessing images, bump to 7x-docs.128

* broken link fixes

* polish sample tutorial

* sampleproject polish

* refactor newproject, remove reportclient doc

* rebuild docs

.vscode/launch.json

@@ -90,7 +90,21 @@
       "cwd": "${workspaceFolder}/packages/template-ocean-eez",
       "runtimeArgs": [
         "--inspect-brk",
-        "${workspaceRoot}/packages/template-ocean-eez/node_modules/.bin/geoprocessing",
+        "${workspaceRoot}/node_modules/.bin/geoprocessing",
+        "test"
+      ],
+      "console": "integratedTerminal",
+      "internalConsoleOptions": "neverOpen",
+      "port": 9229
+    },
+    {
+      "name": "template-blank-project - debug tests",
+      "type": "node",
+      "request": "launch",
+      "cwd": "${workspaceFolder}/packages/template-blank-project",
+      "runtimeArgs": [
+        "--inspect-brk",
+        "${workspaceRoot}/node_modules/.bin/geoprocessing",
         "test"
       ],
       "console": "integratedTerminal",

README.md

@@ -1,15 +1,15 @@
 # Geoprocessing
 
-All-in-one solution for developing low-cost and low-maintenance geoprocessing functions and reports for the web, with Typescript. Simplified publish to cloud with auto-scaling to meet high demand.
+All-in-one solution for developing low-cost and low-maintenance geoprocessing functions and reports for SeaSketch. Simplified publish to cloud with auto-scaling to meet high demand.
 
 This framework is part of the [SeaSketch](https://seasketch.org) ecosystem.
 
-- [Documentation](https://github.com/seasketch/geoprocessing/wiki)
-- [Changelog](https://github.com/seasketch/geoprocessing/blob/dev/CHANGELOG.md)
+- [Documentation](https://seasketch.github.io/geoprocessing)
+- [Releases](https://github.com/seasketch/geoprocessing/releases)
 
 ## Who is this framework for?
 
-This framework is primarily designed for people that want to create and host their own geoprocessing functions and reports and plug them into their SeaSketch project, though it is not dependent on SeaSketch. It is used for all SeaSketch [projects](https://github.com/seasketch/geoprocessing/network/dependents?package_id=UGFja2FnZS0xMTc3OTQ1NDg5).
+This framework is designed for people with coding experience that want to create and host their own geoprocessing functions and reports and plug them into their SeaSketch project. It is used for all SeaSketch [projects](https://github.com/seasketch/geoprocessing/network/dependents?package_id=UGFja2FnZS0xMTc3OTQ1NDg5).
 
 ## Goals
 

eslint.config.mjs

@@ -41,6 +41,7 @@ export default [
     rules: {
       "@typescript-eslint/no-explicit-any": "off",
       "@typescript-eslint/no-unused-vars": "warn",
+      "@typescript-eslint/ban-ts-comment": "off",
       "react/no-unescaped-entities": "off",
       trailingComma: "off",
       "unicorn/prevent-abbreviations": "off",

package.json

@@ -2,7 +2,9 @@
   "name": "root",
   "private": true,
   "workspaces": [
-    "packages/*"
+    "packages/geoprocessing",
+    "packages/base-project",
+    "packages/template-blank-project"
   ],
   "license": "BSD-3-Clause",
   "devDependencies": {
@@ -51,9 +53,10 @@
     "publish:experimental:canary": "BRANCH_NAME=`git rev-parse --symbolic-full-name --abbrev-ref HEAD`; if [ $BRANCH_NAME != \"dev\" -a $BRANCH_NAME != \"main\" ] ; then CI=1 lerna publish --force-publish --canary --preid experimental-${BRANCH_NAME} --dist-tag=experimental-${BRANCH_NAME} ; fi",
     "publish:alpha": "CI=1 lerna publish --force-publish --preid alpha --dist-tag=alpha",
     "publish:beta": "CI=1 lerna publish --force-publish --preid beta --dist-tag=beta",
-    "publish:stable": "npm run test && CI=1 lerna publish",
-    "publish:backport": "npm run test && CI=1 lerna publish --preid backport --dist-tag=backport",
+    "publish:stable": "CI=1 lerna publish",
+    "publish:backport": "CI=1 lerna publish --preid backport --dist-tag=backport",
     "prepare": "husky",
+    "prepublish": "npm test && lerna run --scope=@seasketch/geoprocessing prepare",
     "publish:translation": "npx tsx packages/geoprocessing/src/i18n/bin/publishTerms.ts",
     "import:translation": "npx tsx packages/geoprocessing/src/i18n/bin/importTerms.ts",
     "sync:translation": "npm run extract:translationt && npm run publish:translation && npm run import:translation",

packages/base-project/.prettierignore

@@ -2,4 +2,4 @@
 **/.svn
 **/.hg
 node_modules
-examples/output
+examples

packages/base-project/.storybook/genReportStories.ts

@@ -10,8 +10,6 @@ import {
 import { GpStoryConfig } from "./types.js";
 import { v4 as uuid } from "uuid";
 
-// Not currently used, project space responsible for its own storybook+vite setup
-
 if (!process.env.PROJECT_PATH) {
   throw new Error("PROJECT_PATH environment variable not set");
 }
@@ -162,6 +160,11 @@ for (const storyConfig of storyConfigs) {
       ...(childProperties ? { childProperties } : {}),
     };
 
+    // Convert sketch name to valid variable name, which is displayed as the story name
+    const sketchVariableName = sketch.properties.name
+      .replace(/^[^a-zA-Z$_\p{L}]/u, "_") // Replace invalid starting characters
+      .replaceAll(/[^a-zA-Z0-9$_\p{L}]/gu, "_"); // Replace invalid subsequent characters
+
     const story = `
       import React from "react";
       import { ${storyConfig.componentName} } from '${importFromCacheStr}';
@@ -180,7 +183,7 @@ for (const storyConfig of storyConfigs) {
         language: "en"
       });
 
-      export const ${sketch.properties.name.replaceAll("-", "_")} = () => (
+      export const ${sketchVariableName} = () => (
         <Translator>
           <${storyConfig.componentName} />
         </Translator>

packages/base-project/.vscode/schemas.json

@@ -53,37 +53,67 @@
       ],
       "additionalProperties": false
     },
-    "metricsSchema": {
+    "metricGroupSchema": {
       "type": "array",
       "items": {
-        "type": "object",
-        "properties": {
-          "metricId": { "type": "string" },
-          "type": { "type": "string" },
-          "datasourceId": { "type": "string" },
-          "classKey": { "type": "string" },
-          "classes": {
-            "type": "array",
-            "items": {
-              "type": "object",
-              "properties": {
-                "classId": { "type": "string" },
-                "classKey": { "type": "string" },
-                "datasourceId": { "type": "string" },
-                "display": { "type": "string" },
-                "numericClassId": { "type": "number" },
-                "layerId": { "type": "string" },
-                "objectiveId": { "type": "string" }
+        "oneOf": [
+          {
+            "type": "object",
+            "properties": {
+              "metricId": { "type": "string" },
+              "type": { "type": "string" },
+              "datasourceId": { "type": "string" },
+              "classKey": { "type": "string" },
+              "classes": {
+                "type": "array",
+                "items": {
+                  "type": "object",
+                  "properties": {
+                    "classId": { "type": "string" },
+                    "display": { "type": "string" },
+                    "numericClassId": { "type": "number" },
+                    "layerId": { "type": "string" },
+                    "objectiveId": { "type": "string" }
+                  },
+                  "required": ["classId", "display"],
+                  "additionalProperties": false
+                }
               },
-              "required": ["classId", "display"],
-              "additionalProperties": false
-            }
+              "layerId": { "type": "string" },
+              "objectiveId": { "type": "string" }
+            },
+            "required": ["metricId", "classes", "datasourceId"],
+            "additionalProperties": false
           },
-          "layerId": { "type": "string" },
-          "objectiveId": { "type": "string" }
-        },
-        "required": ["metricId", "type", "classes"],
-        "additionalProperties": false
+          {
+            "type": "object",
+            "properties": {
+              "metricId": { "type": "string" },
+              "type": { "type": "string" },
+              "classes": {
+                "type": "array",
+                "items": {
+                  "type": "object",
+                  "properties": {
+                    "classId": { "type": "string" },
+                    "classKey": { "type": "string" },
+                    "datasourceId": { "type": "string" },
+                    "display": { "type": "string" },
+                    "numericClassId": { "type": "number" },
+                    "layerId": { "type": "string" },
+                    "objectiveId": { "type": "string" }
+                  },
+                  "required": ["classId", "display", "datasourceId"],
+                  "additionalProperties": false
+                }
+              },
+              "layerId": { "type": "string" },
+              "objectiveId": { "type": "string" }
+            },
+            "required": ["metricId", "classes"],
+            "additionalProperties": false
+          }
+        ]
       }
     },
     "objectivesSchema": {
@@ -431,7 +461,7 @@
   "oneOf": [
     { "$ref": "#/definitions/basicSchema" },
     { "$ref": "#/definitions/objectivesSchema" },
-    { "$ref": "#/definitions/metricsSchema" },
+    { "$ref": "#/definitions/metricGroupSchema" },
     { "$ref": "#/definitions/geographiesSchema" },
     { "$ref": "#/definitions/datasourcesSchema" }
   ]

packages/base-project/README.md

@@ -1,5 +1,3 @@
 # Getting Started
 
-This is a geoprocessing project. It was created using the `geoprocessing init` command, as part of the tutorial to [create a new project](https://github.com/seasketch/geoprocessing/wiki/Tutorials#create-a-new-geoprocessing-project).
-
-Once established, you can follow the tutorial to [setup an existing project on a new system](https://github.com/seasketch/geoprocessing/wiki/Tutorials#setup-an-exising-project-on-your-local-system)
+This is a geoprocessing project created using the SeaSketch geoprocessing framework.

packages/base-project/package.json

@@ -35,7 +35,7 @@
     "reimport:data": "geoprocessing reimport:data",
     "start:data": "http-server data/dist -c-1",
     "start:client": "geoprocessing start:client",
-    "storybook": "PROJECT_PATH=$(pwd) npx tsx .storybook/genReportStories.ts && storybook dev -p 6006",
+    "storybook": "PROJECT_PATH=$(pwd) npx tsx .storybook/genReportStories.ts && storybook dev -p 6006 --no-open",
     "sync:translation": "npm run extract:translation && npm run publish:translation && npm run import:translation",
     "synth": "geoprocessing synth",
     "test": "start-server-and-test start:data 8080 'npm run __test'",
@@ -72,6 +72,7 @@
     "storybook": "^8.2.9",
     "vite": "^5.4.2",
     "vitest": "^2.0.5",
+    "vitest-fetch-mock": "^0.4.2",
     "zx": "^8.1.5"
   }
 }