difference mark and shift transform (#1896)

* difference mark

* fix filtering; opacity options

* remove unused import

* withTip; don’t duplicate channels

* difference as a composite mark

* difference tip

* reuse channels

* more composite marks

* apply clip as render transform

* consolidate code

* aria labels

* organize imports

* fix differenceY1 test

* update tests

* better defaults

* handle ChannelValueSpec

* update test

* memoTuple

* checkpoint docs

* fix differenceY1 test

* tip fixes

* **positiveOpacity**, **negativeOpacity** default to **fillOpacity**; **positive**, **negative** are the fill colors for their respective difference areas

* positiveFill

* another test

* positiveFillOpacity & fix test

* swap [xy][12]; default y1 = 0

* shift option

* another difference example

* z

* simpler marks (no need for two differences)

* inferScaleOrder

* simpler chart

* enhanced group extent; findX sketch

* shift transform

* shift domain hint

* promote stroke to z

* simpler channel domain hint

* more difference docs

* more difference docs

* more documentation

* call next twice (once for the path, once for the clipPath)

* support clip: frame

* document differenceY

* test ordinal difference

* adopt Plot.find

* more docs

---------

Co-authored-by: Philippe Rivière <[email protected]>

docs/.vitepress/config.ts

@@ -88,6 +88,7 @@ export default defineConfig({
           {text: "Contour", link: "/marks/contour"},
           {text: "Delaunay", link: "/marks/delaunay"},
           {text: "Density", link: "/marks/density"},
+          {text: "Difference", link: "/marks/difference"},
           {text: "Dot", link: "/marks/dot"},
           {text: "Frame", link: "/marks/frame"},
           {text: "Geo", link: "/marks/geo"},
@@ -121,6 +122,7 @@ export default defineConfig({
           {text: "Map", link: "/transforms/map"},
           {text: "Normalize", link: "/transforms/normalize"},
           {text: "Select", link: "/transforms/select"},
+          {text: "Shift", link: "/transforms/shift"},
           {text: "Sort", link: "/transforms/sort"},
           {text: "Stack", link: "/transforms/stack"},
           {text: "Tree", link: "/transforms/tree"},

docs/data/tsa.csv

@@ -0,0 +1 @@
+../../test/data/tsa.csv

docs/marks/difference.md

@@ -0,0 +1,145 @@
+<script setup>
+
+import * as Plot from "@observablehq/plot";
+import * as d3 from "d3";
+import {computed, shallowRef, onMounted} from "vue";
+
+const aapl = shallowRef([]);
+const gistemp = shallowRef([]);
+const tsa = shallowRef([{Date: new Date("2020-01-01")}]);
+const temperature = shallowRef([{date: new Date("2020-01-01")}]);
+
+onMounted(() => {
+  d3.csv("../data/aapl.csv", d3.autoType).then((data) => (aapl.value = data));
+  d3.csv("../data/gistemp.csv", d3.autoType).then((data) => (gistemp.value = data));
+  d3.csv("../data/tsa.csv",d3.autoType).then((data) => (tsa.value = data));
+  d3.csv("../data/sf-sj-temperatures.csv", d3.autoType).then((data) => (temperature.value = data.filter((d) => d.date.getUTCFullYear() === 2020)));
+});
+
+</script>
+
+# Difference mark <VersionBadge pr="1896" />
+
+The **difference mark** puts a metric in context by comparing it. Like the [area mark](./area.md), the region between two lines is filled; unlike the area mark, alternating color shows when the metric is above or below the comparison value.
+
+In the simplest case, the difference mark compares a metric to a constant. For example, the plot below shows the [global surface temperature anomaly](https://data.giss.nasa.gov/gistemp/) from 1880–2016; 0° represents the 1951–1980 average; above-average temperatures are in <span style="border-bottom: solid var(--vp-c-red) 3px;">red</span>, while below-average temperatures are in <span style="border-bottom: solid var(--vp-c-blue) 3px;">blue</span>. (It’s getting hotter.)
+
+:::plot
+```js
+Plot.differenceY(gistemp, {
+  x: "Date",
+  y: "Anomaly",
+  positiveFill: "red",
+  negativeFill: "blue",
+  tip: true
+}).plot({y: {grid: true}})
+```
+:::
+
+A 24-month [moving average](../transforms/window.md) improves readability by smoothing out the noise.
+
+:::plot
+```js
+Plot.differenceY(
+  gistemp,
+  Plot.windowY(12 * 2, {
+    x: "Date",
+    y: "Anomaly",
+    positiveFill: "red",
+    negativeFill: "blue",
+    tip: true
+  })
+).plot({y: {grid: true}})
+```
+:::
+
+More powerfully, the difference mark compares two metrics. For example, the plot below shows the number of travelers per day through TSA checkpoints in 2020 compared to 2019. (This in effect compares a metric against itself, but as the data represents each year as a separate column, it is equivalent to two metrics.) In the first two months of 2020, there were on average <span style="border-bottom: solid #01ab63 3px;">more travelers</span> per day than 2019; yet when COVID-19 hit, there were many <span style="border-bottom: solid #4269d0 3px;">fewer travelers</span> per day, dropping almost to zero.
+
+:::plot
+```js
+Plot.plot({
+  x: {tickFormat: "%b"},
+  y: {grid: true, label: "Travelers"},
+  marks: [
+    Plot.axisY({label: "Travelers per day (thousands, 2020 vs. 2019)", tickFormat: (d) => d / 1000}),
+    Plot.ruleY([0]),
+    Plot.differenceY(tsa, {x: "Date", y1: "2019", y2: "2020", tip: {format: {x: "%B %-d"}}})
+  ]
+})
+```
+:::
+
+If the data is “tall” rather than “wide” — that is, if the two metrics we wish to compare are represented by separate *rows* rather than separate *columns* — we can use the [group transform](../transforms/group.md) with the [find reducer](../transforms/group.md#find): group the rows by **x** (date), then find the desired **y1** and **y2** for each group. The plot below shows daily minimum temperature for San Francisco compared to San Jose. Notice how the insulating fog keeps San Francisco <span style="border-bottom: solid #01ab63 3px;">warmer</span> in winter and <span style="border-bottom: solid #4269d0 3px;">cooler</span> in summer, reducing seasonal variation.
+
+:::plot
+```js
+Plot.plot({
+  x: {tickFormat: "%b"},
+  y: {grid: true},
+  marks: [
+    Plot.ruleY([32]),
+    Plot.differenceY(
+      temperature,
+      Plot.windowY(
+        14,
+        Plot.groupX(
+          {
+            y1: Plot.find((d) => d.station === "SJ"),
+            y2: Plot.find((d) => d.station === "SF")
+          },
+          {
+            x: "date",
+            y: "tmin",
+            tip: true
+          }
+        )
+      )
+    )
+  ]
+})
+```
+:::
+
+The difference mark can also be used to compare a metric to itself using the [shift transform](../transforms/shift.md). The chart below shows year-over-year growth in the price of Apple stock.
+
+:::plot
+```js
+Plot.differenceY(aapl, Plot.shiftX("+1 year", {x: "Date", y: "Close"})).plot({y: {grid: true}})
+```
+:::
+
+For most of the covered time period, you would have <span style="border-bottom: solid #01ab63 3px;">made a profit</span> by holding Apple stock for a year; however, if you bought in 2015 and sold in 2016, you would likely have <span style="border-bottom: solid #4269d0 3px;">lost money</span>.
+
+## Difference options
+
+The following channels are required:
+
+* **x2** - the horizontal position of the metric; bound to the *x* scale
+* **y2** - the vertical position of the metric; bound to the *y* scale
+
+In addition to the [standard mark options](../features/marks.md#mark-options), the following optional channels are supported:
+
+* **x1** - the horizontal position of the comparison; bound to the *x* scale
+* **y1** - the vertical position of the comparison; bound to the *y* scale
+
+If **x1** is not specified, it defaults to **x2**. If **y1** is not specified, it defaults to 0 if **x1** and **x2** are equal, and to **y2** otherwise. These defaults facilitate sharing *x* or *y* coordinates between the metric and its comparison.
+
+The standard **fill** option is ignored; instead, there are separate channels based on the sign of the difference:
+
+* **positiveFill** - the color for when the metric is greater, defaults to <span style="border-bottom:solid #01ab63 3px;">green</span>
+* **negativeFill** - the color for when the comparison is greater, defaults to <span style="border-bottom:solid #4269d0 3px;">blue</span>
+* **fillOpacity** - the areas’ opacity, defaults to 1
+* **positiveFillOpacity** - the positive area’s opacity, defaults to *opacity*
+* **negativeFillOpacity** - the negative area’s opacity, defaults to *opacity*
+* **stroke** - the metric line’s stroke color, defaults to currentColor
+* **strokeOpacity** - the metric line’s opacity, defaults to 1
+
+These options are passed to the underlying area and line marks; in particular, when they are defined as a channel, the underlying marks are broken into contiguous overlapping segments when the values change. When any of these channels are used, setting an explicit **z** channel (possibly to null) is strongly recommended.
+
+## differenceY(*data*, *options*) {#differenceY}
+
+```js
+Plot.differenceY(gistemp, {x: "Date", y: "Anomaly"})
+```
+
+Returns a new difference with the given *data* and *options*. The mark is a composite of a positive area, negative area, and line. The positive area extends from the bottom of the frame to the line, and is clipped by the area extending from the comparison to the top of the frame. The negative area conversely extends from the top of the frame to the line, and is clipped by the area extending from the comparison to the bottom of the frame.

docs/public/data/sf-sj-temperatures.csv

@@ -0,0 +1 @@
+../../../test/data/sf-sj-temperatures.csv

docs/transforms/shift.md

@@ -0,0 +1,48 @@
+<script setup>
+
+import * as Plot from "@observablehq/plot";
+import * as d3 from "d3";
+import {ref, shallowRef, onMounted} from "vue";
+
+const shift = ref(365);
+const aapl = shallowRef([]);
+
+onMounted(() => {
+  d3.csv("../data/aapl.csv", d3.autoType).then((data) => (aapl.value = data));
+});
+
+</script>
+
+# Shift transform <VersionBadge pr="1896" />
+
+The **shift transform** is a specialized [map transform](./map.md) that derives an output **x1** channel by shifting the **x** channel; it can be used with the [difference mark](../marks/difference.md) to show change over time. For example, the chart below shows the price of Apple stock. The <span style="border-bottom: solid #01ab63 3px;">green region</span> shows when the price went up over the given interval, while the <span style="border-bottom: solid #4269d0 3px;">blue region</span> shows when the price went down.
+
+<p>
+  <label class="label-input" style="display: flex;">
+    <span style="display: inline-block; width: 7em;">Shift (days):</span>
+    <input type="range" v-model.number="shift" min="0" max="1000" step="1">
+    <span style="font-variant-numeric: tabular-nums;">{{shift}}</span>
+  </label>
+</p>
+
+:::plot hidden
+```js
+Plot.differenceY(aapl, Plot.shiftX(`${shift} days`, {x: "Date", y: "Close"})).plot({y: {grid: true}})
+```
+:::
+
+```js-vue
+Plot.differenceY(aapl, Plot.shiftX("{{shift}} days", {x: "Date", y: "Close"})).plot({y: {grid: true}})
+```
+
+When looking at year-over-year growth, the chart is mostly green, implying that you would make a profit by holding Apple stock for a year. However, if you bought in 2015 and sold in 2016, you would likely have lost money. Try adjusting the slider to a shorter or longer interval: how does that affect the typical return?
+
+## shiftX(*interval*, *options*) {#shiftX}
+
+```js
+Plot.shiftX("7 days", {x: "Date", y: "Close"})
+```
+
+Derives an **x1** channel from the input **x** channel by shifting values by the given *interval*. The *interval* may be specified as: a name (*second*, *minute*, *hour*, *day*, *week*, *month*, *quarter*, *half*, *year*, *monday*, *tuesday*, *wednesday*, *thursday*, *friday*, *saturday*, *sunday*) with an optional number and sign (*e.g.*, *+3 days* or *-1 year*); or as a number; or as an implementation — such as d3.utcMonth — with *interval*.floor(*value*), *interval*.offset(*value*), and *interval*.range(*start*, *stop*) methods.
+
+The shiftX also transform aliases the **x** channel to **x2** and applies a domain hint to the **x2** channel such that by default the plot shows only the intersection of **x1** and **x2**. For example, if the interval is *+1 year*, the first year of the data is not shown.

src/index.d.ts

@@ -21,6 +21,7 @@ export * from "./marks/contour.js";
 export * from "./marks/crosshair.js";
 export * from "./marks/delaunay.js";
 export * from "./marks/density.js";
+export * from "./marks/difference.js";
 export * from "./marks/dot.js";
 export * from "./marks/frame.js";
 export * from "./marks/geo.js";
@@ -52,6 +53,7 @@ export * from "./transforms/hexbin.js";
 export * from "./transforms/map.js";
 export * from "./transforms/normalize.js";
 export * from "./transforms/select.js";
+export * from "./transforms/shift.js";
 export * from "./transforms/stack.js";
 export * from "./transforms/tree.js";
 export * from "./transforms/window.js";

src/index.js

@@ -12,6 +12,7 @@ export {Contour, contour} from "./marks/contour.js";
 export {crosshair, crosshairX, crosshairY} from "./marks/crosshair.js";
 export {delaunayLink, delaunayMesh, hull, voronoi, voronoiMesh} from "./marks/delaunay.js";
 export {Density, density} from "./marks/density.js";
+export {differenceY} from "./marks/difference.js";
 export {Dot, dot, dotX, dotY, circle, hexagon} from "./marks/dot.js";
 export {Frame, frame} from "./marks/frame.js";
 export {Geo, geo, sphere, graticule} from "./marks/geo.js";
@@ -38,6 +39,7 @@ export {find, group, groupX, groupY, groupZ} from "./transforms/group.js";
 export {hexbin} from "./transforms/hexbin.js";
 export {normalize, normalizeX, normalizeY} from "./transforms/normalize.js";
 export {map, mapX, mapY} from "./transforms/map.js";
+export {shiftX} from "./transforms/shift.js";
 export {window, windowX, windowY} from "./transforms/window.js";
 export {select, selectFirst, selectLast, selectMaxX, selectMaxY, selectMinX, selectMinY} from "./transforms/select.js";
 export {stackX, stackX1, stackX2, stackY, stackY1, stackY2} from "./transforms/stack.js";

src/marks/axis.js

@@ -3,8 +3,9 @@ import {formatDefault} from "../format.js";
 import {marks} from "../mark.js";
 import {radians} from "../math.js";
 import {arrayify, constant, identity, keyword, number, range, valueof} from "../options.js";
-import {isIterable, isNoneish, isTemporal, isInterval, orderof} from "../options.js";
+import {isIterable, isNoneish, isTemporal, isInterval} from "../options.js";
 import {maybeColorChannel, maybeNumberChannel, maybeRangeInterval} from "../options.js";
+import {inferScaleOrder} from "../scales.js";
 import {offset} from "../style.js";
 import {generalizeTimeInterval, inferTimeFormat, intervalDuration} from "../time.js";
 import {initializer} from "../transforms/basic.js";
@@ -686,13 +687,6 @@ function inferFontVariant(scale) {
   return scale.bandwidth && !scale.interval ? undefined : "tabular-nums";
 }
 
-// Determines whether the scale points in the “positive” (right or down) or
-// “negative” (left or up) direction; if the scale order cannot be determined,
-// returns NaN; used to assign an appropriate label arrow.
-function inferScaleOrder(scale) {
-  return Math.sign(orderof(scale.domain())) * Math.sign(orderof(scale.range()));
-}
-
 // Takes the scale label, and if this is not an ordinal scale and the label was
 // inferred from an associated channel, adds an orientation-appropriate arrow.
 function formatAxisLabel(k, scale, {anchor, label = scale.label, labelAnchor, labelArrow} = {}) {

src/marks/difference.d.ts

@@ -0,0 +1,86 @@
+import type {ChannelValue, ChannelValueSpec} from "../channel.js";
+import type {CurveOptions} from "../curve.js";
+import type {Data, MarkOptions, RenderableMark} from "../mark.js";
+
+/** Options for the difference mark. */
+export interface DifferenceOptions extends MarkOptions, CurveOptions {
+  /**
+   * The comparison horizontal position channel, typically bound to the *x*
+   * scale; if not specified, **x** is used.
+   */
+  x1?: ChannelValueSpec;
+
+  /**
+   * The primary horizontal position channel, typically bound to the *x* scale;
+   * if not specified, **x1** is used.
+   */
+  x2?: ChannelValueSpec;
+
+  /** The horizontal position channel, typically bound to the *x* scale. */
+  x?: ChannelValueSpec;
+
+  /**
+   * The comparison vertical position channel, typically bound to the *y* scale;
+   * if not specified, **y** is used. For differenceY, defaults to zero if only
+   * one *x* and *y* channel is specified.
+   */
+  y1?: ChannelValueSpec;
+
+  /**
+   * The primary vertical position channel, typically bound to the *y* scale;
+   * if not specified, **y1** is used.
+   */
+  y2?: ChannelValueSpec;
+
+  /** The vertical position channel, typically bound to the *y* scale. */
+  y?: ChannelValueSpec;
+
+  /**
+   * The fill color when the primary value is greater than the secondary value;
+   * defaults to green.
+   */
+  positiveFill?: ChannelValueSpec;
+
+  /**
+   * The fill color when the primary value is less than the secondary value;
+   * defaults to blue.
+   */
+  negativeFill?: ChannelValueSpec;
+
+  /** The fill opacity; defaults to 1. */
+  fillOpacity?: number;
+
+  /**
+   * The fill opacity when the primary value is greater than the secondary
+   * value; defaults to **fillOpacity**.
+   */
+  positiveFillOpacity?: number;
+
+  /**
+   * The fill opacity when the primary value is less than the secondary value;
+   * defaults to **fillOpacity**.
+   */
+  negativeFillOpacity?: number;
+
+  /**
+   * An optional ordinal channel for grouping data into series to be drawn as
+   * separate areas; defaults to **stroke**, if a channel.
+   */
+  z?: ChannelValue;
+}
+
+/**
+ * Returns a new vertical difference mark for the given the specified *data* and
+ * *options*, as in a time-series chart where time goes right→ (or ←left).
+ *
+ * The mark is a composite of a positive area, negative area, and line. The
+ * positive area extends from the bottom of the frame to the line, and is
+ * clipped by the area extending from the comparison to the top of the frame.
+ * The negative area conversely extends from the top of the frame to the line,
+ * and is clipped by the area extending from the comparison to the bottom of the
+ * frame.
+ */
+export function differenceY(data?: Data, options?: DifferenceOptions): Difference;
+
+/** The difference mark. */
+export class Difference extends RenderableMark {}