RFC: Tracking feature state for interactivity

[ansis]

Motivation

#6020 provides a way to update the appearance of features. But how do you know if a feature is being hovered on? if is being activated? if it's currently selected? if it was selected at some point? We need to have an approach to tracking state.

we should make all of these possible somehow:

• style the topmost feature on hover
• style the topmost feature on hover, ignoring some layers
• style all the features under the mouse on hover
• style a polygon label and fill when the fill is hovered on
• style the polygon fill, but not label, when the fill is hovered on
• style a feature on activation (:active)
• style a selected feature (only one allowed)
• style all the selected features (multiple allowed)
• style which have been selected at some point

Design Alternatives

Option 1: track state internally

We could define several possible feature interactivity states including hover and active. We'd listen to mousemove events internally, query for the features under the mouse and update state. Users would access the state using a data-driven expression:

"fill-color": ["case",
    ["state": "hover"],
    "#ff0000",
    "#000000"]

Open questions:

• does only the topmost feature get hovered on?
• or all features under the mouse?
• or the topmost excluding layers with no event listeners? with some "pointer-events": "none" equivalent?
• could this handle selection?
• would this highlight the feature in this layer only (fill but not label)? or all layers (fill and label)? could you choose?
• besides hover and active, what would we want to support? clicked? focus? target?

Advantages:

• completely declarative
• makes basic cases easy to add
• handles both mouse and touch events properly automatically
• could potentially have special treatment in Studio

Disadvantages:

• hard/impossible to cover all the use cases described at the top
• can't support things like various possible approaches to selection without expanding the spec a lot
• will never support everything users want to do

Option 2: provide the tools to let users track state themselves

We would have to:

• provide event listeners for individual features!
• provide a way to update feature properties (RFC: Updating the appearance of individual features #6020)
  We would maybe have to:
• provide some way of preventing other event listeners from being called (stopPropagation)
• support data joining

"fill-color": ["case",
    ["get": "isSelected"],
    "#ff0000",
    "#000000"]

multi-selection

map.on('click', 'layerid', featureSelector, (e) => {
    e.feature.properties.isSelected = !e.feature.properties.isSelected;
    e.feature.updateProperties();
});

single-selection

let currentlySelected = null;

map.on('click', 'layerid', featureSelector, (e) => {
    // toggle selection
    e.feature.properties.isSelected = !e.feature.properties.isSelected;
    
    if (currentlySelected) {
        // unselect previously selected feature
        currentlySelected.properties.isSelected = false;
        currentlySelected.updateProperties();
        currentlySelected = null;
    }

    if (e.feature.properties.isSelected) {
        currentlySelected = e.feature;
    }
});

Open questions:

• what order are events called? is this backwards compatible?
• what information are events called with?
• do we want to create "RenderedFeature" objects that are provided by events?
• how are listeners assigned to individual features? with selectors?
• can events be cancelled?
• what does the api for updating feature properties look like?

Advantages:

• can support a huge amount of possibilities with a smaller api

Disadvantages:

• not declarative
• requires more effort from the user
• handling multiple intertwined interactions could get messy
• need to handle desktop/mobile differences yourself
• user bugs can leave you with corrupted feature state

Design

I think we need to implement Option 2 to cover all the cases we want to cover. But I think it makes sense to also implement Option 1 to provide a solid foundation for basic use cases. So... both?

Concepts

Option 1 introduces new concepts like hover and active which would need to be documented. Option 2 would introduce more complexity around events.

What existing precedents support the new concepts? Pseudo-classes from the web are a precedent for Option 1. Interactivity event listeners in Option 2 have precedents both on the web and mobile.

@kkaefer @asheemmamoowala @mollymerp @anandthakker @mourner @lucaswoj

[lucaswoj]

I like the idea of doing both, implementing Option 1 using the primitives defined by Option 2.

If we implement only Option 2, I suspect we will disappoint the majority of users, who just want a good-enough out-of-the-box solution for feature interactivity.

If we implement only Option 1, I suspect we will disappoint our power users, who may run into hard limitations with our feature interactivity system.

I wonder if there isn't an Option 3 that's fully declarative and supports all the listed use cases...

[kkaefer]

/cc @mapbox/studio

[ansis]

Capturing our discussion from last week:

It seems like we're leaning towards doing a combination of 1 and 2:

• 1 is needed so that the common cases are dead simple and code-less
• 2 is needed to support more complex use cases

On 1)

Would states would we track? (hover? active? selected?)
What are the details of when they are set and not?
How does this work or not work on mobile?
Which layers can receive events? Is there a way to explicitly enable/disable this?

We talked about how event though there are a lot of possible types of selectors (checkbox, radio, etc) a simple "zero or one features are selected" would cover a lot of map use cases and might be worth adding.

@mollymerp is looking into this ^

On 2)
We agreed that generally this is needed, but a lot of the questions remain open:
Is there really a clear advantage to per-feature handlers over layer handlers? (I think so, but need to consider this more)
What is the event propagation flow? Can propagation be stopped?
Returning a a feature object that wraps the id vs returning the id directly?

I'm looking into these ^

[mollymerp]

Below is my proposal for the state tracking work. Very interested in folks' feedback/questions/comments:

Goals:

1. Allow users to control features’ style properties in response to interaction events (like hover, touch, selected) without writing Javascript – can be fully declared in the map's stylesheet / configured in Mapbox Studio
2. Should make sense and work cross-platform
3. Should be more performant than current setFilter / setData approaches to styling features based on interactivity.

What states would we track?

• make $state a protected property name for “interactive enabled” features
• behave similarly to CSS pseudo-classes
• $state property value is tracked and mutated only internally (i.e. not available for runtime property updates)
• more complicated interaction-based style updates (i.e. checking a radio button on a DOM element outside the map) will require writing some event handlers in JS (option 2 in OP)
• possible values of $state:
  • default / inactive — don’t actually think this is needed – can use $state: null/undefined or remove the $state key from a features properties entirely
  • hover — triggered by mouseover/mouseleave on web, “short” touch? for mobile
    • consider renaming to something that works cross-platform (possibly highlight)
  • active/selected — triggered by click on web, long(er) touch on mobile, reset when another feature is selected
  • visited — not sure on this one, because multiple features could have this state in a given layer and it might add more complexity than we’re comfortable with

What are the details of when they are set and not?

• only zero or one feature per layer can have $state: hover at any given time
  • current hover feature is reset to inactive / default / null when a new feature is hovered over_or_ the mouse/touch is on an area with no features
• only zero or one feature per layer can have $state: active at any given time
  • previous active feature is reset to inactive / default / null when a new feature is clicked/touched

Which layers can receive events? Is there a way to explicitly enable/disable this?

options:

• automatically enabled on a layer by adding an expression that accesses $state property
• add a property to the layer style-spec (like interactive: true for example) — could add the ability to easily toggle interactivity at runtime?

[ansis]

make $state a protected property name for “interactive enabled” features

Would exposing this as an expression (["state"]) avoid the need to protect a property name?

possible values of $state

With CSS pseudo-classes it is possible for an element to be two things at once, for example both :hover and :visited. Do you think this makes sense or are you thinking it would be better for a feature to have only one at a time? I think the overlap might make sense

Which layers can receive events? Is there a way to explicitly enable/disable this?
options:
...

Which of these do you think we should do? would a combination make sense or would that be weird?

[mollymerp]

Would exposing this as an expression (["state"]) avoid the need to protect a property name?

I was thinking that if we want gl-js to have full control over $state it would make sense to have it protected (e.g. disallow / ignore user-defined feature $state properties) – if users want to programmatically control a state property they could do so but be responsible for managing their event listeners/data updates. But this is an unnecessary limitation 🤔 I was just thinking it could get messy if user-defined event listeners conflicted with internally managed gl-js handlers...

With CSS pseudo-classes it is possible for an element to be two things at once, for example both :hover and :visited. Do you think this makes sense or are you thinking it would be better for a feature to have only one at a time? I think the overlap might make sense

hmm yeah I didn't consider this... do you think it makes sense to store state in multiple property keys then? (e.g. hover: true etc)

Which of these do you think we should do? would a combination make sense or would that be weird?

well option 2 doesn't really make sense by itself – so maybe a combination would make sense for ease-of-use. instead of disabling a hover effect by completely overwriting each interactive property's style rules to remove references to tracked state properties, being able to have a one-liner to enable/disable all interactive properties seems useful

[asheemmamoowala]

Which layers can receive events? Is there a way to explicitly enable/disable this?

If the hover effects are enabled by explicit requests, does this prevent us from having two features with the same ID on different layers be in a given state at the same time? i.e Can I have a polygon and its label (using a point feature on a separate layer) be hovered together using the style-spec, or does this require using event handlers?

[anandthakker]

Would exposing this as an expression (["state"]) avoid the need to protect a property name?

In the same way that we moved away from $type in favor of ["geometry-type"], I think this is the route we should go here.

possible values of $state

Given @ansis 's point above about multiple states being possible simultaneously, what about having a separate boolean expression for each kind of state: ["highlighted"], ["active"], ["visited"], etc.? I'm not convinced that the more general state is adding value.

visited — not sure on this one, because multiple features could have this state in a given layer and it might add more complexity than we’re comfortable with

Yeah, I'm hesitant about this one, too: this feels like a more complicated and less well-defined type of state. Is a feature always visited once it's been clicked? "Visited" since when? Can the user programmatically "reset" the map's set of visited features? Etc. I think we can come back to this in the future if we start to see a clear set of use cases for which a precise design could emerge.

[mollymerp]

@asheemmamoowala

If the hover effects are enabled by explicit requests,

don't understand this bit but,

Can I have a polygon and its label (using a point feature on a separate layer) be hovered together using the style-spec, or does this require using event handlers?

yes – I was assuming so – the symbol layer would just have an expression for whatever paint property would be changed on hover as well.

[mollymerp]

In the same way that we moved away from $type in favor of ["geometry-type"], I think this is the route we should go here.

@anandthakker is the main difference here that geometry-type is a feature-level property and not nested within properties?

[jfirebaugh]

Let's keep state and properties fully distinct and independent, both in how they are accessed in expressions, and how they are stored internally. (AKA follow the React model.)

[anandthakker]

is the main difference here that geometry-type is a feature-level property and not nested within properties?

The difference I was referring to was between having a 'special' feature property $state, accessed like ["get", "$state"] (or ["get", "$highlighted"]), versus having a separate expression like ["state"] (or ["highlighted"]).

[1ec5]

• hover — triggered by mouseover/mouseleave on web, “short” touch? for mobile
  • consider renaming to something that works cross-platform (possibly highlight)
• active/selected — triggered by click on web, long(er) touch on mobile, reset when another feature is selected

I get that hover is highly desirable for the Web, but it’s worth noting the fraught history of hover on the mobile Web, since there’s no concept of hovering in native applications in the vast majority of mobile form factors (styluses notwithstanding). For example, if a webpage applies a :hover ruleset to a link on a webpage, WebKit on iOS will require two taps, one to “hover” the link and another to follow it.

“Hover” is effectively an “active” state. This behavior allows whatever was associated with :hover to take effect, avoiding information loss, but it also frustrates the user by making the interface feel less responsive. Even “active” is questionable on mobile devices. For example, iOS only has a concept of keyboard focus when a keyboard is attached. Otherwise, in WebKit, “active” is either omitted in favor of “hover” or appears only briefly – similar to mousedown on desktops but unlike what tabbing to an element does on a desktop.

Our existing hover examples illustrate the problem well:

• “Create a hover effect” and “Highlight features containing similar data” both highlight the feature in order to make it appear clickable and delineate the hit target. If the hover effect were to be ignored on a mobile device, the map would be perhaps less usable (because it wouldn’t be so clear that you could tap on a feature) but still functional.
• “Display a popup on hover” shows hidden content when hovering over a feature. If the hover effect were to be ignored on a mobile device, there would be a loss of information (because you’d never see the popup contents).

At a minimum, the style specification would need a way to distinguish between these intentions. Even if we rename “hover” to something less obviously desktop-centric, we’d still need some sort of media query syntax so that developers can choose whether to associate this state with an initial tap on mobile devices. When designing for a touch-enabled device, it’s important to make tappable features always appear tappable from the moment they appear, but that means the effect has to be more subtle than a typical highlight effect. For example, on a desktop, “Get features under the mouse pointer” might highlight a feature only on hover, whereas on a phone, it might subtly outline the feature at all times, since it isn’t possible to scrub a cursor over the map to uncover hidden hit targets.

If we go down the route of media queries, why not rework this proposal into a more general framework for state tracking? The style specification would allow arbitrarily named states, and it would be up to the developer to associate these states with predefined events at runtime via an API. The set of events could vary by platform. This would keep the style JSON file format platform-agnostic while also making it possible for mobile platforms to associate states with mobile-specific gestures like force-touch.

/ref #200 (comment)

[ansis]

Let's keep state and properties fully distinct and independent, both in how they are accessed in expressions, and how they are stored internally. (AKA follow the React model.)

The link explains that in React "Props are set by the parent and they are fixed throughout the lifetime of a component. For data that is going to change, we have to use state."

@jfirebaugh Do you think we should disallow updating feature properties completely? If a user wants to implement a custom myselected state/prop would this go in the state namespace? Or could it be a property? I'm seeing two possibilities:

• use the state namespace for internally tracked state and properties for everything the user provides. Allow updates for both.
• use the state namespace for both internally tracked state and updatable user-defined state. Disallow property updates (except with full setData? if we allow updates with setData is there a meaningful distinction between state and props?)

[jfirebaugh]

It would be nice to be able to update feature properties piecemeal, but I see that as a distinct feature, unrelated to feature interactivity (mapbox/geojson-vt#26).

I think there should be an independent API for updating state, under application control. setData should not set any state. (Except that it might reset it to the empty state, unless we come up with a way to track feature identity across calls to setData.) Nor should vector tiles or GeoJSON be allowed to carry any state.

I'm not sure what you mean by "internally tracked state". hover/active/selected? I think @1ec5 made a convincing case that we should not attempt to track these states automatically.

One major reason to keep properties and state fully separated in this way is to avoid introducing for source data the gnarly refresh/merge challenge we've hit with styles in #4225 (comment).

[ansis]

It would be nice to be able to update feature properties piecemeal, but I see that as a distinct feature, unrelated to feature interactivity

Thanks, would there be functional differences between state and properties or would it be mostly just convention? Would state be supported by vectortile and geojson sources while property updates would be geojson-only?

I'm not sure what you mean by "internally tracked state". hover/active/selected? I think @1ec5 made a convincing case that we should not attempt to track these states automatically.

Yep, that's what I meant. State that could potentially be set automatically without the user adding any code

@1ec5 we talked about this a bit a week ago, but could you expand on your thoughts here? I think I remember you saying that these kinds of states might sometimes be platform specific, but that having them could still be useful? and that the iOS sdk has a bit of precedent for this with the built-in annotation selection tracking?

[anandthakker]

I think @1ec5 made a convincing case that we should not attempt to track these states automatically

I think @1ec5 's argument suggests that we shouldn't specify particular interaction states in the spec, not necessarily that we shouldn't track them automatically (possibly in platform-specific ways)

[jfirebaugh]

I'm not sure I understand the distinction. If we track some state automatically, we'll need to document that behavior, including the platform-specific nuances. Is that not equivalent to "specifying" it?

[asheemmamoowala]

We'd necessarily want to track some state internally so that a style can be used across platforms without having to write code for each.

[anandthakker]

If we track some state automatically, we'll need to document that behavior, including the platform-specific nuances. Is that not equivalent to "specifying" it?

I don't think it's exactly equivalent. One way or another a user writing their style sheet has to refer somewhere to know that they can use the word "hover" in fill-color: ["case", ["state"], "hover", "red", "blue"]. Putting hover in the style spec is problematic because it's desktop-centric, but having each SDK document the set of state values it provides seems similar to having each SDK document the types of events it supports.

The alternative of having the SDKs only provide an API for updating state is appealing at one level, but I think part of the problem we're trying to solve here is that it feels like it more lines of code than it should to "just" have features be styled differently on hover/tap/click. To what extent would the lower-level state update API deliver on that problem?

[asheemmamoowala]

I think there should be an independent API for updating state, under application control.

It sounds like there's agreement on having an arbitrary list of named states, with some being predefined (and implemented) for each platform.

Applications should be responsible for enforcing any rules related to these states. For instance, internal mouse event handlers would be responsible for the following hover behavior:

only zero or one feature per layer can have $state: hover at any given time

what about having a separate boolean expression for each kind of state: ["highlighted"], ["active"], ["visited"], etc.? I'm not convinced that the more general state is adding value.

Whether or not we allow externally defined states, the need for different states per-platform makes it hard to support a separate expression for each kind of state. If features could be in multiple states at the same time, then the $state property would need to be treated as an array, which might help with expressing expressions 😁.

[1ec5]

I think I remember you saying that these kinds of states might sometimes be platform specific, but that having them could still be useful? and that the iOS sdk has a bit of precedent for this with the built-in annotation selection tracking?

Yes, there is some precedent for an SDK tracking state: the iOS and macOS SDKs track which annotation is currently selected. Selection means that the annotation dons its selected appearance and any associated callout (popup) is shown. However, that’s the extent of it; the other built-in states proposed above get into behaviors that would differ from platform to platform, which is not a problem for selection.

The alternative of having the SDKs only provide an API for updating state is appealing at one level, but I think part of the problem we're trying to solve here is that it feels like it more lines of code than it should to "just" have features be styled differently on hover/tap/click. To what extent would the lower-level state update API deliver on that problem?

Associating a particular state with a particular event handler could be a one-liner on each platform, no? At any rate, it would be less code and hopefully more performant than setting up a gesture recognizer/mouse event listener and querying visible features every time it fires.

If the style specification allows a library to reserve certain states to be associated with platform-specific behaviors (as with hover), then I think my examples above show that we’d need media query expressions to go with that. And media queries won’t solve the problem that a designer in Studio, on the desktop Web, uses states in ways that only make sense on the desktop Web.

[anandthakker]

Associating a particular state with a particular event handler could be a one-liner on each platform, no?

Hm, yeah I suppose so (maybe a two-liner for hover -- mouseenter / mouseleave), as long as the event listening API and state-updating API were designed with this in mind.

[lucaswoj]

Are there any downsides to

1. defining canonical set of feature states (i.e. hovered, pressed, selected, ...)
2. exposing them all in Studio
3. allowing each platform to support a subset of those states?

This would allow users to design feature interactivity in Studio (which I understand to be a major design goal) and each platform to behave idiomatically.

• web: hovered, pressed, selected
• iOS: pressed, selected
• Android: pressed, selected

[ansis]

I'm splitting off my thoughts on per-feature event listeners into a separate issue #6215 to avoid breaking up the "what is feature state" conversation here.

[asheemmamoowala]

Bringing over from #6020(comment)

Setting state

State should be tracked independently of feature data and assigned per source. This would look like:

//Set a feature_id to a named state
map.setState("source", "state", feature_id);

//Clear state
map.setState("source", "state");

//Set one or more feature Ids to a named state
map.setState("source", "state", [feature_id]);

States are tracked per source in the SourceCache where they can be applied to tiles when preparing them for upload before every frame.

Question: Is there a need for an API to un-set a named state on a single feature or set of features , while preserving other features in that state?

Using state

State can be referenced in expressions as part of layer paint properties

// Increase opacity for features in the 'highlight' state
"fill-opacity": [ "case", ["state", "highlight"], ["number", 0.9], ["number", 0.5]] 

or queried through an API:

map.getState("state");
// Where the returned object looks like:
{ 
 "source_A" : { "highlight": [1234, ...], "dragging" : [...], ... } ,
 "source_B": { "visited" : ["foo", "bar"] }
}

[1ec5]

Are there any downsides to

1. defining canonical set of feature states (i.e. hovered, pressed, selected, ...)
2. exposing them all in Studio
3. allowing each platform to support a subset of those states?

This would be reminiscent of how CSS works today: there are certain desktop-centric pseudoclasses built into the language, Web design tools expose them all, and each platform figures out how to map them in appropriate ways. Unfortunately, this approach disadvantages mobile platforms, often making it inconvenient or impossible to access content on a multitouch device. #6021 (comment) demonstrates that interactive map features would tend to suffer the same incompatibilities between desktop and mobile devices.

One mitigating factor is that CSS has any-pointer and any-hover media queries (and proprietary predecessors like -moz-touch-enabled), which at least in theory allows developers to remap hovered or pressed behaviors to more touch-friendly affordances. It’s unclear to me how commonly these media queries are used.

If we were to introduce media query expressions into the style specification, we’d have to coerce designers to take advantage of them. For example, Studio could disallow hover state usage outside a media query expression when the style declares compatibility with the mobile SDKs. Otherwise, I don’t see how a GL JS–powered WYSIWYG environment could possibly encourage designers to give due consideration to mobile users.

As I see it, the choice is essentially to make developers bind states to events:

• Once for the current platform, at runtime, globally for all style properties
• Once for each platform, at design time, on each individual style property using media query expressions

The first option seems to me like it would ultimately involve less code (if you count expressions as code). I don’t know if there’s been any thought towards how Studio would simulate feature interactivity, but I think the answer would look very different depending on which option we go with.