I2I: "protocol adapters" for client side data massaging. #26474

samouri · 2020-01-24T17:05:03Z

summary
The objective is to solve the protocol adapters use case. We want to allow developers to perform arbitrary client side transformations to an api response before handing it to an amp component, essentially trading UX for DX.

Since amp-script is already capable of performing xhr requests, data manipulations, and can call AMP.setState, it is close to being fully capable of servicing this use-case.

assumptions

I2I: Allow amp-list to render from amp-state initially #26473 lands
We are willing to make initialization from amp-state wait a window of time for data to appear.

basic amp-list w/ refresh example

<html>
  <head>...</head>
  <body>
    <amp-script script="animals-script">
      <button id="refresh-button"> Refresh data </button>
      <amp-list src="amp-state:animals">...</amp-list>
    </amp-script>
    <script type="text/plain" target="amp-script" id="animals-script">
       function fetchAnimalData() {
         fetch('url-with-data.json')
           .then(res => res.json())
           .then(transform)
           .then(array => AMP.setState({ animals: { items: array } })); 
       }

       document.getElementById('refresh-button').addEventListener('click', fetchAnimalData);
       fetchAnimalData(); // first load
    </script>
  </body>
</html>

motivation

Many frontend developers do not have control over their backend APIs. In order to play nicely with amp-list, they create proxy servers. This is a poor DX for something that could easily be solved via a sprinkling of JS.
It is currently impossible to take advantage of alternative data sources, i.e. websockets or rtc data connections.

caveats

amp-script has a nontrivial initialization cost. If after we perform measurements we deem it too high to release this feature, there are opportunities we can pursue for speeding up init, especially for this use case.
- inlining the worker js into a string variable.
- creating a lightweight version of worker-dom (without dom access, but with the ability to perform xhr and call AMP.setState).
We'll need further work to allow for reset-on-refresh behavior (solvable by exposing amp actions within amp-script).

The text was updated successfully, but these errors were encountered:

samouri · 2020-01-30T23:07:25Z

There is one thing about this proposal thats been bugging me. The inversion of control from amp-list to amp-script means that much of the edge-case handling inside of amp-list has to be recreated in userland.

Edge cases:

When hitting a refresh button, they'd need to call an action we expose on amp-list in order to trigger reset-on-refresh
infinite scrolling support would require the invention of convoluted new APIs to support.
Error cases? In the case of a failed fetch, how should the amp-script communicate that to the amp-list?

We might be better served by allowing the src of amp-list to reference an amp-script function. That way amp-list still gets to decide when to call the fetch and what to do about the result.

For the same use case above:

<html>
  <head>...</head>
  <body>
    <button on="click:amp-list.refresh()"> Refresh data </button>
    <amp-list src="fn:animals-script.fetchAnimalData">...</amp-list>
    <script type="text/plain" target="amp-script" id="animals-script">
       function fetchAnimalData() {
         return fetch('url-with-data.json')
           .then(res => res.json())
           .then(data => { 
             let next = data.next ? `http://url-with-data-${data.next}` : undefined;
             return { items: data.animals, next };
           });
       }
    </script>
  </body>
</html>

Note: no use case should ever need to pass data into these functions since they can call AMP.getState().

Also note: this solution for protocol adapters would potentially create a clean solution for #25684

morsssss · 2020-02-03T20:15:12Z

Your second example is what I was personally thinking about and hoping for:

<amp-list src="fn:animals-script.fetchAnimalData">

Generally speaking, I've been hoping that <amp-script> or a similar mechanism could simply host a collection of helper methods for both <amp-list> and <amp-bind>. I didn't even imagine that the components would need to be surrounded by <amp-script> - simply that we'd have a repository or class of these helper methods, living in a Worker.

So, devs would get to use all the features of <amp-list>, but just deploy a little JS to help them out.

If this works, you could get closer to my dream for #25684 - as you have pointed out!

/cc @alankent for his opinion

morsssss · 2020-02-03T20:16:53Z

In other words, we keep the simplicity of our interactive components, but we let developers inject a little logic of their own in nice, native JavaScript.

samouri · 2020-02-03T22:00:09Z

primary question for design review

Should amp components or amp-script have control over the flow. What do web developers want more?
- Giving amp components control means all of their edge cases / extended options are usable for this use case and amp-script is used merely as a data source.
- Giving amp-script full control means we may need to expose more of the internals of components to be accessible by amp-script. Depending on their use case, developers would need to create their own success/error handlers, pagination, inf load, etc.

alankent · 2020-02-04T04:40:10Z

I am not sure I follow "should AMP components or amp-script have control of the flow?" So trying to rephrase my thoughts.

I think what Ben said expresses my thoughts too. If I need to use amp-script to massage the request and response, I don't want to do a major page redesign thinking about different control flows. AMP pages for me should be declarative (as much as possible anyway). So just because I need a protocol adapter, I should not start needing amp-script wrappers around parts of the page (if possible).

I don't understand all the refresh etc edge conditions. But I think about it more simply. Consider a product page that has 2 selectors for shirt size and color. Updating either one should cause the page to refresh. I imagine AMP state would be used to remember the current values. Clicking a button should update the AMP state to the new selection. If there was no protocol adapter, that would trigger an API call and page render. That is the goal - update state and amp-bind causes the amp-list to refresh.

The only extra part to the mix is to say that the update to amp-state instead of causing amp-list to go fetch a [src]=... API call, instead amp-script code will be run that does the XHR call. The JavaScript also converts the response into JSON. So I provide a little bit of logic that fits into the current normal page flow.

So reloading a page should add like what a page would do without amp-script (just using amp-list and [src]=. As soon as the code on an AMP page has to change too much, I think things might be heading the wrong direction. (It might be unavoidable, but keep thinking declarative - AMP state remembers the current state. You make changes to the state to trigger other side effects, like the DOM updating or API calls being made.)

mdmower · 2020-02-04T06:23:05Z

Should amp components or amp-script have control over the flow. What do web developers want more?

If I understand the question correctly, giving amp-script control over the page (AMP.setState) while in the middle of an amp-list data fetch sounds like a rats nest of race conditions. I'm strongly in favor of the simplest black box approach possible for this I2I. The data massage amp-script should have the ability to output JSON and otherwise have no ability to interact with the AMP document. I think the only features worth retaining in amp-script are the ability to inline a script and the script integrity validation (hash check).

samouri · 2020-02-04T17:48:17Z

@alankent, I'll write out the shirts example (reminds me of ampcamp!) through the lens of the two proposals.

(1) amp-script w/ most of the control

<html>
  <head>...</head>
  <body>
    <amp-script script="store-script">
      <button id="shirt-size-button"> rotate shirt size</button>
      <button id="shirt-color-button"> rotate shirt color </button>
      <amp-list src="amp-state:shirtData" [src]="shirtData">...</amp-list>
    </amp-script>
    <script type="text/plain" target="amp-script" id="store-script">
       function fetchShirtData() {
         const size = AMP.getState().selectedSize;
         const color = AMP.getState().selectedColor;
         fetch(`/api?size=${size}&color=${color}`)
           .then(res => res.json())
           .then(transform)
           .then(data => AMP.setState({ shirtData: data })); 
       }

       document.getElementById('shirt-size-button').addEventListener('click', () => {
         AMP.setState({selectedSize: ... }).then(fetchShirtData); 
       });
       document.getElementById('shirt-color-button').addEventListener('click', () => {
         AMP.setState({selectedColor: ... }).then(fetchShirtData); 
       });
       fetchShirtData(); // first load
    </script>
  </body>
</html>

(2) amp-list w/ most of the control

<html>
  <head>...</head>
  <body>
    <button id="shirt-size-button" on="tap:AMP.setState(...),items.refresh()"> rotate shirt size</button>
    <button id="shirt-color-button" on="tap:AMP.setState(...),items.refresh()"> rotate shirt color </button>
    <amp-list id="items" src="fn:store-script.fetchShirtData">...</amp-list>
    
    <amp-script script="store-script"></amp-script>
    <script type="text/plain" target="amp-script" id="store-script">
       function fetchShirtData() {
         const size = AMP.getState().selectedSize;
         const color = AMP.getState().selectedColor;
         return fetch(`/api?size=${size}&color=${color}`)
           .then(res => res.json())
           .then(transform)
       }
    </script>
  </body>
</html>

My take is that something more like (2) is significantly more preferable.

morsssss · 2020-02-04T19:29:08Z

I agree that (2) is more preferable... preferabler?

I would even go further. I imagined that AMP would perform the fetch() as usual, simply passing along the raw data to the user JavaScript. I realize this is something different than what you're imagining. Something like this:

<amp-list id="items" src="'/api/fetchShirtData?size=' + products.size + '&color=' + products.color" fn="store-script"> ... </amp-list>

<amp-script script="store-script"></amp-script>
<script type="text/plain" target="amp-script" id="store-script">
    function massageShirtData(rawData) {
         let transformedData = ... ;
         return transformedData;
    }
</script>

What do you think?

samouri · 2020-02-04T19:38:16Z

@morsssss: I'm okay with that idea, it definitely simplifies the 80% use-case. My reservations would be:

@mdmower has requested the ability to modify requests before they happen (i.e. adding a header or special encoding a parameter)
(less important): this would close off the possibility of using alternative data sources. e.g. websockets or even random js logic (without a network request).

alankent · 2020-02-04T19:49:04Z

Hi @samouri - no detailed comment from me just now, but I think you are understanding the issues correctly and heading in the right direction. I definitely want to add special headers etc (e.g. generate a SOAP request with special authentication headers, or generate 3 separate API requests and merge the results).

mdmower · 2020-02-04T20:41:58Z

@mdmower has requested the ability to modify requests before they happen (i.e. adding a header or special encoding a parameter)

I can't recall my requests exactly, but I suppose these suggestions depend on whether AMP creates the xhr or the worker. If the worker performs the xhr, you can leave it up to the publisher to encode url params as necessary and specify headers (notably, the accept header). If AMP performs the xhr, then yes, at a minimum, this feature should allow modifying the accept header.

samouri · 2020-02-05T22:40:08Z

The consensus in the design review was to pursue option (2) instead of option (1).

morsssss · 2020-02-06T05:56:11Z

hooray!

(@alankent , I'm now getting hives at the memory of making SOAP services)

samouri · 2020-08-21T16:47:17Z

Closing as this feature has launched in #29689

samouri added the INTENT TO IMPLEMENT Proposes implementation of a significant new feature. https://bit.ly/amp-contribute-code label Jan 24, 2020

samouri mentioned this issue Jan 24, 2020

I2I: Allow amp-list to have its json data sourced from amp-state #26009

Closed

jridgewell added the WG: runtime label Jan 24, 2020

samouri self-assigned this Jan 31, 2020

samouri mentioned this issue Feb 3, 2020

wg-runtime Status Update 2020-02-03 ampproject/wg-runtime#28

Closed

This was referenced Feb 3, 2020

I2I: Allow amp-list to render from amp-state initially #26473

Closed

Design Review 2019-02-05 21:00 UTC (Americas) #26348

Closed

samouri mentioned this issue Feb 10, 2020

✨ protocol adapters prototype (warning: do not look at the code) #26715

Closed

jridgewell added the P1: High Priority label Mar 6, 2020

samouri mentioned this issue Apr 23, 2020

Support callFunction API ampproject/worker-dom#850

Merged

3 tasks

dreamofabear mentioned this issue Jun 26, 2020

Create "protocol adapters" to allow client-side data massaging #25719

Closed

samouri mentioned this issue Jul 16, 2020

Introduce "nodom" binary ampproject/worker-dom#882

Merged

This was referenced Aug 5, 2020

🚮 amp-list: remove experiment 'protocol-adapters' #29688

Merged

amp-list: launch protocol adapters in canary and prod #29689

Merged

samouri closed this as completed Aug 21, 2020

samouri mentioned this issue Sep 30, 2020

I2I: Run 3p service scripts in AMP #30193

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I2I: "protocol adapters" for client side data massaging. #26474

I2I: "protocol adapters" for client side data massaging. #26474

samouri commented Jan 24, 2020 •

edited

Loading

samouri commented Jan 30, 2020 •

edited

Loading

morsssss commented Feb 3, 2020

morsssss commented Feb 3, 2020

samouri commented Feb 3, 2020 •

edited

Loading

alankent commented Feb 4, 2020

mdmower commented Feb 4, 2020

samouri commented Feb 4, 2020 •

edited

Loading

morsssss commented Feb 4, 2020

samouri commented Feb 4, 2020

alankent commented Feb 4, 2020

mdmower commented Feb 4, 2020

samouri commented Feb 5, 2020

morsssss commented Feb 6, 2020

samouri commented Aug 21, 2020

I2I: "protocol adapters" for client side data massaging. #26474

I2I: "protocol adapters" for client side data massaging. #26474

Comments

samouri commented Jan 24, 2020 • edited Loading

samouri commented Jan 30, 2020 • edited Loading

morsssss commented Feb 3, 2020

morsssss commented Feb 3, 2020

samouri commented Feb 3, 2020 • edited Loading

alankent commented Feb 4, 2020

mdmower commented Feb 4, 2020

samouri commented Feb 4, 2020 • edited Loading

morsssss commented Feb 4, 2020

samouri commented Feb 4, 2020

alankent commented Feb 4, 2020

mdmower commented Feb 4, 2020

samouri commented Feb 5, 2020

morsssss commented Feb 6, 2020

samouri commented Aug 21, 2020

samouri commented Jan 24, 2020 •

edited

Loading

samouri commented Jan 30, 2020 •

edited

Loading

samouri commented Feb 3, 2020 •

edited

Loading

samouri commented Feb 4, 2020 •

edited

Loading