Implement Glimmer Engine

[wycats]

Glimmer is the next-generation rendering engine for Ember.js. Built on top of HTMLBars, streams, and an entirely rewritten view layer, Glimmer offers the best performance of any of the major JavaScript frameworks while remaining backwards compatible with Ember 1.x apps.

---

The Virtual DOM showed the world that render-time diffing delivers awesome performance by minimizing DOM updates and keeping unchanged elements stable. But like any abstraction, it comes with a cost. In this case, constructing the virtual DOM to compare requires at least some subset of components do a full re-render, and building virtual DOM nodes for static areas of your markup that will never change for every change requires many more allocations and increases GC pressure.

Glimmer analyzes templates at compile time, relying on the fact that Handlebars' declarative syntax clearly differentiates between static areas, which make up the majority of a template, and the dynamic areas that can change. Instead of invoking a render() method and diffing the result, Glimmer only walks the (much smaller) existing dynamic tree.

---

Handlebars' declarative syntax means that from the perspective of an app, the template is being "re-rendered every time", but we can take advantage of static knowledge to reduce the work that we need to do. User code, like helpers and computed properties, are executed during the walk, but only to get (primitive) values which can be === compared, not to build up a new tree. In other words, the programming model is equivalent to "render every time", but we take advantage of the declarative nature of Ember's APIs to reduce work.

Internally, instead of creating a virtual DOM, Glimmer builds a tree of streams, each stream pointing to a node in the DOM. On initial render, we track the last value (always a primitive) that we inserted into the DOM. The streams are not exposed to application code; Glimmer executes user code when necessary and pushes the new value into the stream.

When re-rendering, we walk the tree and flush the streams. If the primitive value produced by the stream has not changed, we do nothing. If it has changed, we write the change to the DOM. Like Virtual DOM approaches, this is a "write-only" algorithm.

One of the benefits of this approach is that we can naturally fuse the Ember/Polymer model of observing individual properties with the React model of explicitly re-rendering areas of the template. No matter what happens, the process of revalidation walks the tree, looking for dirty nodes, and revalidates any dirty nodes it finds.

The only difference is how much of dynamic tree is marked dirty before revalidation begins. Also, because observation can only dirty dynamic nodes (and schedule a top-down revalidation), multiple observers and re-renders that occur during a single run-loop still only produce a single batch of DOM updates.

In essence, the biggest difference between Glimmer and traditional virtual DOM approaches is that we diff values, not DOM. This approach not only lets us avoid the algorithmic complexity of tree diffing (instead, we just === check two values), it also lets us avoid doing many render-time allocations.

Slides from the talk at EmberConf

Work List

(this looks like more things than it actually is)

• merge subscription change into our branch
• linkRenderNode can mark stability
• Ember should return true from linkRenderNode
• make sure all child streams correctly disconnect from parents
• add stream pruning logic that correctly prunes ancestors
• add scope teardown hook to HTMLBars (recursive upon clear or remove)
  • this is also responsible for view layer teardown
• Propagate correct visitor through descendent hooks (so dirty checks can be avoided for a subtree)
• Do not dirty check render nodes that were just created (fixes regression)
• Add "classify" hook to decide which hook content et al should delegate to
• Make sure that #view is idempotent
• Components and views should create a new scope frame
  • Implement custom scope in Ember (so that ambient view, controller, etc. can be stored)
  • Eliminate *component* hacks
• Lifecycle hooks on the renderer
• Move code duplicated between #view, #component and Ember.View into the Renderer
• Support reflecting attributes onto angle-bracket components
• Pass mutable bindings as "mutator" objects
• Support foo={{action "bar"}}
• Legacy support: Implement unknownProperty/setUnknownProperty to work with mutator objects
  • Deprecate setting non-mut attributes, and support (mut foo) sexprs in legacy Handlebars syntax
• Investigate executing teardown callbacks at a "quiescent" state ("vent plasma")
  • Make sure to consider the rpflo stress test situation
• Implement a zip and map abstraction for streams to clean up ad hoc stream creation
• Add tests for memory leaks
• Make sure that custom render functions that mutate the buffer continue to work
• options.template should not exist if there is no template
• feature flag all the things
• Handlebars helper compat
• Template function that returns a string (legacy)
• Context shifting #each
• collection helper
• {{input}}
  • TextField
  • Checkbox
  • TextArea
• Ember.Select
• unbound
  • non-block form
  • block form (partially done, but not completely)
• #view and view helper
• #view and view helper with back-compat edge cases
• #with helper
• with helper with back-compat edge-cases
• yield without a block
• {{render}}
• ContainerView
• Top-level view APIs
  • createElement
  • appendTo
  • replaceIn
  • destroyElement
• isVisible

[wycats]

React Hook	Ember Hook	Server?	Initial Render?	Rerender?	Purpose
componentWillMount	init	Yes	Yes	No	Set initial component state without triggering re-render
componentDidMount	didInsertElement	No	Yes	No	Provides opportunity for manual DOM manipulation
componentWillReceiveProps	willReceiveAttrs	No	No	Yes	React to changes in component attributes, so that setState can be invoked before render
shouldComponentUpdate	Maybe N/A	No	No	Yes	Gives a component an opportunity to reject downstream revalidation
componentWillUpdate	willUpdate	No	No	Yes	Invoked before a template is re-rendered to give the component an opportunity to inspect the DOM before updates have been applied (example)
componentDidUpdate	didUpdate	No	No	Yes	Invoked after a template is re-rendered to give the component an opportunity to update the DOM (example)
componentWillUnmount	willDestroyElement	No	No	Yes	The inverse of componentDidMount/didInsertElement; clean up anything set up in that hook
N/A	willRender	No	Yes	Yes	In Ember, executed both after init and after willUpdate*
N/A	didRender	No	Yes	Yes	In Ember, executed both after didInsertElement and didUpdate*

* These hooks can be used in cases where the setup for initial render and subsequent re-renders is idempotent (e.g. $().addClass) instead of duplicating the logic in both places. In most cases, it is better to try to make these hooks idempotent, in keeping with the spirit of "re-render from scratch every time".

[wycats]

The above comment is an attempt to classify the hooks in React and Ember so that we can make sure that the new Ember components implement at least the hooks that React folks have identified as being important in the programming model.

[rwjblue]

All new API's will also need to be feature flagged.

[wycats]

@rwjblue yessir

[Linicks]

The init hook name is a little unclear compared to componentWillMount. Could it be named initElelement, or something similar ?

[stefanpenner]

The init hook name is a little unclear compared to componentWillMount. Could it be named initElelement, or something similar ?

I would prefer leaving it as init, as it clearly describes what it handles, initialization of the current entity.

That entity just happens to be a component. initFoo as a pattern feels like an exercise in over specification. Especially when you consider the rest of the collaborators all having a unique initFoo method, initModel initRoute etc.

Also worth noting, as we continue to align ourselves with >= ES2015 init will likely become constructor

[stefanpenner]

@wycats @tomdale good job guys :)

[alexspeller]

😍

[alexdiliberto]

This is amazing 👍 👍 👍

[Linicks]

The problem with just using init is that init is used in so many places in the tech world. In an Ember only world it would be clear and concise, but in an ever increasing complex tech stack it's nice to have that extra bit of clarity. I think that's why FB chose an even more verbose naming scheme than my proposal. I have had the chance/misfortune of maintaining allot of legacy software, and appreciate when the code is self describeing as practical. This is really a preference in either direction, but after being in this game for so many years I'm getting a little opionated :) Either way this is exciting stuff, and look forward to seeing this come together in Ember.

[ebryn]

@Linicks if you look at their ES6 syntax, you'll see they're utilizing the constructor to initialize this.state

[fivetanley]

What is a "ShadowRoot"?

[gonvaled]

This being a "DOM write-only" implementation, how does Ember deal with changes to the DOM done by third-party components, like jQuery based changes, which are not controlled by Ember?

[davidlormor]

"the Dynamic Tree only includes nodes that may have changed (...) and never needs to compare static content that will never change"...sounds fast! 🚀 👍

[stefanpenner]

This being a "DOM write-only" implementation, how does Ember deal with changes to the DOM done by third-party components, like jQuery based changes, which are not controlled by Ember?

It doesn't and likely cannot support this. It would be impossible to know what those plugins are doing

[mgenev]

@stefanpenner that means we can no longer use bootstrap (jQiery dependent) in an ember app unless a bootstrap-for-ember version is written, right? Same goes for foundation etc...

[stefanpenner]

@stefanpenner that means we can no longer use bootstrap (jQiery dependent) in an ember app unless a bootstrap-for-ember version is written, right? Same goes for foundation etc...

No, this is no different then today. If you component is re-rendered your bootstrap and jquery code needs to re-run.

[wycats]

@stefanpenner @mgenev I think there's a little bit of confusion here.

From new-world Ember's perspective, as long as the element for a component still exists in the DOM, it doesn't matter where it has been moved, whether it has been decorated, etc.

There are a few restrictions on moving elements around, but they're very minor and would be easy to abstract. Stay tuned as we continue to make more progress.

[ef4]

why is {{#each people as |p|}} marked as legacy? Are we not keeping that syntax?

[rwjblue]

Updated title + description based on EmberConf presentation.

[toblender]

Minor correction: teack === track?

[morenoh149]

@toblender yeah. I thought he meant teach

[darkbaby123]

@wycats You mentioned a lot about diffing values to check if they're dirty. I'm curious what's the difference between Glimmer's diffing logic and Angular's dirty-checking.

As I know Angular dirty-checking also checks values (expressions) one by one and execute corresponding callbacks (mostly change doms). But:

1. Angular doesn't have run loop so it's can not batch update doms.
2. watchers need to be triggered by $apply() manually sometimes.

[drogus]

I'm curious what's the difference between Glimmer's diffing logic and Angular's dirty-checking.

These two diffing situations are not happening at the same level. Angular's diffing happens outside of a view layer in order to check if a template change is needed. So let's say that you have a user = { login: 'drogus' } object. And you use it in a template as <span class="login">{{user.login}}</span>. Now, in order to update the template automatically you need to know if a value of the user object was changed. In Angular it's done by dirty checking all of the properties used in templates. In Ember properties are observed, so it knows when a property is changed.

This PR and diffing described here happen in a view layer when a framework already knows that it needs to rerender something, but it needs to figure out what to rerender exactly. DOM is slow, so regenerating the entire template may be costly and here comes the diffing algorithm that React started with.

[bitinn]

One interesting side-effects of React (or other similar dom-diffing) approach is you can worry less about what's being updated in DOM (as oppose to update them manually on value change).

What sort of limit will value-diffing approach impose on DOM manipulation? Would you say this approach is closer to something like Riot than React?

Update: to me, it seems the key differences here are:

1. dom-diff approach allow user to define almost arbitrary dom update logics, then perform it through a full render.
2. value-diff approach capture those logic via template/helpers, and can minimize the need to go through static dom tree.
3. value-diff is conceptually closer to traditional template renderer, though Glimmer use batch dom update (what dom-diff commonly do) to improve performance.

Apologize if I am totally mistaken :)

[btecu]

Great work!

[Adriaaaaan]

Great work! although our app is currently very broken still I'm already seeing a huge improvement in performance in my list views. Now i just have a million deprecation warnings to fix

[wycats]

@Adriaaaaan That's great news! Please do report any notable breakage. We want things to be pretty clean by the time we ship 1.13 final :)

You might also want to run in prod mode to check out perf. Those deprecation warnings aren't free to generate ;)

[chrism]

Great Work!

Anyone using Liquid Fire should check that ember-animation/liquid-fire#276 has been resolved before updating as it is not yet glimmer compatible.

[jmurphyau]

@wycats any help you can provide on unbound helpers? A helper that produces a stream like value that can be invalidated causing the value in the template to update? Trying to get ember-get-helper to work with glimmer.. Simply returning a new child stream (e..g objectStream.get(key)) doesn't cause a new value to render in the template when that value changes.. Do I need to look into some of the hooks or somehow subscribe to the new stream?

[jmurphyau]

I can't seem to find that really detailed description of all the HTMLBars lifecycle hooks etc.. Do you have a link to that that you could provide?

[wycats]

@jmurphyau right now there is no public way for a helper to express dynamic dependencies, but that's a good idea.

Glimmer intentionally doesn't expose the internal stream machinery; it's changing quite a bit still, but it makes sense for a helper to be able to tell Ember when to invalidate it.

It also kind of makes sense for us to include this get helper in Ember itself honestly. It's a primitive, like {{component}}.

[wycats]

@jmurphyau as I said in the blog post, more info on the life cycle hooks is forthcoming :)

[Adriaaaaan]

@wycats Hmmm have looked into it more, I'm now seeing a rather significant performance regression. related to interacting with sub resources templates from a list. It seems that interacting with it causes the the whole tree to redraw making it unusable if there are a lot of rows per page. 1.11 is faster, its most notable when selecting the rows (the detail page is very laggy at appearing, the more rows in the list the longer it takes). Although it is clearly much faster at rendering (the page appears much faster) interacting with it is much worse as it seems to be doing more work everytime something changes. I'll see If I can make a simple jsbin

[EnglishCriminal]

[tiye]

Hi, you mentioned there's slide in EmberConf 2015. But is there a video?
I can't find it in the list "EmberConf 2015 by Confreaks"
https://www.youtube.com/playlist?list=PLE7tQUdRKcyacwiUPs0CjPYt6tJub4xXU

[igorT]

@jiyinyiyong it's the keynote, towards the end

[tiye]

@igorT I don't get it. Could you point it out in the video?

found it, thx https://youtu.be/o12-90Dm-Qs?t=3077

[blessanm86]

Stupid question. So there is no virtual DOM in ember right? Just value diffing to find which part of the DOM needs to be manipulated.

[axemclion]

Glimmer is pretty fast - almost 25% faster !! Ran some tests and results here - http://blog.nparashuram.com/2015/05/performance-boost-in-emberjs-from.html

[jdurand]

@Blessenm instead of diffing DOM it diffs a tree of possibly mutable states that are bound to the real DOM. It doesn't need to maintain a virtual DOM because it has a deeper knowledge of your apps state.

[blessanm86]

@jdurand thanks for the clarification.

[ballPointPenguin]

[jmurphyau]

@wycats someone has asked a relatively simple question on stackoverflow and in slack - 'what is a stream'.. I answered as best I could but wanted to bring it up here just so I could get a better understanding - what is a stream? Did the name 'stream' come from anywhere/anything in particular?

I mean - I do know what it is (I think) - just not sure how to best describe it..

[matthewp]

Is there somewhere that explains how Glimmer works in more detail, with links to code maybe? Given how many commits are included in this PR it's hard to figure out how the details are implemented in practice. For example you say:

On initial render, we track the last value (always a primitive) that we inserted into the DOM.

But I can't find in the code where this happens. And I don't know what you mean by a primitive value. I assume you mean that you track attribute string values and text node's string nodeValues but am having a difficult time finding where this happens in the code.

Is there maybe one or two commits that contain most of the work that I can look at?

[ef4]

@matthewp Much of the infrastructure is actually in https://github.com/tildeio/htmlbars

There's not really a single commit to point to -- the work was extensive and many people put in hundreds of commits.