Many improvements

Summary:
These got smashed together with some weird rebase snafu. They are pretty intertwined anyway so the value of
separate commits is minimal (e.g. separate commits would not revert cleanly anyway).

== [lists] better fill rate logging (previously D4907958)

After looking through some production data, I think this will address all the issues we're seeing. Now:

- Header/Footer getting no longer counted as blank.
- Avoid floating point for Scuba.
- Compare actual time of blankness, not just samples.
- Include both "any" vs. "mostly" blank (similar to 1 and 4 frame drops).
- Include events where there is no blankness so we have a baseline.
- Remove events with too few samples

**Test Plan: **

A bunch of scrolling in FlatListExample

T17384966

== [Lists] Update SectionSeparatorItem docs (previously D4909526)

Forgot to update the language here when we modified the behavior with the introduction of separator
highlighting support.

** Test Plan: **
nope.

== [Lists] Add renderSectionFooter prop to SectionList (previously D4923353)

Handy for things like "see more" links and such.

The logic here is to render the footer last, *after* the bottom section separator. This is to preserve
the highlighting behavior of the section separator by keeping it adjacent to the items.

**Test Plan: **
Added to snapshot test and example:

{F66635525}

{F66635526}

== [SectionList] Add a bunch more info for rendering items and separators (previously D4923663)

This extra info can be helpful for rending more complex patterns.

**Test Plan: **
Made snapshot test more comprehensive and inspected the output.

== [Lists] reduce render churn (previously D4924639)

I don't think the velocity based leadFactor is helping and might actually be hurting because
it causes a lot of churn in the items we render.

Instead, this diff introduces fillPreference which biases the window expansion in the direction of scroll,
but doesn't actually affect the final bounds of the window at all, so items that are already rendered are
more likely to stay rendered.

**Test Plan: **

Played around in debug mode and watched the overlay - seems better. Also tests all pass.

T16621861

== [Lists] Add initialScrollIndex prop

Makes it easy to load a VirtualizedList at a location in the middle of the content without
wasting time rendering initial rows that aren't relevant, for example when opening an infinite calendar
view to "today".

**Test Plan: **
With debug overlay, set `initialScrollIndex={52}` prop in `FlatListExample` and
and see it immediately render a full screen of items with item 52 aligned at the top of the screen. Note
no initial items are mounted per debug overlay. Scroll around a bunch and everything else seems to work
as normal.

No SectionList impl since `getItemLayout` isn't easy to use there.

T17091314

Reviewed By: bvaughn

Differential Revision: D4907958

fbshipit-source-id: 8b9f1f542f9b240f1e317f3fd7e31c9376e8670e

Examples/UIExplorer/js/SectionListExample.js

@@ -65,7 +65,14 @@ const renderSectionHeader = ({section}) => (
   </View>
 );
 
-const CustomSeparatorComponent = ({text, highlighted}) => (
+const renderSectionFooter = ({section}) => (
+  <View style={styles.header}>
+    <Text style={styles.headerText}>SECTION FOOTER: {section.key}</Text>
+    <SeparatorComponent />
+  </View>
+);
+
+const CustomSeparatorComponent = ({highlighted, text}) => (
   <View style={[styles.customSeparator, highlighted && {backgroundColor: 'rgb(217, 217, 217)'}]}>
     <Text style={styles.separatorText}>{text}</Text>
   </View>
@@ -128,11 +135,11 @@ class SectionListExample extends React.PureComponent {
         <AnimatedSectionList
           ListHeaderComponent={HeaderComponent}
           ListFooterComponent={FooterComponent}
-          SectionSeparatorComponent={({highlighted}) =>
-            <CustomSeparatorComponent highlighted={highlighted} text="SECTION SEPARATOR" />
+          SectionSeparatorComponent={(info) =>
+            <CustomSeparatorComponent {...info} text="SECTION SEPARATOR" />
           }
-          ItemSeparatorComponent={({highlighted}) =>
-            <CustomSeparatorComponent highlighted={highlighted} text="ITEM SEPARATOR" />
+          ItemSeparatorComponent={(info) =>
+            <CustomSeparatorComponent {...info} text="ITEM SEPARATOR" />
           }
           debug={this.state.debug}
           enableVirtualization={this.state.virtualized}
@@ -142,15 +149,23 @@ class SectionListExample extends React.PureComponent {
           refreshing={false}
           renderItem={this._renderItemComponent}
           renderSectionHeader={renderSectionHeader}
+          renderSectionFooter={renderSectionFooter}
           stickySectionHeadersEnabled
           sections={[
-            {renderItem: renderStackedItem, key: 's1', data: [
-              {title: 'Item In Header Section', text: 'Section s1', key: 'header item'},
-            ]},
-            {key: 's2', data: [
-              {noImage: true, title: '1st item', text: 'Section s2', key: 'noimage0'},
-              {noImage: true, title: '2nd item', text: 'Section s2', key: 'noimage1'},
-            ]},
+            {
+              renderItem: renderStackedItem,
+              key: 's1',
+              data: [
+                {title: 'Item In Header Section', text: 'Section s1', key: 'header item'},
+              ],
+            },
+            {
+              key: 's2',
+              data: [
+                {noImage: true, title: '1st item', text: 'Section s2', key: 'noimage0'},
+                {noImage: true, title: '2nd item', text: 'Section s2', key: 'noimage1'},
+              ],
+            },
             ...filteredSectionData,
           ]}
           style={styles.list}

Libraries/Lists/FillRateHelper.js

@@ -10,38 +10,35 @@
  * @flow
  */
 
+/* eslint-disable no-console-disallow */
+
 'use strict';
 
 const performanceNow = require('fbjs/lib/performanceNow');
 const warning = require('fbjs/lib/warning');
 
-export type FillRateExceededInfo = {
-  event: {
-    sample_type: string,
-    blankness: number,
-    blank_pixels_top: number,
-    blank_pixels_bottom: number,
-    scroll_offset: number,
-    visible_length: number,
-    scroll_speed: number,
-    first_frame: Object,
-    last_frame: Object,
-  },
-  aggregate: {
-    avg_blankness: number,
-    min_speed_when_blank: number,
-    avg_speed_when_blank: number,
-    avg_blankness_when_any_blank: number,
-    fraction_any_blank: number,
-    all_samples_timespan_sec: number,
-    fill_rate_sample_counts: {[key: string]: number},
-  },
-};
+export type FillRateInfo = Info;
+
+class Info {
+  any_blank_count = 0;
+  any_blank_ms = 0;
+  any_blank_speed_sum = 0;
+  mostly_blank_count = 0;
+  mostly_blank_ms = 0;
+  pixels_blank = 0;
+  pixels_sampled = 0;
+  pixels_scrolled = 0;
+  total_time_spent = 0;
+  sample_count = 0;
+}
 
 type FrameMetrics = {inLayout?: boolean, length: number, offset: number};
 
-let _listeners: Array<(FillRateExceededInfo) => void> = [];
-let _sampleRate = null;
+const DEBUG = false;
+
+let _listeners: Array<(Info) => void> = [];
+let _minSampleCount = 10;
+let _sampleRate = DEBUG ? 1 : null;
 
 /**
  * A helper class for detecting when the maximem fill rate of `VirtualizedList` is exceeded.
@@ -52,20 +49,19 @@ let _sampleRate = null;
  * `SceneTracker.getActiveScene` to determine the context of the events.
  */
 class FillRateHelper {
+  _anyBlankStartTime = (null: ?number);
+  _enabled = false;
   _getFrameMetrics: (index: number) => ?FrameMetrics;
-  _anyBlankCount = 0;
-  _anyBlankMinSpeed = Number.MAX_SAFE_INTEGER;
-  _anyBlankSpeedSum = 0;
-  _sampleCounts = {};
-  _fractionBlankSum = 0;
-  _samplesStartTime = 0;
-
-  static addFillRateExceededListener(
-    callback: (FillRateExceededInfo) => void
+  _info = new Info();
+  _mostlyBlankStartTime = (null: ?number);
+  _samplesStartTime = (null: ?number);
+
+  static addListener(
+    callback: (FillRateInfo) => void
   ): {remove: () => void} {
     warning(
       _sampleRate !== null,
-      'Call `FillRateHelper.setSampleRate` before `addFillRateExceededListener`.'
+      'Call `FillRateHelper.setSampleRate` before `addListener`.'
     );
     _listeners.push(callback);
     return {
@@ -79,16 +75,62 @@ class FillRateHelper {
     _sampleRate = sampleRate;
   }
 
-  static enabled(): boolean {
-    return (_sampleRate || 0) > 0.0;
+  static setMinSampleCount(minSampleCount: number) {
+    _minSampleCount = minSampleCount;
   }
 
   constructor(getFrameMetrics: (index: number) => ?FrameMetrics) {
     this._getFrameMetrics = getFrameMetrics;
+    this._enabled = (_sampleRate || 0) > Math.random();
+    this._resetData();
+  }
+
+  activate() {
+    if (this._enabled && this._samplesStartTime == null) {
+      DEBUG && console.debug('FillRateHelper: activate');
+      this._samplesStartTime = performanceNow();
+    }
+  }
+
+  deactivateAndFlush() {
+    if (!this._enabled) {
+      return;
+    }
+    const start = this._samplesStartTime; // const for flow
+    if (start == null) {
+      DEBUG && console.debug('FillRateHelper: bail on deactivate with no start time');
+      return;
+    }
+    if (this._info.sample_count < _minSampleCount) {
+      // Don't bother with under-sampled events.
+      this._resetData();
+      return;
+    }
+    const total_time_spent = performanceNow() - start;
+    const info: any = {
+      ...this._info,
+      total_time_spent,
+    };
+    if (DEBUG) {
+      const derived = {
+        avg_blankness: this._info.pixels_blank / this._info.pixels_sampled,
+        avg_speed: this._info.pixels_scrolled / (total_time_spent / 1000),
+        avg_speed_when_any_blank: this._info.any_blank_speed_sum / this._info.any_blank_count,
+        any_blank_per_min: this._info.any_blank_count / (total_time_spent / 1000 / 60),
+        any_blank_time_frac: this._info.any_blank_ms / total_time_spent,
+        mostly_blank_per_min: this._info.mostly_blank_count / (total_time_spent / 1000 / 60),
+        mostly_blank_time_frac: this._info.mostly_blank_ms / total_time_spent,
+      };
+      for (const key in derived) {
+        derived[key] = Math.round(1000 * derived[key]) / 1000;
+      }
+      console.debug('FillRateHelper deactivateAndFlush: ', {derived, info});
+    }
+    _listeners.forEach((listener) => listener(info));
+    this._resetData();
   }
 
-  computeInfoSampled(
-    sampleType: string,
+  computeBlankness(
     props: {
       data: Array<any>,
       getItemCount: (data: Array<any>) => number,
@@ -99,30 +141,45 @@ class FillRateHelper {
       last: number,
     },
     scrollMetrics: {
+      dOffset: number,
       offset: number,
       velocity: number,
       visibleLength: number,
     },
-  ): ?FillRateExceededInfo {
-    if (!FillRateHelper.enabled() || (_sampleRate || 0) <= Math.random()) {
-      return null;
+  ): number {
+    if (!this._enabled || props.getItemCount(props.data) === 0 || this._samplesStartTime == null) {
+      return 0;
     }
-    const start = performanceNow();
-    if (props.getItemCount(props.data) === 0) {
-      return null;
+    const {dOffset, offset, velocity, visibleLength} = scrollMetrics;
+
+    // Denominator metrics that we track for all events - most of the time there is no blankness and
+    // we want to capture that.
+    this._info.sample_count++;
+    this._info.pixels_sampled += Math.round(visibleLength);
+    this._info.pixels_scrolled += Math.round(Math.abs(dOffset));
+    const scrollSpeed = Math.round(Math.abs(velocity) * 1000); // px / sec
+
+    // Whether blank now or not, record the elapsed time blank if we were blank last time.
+    const now = performanceNow();
+    if (this._anyBlankStartTime != null) {
+      this._info.any_blank_ms += now - this._anyBlankStartTime;
     }
-    if (!this._samplesStartTime) {
-      this._samplesStartTime = start;
+    this._anyBlankStartTime = null;
+    if (this._mostlyBlankStartTime != null) {
+      this._info.mostly_blank_ms += now - this._mostlyBlankStartTime;
     }
-    const {offset, velocity, visibleLength} = scrollMetrics;
+    this._mostlyBlankStartTime = null;
+
     let blankTop = 0;
     let first = state.first;
     let firstFrame = this._getFrameMetrics(first);
     while (first <= state.last && (!firstFrame || !firstFrame.inLayout)) {
       firstFrame = this._getFrameMetrics(first);
       first++;
     }
-    if (firstFrame) {
+    // Only count blankTop if we aren't rendering the first item, otherwise we will count the header
+    // as blank.
+    if (firstFrame && first > 0) {
       blankTop = Math.min(visibleLength, Math.max(0, firstFrame.offset - offset));
     }
     let blankBottom = 0;
@@ -132,47 +189,38 @@ class FillRateHelper {
       lastFrame = this._getFrameMetrics(last);
       last--;
     }
-    if (lastFrame) {
+    // Only count blankBottom if we aren't rendering the last item, otherwise we will count the
+    // footer as blank.
+    if (lastFrame && last < props.getItemCount(props.data) - 1) {
       const bottomEdge = lastFrame.offset + lastFrame.length;
       blankBottom = Math.min(visibleLength, Math.max(0, offset + visibleLength - bottomEdge));
     }
-    this._sampleCounts.all = (this._sampleCounts.all || 0) + 1;
-    this._sampleCounts[sampleType] = (this._sampleCounts[sampleType] || 0) + 1;
-    const blankness = (blankTop + blankBottom) / visibleLength;
+    const pixels_blank = Math.round(blankTop + blankBottom);
+    const blankness = pixels_blank / visibleLength;
     if (blankness > 0) {
-      const scrollSpeed = Math.abs(velocity);
-      if (scrollSpeed && sampleType === 'onScroll') {
-        this._anyBlankMinSpeed = Math.min(this._anyBlankMinSpeed, scrollSpeed);
+      this._anyBlankStartTime = now;
+      this._info.any_blank_speed_sum += scrollSpeed;
+      this._info.any_blank_count++;
+      this._info.pixels_blank += pixels_blank;
+      if (blankness > 0.5) {
+        this._mostlyBlankStartTime = now;
+        this._info.mostly_blank_count++;
       }
-      this._anyBlankSpeedSum += scrollSpeed;
-      this._anyBlankCount++;
-      this._fractionBlankSum += blankness;
-      const event = {
-        sample_type: sampleType,
-        blankness: blankness,
-        blank_pixels_top: blankTop,
-        blank_pixels_bottom: blankBottom,
-        scroll_offset: offset,
-        visible_length: visibleLength,
-        scroll_speed: scrollSpeed,
-        first_frame: {...firstFrame},
-        last_frame: {...lastFrame},
-      };
-      const aggregate = {
-        avg_blankness: this._fractionBlankSum / this._sampleCounts.all,
-        min_speed_when_blank: this._anyBlankMinSpeed,
-        avg_speed_when_blank: this._anyBlankSpeedSum / this._anyBlankCount,
-        avg_blankness_when_any_blank: this._fractionBlankSum / this._anyBlankCount,
-        fraction_any_blank: this._anyBlankCount / this._sampleCounts.all,
-        all_samples_timespan_sec: (performanceNow() - this._samplesStartTime) / 1000.0,
-        fill_rate_sample_counts: {...this._sampleCounts},
-        compute_time: performanceNow() - start,
-      };
-      const info = {event, aggregate};
-      _listeners.forEach((listener) => listener(info));
-      return info;
+    } else if (scrollSpeed < 0.01 || Math.abs(dOffset) < 1) {
+      this.deactivateAndFlush();
     }
-    return null;
+    return blankness;
+  }
+
+  enabled(): boolean {
+    return this._enabled;
+  }
+
+  _resetData() {
+    this._anyBlankStartTime = null;
+    this._info = new Info();
+    this._mostlyBlankStartTime = null;
+    this._samplesStartTime = null;
   }
 }
 

Libraries/Lists/FlatList.js

@@ -116,6 +116,13 @@ type OptionalProps<ItemT> = {
    * to improve perceived performance of scroll-to-top actions.
    */
   initialNumToRender: number,
+  /**
+   * Instead of starting at the top with the first item, start at `initialScrollIndex`. This
+   * disables the "scroll to top" optimization that keeps the first `initialNumToRender` items
+   * always rendered and immediately renders the items starting at this initial index. Requires
+   * `getItemLayout` to be implemented.
+   */
+  initialScrollIndex?: ?number,
   /**
    * Used to extract a unique key for a given item at the specified index. Key is used for caching
    * and as the react key to track item re-ordering. The default extractor checks `item.key`, then