feat: adaptive progressive image loading for photo viewer#26636
feat: adaptive progressive image loading for photo viewer#26636alextran1502 merged 8 commits intomainfrom
Conversation
394d962 to
fb719d4
Compare
|
Is it possible to swap the image bitmaps as they're loaded rather than doing z-levels? |
The thumbhash (canvas) actually does get removed after the 'thumbnail' quality image is loaded successfully. Something similar should be possible when the 'preview' image fully loads on top of the 'thumbnail' - once preview is loaded, thumbnail can be safely removed. On zoom, same thing - once full/original is fully loaded, the thumbnail can be removed. In general, we need to have 2 images while the higher quality is loading - so you see the lower quality 'under' the higher quality one. Once loaded, the lower quality one can be removed from DOM. However, this does add overall complexity. Why are you asking? Is it for performance or memory reasons? For managing the z-index? Or something else? |
fb719d4 to
dffc150
Compare
|
Swapping the bitmap is how it's handled in the mobile app, which is generally better for performance (less compositing, lower memory usage, etc.) and handles alpha. It basically needs an API that decouples decoding the image from displaying it. Not a huge deal if there's no good way to do that in web, but it's better if it can. |
6345dfd to
50ee10d
Compare
dffc150 to
1bf67a0
Compare
50ee10d to
cb26611
Compare
1bf67a0 to
25a0c8f
Compare
cb26611 to
11a8a57
Compare
4d0ae32 to
9ecd6d8
Compare
|
Preview environment has been removed. |
11a8a57 to
af8bf90
Compare
9ecd6d8 to
c1308e5
Compare
c1308e5 to
5c478cd
Compare
3107b83 to
6fdf99c
Compare
Split out from #26636 (adaptive image loading). Leverages the ContentMetrics extraction from #26310. Moves face bounding boxes and OCR overlays from viewport-relative to image-relative coordinates. New features: zoom with face editor open, zoom with OCR boxes visible, accurate face hover hit-testing at any zoom level. Bug fixes: face tagging on stacked assets, faces loading when browsing stacks, face editor auto-close on last face deletion, zoomTarget was null, disablePointer option name mismatch.
Split out from #26636 (adaptive image loading). Leverages the ContentMetrics extraction from #26310. Moves face bounding boxes and OCR overlays from viewport-relative to image-relative coordinates. New features: zoom with face editor open, zoom with OCR boxes visible, accurate face hover hit-testing at any zoom level. Bug fixes: face tagging on stacked assets, faces loading when browsing stacks, face editor auto-close on last face deletion, zoomTarget was null, disablePointer option name mismatch.
Replace ImageManager with a new AdaptiveImageLoader that progressively loads images through quality tiers (thumbnail → preview → original). New components and utilities: - AdaptiveImage: layered image renderer with thumbhash, thumbnail, preview, and original layers with visibility managed by load state - AdaptiveImageLoader: state machine driving the quality progression with per-quality callbacks and error handling - ImageLayer/Image: low-level image elements with load/error lifecycle - PreloadManager: preloads adjacent assets for instant navigation - AlphaBackground/DelayedLoadingSpinner: loading state UI Zoom is handled via a derived CSS transform applied to the content wrapper in AdaptiveImage, with the zoom library (zoomTarget: null) only tracking state without manipulating the DOM directly. Also adds scaleToCover to container-utils and getAssetUrls to utils.
6fdf99c to
3a3919c
Compare
michelheusschen
left a comment
There was a problem hiding this comment.
I gave it another round of testing and no more issues, so all good from me. Thank you for the effort you put into this!
I am thinking your approval somehow block the check job to run lol
In this screenshot, the load times are artificially slowed down so you can see all the different quality levels slowly stream into the browser. The artificial delays are as follows: 500ms delay before sending first byte, then throttle each photo as follows:
thumbnails=1s, preview=4s, original=10s
Initially, (b/c of 500ms delay) you'll set the thumbhash, then you'll see the thumb for 1s, then you'll see the preview stream in over 4s, and then when you zoom (which auto-loads the original quality) you'll see that stream in as well.
The demo video is repeated a 2nd time. The second time I start to zoom while preview is still loading, and you'll see the original start to load before the preview finishes.
Note, this is also using progressive-jpegs for preview. Non-progressive jpegs/webp files will stream/render from top to bottom.
Additionally, while all these images are loading, if you click next/prev, all of them are canceled, which is new feature too - this keeps the navigation responsive. If you are coming from the timeline, the preview image from the timeline is available immediately, so the delay between timeline and the full screen viewer should be almost imperceptible in most cases.
This video is an EXAGGERATION showing the worst case - very slow links/mobile devices. With preloading, and reusing thumbs from timeline, in practice, you won't see this at all. But, if you are on a slow link, you'll get faster feedback, better experience than staring at a black screen with a spinner like you currently do.
adaptive.mp4
The photo viewer currently loads images by swapping by showing a loading spinner until the 'preview' size image loads. And then on zoom, replacing that single
<img>element'ssrcfrom preview to original. This causes visible flicker between quality levels because the browser shows a blank state while the next resolution loads.This PR replaces that with a layered progressive loading system. Three
<img>elements (thumbnail, preview, original) are always present in the DOM, stacked by z-index. Each layer fades in on top of the previous one as it loads, so there's never a blank flash between quality transitions. A thumbhash canvas sits underneath everything as the initial placeholder.The end result is - no spinner at all. The 'thumbnail' size image is VERY likely to already have been loaded (if your navigating from the timeline) - so you'll likely instantly see the thumbnail in most cases. However, if you do not (because this is a direct nav) you'll at least see the thumbhash, which is part of the asset response metdata and does not need a url fetch to get the image.
After the thumb is loaded, preview (or fullsize - or original - based on various conditions) image starts to load. If you start to zoom in, the fullsize (or original) image is loaded.
This change is fully transparent compatible. Transparent images (WEBP) have gray background set behind the image. Note - thumbhashes do support a transparent alpha channel, and the thumbhash will be drawn on top of the gray background.
The
AdaptiveImageLoaderis a state machine that manages cancelation and quality progression between:basic → loading-thumbnail → thumbnail → loading-preview → preview → loading-original → originalThe asset viewer preloads and cancels the next/previous images using the same (headless) state-machine.
Because the "image" is now actually 3 layers - zoom-image was modified (upstream) and can now apply to an arbitary element instead of just
tags - we now zoom the
<AdaptiveImage>root div.Added lots of e2e tests.