feat: Add custom placeholder responses for scale-from-zero scenarios

[malpou]

Allows HTTPScaledObjects to serve configurable content (HTML, JSON, XML, plain text, etc.) while workloads scale up from zero, improving user experience during cold starts.

Description

This PR implements customizable placeholder responses that are displayed to users while applications are scaling up from zero. This significantly improves the user experience during cold starts by providing immediate feedback instead of connection timeouts or errors.

The implementation is content-agnostic, allowing users to serve any type of response (HTML pages, JSON API responses, XML, plain text, etc.) without automatic modifications or assumptions about the content format.

Key features:

• Adds placeholderConfig section to HTTPScaledObject CRD for configuring placeholder responses
• Content-type agnostic: Support for any response format (HTML, JSON, XML, plain text, etc.)
• Inline content or ConfigMap-based templates via environment variables
• Go template support with variables like {{.ServiceName}}, {{.Namespace}}, and {{.RefreshInterval}}
• Custom Content-Type headers to match your API format
• Custom HTTP status codes (default: 200 OK)
• No automatic content injection - your content is served exactly as provided
• Configurable via global defaults (ConfigMap) or per-HTTPScaledObject settings

Implementation details:

• New placeholder handler in the interceptor that serves configured content when backends are unavailable
• Template variable substitution works with any content format
• Graceful fallback to upstream requests once the service is ready
• Error handling for endpoint cache failures returns 503 Service Unavailable
• Full backward compatibility - disabled by default

Use cases:

• HTML pages: Show a "Service starting..." page with auto-refresh
• JSON APIs: Return {"status": "initializing", "message": "Service is starting..."} responses
• XML services: Provide XML-formatted status responses
• Plain text: Simple text-based status messages

Checklist

• Commits are signed with Developer Certificate of Origin (DCO)
• Changelog has been updated and is aligned with our changelog requirements
• Any necessary documentation is added, such as:
  • README.md
  • The docs/ directory
  • The docs repo

[JorTurFer]

This is a great feature! Thanks for the contribution 🙇

[JorTurFer]

Should we inject this value instead of expect that users provide it? I mean, this looks as something that we want to enforce. @wozniakjan ?

[malpou]

I was thinking about injecting it but wanted some feedback first.

I think it also would be possible to add some JavaScript instead which calls the URL the browser is on and checks if the X-KEDA-HTTP-Placeholder-Served: true header is there. If it's not we should do a full refresh.
This avoids having the browser do an actual refresh every X seconds. This would mostly be an UX improvement.

Both approaches could be injected into whatever template the user provides, if that's the approach you would prefer.

Example:

<script>
(function() {
    const checkInterval = {{.RefreshInterval}} * 1000; // Convert seconds to milliseconds
    
    async function checkServiceStatus() {
        try {
            // Make a HEAD request to the current URL to check headers
            const response = await fetch(window.location.href, {
                method: 'HEAD',
                cache: 'no-cache'
            });
            
            // Check if the placeholder header is present
            const placeholderHeader = response.headers.get('X-KEDA-HTTP-Placeholder-Served');
            
            if (placeholderHeader !== 'true') {
                // Service is ready! Do a full page refresh
                window.location.reload();
            } else {
                // Still showing placeholder, check again later
                setTimeout(checkServiceStatus, checkInterval);
            }
        } catch (error) {
            // On error, continue checking
            console.error('Error checking service status:', error);
            setTimeout(checkServiceStatus, checkInterval);
        }
    }
    
    // Start checking after the interval
    setTimeout(checkServiceStatus, checkInterval);
})();
</script>

[JorTurFer]

I like this approach! it's quite elegant IMHO, the only thing is that JS needs to be injected as well, doesn't

[JorTurFer]

@wozniakjan @zroubalik WDYT?

[malpou]

@JorTurFer I've tried to make a take on the injection with a <script> tag and some JS to figure out when to refresh the browser.

[wozniakjan]

yeah, I like this as a default behavior, but I have mentioned a few additional points regarding this in my review - #1303 (review)

[Copilot]

Pull Request Overview

This PR introduces configurable placeholder pages for scale-from-zero scenarios, enhancing the user experience during cold starts. Key changes include:

• Adding a new placeholder handler and supporting API changes in the HTTPScaledObject CRD.
• Adjusting proxy handler logic and test cases to incorporate placeholder page responses.
• Updating documentation, examples, and deepcopy functions to reflect the new placeholderConfig functionality.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/checks/placeholder_pages/placeholder_pages_test.go	Added E2E tests for placeholder page responses and script injection.
operator/apis/http/v1alpha1/zz_generated.deepcopy.go	Added DeepCopy functions for the new PlaceholderConfig type.
operator/apis/http/v1alpha1/httpscaledobject_types.go	Introduced the PlaceholderConfig type in the CRD schema.
interceptor/proxy_handlers_test.go, proxy_handlers_integration_test.go	Updated tests to pass new parameters for placeholder handling.
interceptor/proxy_handlers.go	Integrated the placeholder handler into the forwarding logic.
interceptor/main.go, main_test.go	Updated server initialization to include the placeholder handler.
interceptor/handler/placeholder.go	Implemented the placeholder page rendering and caching logic.
examples/vX.X.X/httpscaledobject.yaml	Provided example configuration for placeholder pages.
docs/ref/vX.X.X/http_scaled_object.md	Documented the placeholderConfig section details.
config/crd/bases/http.keda.sh_httpscaledobjects.yaml	Updated the CRD with placeholderConfig properties.
CHANGELOG.md	Added an entry for the custom placeholder pages feature.

Comments suppressed due to low confidence (1)

interceptor/handler/placeholder.go:206

• [nitpick] Consider logging the error returned from getTemplate before falling back to the inline response to improve debuggability of template parsing issues.

tmpl, err := h.getTemplate(r.Context(), hso)

[JorTurFer]

awesome job! Only one small nit inline and it's okey to merge ❤️

[Nico385412]

Hello any news on this feature ? 🙏

[JorTurFer]

Sorry, @wozniakjan told me that he wanted to review the feature and I just kept this to him.
Did you take a look @wozniakjan ?

[wozniakjan]

this looks really great, thank you for contributing!

I do have a couple of discussion points before it gets merged

• usage of ConfigMaps for storing content for placeholder pages - imho defaults would be better configured globally as a statically mounted ConfigMap or env variable on Deployment
• simplifying content processing - the default as a nicely formatted HTML page makes great sense but then we should allow arbitrary payloads with arbitrary headers, not all usage of this tool is aimed at web browsers, sometimes it's microservices talking between each other

[wozniakjan]

wdyt about moving these default templates to configuration instead of having them hardcoded in the binary? I can imagine setting it through environment variables in the Deployment or alternatively, which might be even better, mounted as a ConfigMap.

http-add-on/config/interceptor/deployment.yaml

Line 30 in bf35564

env:

These default pages are pretty good examples, but I'd imagine people may want to tweak them a little bit or centrally replace with different defaults. As implemented, iiuc, users would have to copy the same ConfigMap into each namespace and then reference it in the HTTPScaledObject or inline the same string in each HTTPScaledObject.

[wozniakjan]

what if users don't want HTML content? The client doesn't have to be web browser, it could be two microservices talking to each other and they may have other L7 protocols (json, protobuf, etc) with their own mechanisms on how to handle retries.

[wozniakjan]

mind the fact interceptor doesn't have RBAC for ConfigMaps. Is it worth adding clusterwide read access to ConfigMaps so users can configure content of a placeholder page per namespace? Imho it's a bit too much and I would rather just add defaults as mounted ConfigMaps to interceptor or env variables on the Deployment as mentioned above.

[wozniakjan]

I would rather not use ConfigMaps at the HTTPScaledObject level but if we decide to go with them, please use cached client instead of the direct client. Fetching ConfigMap for each request on zero-scaled apps might be tricky with kube-client QPS and could easily cause a lot of load for kube-apiserver if users increase the burst config to "deal with errors".

[wozniakjan]

situation when placeholder pages are configured and the endpoints cache returns error is not handled, which means the code will fall through to waiting on cold-start. Is that desired, or should the interceptor instead return a 503 error indicating an internal problem?

[malpou]

@wozniakjan I've looked at the feedback and tried to adjust it accordingly.

Haven't had time to do the E2E tests, so put the PR in draft until it's ready for review.