Fix defer_load handling

[philippjfr]

Fixes #5105

[codecov]

Codecov Report

Merging #5107 (e3b6f3b) into main (d04fbce) will decrease coverage by 9.89%.
The diff coverage is 91.30%.

@@            Coverage Diff             @@
##             main    #5107      +/-   ##
==========================================
- Coverage   83.69%   73.81%   -9.89%     
==========================================
  Files         274      274              
  Lines       39254    39268      +14     
==========================================
- Hits        32854    28986    -3868     
- Misses       6400    10282    +3882     

Flag	Coverage Δ
ui-tests	?
unitexamples-tests	73.81% <91.30%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
panel/io/state.py	72.20% <ø> (-1.20%)	⬇️
panel/tests/test_config.py	99.15% <88.88%> (-0.85%)	⬇️
panel/tests/test_param.py	99.66% <90.90%> (-0.09%)	⬇️
panel/param.py	88.40% <100.00%> (+0.03%)	⬆️

... and 59 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more