[sglang-miles] Cherry-pick #24767: Make request dump robust to unpicklable server_args and large meta_info

[ByronHsu]

Summary

Cherry-pick of #24767 (merge commit 1e6c6d1) onto sglang-miles.

• Wraps pickle dump in try/except; on failure retries with server_args=None so request data is still persisted
• Adds configurable dump_requests_exclude_meta_keys to strip bulky keys (routed_experts, hidden_states) from meta_info in dumps
• Surfaces the option in the configure_logging CLI as --dump-requests-exclude-meta-keys

Conflict resolution: None — clean cherry-pick (auto-merged).

Test plan

• Existing request dump tests should pass
• Manual verification with --trust-remote-code models that previously caused pickle failures

Made with Cursor

[gemini-code-assist]

Code Review

This pull request introduces a feature to exclude specific keys from request metadata dumps to optimize storage and avoid serialization issues. It also adds a retry mechanism that attempts to dump data without server_args if the initial pickling fails. Feedback highlights the need for consistent metadata filtering across all dump types and suggests making the retry logic more resilient by catching potential errors during the fallback serialization step.

[gemini-code-assist]

The meta_info filtering logic is correctly implemented here for periodic request dumps. However, this filtering is not applied to unfinished requests in dump_requests_before_crash or to requests recorded in record_request_for_crash_dump. If the goal is to exclude bulky keys from all dumps to save space and avoid pickling issues, consider applying this filtering consistently across all dumping paths. Since these functions are not all within the current diff hunks, ensure the logic is shared or repeated where necessary.

[gemini-code-assist]

Consider making the retry logic more robust by wrapping the second pickle.dump in a try...except block. If the failure is caused by something other than server_args (e.g., unpicklable data in the requests themselves), the current implementation will raise an unhandled exception in the background thread. Also, note that this logic is duplicated in dump_requests_before_crash.

Suggested change

	try:
	pickle.dump(to_dump_with_server_args, f)
	except Exception as e:
	# When the server is launched with --trust-remote-code,
	# server_args sometimes fails to pickle. Retry without
	# server_args so the request data still gets persisted.
	logger.error(
	f"Failed to pickle dump with server_args: {e!r}; "
	"retrying without server_args"
	)
	f.seek(0)
	f.truncate()
	to_dump_with_server_args["server_args"] = None
	pickle.dump(to_dump_with_server_args, f)
	try:
	pickle.dump(to_dump_with_server_args, f)
	except Exception as e:
	# When the server is launched with --trust-remote-code,
	# server_args sometimes fails to pickle. Retry without
	# server_args so the request data still gets persisted.
	logger.error(
	f"Failed to pickle dump with server_args: {e!r}; "
	"retrying without server_args"
	)
	f.seek(0)
	f.truncate()
	to_dump_with_server_args["server_args"] = None
	try:
	pickle.dump(to_dump_with_server_args, f)
	except Exception as e2:
	logger.error(f"Failed to pickle dump even without server_args: {e2!r}")

[gemini-code-assist]

Similar to the periodic dump, consider wrapping the second pickle.dump in a try...except block here. Since this is called during crash handling, an unhandled exception here could interfere with the shutdown process or prevent other cleanup tasks from completing.

Suggested change

	try:
	pickle.dump(data_to_dump_with_server_args, f)
	except Exception as e:
	# When the server is launched with --trust-remote-code,
	# server_args sometimes fails to pickle. Retry without
	# server_args so the request data still gets persisted.
	logger.error(
	f"Failed to pickle dump with server_args: {e!r}; "
	"retrying without server_args"
	)
	f.seek(0)
	f.truncate()
	data_to_dump_with_server_args["server_args"] = None
	pickle.dump(data_to_dump_with_server_args, f)
	try:
	pickle.dump(data_to_dump_with_server_args, f)
	except Exception as e:
	# When the server is launched with --trust-remote-code,
	# server_args sometimes fails to pickle. Retry without
	# server_args so the request data still gets persisted.
	logger.error(
	f"Failed to pickle dump with server_args: {e!r}; "
	"retrying without server_args"
	)
	f.seek(0)
	f.truncate()
	data_to_dump_with_server_args["server_args"] = None
	try:
	pickle.dump(data_to_dump_with_server_args, f)
	except Exception as e2:
	logger.error(f"Failed to pickle dump even without server_args: {e2!r}")