feat(sdk): optional video frame extraction on read_file

[John Kennedy (jkennedyvz)]

Adds experimental media handling around read_file, centered on video frame extraction while consolidating the higher-level optional install surface behind media extras.

Important

Per the LangChain alpha API policy, this feature is experimental and subject to changes. It should be used cautiously in production environments while we learn from real media-read workflows.

Why

Video support now spans more than the SDK's low-level frame extractor. The PR keeps the decoder lazy and optional, but aligns user-facing packages around a single media capability:

• deepagents-code[media] installs the media dependencies used for video frame extraction (av and pillow).
• deepagents-talon[media] replaces the separate speech/video install paths by combining local voice transcription dependencies with deepagents-code[media].
• Deep Agents Code extras discovery classifies media as a standalone extra, so /install and extras status output point users at one media install target.
• Talon docs, the Talon Docker image, and affected lockfiles now use --extra media.

Direct SDK usage still gates the frame extractor on optional video dependencies, but the higher-level package story is now media wherever media features are bundled.

Behavior with media/video dependencies installed

When the optional media/video dependencies are available, read_file treats video reads as frame-extraction requests:

• .mp4, .mov, .webm, and .mkv are decoded into sampled JPEG frames instead of sending raw video bytes to the model.
• offset is interpreted as seconds into the source video.
• limit is interpreted as the number of seconds to sample; omitting it uses the existing default read_file window of 100 seconds.
• Sampling remains fixed at the middleware level, so agents narrow or widen the inspected window through offset/limit rather than choosing a sampling rate per call.
• The ToolMessage stays text-only and reports the sampled frame count; sampled image blocks are attached in a synthetic follow-up human message so provider tool-output rules stay intact.

Resource controls cover raw input size, frame count, emitted bytes, decode time, and frame dimensions. Oversized sampled frames are downscaled before JPEG encoding instead of rejecting normal high-resolution videos.

Behavior without the dependencies installed

When the optional dependencies are unavailable, frame extraction stays dormant:

• read_file does not attempt video decoding.
• Existing video files continue through the generic multimodal content-block path.
• .mkv is read as binary by backends and returned as a generic file block instead of sampled video frames.
• The read_file tool help keeps the previous text/image/PDF guidance instead of describing video seconds-based pagination.

The goal is to preserve existing behavior unless the media/video dependencies are present.

Example