[HW Accel Support]: Frigate + WebRTC + h264/cuda transcoding results in inconsistent non-live playback

[spikeygg]

Describe the problem you are having

I have a Frigate machine with a Nvidia RTX 3060 for transcoding and a Reolink Duo2 camera. The duo2 has an h.265 video stream but that doesn't play nicely with WebRTC so I set up a transcode to convert it to h.264, like this:

    front_drive:
      - ffmpeg:rtsp://{FRIGATE_RTSP_USERNAME}:{FRIGATE_RTSP_PASSWORD}@rlc-duo-1:554/h264Preview_01_main#audio=copy#video=h264/cuda
      - ffmpeg:front_drive#audio=opus

Since setting up the transcode, the live stream is clear and seems to work well (smooth video feed playback) but any past or recorded footage comes back with very inconsistent framerate. It seems to play smoothly for like 500ms, then slow down to half speed for about 500ms, then speed up. This pattern repeats the whole time the video plays. This happens in the browser playback as well as exported videos, I can supply one here if requested (however, they are VERY large so it will have to be quite short). Smooth motion like cars driving down the road or people walking look like they are going/stopping/going/stopping.

I'm curious if there is something I can do to the process (insert a command perhaps) to make the transcoded result smooth, like it should be.

When viewing an exported video with VLC, I found that I could open "Current Media Information" and click on the "Statistics" tab, and open the "Input/Read"-->"Input bitrate" and while playing the video, the chart looks like this:

which appears to track with the footage slowing down and speeding up. Here's another example:

Version

0.14.0-da913d8

Frigate config file

detectors:
  tensorrt:
    type: tensorrt
    device: 0

model:
  path: /config/model_cache/tensorrt/yolov7-tiny-416.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

tls:
  enabled: false

detect:
  fps: 10
  # Optional: Number of frames without a detection before Frigate considers an object to be gone. (default: 5x the frame rate)
  max_disappeared: 50
  stationary:
    threshold: 100

mqtt:
  host: {homeassistant_host}
  user: {username}
  password: {password}
  port: 1883

logger:
  logs:
    frigate.video: info
    frigate.event: info

ffmpeg:
  hwaccel_args: preset-nvidia-h264
  output_args:
    record: preset-record-generic-audio-copy

record:
  enabled: true
  retain:
    days: 2
    mode: all
  events:
    retain:
      default: 10
      mode: active_objects

snapshots:
  enabled: true
  timestamp: false
  bounding_box: true
  height: 800

go2rtc:
  webrtc:
    listen: :8555
    candidates:
      - stun:8555
      - 172.27.10.10:8555
  streams:
    front_door:
      - ffmpeg:rtmp://rlc-520-1/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}
      - ffmpeg:front_door#audio=opus
    west_yard:
      - ffmpeg:rtmp://rlc-410-4/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}
      - ffmpeg:west_yard#audio=opus
    east_yard:
      - ffmpeg:rtmp://rlc-410-3/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}
      - ffmpeg:east_yard#audio=opus
    front_drive:
      - ffmpeg:rtsp://{FRIGATE_RTSP_USERNAME}:{FRIGATE_RTSP_PASSWORD}@rlc-duo-1:554/h264Preview_01_main#audio=copy#video=h264/cuda
      - ffmpeg:front_drive#audio=opus
    back_yard:
      - ffmpeg:rtmp://rlc-410-1/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}
      - ffmpeg:back_yard#audio=opus
    out_the_back:
      - ffmpeg:rtmp://rlc-410-2/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}
      - ffmpeg:out_the_back#audio=opus
    garage:
      - ffmpeg:rtmp://rlc-511-1/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}
      - ffmpeg:garage#audio=opus
    storage_room:
      - ffmpeg:rtmp://rlc-e1-zoom-1/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}#audio=copy#video=h264#hardware#rotate=90
      - ffmpeg:storage_room#audio=opus

objects:
  track:
    - person
    - dog
    - cat

cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_door
          roles:
            - detect
            - record
    detect:
      width: 2048
      height: 1536
    mqtt:
      crop: false
    # Mask out date/timestamp
    motion:
      mask:
        - 865,0,0,0,0,75,865,75
    zones:
      front_door_zone_near:
        coordinates: 0,0.047,0.424,0.049,0.421,0,0.753,0,1,0.124,1,1,0,1
        inertia: 3
        loitering_time: 0
    ui:
      order: 0
    objects:
      filters:
        person:
          min_area: 100000
          max_area: 1000000

    review:
      alerts:
        required_zones: front_door_zone_near
  west_yard:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/west_yard
          roles:
            - detect
            - record
    detect:
      width: 2048
      height: 1536
    mqtt:
      crop: false
    motion:
      mask:
        - 2048,1536,2048,842,1819,800,1778,901,1447,779,1263,971,1150,1349,1550,1536
        - 846,0,846,78,0,78,0,0
    zones:
      west_yard_zone_close:
        coordinates: 
          0.754,1,0.561,0.873,0.617,0.632,0.708,0.506,0.868,0.585,0.887,0.52,0.941,0.534,1,0.106,0.86,0.192,0.608,0.081,0.593,0.374,0.298,0.578,0,0.831,0,1
        inertia: 3
        loitering_time: 0
      west_yard_zone_neighbor_pad:
        coordinates: 0,0.054,0.416,0.051,0.416,0.035,0.605,0.081,0.592,0.372,0,0.829
        inertia: 3
        loitering_time: 0
      west_yard_zone_neighbor_back_yard:
        coordinates: 0.413,0,0.413,0.032,0.609,0.083,0.861,0.189,1,0.107,1,0
        loitering_time: 0
    ui:
      order: 1
    objects:
      filters:
        person:
          min_area: 8000
          mask: 2048,1536,1399,1536
        cat:
          mask: 2048,1536,1399,1536
        dog:
          mask: 2048,1536,1399,1536

    review:
      alerts:
        required_zones:
          - west_yard_zone_close
          - west_yard_zone_neighbor_pad
          - west_yard_zone_neighbor_back_yard
      detections: {}
  east_yard:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/east_yard
          roles:
            - detect
            - record
    detect:
      width: 2048
      height: 1536
    motion:
      mask: 0,0,0,0.805,0.735,0.092,0.635,0
    zones:
      east_yard_zone_close:
        coordinates: 
          1,1,1,0.237,0.763,0.167,0.651,0.152,0.576,0.211,0.352,0.387,0.117,0.615,0.082,0.71,0,0.808,0,1
        inertia: 3
        loitering_time: 0
      east_yard_zone_far:
        coordinates: 1,0,0.418,0,0.422,0.048,0.62,0.057,0.734,0.094,0.651,0.152,0.763,0.164,1,0.237
        inertia: 3
        loitering_time: 0
      east_zone_neighbors_yard:
        coordinates: 
          0.357,0.385,0.646,0.153,0.735,0.095,0.616,0.058,0.418,0.047,0,0.047,0.001,0.797,0.08,0.707,0.117,0.62
        loitering_time: 0
        inertia: 3
    ui:
      order: 2
    objects:
      filters:
        person:
          min_area: 5000
          max_ratio: 1.4

    review:
      alerts:
        required_zones:
          - east_yard_zone_close
          - east_yard_zone_far
          - east_zone_neighbors_yard
      detections: {}
  front_drive:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_drive
          roles:
            - detect
            - record
    detect:
      width: 4096
      height: 1536

    mqtt:
      crop: false
    motion:
      mask:
        - 0,0,4096,0,4096,403,4096,644,3898,705,3294,440,2905,306,2505,203,2144,146,1636,171,1056,303,578,464,0,678
        - 4096,1536,4096,1371,3054,994,1950,1101,798,1010,0,1536
    objects:
      track:
        - person
        - dog
        - cat
        - car
      filters:
        car:
          min_area: 20000
          mask:
            - 675,590,0,753,0,0,4096,0,4096,716,3873,751,2956,553,2079,408
        cat:
          mask:
            - 4096,1536,4096,1371,3054,994,1950,1101,798,1010,0,1536
        dog:
          mask:
            - 4096,1536,4096,1371,3054,994,1950,1101,798,1010,0,1536
        person:
          min_area: 12000
          max_ratio: 1.4
          mask:
            - 1984,521,2202,521,2202,193,1984,193
            - 4096,1536,4096,1371,3054,994,1950,1101,798,1010,0,1536
    zones:
      front_drive_zone_near:
        coordinates: 
          4096,1536,4096,757,3946,704,3735,976,2929,589,2075,477,1397,508,580,654,0,914,0,1536
      front_drive_zone_sidewalk:
        coordinates: 
          531,556,1263,404,1726,358,2093,387,2603,417,3617,622,3595,696,2616,485,2100,586,1498,479,562,622
    ui:
      order: 3

    review:
      alerts:
        required_zones:
          - front_drive_zone_near
          - front_drive_zone_sidewalk
  back_yard:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back_yard
          roles:
            - detect
            - record
    detect:
      width: 2304
      height: 1296
    motion:
      mask:
        - 2304,1296,2304,420,2039,390,2051,236,1872,124,1628,285,1611,655,1300,785,1970,1296
        - 884,0,884,94,0,94,0,0
    zones:
      back_yard_zone_canal:
        coordinates: 2304,0,2304,427,2100,380,1030,0
      back_yard_zone_grass:
        coordinates: 
          2100,380,1300,785,1970,1296,1123,1296,873,984,839,685,486,282,705,213,693,0,1030,0
      back_yard_zone_garden:
        coordinates: 2304,1296,2304,427,2100,380,1300,785,1970,1296
      back_yard_zone_deck_and_patio:
        coordinates: 0,966,839,685,625,431,486,282,705,213,693,0,0,0
    ui:
      order: 4

    review:
      alerts:
        required_zones:
          - back_yard_zone_garden
          - back_yard_zone_grass
          - back_yard_zone_deck_and_patio
          - back_yard_zone_canal
      detections: {}
  out_the_back:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/out_the_back
          roles:
            - detect
            - record
    detect:
      width: 2304
      height: 1296
    motion:
      mask:
        - 887,0,887,91,0,91,0,0
    objects:
      track:
        - person
        - dog
        - cat
        - car
      filters:
        car: {}
        cat: {}
        dog: {}
        person: {}
      mask:
        - 0,0,0,0.268,0.483,0.229,1,0.256,1,0
        - 0.96,0.631,0.963,0.881,0.999,0.881,0.999,0.633
    zones:
      out_the_back_zone_far:
        coordinates: 1,0,1,0.402,0,0.404,0,0.071,0.385,0.071,0.386,0
        inertia: 3
        loitering_time: 0
      out_the_back_zone_near:
        coordinates: 0,1296,2304,1296,2304,522,0,522

    ui:
      order: 5

    review:
      alerts:
        required_zones:
          - out_the_back_zone_near
          - out_the_back_zone_far
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage
          roles:
            - detect
            - record
    detect:
      width: 2048
      height: 1536
    motion:
      mask:
        - 852,0,852,80,0,80,0,0
    objects:
      track:
        - person
      filters:
        person:
          min_area: 6000
          threshold: 0.8
    ui:
      order: 6

  storage_room:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/storage_room
          roles:
            - detect
            - record
    detect:
      width: 1536
      height: 2048
    motion:
      mask:
        - 1452,854,1536,854,1536,0,1452,0
    objects:
      track:
        - person
    ui:
      order: 7

version: 0.14

docker-compose file or Docker CLI command

version: "3.9"
name: frigate
services:
  frigate:
    container_name: frigate
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
      - "1984:1984"
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    shm_size: "1gb"
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128:/dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /all_media/frigate/config:/config
      - /all_media/frigate/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    environment:
      UID: 1000
      GID: 100
      FRIGATE_RTSP_PASSWORD: "password"
      FRIGATE_RTSP_USERNAME: "user"
    healthcheck:
      test: nvidia-smi || exit 1
      interval: 1m
      timeout: 5s
      retries: 1
      start_period: 1m
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Relevant Frigate log output

None

Relevant go2rtc log output

None

FFprobe output from your camera

I tried this but it hung and never came back to the command prompt.

# ffprobe rtsp://username:password@rlc-duo-1:554/h264Preview_01_main
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
^C
#

Install method

Docker Compose

Object Detector

TensorRT

Network connection

Wired

Camera make and model

Reolink Duo2

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

might be an issue of the cameras timestamps coming back incorrectly, I had trouble with that camera when I had one. Maybe also try checking for firmware updates on reolink's site (not the app)

[spikeygg]

That was a good suggestion. I checked the Reolink website and I found an updated FW package, I flashed the FW on the camera but it didn't change the result. 🤔