mirror of
https://github.com/nyanmisaka/ffmpeg-rockchip.git
synced 2026-01-24 07:31:22 +01:00
Filter chain introduces 1s latency between each sub-picture #189
Labels
No labels
bug
enhancement
help wanted
invalid
pull-request
question
upstream
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: starred/ffmpeg-rockchip#189
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @liangtaikonaut on GitHub (Jul 3, 2025).
Dear Nyanmisaka,
The filter chain, maybe the overlay filters, introduce 1s latency between one sub-picture and the one before it, showing in the snapshot below.
How can I remove the latency?
Thank you!
My command is:
ffmpeg -init_hw_device rkmpp=hw -filter_hw_device hw -f lavfi -i color=c=cyan:s=1920x1080,format=nv12
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://admin:xxxx@10.0.24.63:554/h264/ch1/main/av_stream
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://admin:xxxx@10.0.24.63:554/h264/ch1/main/av_stream
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://admin:xxxx@10.0.24.63:554/h264/ch1/main/av_stream
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://admin:xxxx@10.0.24.63:554/h264/ch1/main/av_stream
-filter_complex "
[0:v]hwupload[main];
[1:v]scale_rkrga=w=960:h=540:format=nv12,hwdownload,format=nv12,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwupload,scale_rkrga=iw:h=ih:format=bgra:afbc=1[overlay1];
[2:v]scale_rkrga=w=960:h=540:format=nv12,hwdownload,format=nv12,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwupload,scale_rkrga=iw:h=ih:format=bgra:afbc=1[overlay2];
[3:v]scale_rkrga=w=960:h=540:format=nv12,hwdownload,format=nv12,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwupload,scale_rkrga=iw:h=ih:format=bgra:afbc=1[overlay3];
[4:v]scale_rkrga=w=960:h=540:format=nv12,hwdownload,format=nv12,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwupload,scale_rkrga=iw:h=ih:format=bgra:afbc=1[overlay4];
[main][overlay1]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=0:y=0[temp1];
[temp1][overlay2]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=0:y=H/2[temp2];
[temp2][overlay3]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=W/2:y=0[temp3];
[temp3][overlay4]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=W/2:y=H/2"
-c:v h264_rkmpp -b:v 4M -maxrate 8M -an -f flv rtmp://127.0.0.1:10050/etest/mosaic_04z
@nyanmisaka commented on GitHub (Jul 3, 2025):
Apply
:async_depth=0to each rkrga filter to disable async processing to reduce latency at the expense of reduced performance.https://github.com/nyanmisaka/ffmpeg-rockchip/wiki/Filter
Also, the overlay filter is not designed to stitch images together. So this pipeline is not the best solution. It is better to implement a stitching filter.
@liangtaikonaut commented on GitHub (Jul 3, 2025):
I added
:async_depth=0to all scale_rkrga filters and overlay_rkrga filters, but the latency still exists, maybe there's other reason?@nyanmisaka commented on GitHub (Jul 3, 2025):
Do you mean that this problem does not occur when using pure software codecs and filters?
@liangtaikonaut commented on GitHub (Jul 3, 2025):
Indeed!
Pure software processing has the same problem, havn't noticed it before.
And I did one more test just now, changed input from rtsp stream to local file, the problem disapeared, it seems the cause is not from the filter chain!
@nyanmisaka commented on GitHub (Jul 3, 2025):
The video bitstream is out of sync before being fed to the decoder. It may be related to your RTSP server and FFmpeg's RTSP demuxer.
@liangtaikonaut commented on GitHub (Jul 4, 2025):
Dear Nyanmisaka,
You're right, I found the root cause, the network camera sends video bitstreams differently when ffmpeg pulls 4 streams with one same url.
I set up a RTSP server, publish 4 rtsp streams separately on it, and ffmpeg pulls these 4 streams, there is no timestamp differences between sub-pictures now.
Thank you!
@nyanmisaka commented on GitHub (Jul 4, 2025):
That's expected. The pts synchronization inside ffmpeg is usually quite robust.
Also, this should give you a better drawtext performance:
scale_rkrga=w=960:h=540:format=rgb24,hwmap,format=rgb24,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwmap,format=drm_prime@liangtaikonaut commented on GitHub (Jul 4, 2025):
I had a try:
ffmpeg -init_hw_device rkmpp=hw -filter_hw_device hw -f lavfi -i color=c=cyan:s=1920x1080,format=nv12
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://10.0.24.42/etest000/program000
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://10.0.24.42/etest001/program001
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://10.0.24.42/etest002/program002
-rtsp_transport tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -re -i rtsp://10.0.24.42/etest003/program003
-filter_complex "
[0:v]hwupload[main];
[1:v]scale_rkrga=w=960:h=540:format=rgb24,hwmap,format=rgb24,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwmap,format=drm_prime[overlay1];
[2:v]scale_rkrga=w=960:h=540:format=rgb24,hwmap,format=rgb24,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwmap,format=drm_prime[overlay2];
[3:v]scale_rkrga=w=960:h=540:format=rgb24,hwmap,format=rgb24,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwmap,format=drm_prime[overlay3];
[4:v]scale_rkrga=w=960:h=540:format=rgb24,hwmap,format=rgb24,drawtext=text=TEST:fontcolor=red:fontsize=128:x=w/2:y=h/2,hwmap,format=drm_prime[overlay4];
[main][overlay1]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=0:y=0[temp1];
[temp1][overlay2]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=0:y=H/2[temp2];
[temp2][overlay3]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=W/2:y=0[temp3];
[temp3][overlay4]overlay_rkrga=eof_action=pass:repeatlast=0:format=nv12:x=W/2:y=H/2"
-c:v h264_rkmpp -b:v 4M -maxrate 8M -an -f flv rtmp://127.0.0.1:10050/etest/mosaic_04z
but ffmpeg reported error:
[Parsed_hwmap_5 @ 0x558e75c050] Failed to map frame: -38.
[fc#0 @ 0x558e722b50] Error while filtering: Function not implemented
Failed to inject frame into filter network: Function not implemented
Error while filtering: Function not implemented
[Parsed_hwmap_11 @ 0x558e73a010] Failed to map frame: -38.
[fc#0 @ 0x558e722b50] Error while filtering: Function not implemented
Failed to inject frame into filter network: Function not implemented
[Parsed_hwmap_17 @ 0x558e74d670] Failed to map frame: -38.
[fc#0 @ 0x558e722b50] Error while filtering: Function not implemented
Failed to inject frame into filter network: Function not implemented
[Parsed_hwmap_23 @ 0x558e7991a0] Failed to map frame: -38.
[fc#0 @ 0x558e722b50] Error while filtering: Function not implemented
Failed to inject frame into filter network: Function not implemented
[out#0/flv @ 0x558ffe2e30] Nothing was written into output file, because at least one of its streams received no packets.
Is there any error in may command?
@nyanmisaka commented on GitHub (Jul 4, 2025):
Your ffmpeg-rockchip is outdated.
@liangtaikonaut commented on GitHub (Jul 4, 2025):
Noted.
I'll pull the latest code.
@liangtaikonaut commented on GitHub (Jul 4, 2025):
I updated the rkmpp, rkrga and ffmpeg-rockchip, run the command with
hwmap, the output loses cyan background and first 3 sub-pictures, the bitrate is about 2700Mb/s, the speed rasing slowly from 0.500x to 0.988x:and tried command using
hwdownload|hwupload, the output is full with 4 sub-pictures, the bitrate is about 4800Mb/s, the speed rasing slowly from 0.500x to 0.988x, too.@nyanmisaka commented on GitHub (Jul 4, 2025):
Well, I forgot that you need alpha for overlay. So change rgb24 to rgba/bgra.
Also, the bottleneck seems to be the input speed, so this optimization doesn't really help much.
@liangtaikonaut commented on GitHub (Jul 4, 2025):
rgba and bgra work fine together with
hwmapandoverlay_rkrgafilter, thanks.New version ffmpeg-rockchip seems startuping slower than before, both in speed showing and player opening, because I miss any option?
@nyanmisaka commented on GitHub (Jul 4, 2025):
I don't think I've made any changes to that.
@liangtaikonaut commented on GitHub (Jul 7, 2025):
I removed "-re" option and added "-g 25" option, it's much better now, ffmpeg reaches 1x speed quickly, and player opens quickly, too.
@liangtaikonaut commented on GitHub (Jul 9, 2025):
Dear Nyanmisaka,
I'm trying translate the command to code, implementing a multi-viewer function with ffmpeg-rockchip API，and met the problem that:
I tested again with original ffmpeg command, it reached 9 streams.
What's the most possible mistake I made?
Thank you!
@liangtaikonaut commented on GitHub (Jul 9, 2025):
Sorry for bothering you.
It's because I initialized
rtsp_transport tcpAVDictionary before theavformat_open_input()cycle, then only the first RTSP stream is opened with TCP protocol, the others are UDP transport.@nyanmisaka commented on GitHub (Jul 9, 2025):
https://github.com/search?q=repo%3AFFmpeg%2FFFmpeg+%22max+delay+reached.+need+to+consume+packet%22&type=code
The warning means that the rtsp demuxer buffer is full, which may be caused by inefficient downstream pipelines in your program. Increasing the buffer size and the maximum delay on the receiving end may also help.
Mali GPU can only render images. Video encoding and decoding has nothing to do with it.
@nyanmisaka commented on GitHub (Jul 9, 2025):
Using UDP may result in packet loss and make the decoder complain.
@liangtaikonaut commented on GitHub (Jul 9, 2025):
Understood, thank you.
And the GPU topic is also my misunderstanding, for load checking, I need to check VPU and RGA.