Gsteramer使用mppvideodec硬解码依旧延迟过大

oranger

我希望能够使用RKNN推理无线相机的Rtsp流，格式为H264。由于延迟过大我尝试使用Gsteramer的mppvideodec进行硬解码，针对这部分进行了尝试。
我尝试使用如下程序进行收流：

1. import gi
   
2. gi.require_version('Gst', '1.0')
   
3. from gi.repository import Gst, GObject, GLib
   
4. 
   
5. # 初始化GStreamer
   
6. Gst.init(None)
   
7. loop = GLib.MainLoop()
   
8. 
   
9. pipeline = Gst.parse_launch("rtspsrc location=rtsp://192.168.0.133:554/test ! rtph264depay ! h264parse ! mppvideodec ! videorate ! video/x-raw,framerate=60/1 ! videoscale ! videoconvert ! video/x-raw,width=1280,height=720 ! autovideosink sync=false")
   
10. 
    
11. 
    
12. # 设置管道状态为播放
    
13. pipeline.set_state(Gst.State.PLAYING)
    
14. 
    
15. # 开始主循环
    
16. try:
    
17.     loop.run()
    
18. except KeyboardInterrupt:
    
19.     # 捕获键盘中断事件，停止管道并退出
    
20.     pipeline.set_state(Gst.State.NULL)
    
21.     loop.quit()

复制代码

此时的延迟是比较小的，约为0.2秒，但是我用下面的管道和程序进行测试则有非常大的延迟，最大时延迟会有一分钟左右：

1. import heartrate; heartrate.trace(browser=True)
   
2. 
   
3. import gi
   
4. import cv2
   
5. import numpy as np
   
6. from gi.repository import Gst
   
7. gi.require_version('Gst', '1.0')
   
8. # 初始化GStreamer
   
9. Gst.init(None)
   
10. import time  # 导入time模块
    
11. 
    
12. def gst_to_opencv(sample):
    
13.     start_time = time.time()  # 获取函数开始的时间
    
14.     buf = sample.get_buffer()
    
15.     caps = sample.get_caps()
    
16.     print("图像数据格式:", caps.to_string())
    
17.     array = np.ndarray(
    
18.         (caps.get_structure(0).get_value('height'),
    
19.          caps.get_structure(0).get_value('width'),
    
20.          3),
    
21.         buffer=buf.extract_dup(0, buf.get_size()),
    
22.         dtype=np.uint8)
    
23.     end_time = time.time()  # 获取函数结束的时间
    
24.     print("gst_to_opencv耗时: {:.5f}秒".format(end_time - start_time))  # 打印函数耗时
    
25.     return array
    
26. 
    
27. def new_sample(sink, data):
    
28.     sample = sink.emit('pull-sample')
    
29.     start_time = time.time()  # 获取OpenCV处理开始的时间
    
30.     frame = gst_to_opencv(sample)
    
31.     cv2.imshow('Live Video', frame)
    
32.     cv2.waitKey(1)
    
33.     end_time = time.time()  # 获取显示结束的时间
    
34.     print("OpenCV显示耗时: {:.5f}秒".format(end_time - start_time))  # 打印OpenCV处理+显示耗时
    
35.     return Gst.FlowReturn.OK
    
36. 
    
37. # 设置GStreamer管道
    
38. pipeline = Gst.parse_launch('rtspsrc location=rtsp://192.168.0.133:554/test ! rtph264depay ! h264parse ! mppvideodec ! videorate ! video/x-raw,framerate=60/1 ! videoscale ! videoconvert ! video/x-raw,format=BGR,width=1280,height=720 ! appsink emit-signals=True name=mysink')
    
39. 
    
40. sink = pipeline.get_by_name('mysink')
    
41. sink.set_property('sync', False)
    
42. sink.connect('new-sample', new_sample, None)
    
43. pipeline.set_state(Gst.State.PLAYING)
    
44. 
    
45. try:
    
46.     while True:
    
47.         pass  # 循环运行，直到用户关闭窗口
    
48. except KeyboardInterrupt:
    
49.     print("Interrupted by user")
    
50. 
    
51. cv2.destroyAllWindows()
    
52. pipeline.set_state(Gst.State.NULL)
    

复制代码

gst_to_opencv以及new_sample打印出的两个耗时均为毫秒级耗时。
我想请问的是两个程序比较，为什么第二个程序的延迟增加的如此严重，如何能够减小延迟？

zhangzhenzhao

用RGA不要用opencv或者直接用我们的ffmedia
https://wiki.t-firefly.com/zh_CN ... ubuntu.html#ffmedia

oranger

zhangzhenzhao 发表于 2024-4-11 09:04
用RGA不要用opencv或者直接用我们的ffmedia
https://wiki.t-firefly.com/zh_CN/Firefly-Linux-Guide/manua ...

您好，我按照wiki文档里操作后执行demo_opencv.py报错：
Firefly FFMedia: v2.2.4
input source is a rtsp url
INFO: ff_media: connectToServer: connected to server 192.168.0.131:554
INFO: ff_media: RTPSource: RTPSock current buffer size is(425984)
INFO: 16ModuleRtspClient: init: rtsp para w h: 1280 720, ws hs: 1280 720, format: H264
wait...INFO: ff_media: parseRTSPMessage: RTSP message was fragmented
INFO: 10MppDecoder: getTimeoutSample: 0x2d66dd80 frame info changed 1 error 0 discard 0
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 12ModuleMppDec: work: wait for productor ModuleRtspClient timeout
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
INFO: 10RTSPClient: teardownMediaSession: teardown session success
INFO: ff_media: connectToServer: connected to server 192.168.0.131:554
INFO: ff_media: RTPSource: RTPSock current buffer size is(425984)
INFO: ff_media: parseRTSPMessage: RTSP message was fragmented
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 12ModuleMppDec: work: wait for productor ModuleRtspClient timeout
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
INFO: 10RTSPClient: teardownMediaSession: teardown session success
INFO: ff_media: connectToServer: connected to server 192.168.0.131:554
INFO: ff_media: RTPSource: RTPSock current buffer size is(425984)
INFO: ff_media: parseRTSPMessage: RTSP message was fragmented
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 12ModuleMppDec: work: wait for productor ModuleRtspClient timeout
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
ERROR: 10RTSPClient: getOutputDataTimeOut: wait for session to get buffer time out
INFO: 10RTSPClient: teardownMediaSession: teardown session success
INFO: ff_media: connectToServer: connected to server 192.168.0.131:554
INFO: ff_media: RTPSource: RTPSock current buffer size is(425984)
INFO: ff_media: parseRTSPMessage: RTSP message was fragmented