RTSP视频流处理方法

这里使用Gstreamer + OpenCV来处理RTSP视频流，因此对Gstreamer进行调查。

1. Gstreamer整体框架

Gstreamer是一个用于开发流式多媒体应用的开源框架，采用了基于插件（plugin）和管道（pipeline）的体系结构，框架中的所有的功能模块都被实现成可以插拔的组件（component）， 并且能够很方便地安装到任意一个管道上。由于所有插件都通过管道机制进行统一的数据交换，因此很容易利用已有的各种插件“组装”出一个功能完善的多媒体应用程序。
Nvidia为Gstreamer开发了许多plugin，这些plugin能够利用Nvidia硬件进行加速。Nvidia的deepstream就是基于gstreamer开发的。

下图是对基于Gstreamer框架的应用的简单分层：

1.1 Media Applications

最上面一层为应用，比如gstreamer自带的一些工具（gst-launch，gst-inspect等），以及基于gstreamer封装的库（gst-player，gst-rtsp-server，gst-editing-services等)根据不同场景实现的应用。

1.2 Core Framework

中间一层为Core Framework，主要提供：

• 上层应用所需接口
• Plugin的框架
• Pipline的框架
• 数据在各个Element间的传输及处理机制
• 多个媒体流（Streaming）间的同步（比如音视频同步）
• 其他各种所需的工具库

1.3 Plugins

最下层为各种插件，实现具体的数据处理及音视频输出，应用不需要关注插件的细节，会由Core Framework层负责插件的加载及管理。主要分类为：

• Protocols：负责各种协议的处理，file，http，rtsp等。
• Sources：负责数据源的处理，alsa，v4l2，tcp/udp等。
• Formats：负责媒体容器的处理，avi，mp4，ogg等。
• Codecs：负责媒体的编解码，mp3，vorbis等。
• Filters：负责媒体流的处理，converters，mixers，effects等。
• Sinks：负责媒体流输出到指定设备或目的地，alsa，xvideo，tcp/udp等。

2. Gstreamer组件

Gstreamer由许多基础的组件构成。

2.1 Element

Element是Gstreamer中最重要的对象类型之一。一个element实现了一个功能（读取文件，解码，输出等），程序需要创建多个element，并按顺序将其串连起来，构成一个完整的pipeline。

• Source Element 数据源元件　只有输出端，它仅能用来产生供管道消费的数据，而不能对数据做任何处理。一个典型的数据源元件的例子是音频捕获单元，它负责从声卡读取原始的音频数据，然后作为数据源提供给其它模块使用。
  
• Filter Element 过滤器元件　既有输入端又有输出端，它从输入端获得相应的数据，并在经过特殊处理之后传递给输出端。一个典型的过滤器元件的例子是音频编码单元，它首先从外界获得音频数据，然后根据特定的压缩算法对其进行编码，最后再将编码后的结果提供给其它模块使用。
  
• Sink Element 接收器元件　只有输入端，它仅具有消费数据的能力，是整条媒体管道的终端。一个典型的接收器元件的例子是音频回放单元，它负责将接收到的数据写到声卡上，通常这也是音频处理过程中的最后一个环节。
  

2.2 Pad

Pad是一个element的输入/输出接口，分为src pad（生产数据）和sink pad（消费数据）两种。
两个element必须通过pad才能连接起来，pad拥有当前element能处理数据类型的能力（capabilities），会在连接时通过比较src pad和sink pad中所支持的能力，来选择最恰当的数据类型用于传输。如果element不支持，程序会直接退出。
在element通过pad连接成功后，数据会从上一个element的src pad传到下一个element的sink pad然后进行处理。当element支持多种数据处理能力时，我们可以通过Cap来指定数据类型.

例如，下面的命令通过Cap指定了视频的宽高，videotestsrc会根据指定的宽高产生相应数据：

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1280,height=720" ! autovideosink

2.3 Bin和Pipeline

Bin是一个容器，用于管理多个element，改变bin的状态时，bin会自动去修改所包含的element的状态，也会转发所收到的消息。如果没有bin，我们需要依次操作我们所使用的element。通过bin降低了应用的复杂度。
Pipeline继承自bin，为程序提供一个bus用于传输消息，并且对所有子element进行同步。当将pipeline的状态设置为PLAYING时，pipeline会在一个/多个新的线程中通过element处理数据。

下面通过一个文件播放的例子来熟悉上述提及的概念：
测试文件：sintel_trailer-480p.ogv

gst-launch-1.0 filesrc location=sintel_trailer-480p.ogv ! oggdemux name=demux ! queue ! vorbisdec ! autoaudiosink demux. ! queue ! theoradec ! videoconvert ! autovideosink

通过上面的命令播放文件时，会创建如下pipeline：

可以看到这个pipeline由8个element构成，每个element都实现各自的功能：

• filesrc读取文件
• oggdemux解析文件，分别提取audio，video数据
• queue缓存数据
• vorbisdec解码audio
• autoaudiosink自动选择音频设备并输出
• theoradec解码video
• videoconvert转换video数据格式
• autovideosink自动选择显示设备并输出

不同的element拥有不同数量及类型的pad，只有src pad的element被称为source element，只有sink pad的被称为sink element。

element可以同时拥有多个相同的pad，例如oggdemux在解析文件后，会将audio，video通过不同的pad输出。

3. gstreamer tools

Gstreamer自带了gst-inspect-1.0和gst-launch-1.0等其他命令行工具，我们可以使用这些工具完成常见的处理任务。

3.1 gst-inspect-1.0

查看gstreamer的plugin、element的信息。直接将plugin/element作为参数，会列出其详细信息，包括plugin的功能、Pad的输入输出类型、plugin的属性等。
如果不跟任何参数，会列出当前系统gstreamer所能查找到的所有插件。

# gst-inspect-1.0 rtspsrc
Factory Details:Rank                     none (0)Long-name                RTSP packet receiverKlass                    Source/NetworkDescription              Receive data over the network via RTSP (RFC 2326)Author                   Wim Taymans <wim@fluendo.com>, Thijs Vermeir <thijs.vermeir@barco.com>, Lutz Mueller <lutz@topfrose.de>Plugin Details:Name                     rtspDescription              transfer data via RTSPFilename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstrtsp.soVersion                  1.14.5License                  LGPLSource module            gst-plugins-goodSource release date      2019-05-29Binary package           GStreamer Good Plugins (Ubuntu)Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0GObject+----GInitiallyUnowned+----GstObject+----GstElement+----GstBin+----GstRTSPSrcImplemented Interfaces:GstChildProxyGstURIHandlerPad Templates:SRC template: 'stream_%u'Availability: SometimesCapabilities:application/x-rtpapplication/x-rdtElement has no clocking capabilities.URI handling capabilities:Element can act as source.Supported URI protocols:rtsprtspurtsptrtsphrtsp-sdprtspsrtspsurtspstrtspshPads:noneElement Properties:name                : The name of the objectflags: readable, writableString. Default: "rtspsrc0"parent              : The parent of the objectflags: readable, writableObject of type "GstObject"async-handling      : The bin will handle Asynchronous state changesflags: readable, writableBoolean. Default: falsemessage-forward     : Forwards all children messagesflags: readable, writableBoolean. Default: falselocation            : Location of the RTSP url to readflags: readable, writableString. Default: nullprotocols           : Allowed lower transport protocolsflags: readable, writableFlags "GstRTSPLowerTrans" Default: 0x00000007, "tcp+udp-mcast+udp"(0x00000000): unknown          - GST_RTSP_LOWER_TRANS_UNKNOWN(0x00000001): udp              - GST_RTSP_LOWER_TRANS_UDP(0x00000002): udp-mcast        - GST_RTSP_LOWER_TRANS_UDP_MCAST(0x00000004): tcp              - GST_RTSP_LOWER_TRANS_TCP(0x00000010): http             - GST_RTSP_LOWER_TRANS_HTTP(0x00000020): tls              - GST_RTSP_LOWER_TRANS_TLSdebug               : Dump request and response messages to stdout(DEPRECATED: Printed all RTSP message to gstreamer log as 'log' level)flags: readable, writable, deprecatedBoolean. Default: falseretry               : Max number of retries when allocating RTP ports.flags: readable, writableUnsigned Integer. Range: 0 - 65535 Default: 20timeout             : Retry TCP transport after UDP timeout microseconds (0 = disabled)flags: readable, writableUnsigned Integer64. Range: 0 - 18446744073709551615 Default: 5000000tcp-timeout         : Fail after timeout microseconds on TCP connections (0 = disabled)flags: readable, writableUnsigned Integer64. Range: 0 - 18446744073709551615 Default: 20000000latency             : Amount of ms to bufferflags: readable, writableUnsigned Integer. Range: 0 - 4294967295 Default: 2000drop-on-latency     : Tells the jitterbuffer to never exceed the given latency in sizeflags: readable, writableBoolean. Default: falseconnection-speed    : Network connection speed in kbps (0 = unknown)flags: readable, writableUnsigned Integer64. Range: 0 - 18446744073709551 Default: 0nat-method          : Method to use for traversing firewalls and NATflags: readable, writableEnum "GstRTSPNatMethod" Default: 1, "dummy"(0): none             - None(1): dummy            - Send Dummy packetsdo-rtcp             : Send RTCP packets, disable for old incompatible server.flags: readable, writableBoolean. Default: truedo-rtsp-keep-alive  : Send RTSP keep alive packets, disable for old incompatible server.flags: readable, writableBoolean. Default: trueproxy               : Proxy settings for HTTP tunneling. Format: [http://][user:passwd@]host[:port]flags: readable, writableString. Default: nullproxy-id            : HTTP proxy URI user id for authenticationflags: readable, writableString. Default: nullproxy-pw            : HTTP proxy URI user password for authenticationflags: readable, writableString. Default: nullrtp-blocksize       : RTP package size to suggest to server (0 = disabled)flags: readable, writableUnsigned Integer. Range: 0 - 65536 Default: 0user-id             : RTSP location URI user id for authenticationflags: readable, writableString. Default: nulluser-pw             : RTSP location URI user password for authenticationflags: readable, writableString. Default: nullbuffer-mode         : Control the buffering algorithm in useflags: readable, writableEnum "GstRTSPSrcBufferMode" Default: 3, "auto"(0): none             - Only use RTP timestamps(1): slave            - Slave receiver to sender clock(2): buffer           - Do low/high watermark buffering(3): auto             - Choose mode depending on stream live(4): synced           - Synchronized sender and receiver clocksport-range          : Client port range that can be used to receive RTP and RTCP data, eg. 3000-3005 (NULL = no restrictions)flags: readable, writableString. Default: nulludp-buffer-size     : Size of the kernel UDP receive buffer in bytes, 0=defaultflags: readable, writableInteger. Range: 0 - 2147483647 Default: 524288short-header        : Only send the basic RTSP headers for broken encodersflags: readable, writableBoolean. Default: falseprobation           : Consecutive packet sequence numbers to accept the sourceflags: readable, writableUnsigned Integer. Range: 0 - 4294967295 Default: 2udp-reconnect       : Reconnect to the server if RTSP connection is closed when doing UDPflags: readable, writableBoolean. Default: truemulticast-iface     : The network interface on which to join the multicast groupflags: readable, writableString. Default: nullntp-sync            : Synchronize received streams to the NTP clockflags: readable, writableBoolean. Default: falseuse-pipeline-clock  : Use the pipeline running-time to set the NTP time in the RTCP SR messages(DEPRECATED: Use ntp-time-source property)flags: readable, writable, deprecatedBoolean. Default: falsesdes                : The SDES items of this sessionflags: readable, writableBoxed pointer of type "GstStructure"tls-validation-flags: TLS certificate validation flags used to validate the server certificateflags: readable, writableFlags "GTlsCertificateFlags" Default: 0x0000007f, "validate-all"(0x00000001): unknown-ca       - G_TLS_CERTIFICATE_UNKNOWN_CA(0x00000002): bad-identity     - G_TLS_CERTIFICATE_BAD_IDENTITY(0x00000004): not-activated    - G_TLS_CERTIFICATE_NOT_ACTIVATED(0x00000008): expired          - G_TLS_CERTIFICATE_EXPIRED(0x00000010): revoked          - G_TLS_CERTIFICATE_REVOKED(0x00000020): insecure         - G_TLS_CERTIFICATE_INSECURE(0x00000040): generic-error    - G_TLS_CERTIFICATE_GENERIC_ERROR(0x0000007f): validate-all     - G_TLS_CERTIFICATE_VALIDATE_ALLtls-database        : TLS database with anchor certificate authorities used to validate the server certificateflags: readable, writableObject of type "GTlsDatabase"tls-interaction     : A GTlsInteraction object to promt the user for password or certificateflags: readable, writableObject of type "GTlsInteraction"do-retransmission   : Ask the server to retransmit lost packetsflags: readable, writableBoolean. Default: truentp-time-source     : NTP time source for RTCP packetsflags: readable, writableEnum "GstRTSPSrcNtpTimeSource" Default: 0, "ntp"(0): ntp              - NTP time based on realtime clock(1): unix             - UNIX time based on realtime clock(2): running-time     - Running time based on pipeline clock(3): clock-time       - Pipeline clock timeuser-agent          : The User-Agent string to send to the serverflags: readable, writableString. Default: "GStreamer/1.14.5"max-rtcp-rtp-time-diff: Maximum amount of time in ms that the RTP time in RTCP SRs is allowed to be ahead (-1 disabled)flags: readable, writableInteger. Range: -1 - 2147483647 Default: 1000rfc7273-sync        : Synchronize received streams to the RFC7273 clock (requires clock and offset to be provided)flags: readable, writableBoolean. Default: falsemax-ts-offset-adjustment: The maximum number of nanoseconds per frame that time stamp offsets may be adjusted (0 = no limit).flags: readable, writableUnsigned Integer64. Range: 0 - 18446744073709551615 Default: 0max-ts-offset       : The maximum absolute value of the time offset in (nanoseconds). Note, if the ntp-sync parameter is set the default value is changed to 0 (no limit)flags: readable, writableInteger64. Range: 0 - 9223372036854775807 Default: 3000000000default-rtsp-version: The RTSP version that should be tried first when negotiating version.flags: readable, writableEnum "GstRTSPVersion" Default: 16, "1-0"(0): invalid          - GST_RTSP_VERSION_INVALID(16): 1-0              - GST_RTSP_VERSION_1_0(17): 1-1              - GST_RTSP_VERSION_1_1(32): 2-0              - GST_RTSP_VERSION_2_0backchannel         : The type of backchannel to setup. Default is 'none'.flags: readable, writableEnum "GstRTSPBackchannel" Default: 0, "none"(0): none             - No backchannel(1): onvif            - ONVIF audio backchannelElement Signals:"pad-added" :  void user_function (GstElement* object,GstPad* arg0,gpointer user_data);"pad-removed" :  void user_function (GstElement* object,GstPad* arg0,gpointer user_data);"no-more-pads" :  void user_function (GstElement* object,gpointer user_data);"handle-request" :  void user_function (GstElement* object,gpointer arg0,gpointer arg1,gpointer user_data);"on-sdp" :  void user_function (GstElement* object,GstSDPMessage* arg0,gpointer user_data);"select-stream" :  gboolean user_function (GstElement* object,guint arg0,GstCaps* arg1,gpointer user_data);"new-manager" :  void user_function (GstElement* object,GstElement* arg0,gpointer user_data);"request-rtcp-key" :  GstCaps * user_function (GstElement* object,guint arg0,gpointer user_data);"accept-certificate" :  gboolean user_function (GstElement* object,GTlsConnection* arg0,GTlsCertificate* arg1,GTlsCertificateFlags arg2,gpointer user_data);"before-send" :  gboolean user_function (GstElement* object,GstRTSPMessage* arg0,gpointer user_data);Element Actions:"push-backchannel-buffer" :  GstFlowReturn user_function (GstElement* object,guint arg0,GstBuffer* arg1);

3.2 gst-launch-1.0

用于创建及执行一个Pipline，因此通常使用gst-launch先验证相关功能，然后再编写相应应用。
通过上面ogg视频播放的例子，我们已经看到，一个pipeline的多个element之间通过 “!" 分隔，同时可以设置element及Cap的属性。
下面是解析RTSP视频流的pipeline：

gst-launch-1.0 -v rtspsrc location=rtsp://10.201.0.158:8554/vlc ! rtph264depay !  h264parse ! omxh264dec ! nvvidconv ! video/x-raw, width=(int)2048, height=(int)1536, format=(string)BGRx ! videoconvert !  ximagesink sync=false

我们可以通过gst-inspect-1.0工具查看每个plugin的功能和属性，选择合适的插件来构成pipeline。
具体在python实现OpenCV+Gstreamer的方法是：OpenCV提供了cv2.VideoCapture()函数，只需把Gstreamer参数传给该函数即可。
具体代码如下：

def open_cam_rtsp(uri, width, height, latency):gst_str = ("rtspsrc location={} latency={} ! rtph264depay ! h264parse ! omxh264dec ! ""nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! ""videoconvert ! appsink").format(uri, latency, width, height)return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)def open_cam_usb(dev, width, height):# We want to set width and height here, otherwise we could just do:#     return cv2.VideoCapture(dev)gst_str = ("v4l2src device=/dev/video{} ! ""video/x-raw, width=(int){}, height=(int){}, format=(string)RGB ! ""videoconvert ! appsink").format(dev, width, height)return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)def open_cam_onboard(width, height):# On versions of L4T previous to L4T 28.1, flip-method=2# Use Jetson onboard cameragst_str = ("nvcamerasrc ! ""video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)30/1 ! ""nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! ""videoconvert ! appsink").format(width, height)return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

4. 参考链接

https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html
https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html
https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html