I recently had a need to stream multiple consumer video sources (e.g. USB-connected webcams) over the internet with some interesting conditions. Jump down to the last section if you want an overview of the final solution.
A modest proposal
- The video sources and any audio sources need to be in sync for the viewer.
- The latency from video capture to viewer’s eyeball needs to be low latency enough for useful real-time communications
- Video sources may be added/removed occasionally
- Some user-side flexibility in presentation is desirable since this is semi-experimental
- Future users of this setup are familiar with only Windows
Given that this is all going to end up on one screen on the viewer’s side anyway, I deduced that emitting one stream would be the easiest way to keep the frames in sync across the internet. From there, the choice of OBS (and/or variants) was easy – worked like a charm for actually rendering the frames!
What didn’t work like a charm? Delivering the video over the internet with low latency.
This should be easy
Armed with a hardware-encoding capable machine that’s supported by OBS out of the box, there’s just a little configuration left.
Some quick forum skimming reveals that OBS supports streaming over UDP via configuring “Custom Output” recording. Our problems are solved! It even mentions playing via VLC.
And who doesn’t like VLC? It’s easily one of the most reliable players around. While it won’t win any awards for UI, it just works and has for well over a decade.
The journey begins
A few minutes later after relatively easy configuration, I have a video stream functioning over the VPN! I’m across the country from the broadcasting computer, so I’ve already got live test conditions. A few things become immediately clear:
- The video stream degrades spuriously after 40-60 seconds into green or gray motion-compensated garbage.
- If I move the video source on the frame in OBS via TeamViewer, it takes a long time to appear in the stream – 6 to 20 seconds.
I fire up a command line and run ‘ping -l 1024 8.8.4.4’ – ping with 1KB payloads to Google’s secondary public name server – and let it sizzle for a few minutes. Latency is pretty good (~20ms), but we’re dropping packets every 30-60 seconds. (Side note: I’ve seen plenty of circumstances where small, default 32 or 64 byte ICMP payloads make it through just fine, but larger payloads die much more often – I always recommend larger ping payloads when testing for applications that transfer large packets.)
That explains the spurious degradation. The remote end doesn’t have the best connection, and there’s not many options in that location – it’s likely something I can’t fix. “Packet loss tolerance” just got added to my list of requirements.
The latency, however, was puzzling – that’s a very long time for such a low latency connection. Even setting the encoder at 256kbps, well below the tested upstream bandwidth of the connection, the latency remained.
A stream you can rely on
At this point, I decided to tackle the reliability issue first since laggy video was better than hopelessly-corrupted video. If you know anything about UDP, it’s that packet delivery isn’t guaranteed. There’s plenty of stream control protocols out there: RTP, RTSP, RTMP, RTMFP, FTL, SRT, and WebRTC.
For reasons of documented latency properties, RTP and RTSP were easy to eliminate. RTMFP sounded pretty good and is open, but it has limited supporting software and seems to be mostly for Flash – no thanks. FTL seemed to also be too proprietary. That left RTMP, SRT, and WebRTC. And RTMP is supported by practically everything. Despite not having perfect latency properties, that seemed like the way to go.
Middlemen, please apply
VLC wants to be an RTMP client that receives a stream. OBS wants to be an RTMP client that publishes a stream. Neither one wants to act as a server.
If the video streaming domain is rife with anything, it’s people trying to sell you SaaS solutions. After a hacking through the Googletisements (hm, maybe I should switch to Duckduckgo?), I found MonaServer — surprisingly decent software for how awkward the project and interface is. There’s some bizarre caveats, though:
- There are no precompiled binaries. Google helpfully filled in the blank with this link which, I thought, was what I wanted. More on that later.
- This helpful guide didn’t quite work out for me. After following the guide, I had to carefully watch this video to get it to finally work.
- In case this helps someone: the key insight I gained from the video is that I needed to create a folder that matched the stream key in the www folder.
Great, OBS connects to MonaServer as a publisher. VLC connects to MonaServer as a subscriber. I can see the video stream. It’s actually much more reliable than UDP streaming, despite the same packet loss properties!
Victory! Bring out the band and fireworks! … wait, where are the band and fireworks? … 6 to 20 seconds later, the band and fireworks finally arrive.
Glimpsing the end of the tunnel
At this point, I decided to inspect the network’s properties a bit more carefully. Running the same ping test, but this time over VPN routes, it becomes clear the latency properties of the VPN aren’t nearly as good as they ought to be. At least it’s not dropping packets. Wait a minute…
Tunneling VPN data can happen over UDP or TCP. Generally, you should tunnel over UDP since any protocol you’re using should handle packet loss if it needs anyway, and protocols relying on UDP for speed (like, say, various video streaming protocols) won’t get it if tunneling over UDP.
When tunneling over UDP if the network you’re tunneling over has packet loss, you should also have packet loss on your VPN. A quick check in my VPN’s configuration file on the client side, and it’s clear to see it’s using TCP. No problem, easy fix! I’ll just switch that to UDP…
Or not! My VPN server is actually a consumer router, and it only supports tunneling over TCP. This is somewhat ironic, since I burned quite a bit of time ensuring the VPN configuration on the router survived router reboots properly. (Turned out it didn’t like nist.gov’s NTP servers, so after reboots it wouldn’t recover its clock which secure sockets rely upon — switched the NTP server to the NTP pool servers, problem solved.)
Dangerously fast is too slow
I configured the router to forward the port to the encoding computer for testing purposes. Previously, behind the VPN, this was theoretically secure. Now anyone who port scans the external IP can figure out that there’s an interesting port open that serves video. This obviously isn’t a long term solution, but good enough for testing.
I point VLC at the external IP and RTMP port — and it works! And it’s around 2 seconds consistently, too! That’s much better, but still 4x where I want to be.
At this point, there’s roughly five sources of latency I can think of:
- Frame rendering in OBS
- Frame encoding in OBS
- OBS publishing to MonaServer
- MonaServer streaming to subscribers
- Any buffering done by the subscriber’s player
Frame rendering and encoding seem unlikely. I’m using hardware encoding designed for 4k 60fps streaming, but for 1080p 30fps. In fact, OBS has helpful statistics in View >> Stats which make it pretty clear it’s not getting frames to output that’s the issue.
All three
Fast, cheap, and good. You can’t have all three, right?
Despite RTMP being well supported, according to people more versed with the particulars, it still generally isn’t suitable for less than half a second of latency. And in this case, I have two serial RTMP pathways. I need a faster protocol. I need all three.
The remaining protocol challengers are SRT and WebRTC, neither of which released OBS supports out of the box. Gstreamer supports both. OBS doesn’t interface with Gstreamer but through a plugin that I’d have to compile.
And then I noticed something. MonaServer’s logs mentioned something about SRT. That’s weird — MonaServer didn’t mention anything about SRT in its source code…
Free upgrade, but you’ll never guess where
Not only had Google directed me to precompiled binaries, but in fact MonaServer 2 precompiled binaries! How did I miss this? Neither the Github or forum post mentioned MonaServer 2, I think?
But this is for the best! That means I can start using SRT right away – RTMP to MonaServer 2, STP to the end client. In fact, it’s already opening an SRT port — it should be as simple as pointing VLC at the right SRT-formatted URL. Right?
Wrong. MonaServer 2 requires SRT streams to be hard-coded if your player doesn’t request a streamid through the SRT protocol. VLC doesn’t support this.
But, the error messages does mention you can configure a static route. (Routing like web framework routing, not like IP routing.) The catch is it doesn’t tell you how.
So, I tooled around in MonaServer 2’s configuration (INI) file. There’s a section headed “SRT” — it must go in there somewhere! But the configuration syntax is not obvious, nor is it documented in the INI file. Time to hit the source code, helpfully on Github as well.
Your own worst troll
Hours later tooling around in C++ source, and I’ve deduced through the layers vaguely named of packet deserialization classes that in fact the SRT section isn’t what I’m looking for. I’m about ready to give up, but I decide to read the verbosely documented configuration file one more time – this time in reverse in case it helps me catch anything.
Embarrassingly, there’s examples of nearly what I’m looking for commented out at the very bottom for configuring static routes. In fact, if I didn’t have the ability to read C++ source, I would have looked at the configuration file harder to begin with.
Nonetheless, with configuration in place, I give it a shot, and what do you know? It’s a little better — somewhere around one and half seconds of latency. If I can get the OBS side off of RTMP, too, it should become even better.
Mirror, mirror
And still, OBS doesn’t support SRT out of the box. However, a pull-request had recently been merged in to OBS to support any protocol the FFMPEG libraries support. Unfortunately, this version wasn’t out yet, so I compiled OBS which took around an hour after getting the development environment set up.
I fired it up on the remote machine, tried configuring SRT, but ironically the provided compiled FFMPEG library dependencies don’t support SRT.
This was a bit annoying since, really, something as robust as SRT is heavyweight for simply streaming some packets over a 100% reliable connection (localhost to localhost). If only there was a protocol that did this out of the box that was supported by MonaServer 2 – heeeey, wait. What else does MonaServer support besides RTMP and SRT?
Our good friend UDP. Not so great for streaming over the internet. But localhost to localhost? Perfect.
My troll clearly wasn’t gone yet — I compiled the latest OBS for no benefit!
A hero steps forward
At this point, the latency is hovering around 1.1 seconds. Still about twice as slow as I’d deem acceptable, but within striking distance.
There’s one piece of software I hadn’t yet scrutinized heavily enough. My trusty friend, VLC. VLC is made to just work. I didn’t need “just works” — I need “fast”.
After tinkering with VLC’s settings and using a nightly build of VLC for proper SRT support, I determined I needed something more like a Swiss army knife and less like an end-user product. I needed something I could tinker with until a solution fell out.
Gstreamer is that Swiss army knife. Like fooling around with modular synthesizers, Gstreamer has so many lovely knobs to twist and switches to toggle until you end up with that perfect sound.
Blazing fast
The magic for Gstreamer looks like:
gst-launch-1.0 srtclientsrc uri=srt://EXTERNALIP:PORT ! decodebin ! autovideosink sync=false
Replace EXTERNALIP:PORT with your own, of course. This has two major flaws.
- No audio.
- Inaccurate playback speed due to sync=false
However, the latency is about the same as TeamViewer which itself is designed for using computers interactively.
My use case is a little different — I need audio and video coming across the wire with a sense of time portrayed reasonably well.
The last coat of paint
Turns our gst-launch has a more straightforward way to play back a stream:
gst-launch-1.0 playbin uri=srt://EXTERNALIP:PORT latency=50000000
This includes handling audio. On the OBS side I chose libopus as the encoder in the Custom Output record settings. The Opus codec has excellent quality and very low latency, so it’s a natural choice for my application. One wrinkle is that libopus is picky about audio frequencies, so be sure to your audio sample rate to 48Khz instead of the default 44.1Khz.
The latency parameter allows me to choose the amount of time to buffer for. If this is zero, it behaves the same as sync=false with autovideosink. With audio this is especially noticeable since there’s lots of crackling related to the audio frames being dumped out at somewhat arbitrary times.
The docs for playbin don’t seem to explicitly state what the unit for the parameter is. Some experimentation was required, but it looks like it’s actually in picoseconds.
In the final setup, the video delay is less than .5 seconds — I suspect it’s around .3 seconds, but it’s getting small enough so it’s hard to measure remotely, and it’s good enough so that I don’t need to measure it carefully at this time.
Out of the woods
Of course, the story isn’t over, but the path forward is much clearer now.
The path forward looks like:
- Encrypt the stream adequately since it’s publicly exposed
- Integrate the rest of the audio sources
- Configure the final presentation to the user so it can be launched from an icon on both ends. I may even use Gstreamer’s API to make the end user solution a little nicer.
The view from above
As of writing this post, this is where I’ve ended up:
Consumer webcams --(USB)--> OBS OBS input --(nvenc h.265 + libopus)--> OBS output OBS output --(UDP custom output)--> MonaServer 2 MonaServer2 --(SRT not over VPN)--> Gstreamer client