Writen rtmp send error 10038

Random RTMP Disconnects #6366

Comments

Operating System Info

Other OS

OBS Studio Version

OBS Studio Version (Other)

OBS Studio Log URL

OBS Studio Crash Log URL

Expected Behavior

No disconnects over a stable local connection.

Current Behavior

Random disconnects over a stable local connection to an Owncast server that is essentially dead idle, no transcoding, nobody watching, 5-10% CPU use.

What I see in the log is this exact set of messages every time, it can happen in 5 minutes, 10 minutes, 45, and hour, 2 hours, etc.

10:49:07.638: ==== Streaming Start ===============================================
10:58:38.205: User added source ‘Image Slide Show’ (slideshow) to scene ‘Test Scene’
13:38:24.421: WriteN, RTMP send error 10054 (861 bytes)
13:38:24.421: WriteN, RTMP send error 10053 (50 bytes)
13:38:24.421: WriteN, RTMP send error 10038 (42 bytes)
13:38:24.421: [rtmp stream: ‘adv_stream’] Disconnected from rtmp://192.168.11.132/live
13:38:24.421: Output ‘adv_stream’: stopping
13:38:24.421: Output ‘adv_stream’: Total frames output: 304502 (304625 attempted)
13:38:24.421: Output ‘adv_stream’: Total drawn frames: 304689 (304704 attempted)
13:38:24.421: Output ‘adv_stream’: Number of lagged frames due to rendering lag/stalls: 15 (0.0%)
13:38:24.421: Output ‘adv_stream’: Number of dropped frames due to insufficient bandwidth/connection stalls: 123 (0.0%)
13:38:24.421: Output ‘adv_stream’: Reconnecting in 1 seconds..
13:38:24.421: [rtmp stream: ‘adv_stream’] Freeing 193 remaining packet
[2022-04-21 10-40-53.txt]

Steps to Reproduce

Start streaming to a standard rtmp endpoint such as a local Owncast server, wait for a disconnect to happen.

Anything else we should know?

The system doing the streaming is essentially idle other than the stream, Ryzen 7 5700G system with 64gb of ram, network is all the same subnet/vlan, all gigabit, no noticeable issues, all hard wired. Not streaming anything but some static images and some background photos as a test

I’ve confirmed my MTU settings are correct, 1500, I’ve tested the window size on both ends, 1500 is the exact setting that works. Here’s a WinMTR run to the rtmp server with a 0.01sec delay and 1472 byte packets which with the 28 byte overhead makes 1500, really threw traffic out there for a while to make sure:

Host — % Sent Recv Best Avrg Wrst Last
192.168.11.132 — 0 507520 507520 8
________________________________________________ ______ ______ ______ ______ ______ ______

Streaming to Owncast 0.0.11 which is only doing video passthrough and the system is essentially dead idle, nobody watching, its doing nothing else, its hard wired, i7-3520M CPU which should be totally overkill.

Same thing happens on another system I’ve tried to stream from, a Ryzen 7 4800H with 32gb of ram doing the same basic static images with some slideshow pictures as a test, also hard wired, absolutely dead idle otherwise and not being touched.

On the Owncast server end, the only entries on the log for the period of time the stream happened and then disconnected is:

info 1:39:03 PM 04/21/2022 Video transcoder started using x264 with 1 stream variants.
info 1:39:03 PM 04/21/2022 Inbound stream connected.
info 1:39:01 PM 04/21/2022 Inbound stream disconnected.
info 10:49:45 AM 04/21/2022 Video transcoder started using x264 with 1 stream variants.
info 10:49:45 AM 04/21/2022 Inbound stream connected.

Been battling with this for a few days, totally at a loss of what the problem is, the Owncast server is running absolutely stable, the network is all wired gigabit, all matching MTU, no jumbo packets, all systems involved are completely overkill for what they’re doing and not at all CPU bound, Owncast doesn’t record any errors on its end but OBS states 3 separate RTMP send error’s, always the same. I’ve seen this exact set of errors reported multiple times with this exact same sort of issue, random disconnects, but no resolution ever posted beyond someone who had an MTU mismatch on their local network and others who fessed up to having connection issues to 3rd party servers.

The text was updated successfully, but these errors were encountered:

What bitrate are you streaming at? I’ve been struggling with this a bunch, particularly anytime I go above 20mbit (even streaming to localhost).

Using VBR with a max of 8Mbit, it hovers around 3-4Mbit usually though, I’ve tried everything up to 10 and down to 2. same result every time, random disconnects. I will say I’ve used CBR too, same results. I just use VBR since its what I plan on using once this is all resolved.

When you have problems what sort of utilization are you seeing for the CPU/GPU/etc? I’m really barley breaking a sweat on a single core when I’m testing at what I’m doing above.

This is likely going to be something with your network route between the server and client, or a bug in the streaming server itself. The high bitrate disconnects mentioned is a long-standing bug in nginx-rtmp (and derivatives), and I wouldn’t be surprised to see it somewhere else, but that usually only triggers above 40Mbit.

Even though it’s unlikely this is anything on the OBS side, I will leave this open, just please understand that any kind of troubleshooting or work on this is likely to be very slow going without more information and a repeatable test setup where the issue is proven across multiple environments.

Error 10054 means the connection was uncleanly terminated, either by the server or as a result of the session timing out. I suggest running tcpdump on the client and server and verifying all packets are making it through properly.

This is likely going to be something with your network route between the server and client, or a bug in the streaming server itself. The high bitrate disconnects mentioned is a long-standing bug in nginx-rtmp (and derivatives), and I wouldn’t be surprised to see it somewhere else, but that usually only triggers above 40Mbit.

Even though it’s unlikely this is anything on the OBS side, I will leave this open, just please understand that any kind of troubleshooting or work on this is likely to be very slow going without more information and a repeatable test setup where the issue is proven across multiple environments.

Curious what specific server would you recommend for further testing to see if its Owncast’s implementation its self? I’m up for testing new things 🙂

Error 10054 means the connection was uncleanly terminated, either by the server or as a result of the session timing out. I suggest running tcpdump on the client and server and verifying all packets are making it through properly.

Gotcha, I was thinking of running a pcap and seeing if there was any funny business, its just one of the few things I haven’t gotten around to doing, that’ll probably be the next step and if/when I do it I’ll post the results here for further analysis.

Was able to catch the cause, it was an /odd/ one! The TL;DR is that Intel nics just aren’t as solid as they used to be, e1000e tso issues with enp0s25.

Caught re-transmissions that happened exactly when the disconnect occurred

102194 446.148371 192.168.11.2 192.168.11.132 RTMP 1517 Video Data|Audio Data|Video Data|Audio Data|Audio Data
102195 446.158060 192.168.11.2 192.168.11.132 TCP 4158 60879 → 1935 [PSH, ACK] Seq=134961422 Ack=4328 Win=2100992 Len=4104
102196 446.158089 192.168.11.2 192.168.11.132 RTMP 3726 Video Data
102197 446.224803 192.168.11.2 192.168.11.132 RTMP 2251 Audio Data|Video Data|Audio Data|Audio Data|Video Data
102198 446.291913 192.168.11.2 192.168.11.132 RTMP 4987 Audio Data|Video Data|Audio Data|Audio Data
102199 446.291947 192.168.11.2 192.168.11.132 RTMP 3951 Video Data
102200 446.358152 192.168.11.2 192.168.11.132 RTMP 2079 Audio Data|Video Data|Audio Data|Audio Data|Video Data
102201 446.387456 192.168.11.2 192.168.11.132 TCP 1514 [TCP Retransmission] 60879 → 1935 [PSH, ACK] Seq=134959947 Ack=4328 Win=2100992 Len=1460
102202 446.992287 192.168.11.2 192.168.11.132 TCP 1514 [TCP Retransmission] 60879 → 1935 [PSH, ACK] Seq=134959947 Ack=4328 Win=2100992 Len=1460
102203 448.200816 192.168.11.2 192.168.11.132 TCP 1514 [TCP Retransmission] 60879 → 1935 [PSH, ACK] Seq=134959947 Ack=4328 Win=2100992 Len=1460
102204 450.607768 192.168.11.2 192.168.11.132 TCP 1514 [TCP Retransmission] 60879 → 1935 [PSH, ACK] Seq=134959947 Ack=4328 Win=2100992 Len=1460
102205 463.173924 192.168.11.132 192.168.11.2 TCP 60 1935 → 60879 [FIN, ACK] Seq=4328 Ack=134959959 Win=2988928 Len=0
102206 463.174340 192.168.11.2 192.168.11.132 TCP 78 [TCP Retransmission] 60879 → 1935 [PSH, ACK] Seq=134961407 Ack=4329 Win=2100992 Len=24
102207 463.174520 192.168.11.132 192.168.11.2 TCP 60 1935 → 60879 [RST] Seq=4329 Win=0 Len=0
102208 464.178653 192.168.11.2 192.168.11.132 TCP 66 58629 → 1935 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
102209 464.179018 192.168.11.132 192.168.11.2 TCP 66 1935 → 58629 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128
102210 464.179052 192.168.11.2 192.168.11.132 TCP 54 58629 → 1935 [ACK] Seq=1 Ack=1 Win=2102272 Len=0
102211 464.179091 192.168.11.2 192.168.11.132 RTMP 1591 Handshake C0+C1
102212 464.179304 192.168.11.132 192.168.11.2 TCP 60 1935 → 58629 [ACK] Seq=1 Ack=1538 Win=63616 Len=0
102213 464.179651 192.168.11.132 192.168.11.2 TCP 1514 1935 → 58629 [ACK] Seq=1 Ack=1538 Win=64128 Len=1460
102214 464.179721 192.168.11.132 192.168.11.2 TCP 1514 1935 → 58629 [ACK] Seq=1461 Ack=1538 Win=64128 Len=1460
102215 464.179721 192.168.11.132 192.168.11.2 RTMP 207 Handshake S0+S1+S2
102216 464.179734 192.168.11.2 192.168.11.132 TCP 54 58629 → 1935 [ACK] Seq=1538 Ack=3074 Win=2102272 Len=0
102217 464.179747 192.168.11.2 192.168.11.132 RTMP 1590 Handshake C2
102218 464.180185 192.168.11.132 192.168.11.2 TCP 60 1935 → 58629 [ACK] Seq=3074 Ack=3074 Win=63616 Len=0
102219 464.180191 192.168.11.2 192.168.11.132 RTMP 271 Set Chunk Size 4096|connect()
102220 464.180615 192.168.11.132 192.168.11.2 TCP 60 1935 → 58629 [ACK] Seq=3074 Ack=3291 Win=64128 Len=0
102221 464.180615 192.168.11.132 192.168.11.2 RTMP 305 Window Acknowledgement Size 2500000|Set Peer Bandwidth 2500000,Dynamic|Set Chunk Size 65536
102222 464.180649 192.168.11.2 192.168.11.132 RTMP 106 releaseStream()
102223 464.180901 192.168.11.132 192.168.11.2 TCP 60 1935 → 58629 [ACK] Seq=3325 Ack=3343 Win=64128 Len=0
102224 464.180909 192.168.11.2 192.168.11.132 RTMP 135 FCPublish()|createStream()
102225 464.181145 192.168.11.132 192.168.11.2 TCP 60 1935 → 58629 [ACK] Seq=3325 Ack=3424 Win=64128 Len=0
102226 464.181209 192.168.11.132 192.168.11.2 RTMP 95 _result()
102227 464.181234 192.168.11.2 192.168.11.132 RTMP 111 publish()

Lead me to catch the bridged adapter the RTMP server was on hanging which only seemed to happen when streaming, no other VM caused it, problem would not manifest unless streaming, something about that workload caused things to break badly.

Apr 28 11:12:13 proxmox kernel: e1000e 0000:00:19.0 enp0s25: Detected Hardware Unit Hang:
TDH
TDT
next_to_use
next_to_clean
buffer_info[next_to_clean]:
time_stamp
next_to_watch
jiffies
next_to_watch.status
MAC Status
PHY Status
PHY 1000BASE-T Status
PHY Extended Status
PCI Status
Apr 28 11:12:13 proxmox kernel: e1000e 0000:00:19.0 enp0s25: Reset adapter unexpectedly
Apr 28 11:12:13 proxmox kernel: vmbr0: port 1(enp0s25) entered disabled state
Apr 28 11:12:13 proxmox kernel: vmbr0v100: port 1(enp0s25.100) entered disabled state
Apr 28 11:12:17 proxmox kernel: e1000e 0000:00:19.0 enp0s25: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Apr 28 11:12:17 proxmox kernel: vmbr0: port 1(enp0s25) entered blocking state
Apr 28 11:12:17 proxmox kernel: vmbr0: port 1(enp0s25) entered forwarding state
Apr 28 11:12:17 proxmox kernel: vmbr0v100: port 1(enp0s25.100) entered blocking state
Apr 28 11:12:17 proxmox kernel: vmbr0v100: port 1(enp0s25.100) entered forwarding state

ExecUpPost=’/usr/bin/ethtool -K enp0s25 tso off’

Did about 6 hours of streaming to test afterwards, no further issues, no further interface hangs, all boils down to flaky drivers for certain Intel nics, bet other with this unsolvable problem are having the same general problem given the proliferation of this particular driver/chipset combination. Not so much a network problem as much as a device driver and hardware issue with TCP segmentation offload.

Closing this out, issue unrelated to OBS, someone else might find this useful though!

Источник

Question / Help RTMP random disconnects

Essie Hellcat

New Member

First, I’d like to say that I have tried my very best to solve this issue alone, and now I’m having to ask for help because I just don’t know where to go next. I test streamed for 6.5-7 hrs yesterday, changing, testing, and trying to solve this, but to no avail. Really, I changed all sorts of things yesterday, so if you read the log, please keep reading because whatever I changed, I changed back when it was apparently not the fix I was looking for. (What I’m hoping for is one of you geniuses to be able to view my log and go, uh, uncheck THAT box, dummy.)

The PROBLEM: For the past few months, I am able to stream (usually) for up to 3 or 4 hours with no issues, but then my connection to RTMP is severed. It does reconnect, but will d/c and reconnect quite often afterwards. (Previously, I was using OBS classic, and did not have this issue at all, but since switching to Studio, even Classic does this.)

The D/C error most often seems to be:

17:28:51.201: WriteN, RTMP send error 10054 (353 bytes)
17:28:51.201: WriteN, RTMP send error 10054 (79 bytes)
17:28:51.201: WriteN, RTMP send error 10038 (42 bytes)

but is sometimes:

17:07:44.149: WriteN, RTMP send error 10054 (529 bytes)
17:07:44.150: WriteN, RTMP send error 10053 (79 bytes)
17:07:44.150: WriteN, RTMP send error 10038 (42 bytes)

My internet is usually not disconnected at the same time this happens (pandora keeps playing, twitch chat still continues, etc), though sometimes it does. I realize that internet connections are flexible and turbulent things, but the sheer number of disconnects is astounding (and IRRITATING), and does not usually seem to be the issue. I have

4up. I’ve tried different Twitch influx servers, as well, but still, the issue remains. The log from yesterday was on at least 3 different influx servers (Virginia, Chicago, and Miami. maybe New York as well, I can’t recall.)

I also realize my computer is not the best, but I’m not using very much CPU running OBS Studio, Stardew Valley, and 1 chrome window. Last night, I even completely uninstalled OBS classic & OBS studio and reinstalled OBS Studio, so all of my settings are now wiped. I’m just afraid to try streaming again because of this frustration.

Please help, OBS-Wan-Kenobis. you’re my only hope.

Источник

Читайте также:  Андроид прошивка для mtk
Smartadm.ru
Adblock
detector