Summary:	ASTERISK-29300: res_rtp_asterisk: When native local bridging the remote SSRC becomes permanent
Reporter:	Sebastian Damm (sdamm)	Labels:	patch
Date Opened:	2021-02-18 04:00:37.000-0600	Date Closed:	2021-03-08 08:13:56.000-0600
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	Resources/res_rtp_asterisk
Versions:	16.10.0 18.2.0	Frequency of Occurrence
Related Issues:	is caused by ASTERISK-28773 Incorrect Sender SSRC in RTCP when p2p rtp bridge is active is caused by ASTERISK-28733 stream: Add support for adding/removing streams during SFU/calls is related to ASTERISK-29295 chan_pjsip: Does not respect SRCUPDATE and SRCCHANGE control frames
Environment:		Attachments:	( 0) 0084_change_ssrc_on_p2p_stop.patch ( 1) debug_log_16.16_29300.txt ( 2) debug_log_16.8_29300.txt ( 3) example_asterisk_16.16.png ( 4) example_asterisk_16.8.png
Description:	This is a regression bug, it was possibly introduced by the change from ASTERISK-28733. When switching between bridges during the call, Asterisk must change the SSRC, too. Until 16.8 it did, after this change it doesn't anymore. In our case, a call starts using native bridge. It just passes the SSRC and sequence number from A to B and vice versa. Then MixMonitor is triggered, forcing Asterisk to switch to simple bridge. In Asterisk 16.8 this lead to a new SSRC and new sequence numbers. In Asterisk 16.9 and up, Asterisk starts using its own sequence numbers, sets a marker bit on the first RTP packet, but does not change the SSRC anymore. Depending on the sequence numbers generated by Asterisk, this can lead to immediate audio loss (when Asterisk seq < other client seq) or audio loss as soon as the call switches back to native bridge (when Asterisk seq > other client seq). The change in behavior can be seen in Wireshark, see the two screenshots attached. Scenario is the same: two clients on my machine call each other, through Asterisk. In-call, MixMon gets engaged and disengaged multiple times. I have filtered for only rtp packets with marker bit set. One hint for reproduction: Even though you can see the fault in Wireshark also on UDP calls, the audio drops appear only with TLS clients.
Comments:	By: Asterisk Team (asteriskteam) 2021-02-18 04:00:38.151-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Sebastian Damm (sdamm) 2021-02-18 04:01:15.812-0600 Example screnshots illustrating the issue By: Joshua C. Colp (jcolp) 2021-02-18 04:10:30.535-0600 Did you bisect the changes to determine ASTERISK-28733 was at fault? By: Joshua C. Colp (jcolp) 2021-02-18 04:13:13.906-0600 We require additional debug to continue with triage of your issue. Please follow the instructions on the wiki [1] for how to collect debugging information from Asterisk. For expediency, where possible, attach the debug with a '.txt' file extension so that the debug will be usable for further analysis. Thanks! [1] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information Also a full debug would be useful for this. The previous behavior is not something I've seen, and I don't see any previous code that would strictly do that kind of stuff so seeing what is going on with the debug level in the old and new version would be helpful as well. By: Joshua C. Colp (jcolp) 2021-02-18 04:20:05.990-0600 As well, which SIP channel driver are you referring to? By: Sebastian Damm (sdamm) 2021-02-18 04:33:33.302-0600 We are using the PJSIP SIP channel driver. And every version since 16.9 has had the bug, and there aren't that many commits between 16.8 and 16.9 that deal with bridge handling, that's why we assume it was introduced by the mentioned change. I'll collect the debug information from both scenarios. By: Joshua C. Colp (jcolp) 2021-02-18 04:36:46.322-0600 Okay, so no bisection occurred it was just an assumption. I wanted to be clear on that or else someone could waste time focusing strictly on that change when it may be elsewhere. By: Joshua C. Colp (jcolp) 2021-02-18 04:38:14.863-0600 Configuration and a test case would also be useful. By: Sebastian Damm (sdamm) 2021-02-18 05:01:01.152-0600 Debug Log of example calls using Asterisk 16.8 and 16.16 By: Sebastian Damm (sdamm) 2021-02-18 05:05:30.266-0600 In our setup, we have TLS clients connected to a Kamailio, which converts TLS to TCP before passing it on to Asterisk. RTPengine encrypts/decrypts the audio, so Asterisk gets TCP SIP and plain UDP RTP. To reproduce and not being forced to enable MixMon via AMI or ARI, you can just setup a call between two clients and then HOLD and unHOLD the call from one side. After that, there is at least one way audio. By: Joshua C. Colp (jcolp) 2021-02-18 05:24:28.961-0600 If you revert the change[1] from ASTERISK-28773 is the issue resolved? [1] https://gerrit.asterisk.org/c/asterisk/+/13924 By: Sebastian Damm (sdamm) 2021-02-18 09:15:33.656-0600 Actually yes, this fixes the issue. And it looks like I was wrong. 16.9 still works fine, but 16.10 was broken. After reverting the mentioned patch, 16.10 works as expected. And 16.16 does as well. Sorry for mentioning the wrong change as a cause. And thanks for finding the right one. By: Joshua C. Colp (jcolp) 2021-02-18 09:19:17.042-0600 [~tsearle] This appears to be a regression caused by your change on ASTERISK-28773 fyi. By: Torrey Searle (tsearle) 2021-02-19 00:47:08.502-0600 Thanks for the ping, I'll take a look into this By: Torrey Searle (tsearle) 2021-02-19 01:43:23.573-0600 [~sdamm] can you test if this patch resolves your issue? By: Sebastian Damm (sdamm) 2021-02-19 04:25:45.776-0600 This fixes the behavior. But Asterisk now behaves differently to reverting your patch from ASTERISK-28773. After reverting the patch (and up to Asterisk 16.9), Asterisk would generate exactly one SSRC for simple bridge. Then it would switch between b-client SSRC in native bridge mode and its own SSRC in simple bridge. Your patch makes Asterisk generate a new SSRC every time the call leaves native bridge. Technically, this is totally okay, but for debugging purposes the former version was nicer. In a trace from yesterday, I had exactly two streams with the same IP/port combination, in the new one, I had 6 of them because I toggled between native and simple bridge a few times. By: Friendly Automation (friendly-automation) 2021-03-08 08:13:58.523-0600 Change 15569 merged by George Joseph: res/res_rtp_asterisk: generate new SSRC on native bridge end [https://gerrit.asterisk.org/c/asterisk/+/15569|https://gerrit.asterisk.org/c/asterisk/+/15569] By: Friendly Automation (friendly-automation) 2021-03-08 08:14:17.846-0600 Change 15490 merged by George Joseph: res/res_rtp_asterisk: generate new SSRC on native bridge end [https://gerrit.asterisk.org/c/asterisk/+/15490|https://gerrit.asterisk.org/c/asterisk/+/15490] By: Friendly Automation (friendly-automation) 2021-03-08 08:14:36.567-0600 Change 15570 merged by George Joseph: res/res_rtp_asterisk: generate new SSRC on native bridge end [https://gerrit.asterisk.org/c/asterisk/+/15570|https://gerrit.asterisk.org/c/asterisk/+/15570]