| Summary: | ASTERISK-28927: Asterisk crash in music on hold | ||||||
| Reporter: | David Cunningham (dcunningham) | Labels: | |||||
| Date Opened: | 2020-06-01 18:54:37 | Date Closed: | 2020-08-12 04:09:33 | ||||
| Priority: | Minor | Regression? | No | ||||
| Status: | Closed/Complete | Components: | Resources/res_musiconhold | ||||
| Versions: | 13.34.0 | Frequency of Occurrence | Frequent | ||||
| Related Issues: |
| ||||||
| Environment: | CentOS 6.10 64 bit | Attachments: | |||||
| Description: | We have an issue with Asterisk crashing in the music on hold code. It happened on Asterisk 11, and is still happening after an upgrade to Asterisk 13. Here is the backtrace:
{noformat} #0 0x00002b7043bd84f5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00002b7043bd9cd5 in abort () at abort.c:92 #2 0x00002b7043c16417 in __libc_message (do_abort=2, fmt=0x2b7043cfec00 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 #3 0x00002b7043c1be5e in malloc_printerr (action=3, str=0x2b7043cfeed8 "double free or corruption (fasttop)", ptr=<value optimized out>, ar_ptr=<value optimized out>) at malloc.c:6360 #4 0x00002b7043c1ecad in _int_free (av=0x2b7064000020, p=0x2b7064048490, have_lock=0) at malloc.c:4846 #5 0x00002b70e0003d54 in moh_scan_files (class=0x2b72cc01c078) at res_musiconhold.c:1101 #6 0x00002b70e0008c34 in local_ast_moh_start (chan=0x2b706401aec8, mclass=<value optimized out>, interpclass=0x0) at res_musiconhold.c:1638 #7 0x00002b70e0005638 in start_moh_exec (chan=0x2b706401aec8, data=0x2b7053776d16 "default") at res_musiconhold.c:904 #8 0x0000000000564535 in pbx_exec (c=0x2b706401aec8, app=0xf4e5c0, data=0x2b7053776d16 "default") at pbx_app.c:494 #9 0x00002b70c8616306 in handle_exec (chan=0x2b706401aec8, agi=0x2b70537787f0, argc=3, argv=0x2b7053776850) at res_agi.c:3140 #10 0x00002b70c8618788 in agi_handle_command (chan=0x2b706401aec8, agi=0x2b70537787f0, buf=<value optimized out>, dead=0) at res_agi.c:4045 #11 0x00002b70c8619025 in run_agi (chan=0x2b706401aec8, request=0x2b7053777560 "agi://127.0.0.1/product?scustomer=511&stype=queue&snumber=505&ctype=phone&cnumber=7049103&append_callerid=&screen=0&loopback=1", agi=0x2b70537787f0, pid=-1, status=0x2b7053778858, dead=0, argc=1, argv=0x2b7053777e28) at res_agi.c:4245 #12 0x00002b70c8619d20 in agi_exec_full (chan=0x2b706401aec8, data=<value optimized out>, enhanced=<value optimized out>, dead=0) at res_agi.c:4531 #13 0x0000000000564535 in pbx_exec (c=0x2b706401aec8, app=0xed26e0, data=0x2b7053778ad0 "agi://127.0.0.1/product?scustomer=511&stype=queue&snumber=505&ctype=phone&cnumber=7049103&append_callerid=&screen=0&loopback=1") at pbx_app.c:494 #14 0x0000000000559da6 in pbx_extension_helper (c=0x2b706401aec8, con=<value optimized out>, context=0x2b706401b880 "product-call-exten", exten=0x2b706401b8d0 "2", priority=1, label=0x0, callerid=0x2b70640460c0 "5129477862", action=E_SPAWN, found=0x2b705377bc9c, combined_find_spawn=1) at pbx.c:2907 #15 0x0000000000561b0d in ast_spawn_extension (c=0x2b706401aec8, args=0x0) at pbx.c:4151 #16 __ast_pbx_run (c=0x2b706401aec8, args=0x0) at pbx.c:4328 #17 0x000000000056325b in pbx_thread (data=<value optimized out>) at pbx.c:4650 #18 0x00000000005d215e in dummy_start (data=<value optimized out>) at utils.c:1239 #19 0x00002b7042fcaaa1 in start_thread (arg=0x2b705377c700) at pthread_create.c:301 #20 0x00002b7043c8ec4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 {noformat} Asterisk is configured with a number of entries in /etc/asterisk/musiconhold.conf with streaming music, like this: [bloomberg] mode=custom application=/usr/bin/mpg123 -q -s --mono -r 8000 -f 1024 --rva-radio --rva-mix --rva-album --rva-audiophile -b 1024 -@ http://tunein.streamguys1.com/bloomberg991 The mpg123 program is at version 1.20.1. Please let us know if you need any other information, thank you. | ||||||
| Comments: | By: Asterisk Team (asteriskteam) 2020-06-01 18:54:37.983-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: George Joseph (gjoseph) 2020-06-02 10:01:33.053-0500 David, can you get us the full output of ast_coredumper with the following options... {{/var/lib/asterisk/scripts/ast_coredumper --tarball-coredumps --no-default-search <path_to_coredump>}}. Don't attach it here, instead upload it to the provider of your choice, like Google Drive, DropBox, etc, and email the link to asteriskteam@digium.com with the subject "ASTERISK-28927: Coredumps". By: Sean Bright (seanbright) 2020-06-02 13:46:54.078-0500 You shouldn't be able to get into that code path unless you were using realtime and files mode. Can you attach your musiconhold.conf and extconfig.conf files? By: David Cunningham (dcunningham) 2020-06-02 17:23:44.903-0500 Thank you, I've sent that email. The Dropbox folder also has the musiconhold.conf and extconfig.conf, and we are using realtime and files. Note that when sing ast_coredumper we did get a warning at the end: Processing core-asterisk-10734-1590780338 Creating ./core-asterisk-10734-1590780338-thread1.txt Creating ./core-asterisk-10734-1590780338-brief.txt Creating ./core-asterisk-10734-1590780338-full.txt Creating ./core-asterisk-10734-1590780338-locks.txt Creating ./core-asterisk-10734-1590780338.tar.gz cp: cannot stat `/usr/lib64/libasterisk*': No such file or directory By: George Joseph (gjoseph) 2020-06-05 07:46:01.267-0500 From #asterisk-dev <@gtjoseph> seanbright: Are you still looking at the MOH issue? <@seanbright> gtjoseph: no. the issue is insufficient locking of the struct mohclass type. <@seanbright> but i will not have time to actually fix it in the near future <@gtjoseph> ok, can you just add your findings to the issue? <@seanbright> sure. two threads in moh_scan_files for the same MoH class at the same time is "bad" By: David Cunningham (dcunningham) 2020-06-21 19:37:25.685-0500 Hello, would you have an idea of an ETA to resolution please? By: Joshua C. Colp (jcolp) 2020-06-22 03:45:10.907-0500 The issue has been accepted and is in queue. There is no ETA or timeframe. By: Friendly Automation (friendly-automation) 2020-08-11 16:07:04.303-0500 Change 14686 merged by Friendly Automation: res_musiconhold.c: Prevent crash with realtime MoH [https://gerrit.asterisk.org/c/asterisk/+/14686|https://gerrit.asterisk.org/c/asterisk/+/14686] By: Friendly Automation (friendly-automation) 2020-08-11 16:59:40.700-0500 Change 14671 merged by Friendly Automation: res_musiconhold.c: Prevent crash with realtime MoH [https://gerrit.asterisk.org/c/asterisk/+/14671|https://gerrit.asterisk.org/c/asterisk/+/14671] By: Friendly Automation (friendly-automation) 2020-08-11 16:59:43.800-0500 Change 14673 merged by Friendly Automation: res_musiconhold.c: Prevent crash with realtime MoH [https://gerrit.asterisk.org/c/asterisk/+/14673|https://gerrit.asterisk.org/c/asterisk/+/14673] By: David Cunningham (dcunningham) 2020-08-11 17:04:48.686-0500 Hello, I see that a change has been committed. Does that mean that it's fixed? Which version should we install to get the fix? Thanks for your help. By: Asterisk Team (asteriskteam) 2020-08-11 17:04:48.994-0500 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. By: Friendly Automation (friendly-automation) 2020-08-11 17:19:39.345-0500 Change 14672 merged by Kevin Harwell: res_musiconhold.c: Prevent crash with realtime MoH [https://gerrit.asterisk.org/c/asterisk/+/14672|https://gerrit.asterisk.org/c/asterisk/+/14672] By: Friendly Automation (friendly-automation) 2020-08-11 17:20:17.177-0500 Change 14684 merged by Kevin Harwell: res_musiconhold.c: Prevent crash with realtime MoH [https://gerrit.asterisk.org/c/asterisk/+/14684|https://gerrit.asterisk.org/c/asterisk/+/14684] By: Joshua C. Colp (jcolp) 2020-08-12 04:09:33.574-0500 The fix has been merged, but a release has not been made. When a release is made the Target Release Versions information on JIRA will be automatically updated with the version numbers it was released in. | ||||||