[Home]

Summary:ASTERISK-30418: Crash in ConfBridge
Reporter:Michael Cargile (mcargile)Labels:
Date Opened:2023-02-07 14:47:51.000-0600Date Closed:2023-02-22 12:00:16.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Applications/app_confbridge
Versions:18.16.0 Frequency of
Occurrence
Related
Issues:
Environment:Server has a 6 core Intel CPU with 16 gigs of RAM. The OS is OpenSuSE 15.3 and is running VicidialAttachments:
Description:My dev system keeps crashing while running Asterisk 18.16.0. I am in the process of validating our software with Asterisk 18. This time it seems to have seg faulted in the ConfBridge code. It crashed previously with a double free, but sadly I had forgotten to compile with DONT_OPTIMIZE so the dump was useless. Here is the back trace for the Seg Fault:

{quote}
Core was generated by `/usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvgcT'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f75374455ae in _int_malloc () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f738acce700 (LWP 11401))]
(gdb) bt
#0  0x00007f75374455ae in _int_malloc () from /lib64/libc.so.6
#1  0x00007f7537447d72 in calloc () from /lib64/libc.so.6
#2  0x0000000000468d7e in __ast_repl_calloc (nmemb=1, size=530, file=0x6a8674 "translate.c", lineno=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at astmm.c:1537
#3  0x0000000000468fa5 in __ast_calloc (nmemb=1, size=530, file=0x6a8674 "translate.c", lineno=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at astmm.c:1607
#4  0x000000000050e3b0 in __ast_frdup (f=0x7f73a40a63b8, file=0x6a8674 "translate.c", line=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at frame.c:346
#5  0x000000000050dd90 in __ast_frisolate (fr=0x7f73a40a63b8, file=0x6a8674 "translate.c", line=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at frame.c:206
#6  0x00000000005d9c92 in ast_trans_frameout (pvt=0x7f73a40a63b0, datalen=0, samples=0) at translate.c:465
#7  0x00000000005d9cb6 in default_frameout (pvt=0x7f73a40a63b0) at translate.c:470
#8  0x00000000005da414 in ast_translate (path=0x7f73a40a63b0, f=0x7f73a403bb00, consume=0) at translate.c:624
#9  0x00000000004ff681 in ast_writestream (fs=0x7f73a408b080, f=0x7f73a403bb00) at file.c:287
#10 0x00000000004b2343 in ast_write_stream (chan=0x7f74f801f010, stream_num=0, fr=0x7f749001df40) at channel.c:5480
#11 0x0000000000496ada in bridge_channel_handle_write (bridge_channel=0x7f73a4095590) at bridge_channel.c:2469
#12 0x000000000049732d in bridge_channel_wait (bridge_channel=0x7f73a4095590) at bridge_channel.c:2782
#13 0x0000000000497c83 in bridge_channel_internal_join (bridge_channel=0x7f73a4095590) at bridge_channel.c:2947
#14 0x000000000047b862 in ast_bridge_join (bridge=0x7f740002b560, chan=0x7f74f801f010, swap=0x0, features=0x7f738acca7d0, tech_args=0x7f738acca7f8, flags=0) at bridge.c:1679
#15 0x00007f74d9189bc1 in confbridge_exec (chan=0x7f74f801f010, data=0x7f738acca9b0 "9600074,vici_agent_bridge,vici_customer_user") at app_confbridge.c:2887
#16 0x00000000005545ed in pbx_exec (c=0x7f74f801f010, app=0x21086f0, data=0x7f738acca9b0 "9600074,vici_agent_bridge,vici_customer_user") at pbx_app.c:492
#17 0x000000000054006d in pbx_extension_helper (c=0x7f74f801f010, con=0x0, context=0x7f74f801f9d0 "default", exten=0x7f74f801fa20 "9600074", priority=3, label=0x0, callerid=0x7f74f801c600 "9999024847", action=E_SPAWN, found=0x7f738accdc4c, combined_find_spawn=1)
   at pbx.c:2948
#18 0x0000000000543e7e in ast_spawn_extension (c=0x7f74f801f010, context=0x7f74f801f9d0 "default", exten=0x7f74f801fa20 "9600074", priority=3, callerid=0x7f74f801c600 "9999024847", found=0x7f738accdc4c, combined_find_spawn=1) at pbx.c:4203
#19 0x0000000000544b34 in __ast_pbx_run (c=0x7f74f801f010, args=0x0) at pbx.c:4377
#20 0x0000000000546499 in ast_pbx_run_args (c=0x7f74f801f010, args=0x0) at pbx.c:4749
#21 0x00000000005464c3 in ast_pbx_run (c=0x7f74f801f010) at pbx.c:4758
#22 0x0000000000486fca in ast_bridge_run_after_goto (chan=0x7f74f801f010) at bridge_after.c:530
#23 0x000000000047bc80 in bridge_channel_ind_thread (data=0x7f74f803c210) at bridge.c:1773
#24 0x00000000005e1708 in dummy_start (data=0x7f74f804f1b0) at utils.c:1574
#25 0x00007f753830594a in start_thread () from /lib64/libpthread.so.0
#26 0x00007f75374bdd0f in clone () from /lib64/libc.so.6
(gdb) bt full
#0  0x00007f75374455ae in _int_malloc () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f7537447d72 in calloc () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000000000468d7e in __ast_repl_calloc (nmemb=1, size=530, file=0x6a8674 "translate.c", lineno=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at astmm.c:1537
No locals.
#3  0x0000000000468fa5 in __ast_calloc (nmemb=1, size=530, file=0x6a8674 "translate.c", lineno=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at astmm.c:1607
       p = 0x7f73a40746a0
       __FUNCTION__ = "__ast_calloc"
#4  0x000000000050e3b0 in __ast_frdup (f=0x7f73a40a63b8, file=0x6a8674 "translate.c", line=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at frame.c:346
       out = 0x0
       len = 530
       srclen = 9
       buf = 0x0
       frames = 0x7f73a40746a0
       __PRETTY_FUNCTION__ = "__ast_frdup"
#5  0x000000000050dd90 in __ast_frisolate (fr=0x7f73a40a63b8, file=0x6a8674 "translate.c", line=465, func=0x6a9190 <__PRETTY_FUNCTION__.17450> "ast_trans_frameout") at frame.c:206
       out = 0x7f73a40a6618
       newdata = 0x7f73a403bb00
       __PRETTY_FUNCTION__ = "__ast_frisolate"
#6  0x00000000005d9c92 in ast_trans_frameout (pvt=0x7f73a40a63b0, datalen=0, samples=0) at translate.c:465
       f = 0x7f73a40a63b8
       __PRETTY_FUNCTION__ = "ast_trans_frameout"
#7  0x00000000005d9cb6 in default_frameout (pvt=0x7f73a40a63b0) at translate.c:470
No locals.
#8  0x00000000005da414 in ast_translate (path=0x7f73a40a63b0, f=0x7f73a403bb00, consume=0) at translate.c:624
       current = 0x0
       p = 0x7f73a40a63b0
       out = 0x7f73a403bb00
       delivery = {tv_sec = 0, tv_usec = 0}
       has_timing_info = 0
       ts = 0
       len = 0
       seqno = 0
       __FUNCTION__ = "ast_translate"
#9  0x00000000004ff681 in ast_writestream (fs=0x7f73a408b080, f=0x7f73a403bb00) at file.c:287
       trf = 0x0
       res = -1
       __FUNCTION__ = "ast_writestream"
       __PRETTY_FUNCTION__ = "ast_writestream"
#10 0x00000000004b2343 in ast_write_stream (chan=0x7f74f801f010, stream_num=0, fr=0x7f749001df40) at channel.c:5480
       jump = -640
       cur = 0x7f73a403bb00
       stream = 0x7f749cd31df0
       default_stream = 0x7f749cd31df0
       res = -1
       f = 0x7f73a403bb00
       count = 0
       hooked = 0
       __PRETTY_FUNCTION__ = "ast_write_stream"
       __FUNCTION__ = "ast_write_stream"
#11 0x0000000000496ada in bridge_channel_handle_write (bridge_channel=0x7f73a4095590) at bridge_channel.c:2469
       fr = 0x7f749001df40
       sync_payload = 0x7f74f801efd0
       num = 0
       msg = 0x7f73a4095578
       __PRETTY_FUNCTION__ = "bridge_channel_handle_write"
       __FUNCTION__ = "bridge_channel_handle_write"
#12 0x000000000049732d in bridge_channel_wait (bridge_channel=0x7f73a4095590) at bridge_channel.c:2782
       ms = -1
       outfd = 400
       chan = 0x0
       __PRETTY_FUNCTION__ = "bridge_channel_wait"
       __FUNCTION__ = "bridge_channel_wait"
#13 0x0000000000497c83 in bridge_channel_internal_join (bridge_channel=0x7f73a4095590) at bridge_channel.c:2947
       res = 0
       indicate_src_change = 0 '\000'
       channel_features = 0x0
       peer = 0x0
       swap = 0x0
       __FUNCTION__ = "bridge_channel_internal_join"
       __PRETTY_FUNCTION__ = "bridge_channel_internal_join"
#14 0x000000000047b862 in ast_bridge_join (bridge=0x7f740002b560, chan=0x7f74f801f010, swap=0x0, features=0x7f738acca7d0, tech_args=0x7f738acca7f8, flags=0) at bridge.c:1679
       bridge_channel = 0x7f73a4095590
       res = 0
       __PRETTY_FUNCTION__ = "ast_bridge_join"
--Type <RET> for more, q to quit, c to continue without paging--c
#15 0x00007f74d9189bc1 in confbridge_exec (chan=0x7f74f801f010, data=0x7f738acca9b0 "9600074,vici_agent_bridge,vici_customer_user") at app_confbridge.c:2887
       res = 0
       volume_adjustments = {0, 0}
       quiet = 0
       async_delete_task_pushed = 0
       parse = 0x7f738acc72d0 "9600074"
       b_profile_name = 0x7f738acc72d8 "vici_agent_bridge"
       u_profile_name = 0x7f738acc72ea "vici_customer_user"
       menu_profile_name = 0x0
       conference = 0x7f740002a190
       user = {conference = 0x7f740002a190, b_profile = {name = "vici_agent_bridge", '\000' <repeats 110 times>, language = "en", '\000' <repeats 37 times>, rec_file = '\000' <repeats 4095 times>, rec_options = '\000' <repeats 127 times>, rec_command = '\000' <repeats 127 times>, flags = 304, max_members = 10, internal_sample_rate = 8000, maximum_sample_rate = 0, mix_interval = 20, sounds = 0x2106b80, regcontext = '\000' <repeats 79 times>, video_update_discard = 2000, remb_send_interval = 0, remb_estimated_bitrate = 0}, u_profile = {name = "vici_customer_user", '\000' <repeats 109 times>, pin = '\000' <repeats 79 times>, moh_class = '\000' <repeats 127 times>, announcement = '\000' <repeats 4095 times>, flags = 1585156, announce_user_count_all_after = 0, talking_threshold = 160, silence_threshold = 2500, timeout = 0}, menu_name = "default_menu", '\000' <repeats 115 times>, name_rec_location = '\000' <repeats 4095 times>, chan = 0x7f74f801f010, features = {dtmf_hooks = 0x7f73a40ac910, other_hooks = 0x7f73a40ac9a0, interval_hooks = 0x7f73a40ac9f0, feature_flags = {flags = 0}, interval_sequence = 0, usable = 0, mute = 0, dtmf_passthrough = 1, inhibit_colp = 0, text_messaging = 1}, tech_args = {talking_threshold = 160, silence_threshold = 2500, drop_silence = 1}, suspended_moh = 0, muted = 0, kicked = 0, playing_moh = 0, talking = 0, post_join_list = {first = 0x0, last = 0x0}, list = {next = 0x0}}
       join_hook_data = 0x7f73a4066130
       leave_hook_data = 0x7f73a404a630
       args = {argc = 3, {argv = {0x7f738acc72d0 "9600074", 0x7f738acc72d8 "vici_agent_bridge", 0x7f738acc72ea "vici_customer_user", 0x0}, {conf_name = 0x7f738acc72d0 "9600074", b_profile_name = 0x7f738acc72d8 "vici_agent_bridge", u_profile_name = 0x7f738acc72ea "vici_customer_user", menu_profile_name = 0x0}}}
       __FUNCTION__ = "confbridge_exec"
       __PRETTY_FUNCTION__ = "confbridge_exec"
#16 0x00000000005545ed in pbx_exec (c=0x7f74f801f010, app=0x21086f0, data=0x7f738acca9b0 "9600074,vici_agent_bridge,vici_customer_user") at pbx_app.c:492
       res = 0
       u = 0x7f73a404e5c0
       saved_c_appl = 0x7f74e9cb60c0 "AppDial"
       saved_c_data = 0x7f74e9cb60c8 "(Outgoing Line)"
       __PRETTY_FUNCTION__ = "pbx_exec"
#17 0x000000000054006d in pbx_extension_helper (c=0x7f74f801f010, con=0x0, context=0x7f74f801f9d0 "default", exten=0x7f74f801fa20 "9600074", priority=3, label=0x0, callerid=0x7f74f801c600 "9999024847", action=E_SPAWN, found=0x7f738accdc4c, combined_find_spawn=1) at pbx.c:2948
       e = 0x21db870
       app = 0x21086f0
       substitute = 0x7f738acca920 "${EXTEN},vici_agent_bridge,vici_customer_user"
       q = {incstack = {0x0 <repeats 512 times>}, stacklen = 0, status = 5, swo = 0x0, data = 0x0, foundcontext = 0x7f74f801f9d0 "default"}
       passdata = "9600074,vici_agent_bridge,vici_customer_user\000----NODEBUG-----16-----ANSWER-----7-----0-----SIP 200 OK)\000\000\020)\006\340s\177\000\000\000\000\000\000\000\000\000\000P\037\006\340s\177\000\000PJSIP/9999022682@dial41,,tTo", '\000' <repeats 3404 times>...
       matching_action = 0
       __FUNCTION__ = "pbx_extension_helper"
#18 0x0000000000543e7e in ast_spawn_extension (c=0x7f74f801f010, context=0x7f74f801f9d0 "default", exten=0x7f74f801fa20 "9600074", priority=3, callerid=0x7f74f801c600 "9999024847", found=0x7f738accdc4c, combined_find_spawn=1) at pbx.c:4203
No locals.
#19 0x0000000000544b34 in __ast_pbx_run (c=0x7f74f801f010, args=0x0) at pbx.c:4377
       digit = 0
       invalid = 0
       timeout = 0
       dst_exten = "\000\373f\000\000\000\000\000\320?g\000\000\000\000\000\260\333̊h\004\000\000\353\366f\000\000\000\000\000\320?g\000\000\000\000\000\353\366f\000\000\000\000\000\260\333̊s\177\000\000\374\230F\000\000\000\000\000\260\333̊\000\000\000\000\240\373f\000\000\000\000\000\031Gf\000h\004\000\000\320?g\000\000\000\000\000\353\366f\000\000\000\000\000\020\360\001\370t\177\000\000\240F\a\244s\177\000\000\320\357\001\370t\177\000\000\000\000\000\000\001\000\000\000\240!\001\370t\177\000\000\260\333̊s\177\000\000\370\357\001\370t\177\000\000KyL\000\000\000\000\000\370\357\001\370t\177\000\000p\334̊s\177\000\000\345RJ", '\000' <repeats 13 times>, "\271)R\000\000\000\000\000"...
       pos = 0
       found = 1
       res = 0
       autoloopflag = 0
       error = 0
       pbx = 0x7f73a405a2b0
       callid = 12830
       __FUNCTION__ = "__ast_pbx_run"
       __PRETTY_FUNCTION__ = "__ast_pbx_run"
#20 0x0000000000546499 in ast_pbx_run_args (c=0x7f74f801f010, args=0x0) at pbx.c:4749
       res = AST_PBX_SUCCESS
       __FUNCTION__ = "ast_pbx_run_args"
#21 0x00000000005464c3 in ast_pbx_run (c=0x7f74f801f010) at pbx.c:4758
No locals.
#22 0x0000000000486fca in ast_bridge_run_after_goto (chan=0x7f74f801f010) at bridge_after.c:530
       goto_failed = 0
#23 0x000000000047bc80 in bridge_channel_ind_thread (data=0x7f74f803c210) at bridge.c:1773
       bridge_channel = 0x7f74f803c210
       chan = 0x7f74f801f010
       __PRETTY_FUNCTION__ = "bridge_channel_ind_thread"
#24 0x00000000005e1708 in dummy_start (data=0x7f74f804f1b0) at utils.c:1574
       __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {0, -2404430039997251268, 140134343594126, 140134343594127, 140134343594416, 17, -2404430039972085444, 2325733980844425532}, __mask_was_saved = 0}}, __pad = {0x7f738accded0, 0x0, 0xc59f9f7b26d69800, 0x7f7538518480 <__pthread_keys+1024>}}
       __cancel_routine = 0x45f066 <ast_unregister_thread>
       __cancel_arg = 0x7f738acce700
       __not_first_call = 0
       ret = 0x11
       a = {start_routine = 0x47baf0 <bridge_channel_ind_thread>, data = 0x7f74f803c210, name = 0x7f74f8025900 "bridge_channel_ind_thread started at [ 1846] bridge.c bridge_impart_internal()"}
       __PRETTY_FUNCTION__ = "dummy_start"
#25 0x00007f753830594a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#26 0x00007f75374bdd0f in clone () from /lib64/libc.so.6
No symbol table info available.
{quote}
Comments:By: Asterisk Team (asteriskteam) 2023-02-07 14:47:55.124-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].
By: Joshua C. Colp (jcolp) 2023-02-07 15:08:29.530-0600

Please attach backtraces as attachments in the future, or else JIRA will eat up formatting and make things confusing.

This shows it happening during translation. What codecs are in use? Does it always happen during translation? What is the scenario and configuration? What is the console output at the time?

You'll also likely need to compile and reproduce under MALLOC_DEBUG since it may be memory corruption.
By: Michael Cargile (mcargile) 2023-02-08 11:24:20.361-0600

Looks like it was configured for g729 which was using the free (intel?) version. I forgot that I had that installed for another test. I have removed that, switched to ULAW, and recompiled with with MALLOC_DEBUG.

The system is running Vicidial, placing thousands of test calls per day as Local channels. When the call connects they go through our Optimize script which transfer the underlying PJSIP channel to our routing script and hangs up the Local channel. The routing script then sends the calls to the dummy agents waiting in ConfBridge conferences. The calls then last a random period of time and then are hung up.

Unfortunately I was unable to get the console output before it got purged. I have increased the log retention so I should hopefully get that if it happens again.
By: Joshua C. Colp (jcolp) 2023-02-08 11:26:48.682-0600

We're unable to support issues that may involve external modules, so will have to wait for the next crash.
By: Michael Cargile (mcargile) 2023-02-08 11:33:58.836-0600

Completely understand.
By: Asterisk Team (asteriskteam) 2023-02-22 12:00:16.076-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines