| Summary: | ASTERISK-14878: core show locks stop responding and crash asterisk | ||
| Reporter: | Atis Lezdins (atis) | Labels: | |
| Date Opened: | 2009-09-24 06:55:15 | Date Closed: | 2011-07-27 13:00:42 |
| Priority: | Critical | Regression? | No |
| Status: | Closed/Complete | Components: | Core/General |
| Versions: | Frequency of Occurrence | ||
| Related Issues: | |||
| Environment: | Attachments: | ( 0) bt.asterisk-dev-mc-2009-09-24T04:15:12-0700.23236.txt ( 1) bt.asterisk-dev-mc-2009-09-24T08:28:11-0700.15205.txt | |
| Description: | I had bunch of testing calls running, but i saw some weird things with queue, so i wanted to check for deadlocks. Issuing "core show locks" returned empty result (no lock headers at all) for few times, and then it hanged for a minute and crashed. ****** ADDITIONAL INFORMATION ****** Core was generated by `/usr/sbin/asterisk -U asterisk -G asterisk -vvvdgt'. Program terminated with signal 11, Segmentation fault. #0 0x0000003a41e795a0 in strlen () from /lib64/libc-2.7.so -------------------------------------------------------------------------------- BT #0 0x0000003a41e795a0 in strlen () from /lib64/libc-2.7.so #1 0x0000003a41e46c3c in _IO_vfprintf_internal (s=0x43a1e3f0, format=<value optimized out>, ap=0x43a1e580) at vfprintf.c:1560 #2 0x0000003a41e6a10a in _IO_vsnprintf (string=0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., maxlen=<value optimized out>, format=0x5b4cc5 "\t%s\n", args=0x43a1e580) at vsnprintf.c:120 #3 0x000000000054c73d in __ast_str_helper (buf=0x43a1e818, max_len=0, append=1, fmt=0x5b4cc5 "\t%s\n", ap=0x43a1e610) at utils.c:1779 #4 0x00000000005483a4 in ast_str_append_va (buf=0x43a1e818, max_len=0, fmt=0x5b4cc5 "\t%s\n", ap=0x43a1e610) at /usr/dist/asterisk-svn-1.6.1.5-iqlabs/include/asterisk/strings.h:642 ASTERISK-1 0x0000000000548588 in ast_str_append (buf=0x43a1e818, max_len=0, fmt=0x5b4cc5 "\t%s\n") at /usr/dist/asterisk-svn-1.6.1.5-iqlabs/include/asterisk/strings.h:686 ASTERISK-2 0x00000000005498cc in append_backtrace_information (str=0x43a1e818, bt=0x7eddc8) at utils.c:743 ASTERISK-3 0x0000000000549ac5 in append_lock_information (str=0x43a1e818, lock_info=0xc15de68, i=0) at utils.c:769 ASTERISK-4 0x0000000000549e8c in handle_show_locks (e=0x815620, cmd=-4, a=0x43a1e870) at utils.c:880 ASTERISK-5 0x0000000000475af2 in ast_cli_command (fd=853, s=0x43a1eb00 "core show locks") at cli.c:1888 ASTERISK-6 0x0000000000475d5d in ast_cli_command_multiple (fd=853, size=16, s=0x43a1ed6f "core show locks") at cli.c:1921 ASTERISK-7 0x0000000000437b52 in netconsole (vconsole=0x836df8) at asterisk.c:1039 ASTERISK-8 0x000000000054a11c in dummy_start (data=0x2aab000ad8e8) at utils.c:968 ASTERISK-9 0x0000003a42a06407 in start_thread () from /lib64/libpthread-2.7.so ASTERISK-10 0x0000003a41ed4b0d in clone () from /lib64/libc-2.7.so -------------------------------------------------------------------------------- BT FULL #0 0x0000003a41e795a0 in strlen () from /lib64/libc-2.7.so mallstream = (FILE *) 0x0 tr_old_memalign_hook = (void *(*)(size_t, size_t, const void *)) 0 tr_old_malloc_hook = (void *(*)(size_t, const void *)) 0 tr_old_realloc_hook = (void *(*)(void *, size_t, const void *)) 0 lock = 0 mallenv = "MALLOC_TRACE" malloc_trace_buffer = 0x0 tr_old_free_hook = (void (*)(void *, const void *)) 0 mallwatch = (void *) 0x0 #1 0x0000003a41e46c3c in _IO_vfprintf_internal (s=0x43a1e3f0, format=<value optimized out>, ap=0x43a1e580) at vfprintf.c:1560 outc = <value optimized out> thousands_sep = 0x0 grouping = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds> done = 1 f = <value optimized out> lead_str_end = (const unsigned char *) 0x5b4cc6 "%s\n" work_buffer = "Eä?C\001\000\000\000\001", '\0' <repeats 11 times>, "\001\000\000\000KM[", '\0' <repeats 13 times>, "\001", '\0' <repeats 23 times>, " å?C\000\000\000\0000?W\000\000\000\000\000\2207\216\000\000\000\000\0000â?s\000\000\000\000pá?C\000\000\000\000\000°ÁA:", '\0' <repeats 11 times>, "????????\220ß?Cb\000\000\000R?W", '\0' <repeats 13 times>, "Hä?C\000\000\000\000\205\000\000\000\003", '\0' <repeats 11 times>, "\001\000\000\000\n", '\0' <repeats 19 times>, "\n", '\0' <repeats 11 times>, "????·O[\000\000\000\000\000 °ÁA\000\000\000\000"... workstart = 0x0 workend = 0x43a1e2b8 "\026" ap_save = {{ gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x43a1e6f0, reg_save_area = 0x43a1e630 }} nspecs_done = 0 save_errno = 0 readonly_format = 0 jump_table = {1, 0, 0, 4, 0, 14, 0, 6, 0, 0, 7, 2, 0, 3, 9, 0, 5, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0, 26, 0, 25, 0, 19, 19, 19, 0, 29, 0, 0, 12, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0, 18, 0, 13, 0, 0, 0, 0, 0, 0, 26, 0, 20, 15, 19, 19, 19, 10, 15, 28, 0, 11, 24, 23, 17, 22, 12, 0, 21, 27, 16, 0, 0, 18, 0, 13} step0_jumps = {0, 2207, 2299, 2389, 2486, 2576, 2686, 2882, 3104, 3277, 4382, 4187, 4277, 4377, 3570, 5014, 3739, 4172, 4574, 4589, 6440, 6641, 4925, 5100, 5211, 1190, -646, -100, -100, 2790} step1_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 3277, 4382, 4187, 4277, 4377, 3570, 5014, 3739, 4172, 4574, 4589, 6440, 6641, 4925, 5100, 5211, 1190, -646, -100, -100, 0} step2_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4382, 4187, 4277, 4377, 3570, 5014, 3739, 4172, 4574, 4589, 6440, 6641, 4925, 5100, 5211, 1190, -646, -100, -100, 0} step3a_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4474, 0, 0, 0, 3570, 5014, 3739, 4172, 4574, 0, 0, 0, 0, 5100, 0, 0, 0, 0, 0, 0} step3b_jumps = {0 <repeats 11 times>, 4277, 0, 0, 3570, 5014, 3739, 4172, 4574, 4589, 6440, 6641, 4925, 5100, 5211, 1190, -646, 0, 0, 0} step4_jumps = {0 <repeats 14 times>, 3570, 5014, 3739, 4172, 4574, 4589, 6440, 6641, 4925, 5100, 5211, 1190, -646, 0, 0, 0} step4_jumps = {5528 <repeats 14 times>, 6250, 7372, 6748, 7477, 5681, 6166, 6763, 6992, 7492, 6322, 7331, 338, 6678, 5528, 5528, 5528} #2 0x0000003a41e6a10a in _IO_vsnprintf (string=0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., maxlen=<value optimized out>, format=0x5b4cc5 "\t%s\n", args=0x43a1e580) at vsnprintf.c:120 sf = { f = { _sbf = { _f = { _flags = -72515583, _IO_read_ptr = 0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., _IO_read_end = 0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., _IO_read_base = 0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., _IO_write_base = 0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., _IO_write_ptr = 0xce91dca "ntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54a"..., _IO_write_end = 0xce93f07 "eï?Þ(ast_func_re?T", _IO_buf_base = 0xce91dc9 "\tntexts+0x23) [0x4fb140]\n\t/usr/sbin/asterisk [0x4e83c3]\n\t/usr/sbin/asterisk(ast_spawn_extension+0x64) [0x4e9e70]\n\t/usr/sbin/asterisk [0x4ea5dd]\n\t/usr/sbin/asterisk [0x4ec47b]\n\t/usr/sbin/asterisk [0x54"..., _IO_buf_end = 0xce93f07 "eï?Þ(ast_func_re?T", _IO_save_base = 0x0, _IO_backup_base = 0x0, _IO_save_end = 0x0, _markers = 0x0, _chain = 0x0, _fileno = -1885910456, _flags2 = 0, _old_offset = 4194304, _cur_column = 0, _vtable_offset = 64 '@', _shortbuf = "", _lock = 0x0, _offset = 140735602444872, _codecvt = 0x400000, _wide_data = 0x0, _freeres_list = 0x0, _freeres_buf = 0x7fff8f974a48, _freeres_size = 4194304, _mode = -1, _unused2 = "\000\000\000\000?\001H\000\000\000\000\000\200ð\226\000\000\000\000" }, _vtable = 0x3a42150460 }, _s = { _allocate_buffer = 0, _free_buffer = 0 } }, overflow_buf = "HJ\227\217?\177\000\000\000\000@\000\000\000\000\000(î@\000\000\000\000\000@uN\000\000\000\000\000HJ\227\217?\177\000\000\000\000@", '\0' <repeats 20 times> } ret = 1134683720 #3 0x000000000054c73d in __ast_str_helper (buf=0x43a1e818, max_len=0, append=1, fmt=0x5b4cc5 "\t%s\n", ap=0x43a1e610) at utils.c:1779 res = 60 need = 54602 offset = 54601 aq = {{ gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x43a1e6f0, reg_save_area = 0x43a1e630 }} __PRETTY_FUNCTION__ = "__ast_str_helper" #4 0x00000000005483a4 in ast_str_append_va (buf=0x43a1e818, max_len=0, fmt=0x5b4cc5 "\t%s\n", ap=0x43a1e610) at /usr/dist/asterisk-svn-1.6.1.5-iqlabs/include/asterisk/strings.h:642 No locals. ASTERISK-1 0x0000000000548588 in ast_str_append (buf=0x43a1e818, max_len=0, fmt=0x5b4cc5 "\t%s\n") at /usr/dist/asterisk-svn-1.6.1.5-iqlabs/include/asterisk/strings.h:686 res = 60 ap = {{ gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x43a1e6f0, reg_save_area = 0x43a1e630 }} ASTERISK-2 0x00000000005498cc in append_backtrace_information (str=0x43a1e818, bt=0x7eddc8) at utils.c:743 frame_iterator = 15 symbols = (char **) 0xcb8ea20 __PRETTY_FUNCTION__ = "append_backtrace_information" ASTERISK-3 0x0000000000549ac5 in append_lock_information (str=0x43a1e818, lock_info=0xc15de68, i=0) at utils.c:769 j = 0 lock = (ast_mutex_t *) 0x0 lt = (struct ast_lock_track *) 0x43a1f950 ASTERISK-4 0x0000000000549e8c in handle_show_locks (e=0x815620, cmd=-4, a=0x43a1e870) at utils.c:880 i = 0 lock_info = (struct thr_lock_info *) 0xc15de68 str = (struct ast_str *) 0xce84868 __PRETTY_FUNCTION__ = "handle_show_locks" ASTERISK-5 0x0000000000475af2 in ast_cli_command (fd=853, s=0x43a1eb00 "core show locks") at cli.c:1888 args = {0x815620 "X\026\204", 0xced54c8 "core", 0xced54cd "show", 0xced54d2 "locks", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x43a1eb80 "", 0x0 <repeats 11 times>, 0x3a4320886a "H\215\225Pý??H\211\225(ý??H\213?(ý??H\213?0ý???\025í??\205?\211\205Lý??t;\203ø\005t6?\205Lý??\003", 0x0, 0x3a432096aa "\017?U\001H\215l$\030\200úRu\036ëBf\017\037D", 0x0, 0xfffffffffffffff8 <Address 0xfffffffffffffff8 out of bounds>, 0xb <Address 0xb out of bounds>, 0x43a1eaf0 "\020", 0x3a41f244dc "\001\033\003;(a", 0x3a41f283c0 "Ä\005û?Ä?\001", 0x3633323332a10288 <Address 0x3633323332a10288 out of bounds>, 0x0, 0x3a42a09ae9 "H\201Ä\200", 0xd386800000823500 <Address 0xd386800000823500 out of bounds>, 0x0, 0x3a4320886a "H\215\225Pý??H\211\225(ý??H\213?(ý??H\213?0ý???\025í??\205?\211\205Lý??t;\203ø\005t6?\205Lý??\003", 0x3000000018 <Address 0x3000000018 out of bounds>, 0x43a1ed20 "\017", 0x43a1ec60 "", 0x2f <Address 0x2f out of bounds>, 0xc24 <Address 0xc24 out of bounds>, 0x3a41f2a608 "\024", 0x0, 0x2aaaa! aaab000 "", 0x0, 0xc2 <Address 0xc2 out of bounds>, 0x43a1ea70 "", 0x3a41c1b000 "", 0x3a41c1b000 "", 0x3a41f0a0d4 "H\203Äx\211?[]A\\A]A^A_Ã", '\220' <repeats 11 times>, "D\017·\217°\002", 0x0, 0x0, 0x43a1f0a0 "", 0x43a1eaf0 "\020", 0x3a4320a6c0 "AWAVAUATUH\211ÕSH\203?X\213B(H\203þ/\017\227ÃL\213G\020L\213\027\205?t\n\204Û\017\205\032\001", 0x0, 0x0, 0x3a41a16bcf "", 0x400040 "\006", 0x43a1eae0 "\001\200û", 0x43a1ed60 "cli quit after core show locks", 0x43a1ef5f "", 0x43a1ed60 "cli quit after core show locks"} e = (struct ast_cli_entry *) 0x815620 x = 3 duplicate = 0xced54c8 "core" retval = 0x0 a = { fd = 853, argc = 3, argv = 0x43a1e8a8, line = 0x0, word = 0x0, pos = 0, n = 0 } __PRETTY_FUNCTION__ = "ast_cli_command" | ||
| Comments: | By: Leif Madsen (lmadsen) 2009-09-24 08:59:20 Your values are optimized out -- you will need to provide a backtrace using DONT_OPTIMIZE. Additionally, please just attach your backtraces in the future as a file. Having them all inline in the bug notes makes the issues difficult to deal with. Thanks! By: Atis Lezdins (atis) 2009-09-24 09:08:34 It is not optimized: MENUSELECT_CFLAGS=DONT_OPTIMIZE DEBUG_THREADS LOADABLE_MODULES G711_NEW_ALGORITHM G711_REDUCED_BRANCHING DEBUG_CHANNEL_LOCKS MALLOC_DEBUG Well, about backtraces, it helps others to search for similar crash. But probably yes, this time I posted too much of stack :) By: Atis Lezdins (atis) 2009-09-24 10:58:53 I did make dist-clean and recompiled everything, but I still get "value optimized out". [atis@asterisk-dev-mc dist]$ asterisk -rx "core show settings" Parsing /etc/asterisk/extconfig.conf PBX Core settings ----------------- Version: 1.6.1.5 Build Options: DONT_OPTIMIZE, DEBUG_THREADS, LOADABLE_MODULES, DEBUG_CHANNEL_LOCKS, MALLOC_DEBUG Maximum calls: Not set Maximum open file handles: Not set Verbosity: 3 Debug level: 1 Maximum load average: 0.000000 Minimum free memory: 0 MB Startup time: 08:29:16 Last reload time: 08:29:16 System: Linux/2.6.24.7-92.fc8 built by atis on x86_64 2009-09-24 14:39:30 UTC System name: Entity ID: 00:08:54:b0:78:bc Default language: en Language prefix: Enabled User name and group: asterisk/asterisk Executable includes: Disabled Transcode via SLIN: Enabled Internal timing: Disabled Transmit silence during rec: Disabled * Subsystems ------------- Manager (AMI): Enabled Web Manager (AMI/HTTP): Disabled Call data records: Enabled Realtime Architecture (ARA): Enabled * Directories ------------- Configuration file: Configuration directory: /etc/asterisk Module directory: /usr/lib/asterisk/modules Spool directory: /var/spool/asterisk Log directory: /var/log/asterisk Asterisk ending (0). By: Atis Lezdins (atis) 2009-09-28 11:09:16 I have found a workaround: ./configure --without-execinfo disables backtraces in "core show locks" and doesn't crash asterisk, even if requesting lock information every minute. By: Leif Madsen (lmadsen) 2009-09-28 13:36:46 Great thanks! By: Russell Bryant (russell) 2011-07-27 13:00:37.426-0500 Per the Asterisk maintenance timeline page at http://www.asterisk.org/asterisk-versions maintenance (bug) support for the 1.4 and 1.6.x branches has ended. For continued maintenance support please move to the 1.8 branch which is a long term support (LTS) branch. For more information about branch support, please see https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions If this is still an issue, please open a new issue so it can be re-triaged appropriately. Thanks! | ||