[Home]

Summary:ASTERISK-00764: Show Memory Allocations hangs asterisk
Reporter:casey0999 (casey0999)Labels:
Date Opened:2004-01-08 21:00:12.000-0600Date Closed:2004-09-25 02:52:04
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20040111__show_memory_fix.diff.txt
Description:This command hangs asterisk (no seg fault) whenever used.  "Show memory summary", on the other hand, works fine.  Commands are enabled as documented using MALLOC_DEBUG switch in Makefile.  Running under Fedora 1.

****** ADDITIONAL INFORMATION ******

I posted a msg on asterisk-users mail list to try and get someone else to confirm this bug under other flavors on Linux - got no takers.
Comments:By: casey0999 (casey0999) 2004-01-08 21:06:55.000-0600

Forgot to mention:  This bug appears on a very recent CVS (1/1/04), and I also tried it on a version about 1 month old - same result.
By: zoa (zoa) 2004-01-10 17:52:09.000-0600

oké, as i didnt see this immediately, you need to uncomment a line in the Makefile for this :)
By: zoa (zoa) 2004-01-10 17:59:20.000-0600

on Asterisk CVS-06/16/03-05:03:25 built by root@caesar on a i686 running Linux
this works sweet...

on another server with Asterisk CVS-12/22/03-12:11:40 it doesnt coredump, but only shows the first line, then deadlocks the server.

cleopatra*CLI> show memory allocations
       84 bytes allocated in        build_context at line  5067 of chan_iax2.c
*DEADLOCK*
By: zoa (zoa) 2004-01-10 18:08:29.000-0600

Thread 19 (Thread 18450 (LWP 7108)):
#0  0x400c787e in sigsuspend () from /lib/libc.so.6
#1  0x40022879 in __pthread_wait_for_restart_signal ()
  from /lib/libpthread.so.0
#2  0x40024a36 in __pthread_alt_lock () from /lib/libpthread.so.0
#3  0x40020fd2 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x0808bf8d in __ast_free (ptr=0x80d9e58, file=0x80afce8 "cli.c",
   lineno=45, func=0x80afce0 "ast_cli") at astmm.c:115
ASTERISK-1  0x0806b227 in ast_cli (fd=145,
   fmt=0x80b50c0 "%10d bytes allocated in %20s at line %5d of %s\n")
---Type <return> to continue, or q <return> to quit---
   at cli.c:45
ASTERISK-2  0x0808c178 in handle_show_memory (fd=145, argc=3, argv=0xbd7ff674)
   at astmm.c:236
ASTERISK-3  0x0806c3e4 in ast_cli_command (fd=145,
   s=0xbd7ff824 "show memory allocations") at cli.c:1007
ASTERISK-4  0x08080e37 in netconsole (vconsole=0x80c9980) at asterisk.c:193
ASTERISK-5  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 18 (Thread 16401 (LWP 7095)):
#0  0x400c787e in sigsuspend () from /lib/libc.so.6
#1  0x40022879 in __pthread_wait_for_restart_signal ()
---Type <return> to continue, or q <return> to quit---
  from /lib/libpthread.so.0
#2  0x4001f102 in pthread_cond_wait () from /lib/libpthread.so.0
#3  0x405f62b2 in poster_worker (arg=0x0) at rate_engine.c:668
#4  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 17 (Thread 15376 (LWP 7094)):
#0  0x4013ede1 in nanosleep () from /lib/libc.so.6
#1  0x4013ed78 in sleep () from /lib/libc.so.6
#2  0x4051b5f9 in qcall (ignore=0x0) at app_qcall.c:167
#3  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

---Type <return> to continue, or q <return> to quit---
Thread 16 (Thread 14351 (LWP 7093)):
#0  0x4013ede1 in nanosleep () from /lib/libc.so.6
#1  0x4013ed78 in sleep () from /lib/libc.so.6
#2  0x404ddfee in scan_thread (unused=0x0) at pbx_spool.c:288
#3  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 15 (Thread 13326 (LWP 7092)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x404db144 in __dso_handle () from /usr/lib/asterisk/modules/pbx_wilcalu.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 14 (Thread 12301 (LWP 7091)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x404a9e84 in __dso_handle () from /usr/lib/asterisk/modules/chan_zap.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 13 (Thread 11276 (LWP 7090)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x404a9e84 in __dso_handle () from /usr/lib/asterisk/modules/chan_zap.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 12 (Thread 10251 (LWP 7089)):
---Type <return> to continue, or q <return> to quit---
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x404a9e84 in __dso_handle () from /usr/lib/asterisk/modules/chan_zap.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 11 (Thread 9226 (LWP 7088)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x404a9e84 in __dso_handle () from /usr/lib/asterisk/modules/chan_zap.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 10 (Thread 8201 (LWP 7087)):
#0  0x401687ee in select () from /lib/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x404a9e84 in __dso_handle () from /usr/lib/asterisk/modules/chan_zap.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 9 (Thread 7176 (LWP 7086)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x40486d14 in __dso_handle () from /usr/lib/asterisk/modules/chan_phone.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 8 (Thread 6151 (LWP 7085)):
#0  0x400c787e in sigsuspend () from /lib/libc.so.6
#1  0x40022879 in __pthread_wait_for_restart_signal ()
---Type <return> to continue, or q <return> to quit---
  from /lib/libpthread.so.0
#2  0x40024a36 in __pthread_alt_lock () from /lib/libpthread.so.0
#3  0x40020fd2 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x0808b6dd in __ast_malloc (size=2340, file=0x4035617d "chan_iax2.c",
   lineno=572, func=0x403561b7 "new_iax") at astmm.c:67
ASTERISK-1  0x40341f95 in new_iax (sin=0xbedffa94, lockpeer=1) at chan_iax2.c:572
ASTERISK-2  0x403428c1 in find_callno (callno=1, dcallno=0, sin=0xbedffa94, new=1,
   lockpeer=1) at chan_iax2.c:787
ASTERISK-3  0x4034b51a in socket_read (id=0x8101070, fd=17, events=1, cbdata=0x0)
   at chan_iax2.c:4051
ASTERISK-4  0x08051720 in ast_io_wait (ioc=0x80f0020, howlong=1000) at io.c:267
---Type <return> to continue, or q <return> to quit---
ASTERISK-5  0x4035075b in network_thread (ignore=0x0) at chan_iax2.c:5052
ASTERISK-6 0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 7 (Thread 5126 (LWP 7082)):
#0  0x40166bb0 in poll () from /lib/libc.so.6
#1  0x080516cb in ast_io_wait (ioc=0x80ebe08, howlong=1000) at io.c:254
#2  0x402e79bb in do_monitor (data=0x0) at chan_sip.c:5548
#3  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 6 (Thread 4101 (LWP 7081)):
#0  0x401687ee in select () from /lib/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x401d54b4 in __dso_handle () from /usr/lib/asterisk/modules/res_parking.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 5 (Thread 3076 (LWP 7080)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x401c0414 in __dso_handle () from /usr/lib/asterisk/modules/chan_modem.so
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 4 (Thread 2051 (LWP 7079)):
#0  0x4016e652 in accept () from /lib/libc.so.6
#1  0x400254fb in accept () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x0807f433 in accept_thread (ignore=0x0) at manager.c:731
#3  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 3 (Thread 1026 (LWP 7078)):
#0  0x401687ee in select () from /lib/libc.so.6
#1  0x00000091 in ?? ()
#2  0x400200ba in pthread_start_thread () from /lib/libpthread.so.0

Thread 2 (Thread 2049 (LWP 7077)):
#0  0x40166bb0 in poll () from /lib/libc.so.6
#1  0x4001fd96 in __pthread_manager () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---

Thread 1 (Thread 1024 (LWP 7076)):
#0  0x401613c4 in read () from /lib/libc.so.6
#1  0x4002dae0 in __DTOR_END__ () from /lib/libpthread.so.0
#2  0x0809231f in read_char (el=0x80ca848, cp=0xbfffe7db "") at read.c:302
#3  0x080923bd in el_getc (el=0x80ca848, cp=0xbfffe7db "") at read.c:347
#4  0x0809222a in read_getcmd (el=0x80ca848, cmdnum=0xbfffe7da "",
   ch=0xbfffe7db "") at read.c:243
ASTERISK-1  0x08092665 in el_gets (el=0x80ca848, nread=0xbfffe820) at read.c:443
ASTERISK-2  0x08083932 in main (argc=3, argv=0xbfffebb4) at asterisk.c:1469
#0  0x401613c4 in read () from /lib/libc.so.6
By: zoa (zoa) 2004-01-10 18:15:03.000-0600

if i uncomment the thread debug option in the makefile, i cannot get it to crash any longer. :(
By: zoa (zoa) 2004-01-10 18:28:47.000-0600

still blocks on CVS-01/11/04-01:10:21. (tonights cvs)
By: Tilghman Lesher (tilghman) 2004-01-11 13:54:45.000-0600

zoa, can you post a 'bt full' for this bug?
By: James Golovich (jamesgolovich) 2004-01-11 14:51:06.000-0600

I know I tracked this down some time in the past, it started happening at some point when ast_cli was modified.  What happens is that in handle_show_memory in astmm.c reglock is locked, but then ast_cli gets called while reglock is locked, and ast_cli goes to allocate some memory but reglock is already locked so its deadlocked at that point.
By: zoa (zoa) 2004-01-11 16:03:59.000-0600

(Stripping trailing CRs from patch.)
patching file astmm.c
Hunk #2 FAILED at 228.
Hunk #3 FAILED at 242.
2 out of 3 hunks FAILED -- saving rejects to file astmm.c.rej
Press any key to continue...cleopatra:/usr/src/asterisk# cat astmm.c.rej
***************
*** 227,233 ****
               fn = argv[3];

       /* try to lock applications list ... */
-       ast_mutex_lock(>reglock);

       for (x=0;x<SOME_PRIME;x++) {
               reg = regions[x];
--- 228,234 ----
               fn = argv[3];

       /* try to lock applications list ... */
+       ast_mutex_lock(>showmemorylock);

       for (x=0;x<SOME_PRIME;x++) {
               reg = regions[x];
***************
*** 241,247 ****
               }
       }
       ast_cli(fd, "%d bytes allocated %d units total\n", len, count);
-       ast_mutex_unlock(>reglock);
       return RESULT_SUCCESS;
 }

--- 242,248 ----
               }
       }
       ast_cli(fd, "%d bytes allocated %d units total\n", len, count);
+       ast_mutex_unlock(>showmemorylock);
       return RESULT_SUCCESS;
 }
By: Tilghman Lesher (tilghman) 2004-01-11 16:16:14.000-0600

zoa, whatever you used to download the patch is smoking crack.  It converted the & to >
By: zoa (zoa) 2004-01-11 16:43:24.000-0600

sorry, looks indeed like i fucked up :)
trying again.

FIXED!
By: Brian West (bkw918) 2004-01-11 16:45:20.000-0600

Fixed in CVS