[Home]

Summary:ASTERISK-04770: Asterisk Crashes under heavy sip load
Reporter:Alberto Fernandez (derkommissar)Labels:
Date Opened:2005-08-03 14:42:56Date Closed:2011-06-07 14:00:33
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:Asterisk afther a couple of hours wit heavy sip calls crashes.

Here is the coredump

Everything is set to heavy amouts of traffic on this box.

# ulimit -a
core file size          (blocks, -c) 8096
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 90000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

and the usage at crash time was...

Tasks: 330 total,   1 running, 329 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.7% us,  9.8% sy,  0.0% ni, 74.6% id,  0.0% wa,  0.1% hi, 11.8% si
Mem:   3112960k total,  2503900k used,   609060k free,    99572k buffers
Swap:  2031608k total,        0k used,  2031608k free,   853328k cached

at crash time there was about 500 sip calls.


****** ADDITIONAL INFORMATION ******

gdb asterisk core.31055
GNU gdb Red Hat Linux (6.1post-1.20040607.41rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `asterisk -vvvg -c'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /usr/lib/libncurses.so.5...done.
Loaded symbols for /usr/lib/libncurses.so.5
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libssl.so.4...done.
Loaded symbols for /lib/libssl.so.4
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcrypto.so.4...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/asterisk/modules/res_agi.so...done.
Loaded symbols for /usr/lib/asterisk/modules/res_agi.so
Reading symbols from /usr/lib/asterisk/modules/res_adsi.so...done.
Loaded symbols for /usr/lib/asterisk/modules/res_adsi.so
Reading symbols from /usr/lib/asterisk/modules/res_features.so...done.
Loaded symbols for /usr/lib/asterisk/modules/res_features.so
Reading symbols from /usr/lib/asterisk/modules/chan_sip.so...done.
Loaded symbols for /usr/lib/asterisk/modules/chan_sip.so
Reading symbols from /usr/lib/asterisk/modules/chan_local.so...done.
Loaded symbols for /usr/lib/asterisk/modules/chan_local.so
Reading symbols from /usr/lib/asterisk/modules/chan_phone.so...done.
Loaded symbols for /usr/lib/asterisk/modules/chan_phone.so
Reading symbols from /usr/lib/asterisk/modules/chan_oh323.so...done.
Loaded symbols for /usr/lib/asterisk/modules/chan_oh323.so
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /usr/lib/libldap-2.2.so.7...done.
Loaded symbols for /usr/lib/libldap-2.2.so.7
Reading symbols from /usr/lib/libldap_r-2.2.so.7...done.
Loaded symbols for /usr/lib/libldap_r-2.2.so.7
Reading symbols from /usr/lib/liblber-2.2.so.7...done.
Loaded symbols for /usr/lib/liblber-2.2.so.7
Reading symbols from /usr/lib/libsasl.so.7...done.
Loaded symbols for /usr/lib/libsasl.so.7
Reading symbols from /usr/lib/libexpat.so.0...done.
Loaded symbols for /usr/lib/libexpat.so.0
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgdbm.so.2...done.
Loaded symbols for /usr/lib/libgdbm.so.2
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/asterisk/modules/pbx_functions.so...done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_functions.so
Reading symbols from /usr/lib/asterisk/modules/pbx_config.so...done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_config.so
Reading symbols from /usr/lib/asterisk/modules/app_disa.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_disa.so
Reading symbols from /usr/lib/asterisk/modules/format_g729.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_g729.so
Reading symbols from /usr/lib/asterisk/modules/format_pcm_alaw.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_pcm_alaw.so
Reading symbols from /usr/lib/asterisk/modules/codec_g729.so...done.
Loaded symbols for /usr/lib/asterisk/modules/codec_g729.so
Reading symbols from /usr/lib/asterisk/modules/app_setcidnum.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcidnum.so
Reading symbols from /usr/lib/asterisk/modules/codec_ilbc.so...done.
Loaded symbols for /usr/lib/asterisk/modules/codec_ilbc.so
Reading symbols from /usr/lib/asterisk/modules/cdr_odbc.so...done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_odbc.so
Reading symbols from /usr/lib/libodbc.so.1...done.
Loaded symbols for /usr/lib/libodbc.so.1
Reading symbols from /usr/lib/libltdl.so.3...done.
Loaded symbols for /usr/lib/libltdl.so.3
Reading symbols from /usr/lib/gconv/ISO8859-1.so...done.
Loaded symbols for /usr/lib/gconv/ISO8859-1.so
Reading symbols from /usr/lib/asterisk/modules/format_g726.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_g726.so
Reading symbols from /usr/lib/asterisk/modules/cdr_custom.so...done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_custom.so
Reading symbols from /usr/lib/asterisk/modules/format_gsm.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_gsm.so
Reading symbols from /usr/lib/asterisk/modules/app_lookupcidname.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_lookupcidname.so
Reading symbols from /usr/lib/asterisk/modules/cdr_manager.so...done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_manager.so
Reading symbols from /usr/lib/asterisk/modules/format_sln.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_sln.so
Reading symbols from /usr/lib/asterisk/modules/app_setcidname.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcidname.so
Reading symbols from /usr/lib/asterisk/modules/codec_alaw.so...done.
Loaded symbols for /usr/lib/asterisk/modules/codec_alaw.so
Reading symbols from /usr/lib/asterisk/modules/app_cdr.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_cdr.so
Reading symbols from /usr/lib/asterisk/modules/format_wav.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_wav.so
Reading symbols from /usr/lib/asterisk/modules/cdr_csv.so...done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_csv.so
Reading symbols from /usr/lib/asterisk/modules/format_wav_gsm.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_wav_gsm.so
Reading symbols from /usr/lib/asterisk/modules/codec_ulaw.so...done.
Loaded symbols for /usr/lib/asterisk/modules/codec_ulaw.so
Reading symbols from /usr/lib/asterisk/modules/format_g723.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_g723.so
Reading symbols from /usr/lib/asterisk/modules/format_pcm.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_pcm.so
Reading symbols from /usr/lib/asterisk/modules/app_verbose.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_verbose.so
Reading symbols from /usr/lib/asterisk/modules/format_ilbc.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_ilbc.so
Reading symbols from /usr/lib/asterisk/modules/codec_gsm.so...done.
Loaded symbols for /usr/lib/asterisk/modules/codec_gsm.so
Reading symbols from /usr/lib/asterisk/modules/app_setcallerid.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcallerid.so
Reading symbols from /usr/lib/asterisk/modules/app_dial.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_dial.so
Reading symbols from /usr/local/lib/libtdsodbc.so...done.
Loaded symbols for /usr/local/lib/libtdsodbc.so
Reading symbols from /usr/lib/libodbcinst.so.1...done.
Loaded symbols for /usr/lib/libodbcinst.so.1
#0  0xf41e61b4 in tds_write_packet (tds=0xa67af58, final=255 '?')
   at write.c:387
387                             FD_SET(tds->s, &fds);
(gdb) bt ful
#0  0xf41e61b4 in tds_write_packet (tds=0xa67af58, final=255 '?')
   at write.c:387
No locals.
#1  0xf41e64b6 in tds_put_n (tds=0xa67af58, buf=0x0, n=1073707558)
   at write.c:98
       left = 0
       bufp = (const unsigned char *) 0x0
       __PRETTY_FUNCTION__ = "tds_put_n"
#2  0xf41e6824 in tds_put_buf (tds=0xa67af58, buf=0x9e37128 "us_english",
   dsize=20, ssize=31) at write.c:200
       cpsize = 10
#3  0xf41e3fbf in tds_put_login_string (tds=0xa67af58,
   buf=0x7ffffff <Address 0x7ffffff out of bounds>, n=30) at login.c:421
       buf_len = 31
#4  0xf41e4f9f in tds_connect (tds=0xa67af58, connect_info=0xab27c98)
   at login.c:547
       sin = {sin_family = 2, sin_port = 39173, sin_addr = {
   s_addr = 49165264}, sin_zero = "?\000\000\000\020?\\"}
       ioctl_blocking = 1
       selecttimeout = {tv_sec = 0, tv_usec = 0}
       fds = {fds_bits = {0 <repeats 32 times>}}
       retval = -199250172
       start = -268562128
---Type <return> to continue, or q <return> to quit---
       now = 174567256
       connect_timeout = 0
       db_selected = 0
       version = "4.2", '\0' <repeats 61 times>, "e?L\000!?\037?\000\000\000\000\000\000\000\000#?\037?TD\000\000\005\000\000\000Xf ?\000\000\000\000\b\235?\n?\022??N?\036?!?\037?", '\0' <repeats 36 times>, "$uP", '\0' <repeats 25 times>, "?n?\n$uP\000\000\000\000\000H?\\\000\000?\\\000\220\000\000\000\000?\\\000\000?\\\000\210\206;\nH?\\\000\020\000\000\000H?\\\000\000?\\\000\030\002\000\000\000?\\\000\000?\\\000\000\000\000\000H?\\\000\000\000\000"
       str = 0x0
       len = 1
       linger_opt = {l_onoff = 1, l_linger = 0}
ASTERISK-1  0xf41d05d7 in odbc_connect (dbc=0xacd9d08, connect_info=0xab27c98)
   at odbc.c:316
       env = (TDS_ENV *) 0x1f
ASTERISK-2  0xf41d181f in SQLConnect (hdbc=0xacd9d08, szDSN=0xeffe24c0 "mssql",
   cbDSN=-15168, szUID=0xf2c09280 "asterisk", cbUID=-3,
   szAuthStr=0xf2c5c4c0 "cynergia", cbAuthStr=-3) at odbc.c:1317
       result = -15168
       connect_info = (TDSCONNECTINFO *) 0xab27c98
ASTERISK-3  0xf421c2b6 in SQLConnect () from /usr/lib/libodbc.so.1
No symbol table info available.
ASTERISK-4  0xf4a8a239 in odbc_init () at cdr_odbc.c:440
---Type <return> to continue, or q <return> to quit---
       ODBC_err = -268556016
       ODBC_mlen = -2904
       ODBC_res = 31
       ODBC_msg = "\0171??", '\0' <repeats 20 times>, "\b\000\000\000\000\000\000\000\b\000\000\000\000\000\000?\000\000\000\000\000\000\000\000s\bR\000\000\000\000\000\000\000\000\000????\000\000\000\000\000\000\000\000\006?Q\000\000\000\000\000\000\000\000\000????\a&??\000\000\000\000\002", '\0' <repeats 11 times>, "?(??\023\000\000\000 \212\\\000\000\000\000\000\024\000\000\000?\001\000\000?1\000\000?\a\000\000\001\000\000\000?\217\\\000`(??h+\213\n &??\002\027R\000h+\213\n????`(??\020&??2002\000\000\000\000?\217\\\000?\217\\\000\020)??"
       ODBC_stat = "\020)???)??\0171"
ASTERISK-5  0xf4a8ac17 in odbc_log (cdr=0xa8b28e8) at cdr_odbc.c:89
       ODBC_err = 0
       ODBC_mlen = 2062
       ODBC_res = 134217727
       ODBC_msg = "????W\000\000\000?)P\000 \214\020\bV\000\000\0000\000\020\b\000???\000\000\000\000?+P\000?\217\\\000?\225\\\000\000\000\000\000x1???\217\\\000?\225\\\000?\213\\\000x1??L\207O\000?\225\\\000?Ha\000?\225\\\000\001\000\000\000?\225\\\000$6\v\b?&??\027\000\000\000\236\000\000\000\000\000\000\000P\000\000\000x?\t?\201\\\000\024\000\000\000\033[1;30;40m  == \033[0;37;40m\000\000\000h\000\000\000?\217\\\000\000?\\\000?????1??\b2??dda\000]\000\000\000?,\v\b?\v\000\000!nP"
       ODBC_stat = "\000\000\000\000P1??6"
---Type <return> to continue, or q <return> to quit---
       sqlcmd = "INSERT INTO cdr_wholesale (calldate,clid,src,dst,dcontext,channel,dstchannel,lastapp,lastdata,duration,billsec,disposition,amaflags,accountcode,uniqueid,userfield) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,"...
       timestr = "2005-08-02 17:36:36", '\0' <repeats 108 times>
       res = 31
       tm = {tm_sec = 36, tm_min = 36, tm_hour = 17, tm_mday = 2, tm_mon = 7,
 tm_year = 105, tm_wday = 2, tm_yday = 213, tm_isdst = 1, tm_gmtoff = -14400,
 tm_zone = 0x9a7d578 "EDT"}
ASTERISK-6 0x0809ee5a in post_cdr (cdr=0xa8b28e8) at cdr.c:808
       i = (struct ast_cdr_beitem *) 0xf0e13a10
       __PRETTY_FUNCTION__ = "post_cdr"
ASTERISK-7 0x0809f387 in ast_cdr_detach (cdr=0xa8b28e8) at cdr.c:995
       newtail = (struct ast_cdr_batch_item *) 0xf341dfc0
       __PRETTY_FUNCTION__ = "ast_cdr_detach"
ASTERISK-8 0x08064fe2 in ast_hangup (chan=0xf341dfc0) at channel.c:875
       res = 0
       __PRETTY_FUNCTION__ = "ast_hangup"
ASTERISK-9 0x0808c137 in __ast_pbx_run (c=0xf341dfc0) at pbx.c:2471
       digit = 0
       exten = '\0' <repeats 255 times>
       pos = 0
       waittime = 183821088
       res = -1
---Type <return> to continue, or q <return> to quit---
       __PRETTY_FUNCTION__ = "__ast_pbx_run"
ASTERISK-10 0x0808d9ce in pbx_thread (data=0xf341dfc0) at pbx.c:2521
No locals.
ASTERISK-11 0x006111d5 in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
ASTERISK-12 0x0056b2da in clone () from /lib/tls/libc.so.6
No symbol table info available.
(gdb)
(gdb)
(gdb)
(gdb)
(gdb)
Comments:By: twisted (twisted) 2005-08-03 14:50:05

for the love of god, attach backtraces as separate files from now on.  Also, please make your subject line a little more clearer.  "Asterisk Crashes" is NOT a good title for the bug.
By: Mark Spencer (markster) 2005-08-03 15:03:32

Uhm, turn off TDS CDR logging.
By: Mark Spencer (markster) 2005-08-07 10:18:51

I'm suspending this one until we get more input from the bug placer since clearly the problem appears to be in the TDS ODBC driver.