[Home]

Summary:ASTERISK-15547: Core Dump on Exit - v1.6.2/FreeBSD
Reporter:Alex Z (ahhyes)Labels:
Date Opened:2010-01-29 23:17:34.000-0600Date Closed:2011-06-07 14:07:27
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/Portability
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) gdb.txt
( 1) valgrind_memleaktest.log
Description:Hi Guys,

I am running Asterisk 1.6.2.1 On FreeBSD 8 64 bit.
I have compiled and installed the application, and it appears to very well, however upon exit (core stop gracefully), it seg faults and a core dump file is created.

I have provided a back trace from gdb.

Any ideas?


****** ADDITIONAL INFORMATION ******

Core was generated by `asterisk'.
Program terminated with signal 11, Segmentation fault.
(gdb) bt
#0  _ao2_find (c=0x0, arg=0x7ffffed58de0, flags=OBJ_POINTER) at astobj2.c:747
#1  0x000000080eacfb37 in pthread_timer_ack (handle=Variable "handle" is not available.
) at res_timing_pthread.c:284
#2  0x0000000802f4d274 in timing_read (id=Variable "id" is not available.
) at chan_iax2.c:8793
#3  0x0000000000494003 in ast_io_wait (ioc=0x80f47ede0, howlong=Variable "howlong" is not available.
) at io.c:288
#4  0x0000000802f4dba4 in network_thread (ignore=Variable "ignore" is not available.
) at chan_iax2.c:11692
ASTERISK-1  0x00000000004f8216 in dummy_start (data=Variable "data" is not available.
) at utils.c:968
ASTERISK-2  0x0000000801451511 in pthread_getprio () from /lib/libthr.so.3
ASTERISK-3  0x0000000000000000 in ?? ()
Cannot access memory at address 0x7ffffed59000
(gdb)
Comments:By: Leif Madsen (lmadsen) 2010-02-01 09:43:17.000-0600

I'm not sure there is enough information to go on here.

1) can you reproduce this on a later version of 1.6.2 (i.e. from the branch?)

2) per the doc/backtrace.txt file in your Asterisk source, please provide:  bt, bt full, and thread apply all bt

Attach the resulting output as a text file to this issue as opposed to pasting it inline.

Also, it appears as though you already have DONT_OPTIMIZE enabled, but make sure your Asterisk has that compiled in.

Thanks!
By: Alex Z (ahhyes) 2010-02-01 20:18:34.000-0600

Have recompiled the application with DONT_OPTIMIZE specified.
Attaching gdb.txt from most recent core dump.

By the way, bt all did not produce any useful information:

(gdb) bt all
No symbol "all" in current context.

The other commands outlined in the doc/backtrace.txt worked fine.

Will check with branch version (from svn) and advise shortly.

Thanks
By: Alex Z (ahhyes) 2010-02-01 21:08:21.000-0600

Tested Asterisk SVN-branch-1.6.2-r244320 -- Can confirm issue still present.
By: Alex Z (ahhyes) 2010-02-01 21:20:03.000-0600

Are you able to change the report description to "Core Dump on Exit - v1.6.2/FreeBSD" -- It just may help incase someone else has observed this issue and may help prevent creation of duplicate issue reports.

Thanks.
By: Leif Madsen (lmadsen) 2010-02-02 08:46:11.000-0600

Note that backtrace.txt (and I) said:  bt full, not bt all.
By: Alex Z (ahhyes) 2010-02-02 13:51:05.000-0600

Ok, thanks :)

Let me know if you need any more information.
By: Leif Madsen (lmadsen) 2010-02-03 10:57:21.000-0600

I had a discussion with Tilghman on IRC, and unfortunately there aren't very many options for debugging this on FreeBSD. Are you able to reproduce this on Linux at all?

I'm going to ping mvanbaak on this as well as he might have an idea, but at this time I'm not sure how we're going to be able to move this forward.
By: Leif Madsen (lmadsen) 2010-02-03 10:57:47.000-0600

Any chance you could take a look at this and help determine how we might get some more useful debugging information?
By: Alex Z (ahhyes) 2010-02-03 16:21:50.000-0600

I don't have access to a linux machine unfortunately. I suspect even if I did, this wouldn't help anyway because the issue is probably BSD specific. If this were to be reproducible on linux it's likely someone would have reported it already for linux.

In regards to saying 'there aren't very many options for debugging this on FreeBSD' could you elaborate on this at all? Were you able to reproduce the issue on FreeBSD? If you don't have access to a freebsd machine, it should be fairly effortless to install FreeBSD under VirtualBox or VMWare, compile the source and test it there. If it's not access to a bsd machine thats the issue, but the information I have provided not being enough I am not sure. Have you been able to reproduce this yourself?

Could the results of the configure script help you at all? Perhaps the code is trying to execute some sort of system call thats either handled differently on bsd, or not supported.
By: Leif Madsen (lmadsen) 2010-02-03 16:31:20.000-0600

The problem is Tilghman states that there is no valgrind or timerfd on FreeBSD, so it's difficult to get information since your backtrace isn't really usable. It kind of appears to me to be a memory corruption issue (which would be debugged via valgrind... but.... it being FreeBSD).

I don't use FreeBSD at all, and will be unable to reproduce this issue myself. If you're able to provide access to the system to a developer, they may be able to move the issue forward, but there isn't anything I can do further here.

I'm not a developer, so I'm not sure what else to ask for here. Any additional information you feel may be useful you're welcome to attach to the issue though.
By: Alex Z (ahhyes) 2010-02-03 16:49:31.000-0600

[root@nas /usr/ports/devel/valgrind]# cat pkg-descr
Valgrind is a system for debugging and profiling un*x programs. With the tools
that come with Valgrind, you can automatically detect many memory management
and threading bugs, avoiding hours of frustrating bug-hunting, making your
programs more stable. You can also perform detailed profiling, to speed up and
reduce memory use of your programs.
By: Alex Z (ahhyes) 2010-02-03 16:50:15.000-0600

seems there it has been ported to freebsd, valgrind-3.5.0-1
By: Alex Z (ahhyes) 2010-02-03 17:02:59.000-0600

Whilst I'd prefer not to give public access to the box that's running asterisk, I can try setting up a VM with FreeBSD 8/64 bit installed, to see if I can reproduce the issue in there; if I can, I am happy to let a developer loose in the VM even with root privs.

Let me know if this is needed and I will try to arrange something.
By: Leif Madsen (lmadsen) 2010-02-03 18:39:02.000-0600

You're likely best to ask your programming question in #asterisk-dev on the Freenode IRC network or the asterisk-dev mailing list (I'm unable to answer it).

As there has been no developer assigned to this issue currently, I'd suggest trying to gather as much information as you can via valgrind and the other tools to provide as much information to a developer as possible. Once this issue is assigned to someone, they may require access to a VM demonstrating the issues if the information is insufficient to determine the issue.

For now I'd use the #asterisk-dev chat or asterisk-dev mailing lists to help gather the information required to debug the issue. Sorry I can't be of any more assistance. Thanks!
By: Alex Z (ahhyes) 2010-02-03 20:02:31.000-0600

I have added a log file generated by ValGrind, for valgrind --leak-check=yes --log-file=valgrind_memleaktest.log asterisk -f

One thing that sticks out in the log is the following:

==11580== Thread 33:
==11580== Invalid read of size 8
==11580==    at 0x4F1866: ast_timer_ack (timing.c:169)
==11580==  Address 0x2d5a0e0 is 16 bytes inside a block of size 24 free'd
==11580==    at 0x25B240: free (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==11580==    by 0x4F1A8F: ast_unregister_timing_interface (timing.c:111)
==11580==    by 0xE02A0D5: unload_module (res_timing_pthread.c:517)
==11580==    by 0x495F21: ast_module_shutdown (loader.c:460)
==11580==    by 0x42D584: quit_handler (asterisk.c:1597)
==11580==    by 0x42D828: handle_stop_gracefully (asterisk.c:1836)
==11580==    by 0x4585D5: ast_cli_command_full (cli.c:2299)
==11580==    by 0x45885A: ast_cli_command_multiple_full (cli.c:2322)
==11580==    by 0x42F967: netconsole (asterisk.c:1231)
==11580==    by 0x4F82D5: dummy_start (utils.c:968)
==11580==    by 0x1B50510: ??? (in /lib/libthr.so.3)
==11580==
==11580== Invalid read of size 8
==11580==    at 0x4363A6: _ao2_find (astobj2.c:747)
==11580==    by 0xE029B36: pthread_timer_ack (res_timing_pthread.c:284)
==11580==    by 0x341E1D3: timing_read (chan_iax2.c:8791)
==11580==    by 0x493F22: ast_io_wait (io.c:288)
==11580==    by 0x341EB03: network_thread (chan_iax2.c:11690)
==11580==    by 0x4F82D5: dummy_start (utils.c:968)
==11580==    by 0x1B50510: ??? (in /lib/libthr.so.3)
==11580==  Address 0x8 is not stack'd, malloc'd or (recently) free'd
==11580==
==11580==
==11580== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==11580== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==11580==  Access not within mapped region at address 0x8
==11580==    at 0x4363A6: _ao2_find (astobj2.c:747)
==11580==    by 0xE029B36: pthread_timer_ack (res_timing_pthread.c:284)
==11580==    by 0x341E1D3: timing_read (chan_iax2.c:8791)
==11580==    by 0x493F22: ast_io_wait (io.c:288)
==11580==    by 0x341EB03: network_thread (chan_iax2.c:11690)
==11580==    by 0x4F82D5: dummy_start (utils.c:968)
==11580==    by 0x1B50510: ??? (in /lib/libthr.so.3)

I am not experienced with Valgrind, perhaps a dev can guide me in what commands to use so I can provide useful debugging data.

If the dev willing to look at this would prefer a VM to use please let me know, and I will see what I can do.


By: Alex Z (ahhyes) 2010-02-03 23:00:25.000-0600

And I have bumped into another issue with asterisk coredumping randomly while leaving a voicemail if a user has set their extension as busy/dnd

Close this bug please. I am going to move this server over to Linux. I don't have the patience to keep dealing with things that don't work.

Perhaps I will have better stability under linux, and better chances of getting a resolution should something go pear shaped.

Sorry for wasting peoples time.
By: Leif Madsen (lmadsen) 2010-02-04 14:23:38.000-0600

No worries -- Asterisk is mostly developed under Linux, so you should have better luck getting things working there, and more help getting issues fixed if need be. Thanks!