Summary:ASTERISK-15109: [patch] Script to automatically email backtrace to admin
Reporter:Atis Lezdins (atis)Labels:patch
Date Opened:2009-11-10 05:51:11.000-0600Date Closed:
Versions:Frequency of
Environment:Attachments:( 0) asterisk_mail_backtrace.sh
( 1) backtrace_small.gdb
( 2) backtrace_threads.gdb
( 3) email_backtraces.patch
( 4) safe_asterisk_backtraces.patch
Description:Attached asterisk_mail_backtrace.sh

It can be added in safe_asterisk with the following lines:

nice -n 10 /usr/share/asterisk/asterisk_mail_backtrace.sh $PID &

So after asterisk crashes it's executed in background while new instance of asterisk starts.

I'm still uncertain what's the best place to put this script and both GDB scripts.
Comments:By: Leif Madsen (lmadsen) 2009-11-10 08:11:33.000-0600

I'd suggest these would go into the /path/to/src/asterisk/contrib/scripts/ directory probably. Or somewhere under the /contrib/ directory anyways.

By: Atis Lezdins (atis) 2009-11-10 09:02:53.000-0600

lmadsen, those files should be put in some standard path, i don't think safe_asterisk is generated as config samples.

On the other hand, they could actually reside in some /path/to/src/asterisk as you will need to compile from sources to get usable backtrace.

By: Leif Madsen (lmadsen) 2009-11-10 10:29:55.000-0600

Not config samples.... contrib, which is a directory of contributed scripts.

By: Leif Madsen (lmadsen) 2010-05-18 13:26:06

I really like this feature, however I think this would be best as a modification to the safe_asterisk script. Do you think you could update your patch to be a modification to safe_asterisk?

By: Atis Lezdins (atis) 2010-05-18 13:42:45

lmadsen, yes it was intended as modification to safe_asterisk - therefore the separate script and background - to not delay starting asterisk

i could submit patch, but some constant path should be used from safe_asterisk and AFAIK it doesn't have path to asterisk source directory, so perhaps some modifications to build scripts

By: Atis Lezdins (atis) 2010-05-18 18:03:11

A little more work and I managed to squeeze everything inside safe_asterisk (actually it's better there, as all the variables are easily accessible)

Now I know bash a little bit more :)

By: Leif Madsen (lmadsen) 2010-05-19 09:51:23

Awesome! I like this approach much better. I'm going to assign this to pabelanger to look through and see that the output being generated jives with our documentation in backtrace.txt. Ideally this information should be useful enough to submit as a bug report.

By: Atis Lezdins (atis) 2010-05-21 15:21:54

Changed conditions to using 'test' utility according to issue ASTERISK-15293

By: Amilcar S Silvestre (amilcar) 2010-05-21 15:53:50

Why do you use "grep -A200"? Almost all my backtraces have more than 200 lines, and using a limit like that will cut valuable information of the backtrace. Why not using awk to filter out symbol loading without cutting the end of the file?

Instead of:
grep -A200 "Core was generated by" $BACKTRACE

awk '/Core was generated by/,0' $BACKTRACE

Other than this, great patch.

By: Atis Lezdins (atis) 2010-05-21 20:24:31

Initial patch had full backtrace, however I decided to join both calls to GDB because of symbol loading stuff without --exec option.

Backtrace script would generate full thread backtrace (thread apply all bt), so it would often exceed email limits.

The idea is to give admin/developer a general idea about the crash, and very often I can tell what's wrong by first 200 lines. If required, full backtrace is also saved next in DUMPDROP directory.