| Summary: | ASTERISK-22455: Asterisk 12 on Ubuntu Lucid deadlocks with DEBUG_THREADS+OPTIONAL_API enabled | ||
| Reporter: | David M. Lee (dlee) | Labels: | |
| Date Opened: | 2013-09-04 09:22:05 | Date Closed: | 2013-09-09 15:00:01 |
| Priority: | Blocker | Regression? | |
| Status: | Closed/Complete | Components: | |
| Versions: | 12.0.0-alpha1 | Frequency of Occurrence | |
| Related Issues: | |||
| Environment: | Attachments: | ||
| Description: | @mmichelson discovered that Asterisk 12 will deadlock when DEBUG_THREADS+OPTIONAL_API is enabled, on Ubuntu Lucid.
It deadlocks consistently on startup with the sample config installed. I've tried various permutations trying to narrow down the problem. * Ubuntu Raring - no deadlock * CentOS 5 - no deadlock * Disable DEBUG_THREADS - no deadlock * Disable OPTIONAL_API - no deadlock * Update to r397988 (before the optional_api changes) - no deadlock | ||
| Comments: | By: David M. Lee (dlee) 2013-09-04 12:46:23.738-0500 Figured it out. When debug threads are enabled, {{ast_pthread_mutex_lock}} invokes {{backtrace()}}. In Lucid's libc (glibc-2.11.1), {{backtrace()}} ends up calling {{dl_iterate_phdr()}}, which "allows an application to inquire at run time to find out which shared objects it has loaded". When log functions are called from {{__constructor__}} functions, they execute during a {{dlopen()}}. If glibc holds a lock while this happens, then it's a classic lock ordering problem. dlopen() locks the module list, then (buried in some calls) it locks the {{logmsgs}} reentrancy lock (for debug threads). The logger thread locks the {{logmsgs}} reentrancy lock, then (buried in {{backtrace()}}) locks the module list. {code} Main thread | Logger thread -----------------------------------+----------------------------- - dlopen() | - AST_LIST_LOCK(&logmsgs) - lock(dl) | - ast_reentrancy_lock(lt) - __constructor__ | - backtrace() - ast_verb() | - lock(dl) - AST_LIST_LOCK(&logmsgs) | *** DEADLOCK *** - ast_reentrancy_lock(lt) | *** DEADLOCK *** | {code} | ||