Summary: | ASTERISK-29071: app_confbridge: Memory rises when jitterbuffer enabled and muting over AMI occurs | ||
Reporter: | Stefan Ruf (stefan20) | Labels: | fax |
Date Opened: | 2020-09-08 06:52:33 | Date Closed: | 2021-03-03 10:07:59.000-0600 |
Priority: | Minor | Regression? | |
Status: | Closed/Complete | Components: | Applications/app_confbridge Core/Bridging |
Versions: | 16.13.0 | Frequency of Occurrence | Constant |
Related Issues: | |||
Environment: | Linux svast20 4.12.14-94.41-default #1 SMP Wed Oct 31 12:25:04 UTC 2018 (3090901) x86_64 x86_64 x86_64 GNU/Linux | Attachments: | ( 0) ami_commands.txt ( 1) confbridge.txt ( 2) debug_log_20200909_1100.txt ( 3) debug_log_20200914_1026_with_asterisk_restart.txt.gz ( 4) mmlog_20200914_1026__with_asterisk_restart.txt ( 5) mmlog.txt |
Description: | Using ConfbridgeMute / ConfbridgeUnMute over AMI and additionally MuteAudio on/off results in increasing values of frame.c - shown by "memory show summary". The values rises constant over time and channel. So a system with 24 GB RAM using 5 confbridges permanent with about 20 channels each ist filled up within a day and asterisk restarts.
The problem ist reprorduceable and starts at a certain point. This point is action-id 1705904476_335#berta-ddab9ea6-dcb4-4e83-b9bf-ff5e44325110. At action-id 1705904476_336#berta-67e059f6-fa67-4d4a-a73a-d5fbd4cf6475 the value frame.c increases with double speed per time. Please see attached file <ami_commands.txt>. Asterisk had been compiled with MALLOC_DEBUG and “cache_media_frames = no” in asterisk.conf For one channel the value increases about 32 per second; for two channels the value increases about 64 per second. Tested versions of asterisk are 16.13.0, 16.8-cert3, 13.36.0, 13.21-cert6, all of them with the described behaviour. | ||
Comments: | By: Asterisk Team (asteriskteam) 2020-09-08 06:52:35.150-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: Stefan Ruf (stefan20) 2020-09-08 06:56:32.690-0500 sent AMI-Commands with comments By: Stefan Ruf (stefan20) 2020-09-08 06:58:47.330-0500 Configuration of confbridge By: Stefan Ruf (stefan20) 2020-09-08 07:00:07.490-0500 Dialplan in extensions.conf: {noformat} [funk_berta] exten => _berta,1,Answer() exten => _berta,2,Set(JITTERBUFFER(fixed)=20,500) same => n, NoOp(${CALLERID}) same => n, Set(VOLUME(RX)=2) same => n, Set(VOLUME(TX)=2) same => n, ConfBridge(berta) {noformat} By: Benjamin Keith Ford (bford) 2020-09-08 09:46:39.278-0500 We're going to need logs[1] that include the issue taking place, as well as memory output[2] that shows the changes in memory values. [1]: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information [2]: https://wiki.asterisk.org/wiki/display/AST/Memory+Leak+Debugging By: Stefan Ruf (stefan20) 2020-09-14 09:27:32.408-0500 Log with restart of asterisk, after lack of memory By: Stefan Ruf (stefan20) 2020-09-14 09:36:09.859-0500 attached files ...20200914... where asterisk restarts, after lack of memory. => memory short time after start of asterisk and 30 channels in use: Mon Sep 14 10:59:39 CEST 2020 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND asterisk 21366 13.4 3.5 2796292 290024 ? Ssl 10:26 4:25 /usr/sbin/asterisk -f -C /etc/asteris => memory before self-restart of asterisk: Mon Sep 14 13:26:09 CEST 2020 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND asterisk 21366 18.0 86.9 8731732 7099416 ? Ssl 10:26 32:20 /usr/sbin/asterisk -f -C /etc/asteris => asterisk restart, after "lack of memory" [Sep 14 13:37:24] Asterisk 16.13.0 built by root @ svast20 on a x86_64 running Linux on 2020-09-08 10:04:07 UTC ... After playing with the dialplan it seems that "Set(JITTERBUFFER...)" causes the problem. Please tell me, is this description helpful for you ? By: Benjamin Keith Ford (bford) 2020-09-15 10:21:09.726-0500 Thanks for the additional information. Were you able to compile with MALLOC_DEBUG enabled? If so, what is the outputs of {{memory show summary}} when Asterisk is using that large amount of memory? By: Stefan Ruf (stefan20) 2020-09-15 11:51:08.213-0500 compiled with MALLOC_DEBUG, doing a "hangup request all", after asterisk used 20 % of memory; no memory is freed by the hangup request. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND asterisk 3189 22.0 20.2 3382528 1651868 ? Ssl 18:07 5:06 /usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf 27484064 bytes in 610 allocations in file ../src/pj/pool_policy_malloc.c 302661 bytes in 454 allocations in file /opt/swm_install/asterisk-16.13.0/include/asterisk/strings.h 4483 bytes in 21 allocations in file /opt/swm_install/asterisk-16.13.0/include/asterisk/threadstorag 288 bytes in 1 allocations in file acl.c 1263 bytes in 67 allocations in file ael/pval.c 216 bytes in 2 allocations in file app.c 400 bytes in 3 allocations in file app_agent_pool.c 1024 bytes in 1 allocations in file app_bridgewait.c 1584 bytes in 3 allocations in file app_confbridge.c 29616 bytes in 2 allocations in file app_followme.c 36011 bytes in 10 allocations in file app_minivm.c 10016 bytes in 2 allocations in file app_queue.c 45502 bytes in 13 allocations in file app_voicemail.c 856 bytes in 4 allocations in file ari/config.c 688 bytes in 1 allocations in file ari/resource_events.c 4188 bytes in 137 allocations in file asterisk.c 1008 bytes in 42 allocations in file astobj2.c 1258 bytes in 16 allocations in file astobj2_container.c 121360 bytes in 1517 allocations in file astobj2_hash.c 352 bytes in 2 allocations in file bridge.c 1528 bytes in 2 allocations in file bucket.c 1152 bytes in 6 allocations in file ccss.c 37888 bytes in 9 allocations in file cdr.c 608 bytes in 16 allocations in file cdr_sqlite3_custom.c 9831 bytes in 7 allocations in file cel.c 176 bytes in 2 allocations in file chan_bridge_media.c 1692072 bytes in 27 allocations in file chan_iax2.c 1916 bytes in 3 allocations in file chan_oss.c 264 bytes in 3 allocations in file chan_phone.c 1136 bytes in 2 allocations in file chan_pjsip.c 176 bytes in 2 allocations in file chan_rtp.c 114831 bytes in 54 allocations in file chan_sip.c 37952 bytes in 14 allocations in file channel.c 16 bytes in 1 allocations in file chanvars.c 15608 bytes in 754 allocations in file cli.c 7072 bytes in 48 allocations in file codec.c 26496 bytes in 1 allocations in file codec_resample.c 157644 bytes in 71 allocations in file confbridge/conf_config_parser.c 91615 bytes in 665 allocations in file config.c 129272 bytes in 666 allocations in file config_options.c 248 bytes in 2 allocations in file core_local.c 1244 bytes in 18 allocations in file devicestate.c 7928 bytes in 16 allocations in file endpoints.c 2257 bytes in 13 allocations in file features_config.c 10320 bytes in 30 allocations in file file.c 15396 bytes in 199 allocations in file format.c 1432 bytes in 1 allocations in file format_cache.c 55136 bytes in 517 allocations in file format_cap.c 1089193619 bytes (1089193619 cache) in 2992535 allocations in file frame.c 1024 bytes in 1 allocations in file func_dialgroup.c 608 bytes in 2 allocations in file hashtab.c 730 bytes in 4 allocations in file http.c 464 bytes in 3 allocations in file iax2/netsock.c 256 bytes in 1 allocations in file iax2/provision.c 46007 bytes in 1200 allocations in file indications.c 24688 bytes in 13 allocations in file io.c 304 bytes in 5 allocations in file json.c 2296 bytes in 1 allocations in file libasteriskssl.c 69674 bytes in 388 allocations in file loader.c 12604 bytes in 8 allocations in file logger.c 215073 bytes in 333 allocations in file manager.c 1600 bytes in 1 allocations in file media_cache.c 96 bytes in 2 allocations in file message.c 3384 bytes in 6 allocations in file mwi.c 1104 bytes in 2 allocations in file named_acl.c 2560 bytes in 1 allocations in file named_locks.c 19408 bytes in 468 allocations in file optional_api.c 128 bytes in 1 allocations in file parking.c 120368 bytes in 849 allocations in file pbx.c 144 bytes in 1 allocations in file pbx_ael.c 524875 bytes in 443 allocations in file pbx_app.c 417728 bytes in 268 allocations in file pbx_functions.c 30 bytes in 3 allocations in file pbx_ignorepat.c 5390 bytes in 19 allocations in file pbx_include.c 13888 bytes in 1 allocations in file pbx_realtime.c 51 bytes in 1 allocations in file pbx_sw.c 276 bytes in 8 allocations in file pbx_variables.c 256 bytes in 2 allocations in file pjsip/cli_commands.c 56 bytes in 1 allocations in file presencestate.c 232 bytes in 1 allocations in file res_ari.c 592 bytes in 1 allocations in file res_calendar.c 3535 bytes in 7 allocations in file res_clialiases.c 304 bytes in 1 allocations in file res_config_sqlite3.c 376 bytes in 1 allocations in file res_fax.c 1093 bytes in 10 allocations in file res_http_websocket.c 1464 bytes in 2 allocations in file res_musiconhold.c 2348 bytes in 9 allocations in file res_parking.c 84597 bytes in 208 allocations in file res_phoneprov.c 2096 bytes in 35 allocations in file res_pjproject.c 192 bytes in 6 allocations in file res_pjsip.c 592 bytes in 2 allocations in file res_pjsip/config_auth.c 465 bytes in 2 allocations in file res_pjsip/config_global.c 96 bytes in 1 allocations in file res_pjsip/config_system.c 1536 bytes in 2 allocations in file res_pjsip/config_transport.c 256 bytes in 2 allocations in file res_pjsip/location.c 504 bytes in 1 allocations in file res_pjsip/pjsip_cli.c 3449 bytes in 10 allocations in file res_pjsip/pjsip_configuration.c 7728 bytes in 5 allocations in file res_pjsip/pjsip_distributor.c 1392 bytes in 3 allocations in file res_pjsip/pjsip_global_headers.c 75984 bytes in 3 allocations in file res_pjsip/pjsip_options.c 1432 bytes in 1 allocations in file res_pjsip/pjsip_scheduler.c 2184 bytes in 21 allocations in file res_pjsip/pjsip_session.c 3184 bytes in 1 allocations in file res_pjsip/pjsip_transport_events.c 3184 bytes in 1 allocations in file res_pjsip/pjsip_transport_management.c 109 bytes in 1 allocations in file res_pjsip_authenticator_digest.c 338 bytes in 7 allocations in file res_pjsip_config_wizard.c 128 bytes in 1 allocations in file res_pjsip_endpoint_identifier_ip.c 880 bytes in 1 allocations in file res_pjsip_exten_state.c 2048 bytes in 1 allocations in file res_pjsip_history.c 4216 bytes in 1 allocations in file res_pjsip_logger.c 2816 bytes in 2 allocations in file res_pjsip_mwi.c 5139 bytes in 43 allocations in file res_pjsip_notify.c 1536 bytes in 2 allocations in file res_pjsip_outbound_registration.c 2048 bytes in 2 allocations in file res_pjsip_pubsub.c 703 bytes in 4 allocations in file res_pjsip_session.c 160 bytes in 1 allocations in file res_smdi.c 35 bytes in 2 allocations in file res_sorcery_astdb.c 6978 bytes in 50 allocations in file res_sorcery_config.c 9856 bytes in 7 allocations in file res_sorcery_memory.c 1408 bytes in 1 allocations in file res_sorcery_memory_cache.c 11600 bytes in 5 allocations in file res_stasis.c 1024 bytes in 1 allocations in file res_stasis_device_state.c 3184 bytes in 1 allocations in file res_stasis_playback.c 3184 bytes in 1 allocations in file res_stasis_recording.c 344 bytes in 2 allocations in file res_statsd.c 13648 bytes in 1 allocations in file res_timing_pthread.c 120 bytes in 1 allocations in file res_timing_timerfd.c 3327 bytes in 61 allocations in file rtp_engine.c 52544 bytes in 359 allocations in file sched.c 376 bytes in 4 allocations in file serializer.c 126527 bytes in 554 allocations in file sorcery.c 56392 bytes in 273 allocations in file stasis.c 3240 bytes in 2 allocations in file stasis/messaging.c 98274 bytes in 50 allocations in file stasis_cache.c 728 bytes in 7 allocations in file stasis_cache_pattern.c 11310 bytes in 238 allocations in file stasis_message.c 3432 bytes in 20 allocations in file stasis_message_router.c 37104 bytes in 2 allocations in file stdtime/localtime.c 216 bytes in 4 allocations in file stream.c 44288 bytes in 1487 allocations in file stringfields.c 20288 bytes in 314 allocations in file strings.c 69643 bytes in 234 allocations in file taskprocessor.c 338 bytes in 9 allocations in file tcptls.c 25016 bytes in 90 allocations in file threadpool.c 200 bytes in 5 allocations in file timing.c 16768 bytes in 34 allocations in file translate.c 176 bytes in 2 allocations in file udptl.c 5650 bytes in 70 allocations in file utils.c 1114537 bytes in 5609 allocations in file xmldoc.c 1123107432 bytes allocated (1089193619 in caches) in 3012530 selected allocations 1123107432 bytes in all allocations 487259 bytes in deferred free large allocations 37097 bytes in deferred free small allocations 524356 bytes in deferred free allocations 1123631788 bytes in all allocations and deferred free allocations By: Benjamin Keith Ford (bford) 2020-09-17 11:55:25.095-0500 Are you able to isolate the scenario down to a simple use case? I tried spinning up multiple channels that went through the dialplan, calling {{Set(JITTERBUFFER(fixed)=200}} just to see if I could get the memory to rise, but could not get the same result. Is there a difference in memory consumption for you between {{fixed}} and {{adaptive}}? If you take out the {{JITTERBUFFER}} line in the dialplan, does memory stay at a sane level? By: Stefan Ruf (stefan20) 2020-09-30 08:56:53.397-0500 I have reduced the dialplan to 1. Set(JITTERBUFFER(adaptive)=default) 2. ConfBridge(berta) Is there a difference in memory consumption for you between fixed and adaptive? => I tried "(FIXED)=DEFAULT", "(ADAPTIVE)=DEFAULT" and "(FIXED)=200" every scenario rises up the memory If you take out the JITTERBUFFER line in the dialplan, does memory stay at a sane level? => Yes, without JITTERBUFFER memory stays sane The rising memory happens only in conjunction with muting/unmuting the channels. Plesae refer to attached file ami_commands.txt. Before passing actionid: 1705904476_335#berta-ddab9ea6-dcb4-4e83-b9bf-ff5e44325110 all works in a sane state. After handling this command, the memory begins to rise. By: Friendly Automation (friendly-automation) 2021-03-03 10:07:59.627-0600 Change 15538 merged by Friendly Automation: channel: Fix memory leak in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15538|https://gerrit.asterisk.org/c/asterisk/+/15538] By: Friendly Automation (friendly-automation) 2021-03-03 10:14:57.140-0600 Change 15537 merged by Joshua Colp: channel: Fix memory leak in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15537|https://gerrit.asterisk.org/c/asterisk/+/15537] By: Friendly Automation (friendly-automation) 2021-03-03 10:15:05.497-0600 Change 15536 merged by Joshua Colp: channel: Fix memory leak in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15536|https://gerrit.asterisk.org/c/asterisk/+/15536] By: Friendly Automation (friendly-automation) 2021-03-03 10:15:13.095-0600 Change 15556 merged by Joshua Colp: channel: Fix memory leak in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15556|https://gerrit.asterisk.org/c/asterisk/+/15556] By: Friendly Automation (friendly-automation) 2021-03-10 11:07:46.229-0600 Change 15605 merged by George Joseph: channel: Fix crash in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15605|https://gerrit.asterisk.org/c/asterisk/+/15605] By: Friendly Automation (friendly-automation) 2021-03-10 11:07:59.551-0600 Change 15606 merged by George Joseph: channel: Fix crash in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15606|https://gerrit.asterisk.org/c/asterisk/+/15606] By: Friendly Automation (friendly-automation) 2021-03-10 11:08:12.206-0600 Change 15604 merged by George Joseph: channel: Fix crash in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15604|https://gerrit.asterisk.org/c/asterisk/+/15604] By: Friendly Automation (friendly-automation) 2021-03-10 11:08:35.724-0600 Change 15621 merged by George Joseph: channel: Fix crash in suppress API. [https://gerrit.asterisk.org/c/asterisk/+/15621|https://gerrit.asterisk.org/c/asterisk/+/15621] |