Summary:	ASTERISK-05854: AEL silently skipping whole contexts.
Reporter:	Steve Murphy (murf)	Labels:
Date Opened:	2005-12-16 12:49:05.000-0600	Date Closed:	2006-07-06 11:15:22
Priority:	Blocker	Regression?	No
Status:	Closed/Complete	Components:	PBX/pbx_ael
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) 20051217__bug6013.diff.txt ( 1) extensions.ael.bug1.sanitized
Description:	I'm feeding an extensions.ael to AEL, which has 94 contexts, and 22 contexts in the middle of the file are empty. The contexts are created and a show dialplan shows they are there, but empty. Contexts both before  and after these are filled in properly. No error messages. Even 'ael debug tokens' just seems to quit after a while, and generates no more messages while in an "asterisk -r" session. ****** STEPS TO REPRODUCE ****** Here it gets sticky. My dialplan has passwords, phone numbers, etc. that I don't really want to send off in a bug report. I'll see if I can make a sanitized copy.
Comments:	By: twisted (twisted) 2005-12-16 12:52:01.000-0600 You just said that they were empty in the file.  This means they will also be empty when loaded.  Where is the bug? By: Steve Murphy (murf) 2005-12-16 13:47:54.000-0600 Sorry, the missing contexts are from the middle of the file. In asterisk, they are empty. In the file, they varied in size from small to large. In the file, none were empty. It's clearly a big bug, as no error message or debug helps to determine where the problem might be. No common item contained in the affected contexts jumps out to say "I might be causing this problem". The contents at the end of the file that were successfully parsed, are fairly simple, just an "s" extension with some Background() calls, all the numeric extensions 0-9 (usually), and the t,i, extensions. No variables at all. Sometimes, if I forget to turn a ; into a //, I get some hints in the "show dialplan", but everything looks clean now. I say 'LOOKS', because at this point in time, I can't see the problem. I'm sure it's there, I've done SOMETHING wrong, but the parser is giving me no clues as to what is bad. And that's the bug. By: Steve Murphy (murf) 2005-12-16 14:42:50.000-0600 OK, it's nasty to submit a bug with no test data. So, I've taken some time to sanitize my extensions.ael, and sure enough, it has the same problems my real one does. Please, no snide comments. I'm feeling a little sensitive right now. ;^) By: Jason Parker (jparker) 2005-12-16 14:47:46.000-0600 Can you list the names of (say, 2 of each?) contexts that do and do not work, from the example? By: Tilghman Lesher (tilghman) 2005-12-16 14:47:59.000-0600 Which contexts, specifically, are showing up blank? By: Steve Murphy (murf) 2005-12-16 17:09:07.000-0600 On my machine, privacyManagerFailed homeline pageall add-to-conference home-introduction home-introduction-record homeline-kids voipworkline workline dialFWD dialiaxtel dialgoiax homefirst workfirst force_cell homeext fromvmhome fromvmwork fromSeanUniden workext wakeup announce-all are all empty after the extensions.ael is loaded. They are in a contiguous block. PrivacyManagerFailed is the first context in the file, all those that precede it are macro definitions. Commenting out PrivacyManagerFailed does not change anything, except reduce the list by one. By: twisted (twisted) 2005-12-16 17:13:38.000-0600 question: if you remove your //comments // more comments before the contexts that are not showing up, do they start showing up properly? By: twisted (twisted) 2005-12-16 17:17:25.000-0600 if not, the only other difference I noted was that your context's at the bottom of the list you sstated that are showing up empty do not have  a '{' at the end of the line where you define the context, but the ones at the bottom do.  Macro's seem to work without it. Try moving the opening '{' to the end of the line context privacyManagerFailed { as opposed to the way you have it.  if that starts solving them, it'll give us a dead on location to look in the parser. By: Tilghman Lesher (tilghman) 2005-12-16 18:50:42.000-0600 Please type the following sequence into the CLI: ael debug tokens ael debug contexts ael reload ael no debug and post the results as a file upload.  You may wish to log verbose output first in logger.conf as: ael_debug => verbose and do a 'logger reload' from the CLI. By: Steve Murphy (murf) 2005-12-17 07:59:19.000-0600 Sorry, went to a christmas party. You've hit the nail on the head, twisted! I moved the curly up to the end of the context declaration line, and all the contexts load fine now. I LOVE ael, but the diagnostics are just plain awful! If the curly was in the wrong place, it should just have told me, not leave me to guess why the contents of the context was ignored. It would have been another day or so before I hit on the placement of the curly as to the cause. And, really, the placement of the curly should not be line-dependent. You have semicolons at the end of each statement, so line breaks could and should be superfluous.  Wait-- does this mean I can't put two statements on the same line? And what will happen if I do? I patched up the contexts and asterisk is running again. It'll take a few days/weeks to fully test and debug all the contexts, but I'm on my way. I don't really know if you can close this bug, tho! Upgrading ael to give better diagnostics should be a priority! By: Tilghman Lesher (tilghman) 2005-12-17 08:35:45.000-0600 As noted in the previous bugnote, there are diagnostics; you just haven't turned them on. By: Steve Murphy (murf) 2005-12-17 11:34:06.000-0600 That's false. I DID turn on ALL the diagnostics (they are documented in the source), and it did NOT help me one bit. Sorry I didn't mention this. Now, if you can take my test case, turn on all the diagnostics, and see where/why the problem occurred, and you are perfectly satisfied that ANY asterisk dialplan developer would be able to easily spot the problem and rectify it from those diagnostics, then go ahead and CLOSE the issue again. I can be exceptionally dense at times, and may have missed some obvious clue. Begin rave. I just sat down and started mapping out a BNF (in bison syntax) for AEL. We could use flex's ability to build contextual lexers. At the very mininum, a bison parser could be built, that would not only generate the information that the compiler would need, but would also do a very rigorous check of the syntax in the file, right down to the innards of the $[ ] expressions, so probs can be found at compile time instead of at run time. Maybe I'm a fool for proposing this, but it would make the detection and reporting of syntax errors easier, make the compiler more robust and easier to maintain and upgrade, get rid of useless/needless line restrictions, provide a clear definition of exactly what the parser expects everywhere. The added benefit would be that the parser and compiler would be separated from each other, which doesn't appear to be the case in pbx_ael.c, but is a better practice in compiler building. The parser would generate a linked data structure, and the compiler could traverse it to generate code (asterisk extension.conf syntax, or straight to internal structures). The code would be simpler, as all parsing details would be separated out into the parser. And you won't have to exhaustively test for all possible syntax problems and report each in the code-- the parser has mechanisms to detect and recover from errors, which can be hand-tuned. Right now, I think you'd have to considerably add code (double the size?) to pbx_ael to do proper error checking, and still miss a ton of cases. End of rave. By: Tilghman Lesher (tilghman) 2005-12-17 11:47:00.000-0600 In that case, I invite you to open a Feature Request with your suggestion on bison and flex, and let us close this one out, since the problems would be resolved with such an implementation.