Summary:	ASTERISK-02030: [patch] Update of ast_expr to use flex scanner - spaces no longer necc.
Reporter:	Steve Murphy (murf)	Labels:
Date Opened:	2004-07-15 23:56:30	Date Closed:	2011-06-07 14:10:09
Priority:	Minor	Regression?	No
Status:	Closed/Complete	Components:	Core/General
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) ast_expr2.fl ( 1) ast_expr2.y ( 2) diff.Makefile-154 ( 3) Makefile.diffu-vercomp-c-exec-realreal ( 4) Makefile-152.diffu ( 5) README.variables.diffu ( 6) vercomp.c ( 7) vercomp.pl ( 8) vercomp.sh
Description:	Even tho my previous patches eased some of the restrictions with spaces between tokens, most unix users and programmers are used to writing expressions where spaces are optional. I've taken the plunge; I put together a flex scanner to interleave with the bison grammar for exprs, and spaces are not required to separate tokens. There are pros and cons, tho! ****** ADDITIONAL INFORMATION ****** Pros:   The lexer is now described in flex format. It will   be easier to extend and maintain. At least, it should be!   You won't have newbies complaining about the way it   behaves, or asking why they are getting syntax errors,   or asking why the thing isn't working the way they   expected. Cons:   Legacy extensions.conf scripts 'depending' on spaces to separate tokens in $[] expressions may break. While such constructs may be rare, it is impossible to predict how many such scripts will be affected.
Comments:	By: Mark Spencer (markster) 2004-07-15 23:59:48 I'd like to keep this around for post 1.0 Changing the syntax for stuff really caused a lot of heartburn the last time even though the change was very minor.  We're so close to a 1.0 release now that I'd rather wait until after that before merging this. Does this also mean we need flex instead of just bison?  Are there Makefile changes as well? By: Steve Murphy (murf) 2004-07-16 00:33:22 The purpose of this upgrade is to make $[] as fool proof as possible. When I first started playing with extensions.conf, and making my own $[] expressions, I didn't know that README.variables existed and explained everything. I didn't know spaces were a big issue. As a long time unix user and programmer, I had a builtin expectation that spaces were just for making things work or just making them pretty. I think enough newbies have the same expectation, that the above will keep them from stubbing their toes as they play. Well, using a flex generated scanner with ast_expr.y fixes this problem, and, I hope, makes the source easier to extend and maintain while we are at it. However, there will be some bumps for users implementing the new code! First of all, they need to have a cutting-edge version of flex to get a re-entrant C scanner. This is a new thing in flex. Ya have to have flex-2.5.31 or later! (32 isn't out yet, BTW). So, this improvement wasn't possible a while back. To help keep down the hundreds of queries that will most surely arise as to why things aren't working (just like the necc. upgrade to bison-1.85 last time around), I've a patch for the Makefile that shows which version of flex and bison you have, and tells you what you need, and where you can get it. I hope this calms the angry waters in this respect. The Makefile patch will add the necc. rules to have both ast_expr2.y and ast_expr2.fl run thru their respective generators, and the resulting C files compiled and linked into asterisk. I've made them different files, so that "legacy" users can revert easily to their existing setup if it breaks their setup too badly. What's new beyond the flex scanner?  It's mostly explained in the README.variables2 file. I've reworded it about spaces separating tokens. I've had to add the unary-minus operator, because a leading minus sign was OK (I think) under the old regime, and just part of the number token, as long as they didn't separate it from the number by a space. I've made the colon operator, so it strips any double quotes from its arguments, so this operator will work as expected, even if one or both of its operands are wrapped in double quotes. I got rid of GE, LE, AND NE operators. They were not documented in the README.variable file. It looks like they were accepted by the previous lexer. The reason for wiping them: They were not documented in the README.variable file. They are not necessary, as there are the <=, >=, and != operators. -- So I made the code agree with the spec... To Build: throw ast_expr2.fl and ast_expr2.y into the asterisk/ dir. Apply the Makefile-expr2-patch file there. Throw the README.variables2 into the doc dir. do a make, make install. restart asterisk, and see how it works, but first you might review your extensions.conf file for any of the issue discussed above or at the end of the README.variables2 file. To test the parser/scanner outside of asterisk: type in "make testexpr2", which will make a small standalone executable. it takes one arg, a string to evaluate. you can try stuff like: ./testexpr2 ' 1+3*7+22 * 34 ' ./testexpr2 ' "Five Hundred" :"Five" ' and whatever else you want. It will output the result of the evaluation. BTW, I had the idea that adding the unary ! (logical complement) operator might be a "GOOD THING", but haven't done it yet. By: Steve Murphy (murf) 2004-07-16 00:37:48 Gee Wiz, you are quick. You got in a comment before I even finished my Bug Note! Hopefully, the verbage answers your questions. I did what I did, and I've placed it before you. It's up to you to decide when (and even if) you want to merge it into asterisk. I won't be upset either way. I'll try to help smooth things over when you do, like last time. Just give me warning, so I can schedule some time to help put out fires, if you do merge it in. By: twisted (twisted) 2004-10-27 15:44:19 okay, digging in the way-back machine here.  This was flagged for post-1.0 review.  Seeing the state of the bugtracker now, This is a call for action. Please check current cvs and make patches apply cleanly, and let's get some more activity going on this bug, before we have to close it for being open w/o activity for too long. -Housekeeping. By: Steve Murphy (murf) 2004-10-27 18:03:33 I haven't changed a byte in either of the flex/bison files since I uploaded them in, what, July. They've been incorporated in my version of asterisk and have worked reliably. I have noted that the Makefile has gone thru a bunch of revisions since then, so I uploaded a new patch for the Makefile. By: twisted (twisted) 2004-11-14 21:25:24.000-0600 Gonna poke at this again.   Note, that to keep these open, we need to keep activity going on them, otherwise, they can get closed due to lack of interest. --Housekeeping By: Steve Murphy (murf) 2004-11-23 08:16:30.000-0600 Just updated CVS on my machine, and no conflicts, so the patches I've turned in should still work. Let me know when you plan to push this into the tree, I'll try to help with the probs, that is, IF you are going to push this into the tree! By: Russell Bryant (russell) 2004-12-10 22:28:26.000-0600 I'm sitting next to Mark right now.  There is still interest in your patch.   Is there a way that we could use this while still maintaining backwards compatibility with older systems?  Could you detect what version of flex is available? Thanks. By: Steve Murphy (murf) 2004-12-11 08:53:22.000-0600 The makefile patch basically does a flex --version and tells the user that they need to run flex-2.5.31 or greater. It doesn't detect the version and either stop the compile or reroute the build. I hoped that this would help reduce the questions raised when they do the builds. What 'xactly do you want? A script to detect the version of flex, and stop the build if they haven't upgraded (with explanation of where to pick up the proper version,etc)?  Or, do you want to make the build conditional, and use the current version of the expr stuff if they have an older bison/flex, and use the newer .y/.fl if they have the newer versions of the bison/flex? By: Steve Murphy (murf) 2004-12-11 15:52:42.000-0600 I just uploaded a new version of ast_expr2.y, because I see that some corrections were made to routines in ast_expr.y, that I carried over. I have made the same mods, mostly in arithmetic error condition handling, in the new code. By: Mark Spencer (markster) 2004-12-11 22:06:51.000-0600 I'm thinking we select the right code based on the version of flex that would be best. By: Steve Murphy (murf) 2004-12-12 17:39:29.000-0600 OK. I've uploaded Makefile.diffunew, which contains some simple mechanisms to detect versions of bison and flex, and route the compilation accordingly. For Flex: if version 2.5.31 is present, the new flex/bison source is used. Otherwise, the current file is used, and verbage printed informing the user that installing a new version will upgrade the code... and where to get the latest source. This is done using make conditional statements and variables. For Bison: if version is less than 1.85, a NOTE is issued, informing the user where to get newer versions, if there is trouble. I am running RH9, and have bison 1.35 installed, and it works fine. For others with the same version, it doesn't work at all. So for now, I just issue a NOTE, and say they "may" have problems. This is done with shell statements. Both approaches are pretty crude, since shell scripts can't handle floating point numbers. But since we know the domain of versions of flex and bison, and that bison versions are 1.x, and the only version of flex that will work under the new regime is 2.5.31, what I've done will work for now. I have to get hold of the scripts that can handle arbitrary version comparisons, like what the GNU configure scripts do. This can wait until the next version of flex. What I've done works with the new version of flex, and the old version of bison. Please, see if they work OK for the other cases for you before pushing into CVS. By: twisted (twisted) 2005-01-04 01:00:13.000-0600 I'd like to clean up the files a bit, as well as shove this back to the top of the list so it's visible again.  murf - which files can go and which need to stay? By: Steve Murphy (murf) 2005-01-04 11:30:08.000-0600 It looks like you could remove Makefile-expr2-patch and Makefile-expr2-patch2, and the 17K ast_expr2.y file (the older of the two). When you're done, you should have: ast_expr2.fl ast_expr2.y README.variables2 Makefile.diffunew By: twisted (twisted) 2005-01-04 12:05:41.000-0600 Cleaned up ;)  thanks murf By: twisted (twisted) 2005-01-28 19:46:48.000-0600 Let's take a trip on the wayback machine... markster, is this going anywhere? By: Russell Bryant (russell) 2005-02-13 23:55:07.000-0600 We should probably merge this (if so) before 1.2 ... We need to get some people testing it. edited on: 02-13-05 23:56 By: Mark Spencer (markster) 2005-03-02 23:09:27.000-0600 Is it possible to document syntax which is compatible with both the old and new systems, and at a minimum to be sure we *can* have common syntax even if it can't for some reason be backwards compatible. By: Steve Murphy (murf) 2005-03-03 14:10:23.000-0600 After sifting thru the source for the "old" and "new" parsers, I came up with the following differences in syntax: Differences: Removed: --- nothing --- Added (or handled differently): Unary Minus now handled as syntax element (-1) would be parsed as before, but not (- 1). --(or will the "old" version handle -1 at all? I haven't tested). Unary (!) complement operator added. Seemed logical. "=~" operator added; This operator is the same as :, but not fixed to the beginning of the string. ===================================== And thatis it. Except: README.variables2 (attached file) goes over the incompatibilities list with the pre-flex/bison grammar, mostly involving the use of spaces and double quotes in the new regime. I've been using the new stuff so long now, I'm forgetting what version of stuff YOU are using... Oh, and I saw today that README.variables has had some additions that I have not tracked in README.variables2. I've just updated my version to include the new variables and paragraph on removing characters (which by the way, was a great addition, much needed, and I hadn't thought to include). I'll attach my updated README.variables2. By: Olle Johansson (oej) 2005-03-17 08:37:43.000-0600 We have docs and a patch - are we waiting for Markster or anything else that needs to be fixed? By: Kevin P. Fleming (kpfleming) 2005-03-17 09:35:35.000-0600 I believe we are waiting on a corrected Makefile patch, but Mark never mentioned that here (only in the conference call). I don't know specifically what is wrong with the current one (haven't had time to look), but I think it has something to do with the flex/bison version checking snippets. By: Michael Jerris (mikej) 2005-04-17 22:54:14 Reminder sent to murf murf-  are you working on an updated makefile patch for this that better handles the version numbers?  The issue was with requiring a specific version vs. a version > x.y.z By: Kevin P. Fleming (kpfleming) 2005-04-29 12:56:55 Mark, can we put this one to bed? I'll fix up the Makefile to do what you want, if you can document it here. By: Michael Jerris (mikej) 2005-04-29 13:58:12 the issue in the makefile was for the testing of the specific version of flex ifeq (${FLEXVER},flex 2.5.31), instead of a > kind of compare that is a little more difficult due to the need to parse the 2.5.31.  The bison check is only checking the x portion of 1.x and that will break as well if bison ever goes to a 2. release. By: Steve Murphy (murf) 2005-04-29 14:49:42 Sorry, didn't see any mail about this until today. The last I remember seeing was the comment dated 17 Mar, to which I couldn't say much. The trouble with the more general version number comparison is that it involves some intricate SH shell coding, or a program. All that trouble isn't even guaranteed to work forever, either, because in some future version 1.x or 2.x or whatever, if they modify the version string more than just a tad, it may break all the fancy code you invent. It turns into just more code to keep maintained. Neither Bison nor flex are being updated very often. It'd be easier to just follow the version history with individual entries in the makefile, than to invent the code to parse and compare version numbers. That's my 2¢. If you require it, I'll write a script or program, whichever you prefer. I may be able to hack something together from the version comparison stuff in the configure script stuff from GNU... whatever. Let me know. By: Kevin P. Fleming (kpfleming) 2005-04-29 19:23:21 I think we can safely rely on the flex/bison version numbering scheme staying consistent, since they are both GNU projects and they tend not to change things like that very often. Given that, I would think that some bash and/or awk scripting could be done to do the proper checks; certainly it's acceptable for these checks to be in a script external to the Makefile that just produces an output file for the remainder of the Makefile to be dependent on. By: Steve Murphy (murf) 2005-05-02 14:49:17 OK, just attached a perl script, vercomp.pl, and a new Makefile based on the latest version (1.152), with the perl script in use. The perl script takes 3 args:  <program>  <comparison-type>  <version#> The program is the program name or path to a program. The comparison type is either =, >, <, <=, or >=. The version# to compare with is a string like 1.875, or 1.5.4.  It can be of any size of version number. By this, I mean it can consist of any number of version numbers separated by dots. It runs the indicated program with the --version option, and collects the output. It tosses all but the first line of the output. So the version number had better be in the first line of output. Bison and flex both do this, maybe a bunch of others do, also, but that's irrelevant. It then plucks the version "word" from the rest of the output. It's expected to be the last word in the first line. Bison and flex do this. At least, for now. The "words" are separated by a space. It takes both version strings and splits them into arrays at the dots. It compares the two versions by starting comparing the first elements, and proceeds to the right until either a difference is found, or all elements are the same. If either the program, or the version # supplied on the arg, have any non-numeric characters associated with any of the numbers in the set, then the comparison between numbers will be alphabetic. If both sides are pure numbers, then the comparison is numeric. The version of bison I'm using is 1.875c, for instance. The perl script prints a "true" or "false" to stdout depending on whether the program's version vs the command-line supplied version follow the relationship indicated by the <comparison-type>. It's as general as I could think of at the moment. To be more powerful, all you have to do is add more code. Ha. It isn't perfect. It's pretty dumb about non-numeric codes. I can't foresee all the possible problems. But if bison and flex stick with their current versioning  system, then there shouldn't be any problems. If you try to use this script on more complicated version strings, be prepared to make it smarter. By: Kevin P. Fleming (kpfleming) 2005-05-04 13:13:26 Nice work! Just to confirm what to do for the commit: - vercomp.pl should be added - ast_expr2.fl should be added - ast_expr2.y should be added - README.variables2 is a replacement for README.variables, which can be merged - Makefile-152.diffu is the diff for the top-level Makefile Anything else? By: Steve Murphy (murf) 2005-05-04 21:51:36 Dang. I really get scatter brained at times. I attached the README.privacy. Wrong bug. Tried to stop it, but it went in, and I don't have authority to remove it. Anyway, yes, as best I can tell, you have the list correct. You have two README.variables2, you should delete the older one. (and the README.privacy, of course, also!). Make sure the vercomp.pl has execute permission, of course. And, you can remove the Makefile.diffunew (replaced by Makefile-152.diffu) And away you go. The existing bison and flex versions should determine the proper ast_expr files to use. Good Luck, and I should be around to help with any fires that may blaze. By: Michael Jerris (mikej) 2005-05-05 07:14:26 Cleaned up files, this looks ready to go. By: Kevin P. Fleming (kpfleming) 2005-05-05 08:50:46 Sorry for the delay in responding, but README.variables2 is missing quite a bit of changes that have happened in README.variables since 2005-03-03. We'll need a diff against the current README.variables that only adds the parts relevant to these patches... By: Steve Murphy (murf) 2005-05-05 11:40:35 Uh, just one problem; the file README.variables2 is a separate file, justified as such by the fact that some users will still be using the "old" parsers, some users will be using the "NEW" parser code, based on their versions of flex, bison. Now, ..., I guess it would be a hassle to keep the two versions up to date. So, I'll synthesize a single version, based on the CVS HEAD version, that includes special notes on the new code. I hope this is  acceptable. By: Michael Jerris (mikej) 2005-05-08 15:13:50 using vercomp.pl means we now have a perl requirement for building asterisk.  Is there a way to do this in shell scripts without adding the new dependency? By: Michael Jerris (mikej) 2005-05-08 15:14:41 By: Steve Murphy (murf) 2005-05-08 16:05:35 OK. I'll see if I can convert the perl script to a shell script. Sending the reminders sends me a copy of the bug notice, but so does the status changes, like confirmed->feedback or viceversa. Seems like it used to send an update if someone attached a note. I've generated a bug about this, and nothing so far invalidates the hypothesis... I should be done in a day or two. I've been sick with a head cold, but am recovering... By: Steve Murphy (murf) 2005-05-11 20:08:50 OK, I've done it. a new "vercomp.sh" script is created, that follows the exact same venue as vercomp.pl, and a new Makefile diff (against now 154), have been attached. Anything else? Actually, the shell script is really a BASH script. Hope that isn't too strict a requirement...? By: Kevin P. Fleming (kpfleming) 2005-05-15 20:31:29 Committed to CVS HEAD, thanks! (And major thanks for sticking with this for so long and going so through so many iterations...) By: Michael Jerris (mikej) 2005-05-15 22:14:34 vercomp.sh is blowing up: [root@phone asterisk]# ./vercomp.sh ./vercomp.sh: line 1: --version: command not found ./vercomp.sh: line 27: conditional binary operator expected ./vercomp.sh: line 27: syntax error near `=~' ./vercomp.sh: line 27: `[[ $progver1 =~ '([^ ]+$)'  ]]' for several diff versions of bison.  File is also seeing this issue.  It seems this script is not well! By: Kevin P. Fleming (kpfleming) 2005-05-15 22:25:29 That is not how it is supposed to be run... look at the Makefile, there are three arguments required for the script to do its job. By: Michael Jerris (mikej) 2005-05-15 22:34:50 from when running make, I get the same error when pulling the arguments right from the make file, sorry, I coppied the wrong one above. [root@phone asterisk]# make ./vercomp.sh: line 27: conditional binary operator expected ./vercomp.sh: line 27: syntax error near `=~' ./vercomp.sh: line 27: `[[ $progver1 =~ '([^ ]+$)'  ]]' By: Steve Murphy (murf) 2005-05-15 22:38:46 OK. I'll look into it. By: Steve Murphy (murf) 2005-05-15 22:41:01 What versions of flex/bison are you using? By: Michael Jerris (mikej) 2005-05-15 22:42:54 [root@phone asterisk]# bison --version bison (GNU Bison) 1.875c Written by Robert Corbett and Richard Stallman. Copyright (C) 2003 Free Software Foundation, Inc. This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. file had the same response on 1.35 and 2.0 By: Kevin P. Fleming (kpfleming) 2005-05-15 23:18:03 Well, you can fix the syntax error report by wrapping the variable reference in quotes so that it won't disappear if it's not set, but obviously there must be some reason why the version is not being properly extracted into the variable. By: Steve Murphy (murf) 2005-05-15 23:32:43 Just pulled out bison versions 1.25, 1.30, 1.34, 1.50, 1.75, and 2.0, which I have not yet played with; all work fine with vercomp.sh... And, on top of that, the flex versions 2.5.4 and 2.5.31, I have previously tested. To get a --version arg not accepted, implies that the program fired up with "bison", isn't bison (or flex, as the case may be). The other possibility is bash. the syntax error might be a result of a bash version not supporting the constructs I use in the script. So, also, let me know which version of bash you are using... I have GNU bash, version 3.00.14(1)-release; GNU has source for bash versions 1.14.7, 2.0, 2.01, 2.01.1, 2.02, 2.03, 2.04, 2.05, 2.05a, and 2.05b, and 3.0... and patches for several intermediate versions.... If this ends up being a bash compatibility problem, this could get a little involved....! By: Steve Murphy (murf) 2005-05-15 23:45:14 Or, it could be "head -1"... try running the command to extract the version by hand, and see what you get, and share it with me. By: Olle Johansson (oej) 2005-05-16 05:08:05 The vercomp.sh script adds a dependence to bash in /bin/bash. Do we want that to happen? I don't like accidentally adding new dependecies... Also egrep -o doesn't work on my system... (FreeBSD 4). Can we create another solution that is a bit more portable to solve this without adding the requirement to install bash, and install it in /bin... By: Michael Jerris (mikej) 2005-05-16 05:53:21 [root@phone asterisk]# bison --version | head -1 bison (GNU Bison) 1.875c bash:  GNU bash, version 2.05b.0(1)-release (i386-redhat-linux-gnu) By: Steve Murphy (murf) 2005-05-16 08:12:33 Perl isn't allowed. Bash isn't allowed. Some egrep opts not allowed... is that how bash does the =~. Can't code a C-program, this has to be avaiable during the make, not afterwards. Maybe gawk will work? I really am having doubts about sh... the tricky part is pulling the version from the first line of the answer to the version request... and breaking it into peices at the '.' -- and egrep -o would have been a natural fallback. Any ideas on a completely portable attack? By: Kevin P. Fleming (kpfleming) 2005-05-16 09:18:22 It should be possible to write this script using /bin/sh and POSIX-shell features (nothing bash-specific). That's a separate issue from why it's failing on some people's systems, though... Can anyone having this complete failure problem get me SSH access to the failing box? I need to get that part of this issue fixed quickly. By: Michael Jerris (mikej) 2005-05-16 09:32:38 Got kevin access to my dev box. By: Kevin P. Fleming (kpfleming) 2005-05-16 09:48:05 OK, here's the problem: regular expression matching is a bash-3.0 feature, not available in bash-2.0x. I've suppressed the calls to vercomp.sh from Makefile for now, until we can decide what the best course of action is. This _could_ be done using a small C program if necessary, as long as the dependencies are set up properly, I believe. By: Olle Johansson (oej) 2005-05-16 11:20:58 Good ol' sed is on every system though, isn't it? By: Kevin P. Fleming (kpfleming) 2005-05-16 11:42:27 Yes, sed is pretty safe to use, as is awk/gawk. murf, whatever path you take, if you need help with Makefile changes to accomodate it let me know. By: Mark Monnin (wrmem) 2005-05-16 12:35:52 FYI - Flex v2.4.31 is really new, and you can't get it from the Gnu website (it's now a sourceforge project).  Also, the version shipped with Fedora Core 3 and RHEL 3 & 4 are all 2.5.4.   Expect a lot of end-user problems getting this as it's not very common right now. By: Michael Jerris (mikej) 2005-05-16 12:45:25 wrmem, the beauty of this patch is that it runs fine if you do not have the newer versions, it just is better if you do.  The problems that people were having was with that version check. By: Steve Murphy (murf) 2005-05-16 13:31:19 Gentlemen-- I've uploaded another attempt at portability. A simple C program to compare the versions and return, as before "true" or "false" to the assertions. A new diffu with Makefile to compile and use the program... + +ifneq ($(wildcard vercomp),vercomp)  ### if vercomp hasn't been compiled yet... +zx=$(shell echo make vercomp)   ### uh, maybe you should take this out +zx=$(shell gcc -o vercomp vercomp.c)  ## compile it silently, very silently +endif + +FLEXVER_GT_2_5_31=$(shell ./vercomp.sh flex \>= 2.5.31) +BISONVER=$(shell bison --version | grep \^bison | egrep -o '[0-9]+\.[-0-9.]+[a-z]?' ) +BISONVERGE_85=$(shell ./vercomp.sh bison \>= 1.85 ) + +ifeq (${FLEXVER_GT_2_5_31},true) +FLEXOBJS=ast_expr2.o ast_expr2f.o +else +FLEXOBJS=ast_expr.o +endif As you can see, I buried the gcc for vercomp in the conditional statements of the Makefile, so it gets done in-line with the tests and variable assignments, before the actual "making" is done. I've tested it on bison 2.0, 1.875c, and flex 2.5.4 and 2.5.31. All seems well.... The program uses string.h and stdio.h. That's all, folks. So, the dependency is gcc (hardwired), but if you don't like that, you can tweek it to say cc, or $(CC) or whatever you think best... Sed, and gawk, might be portable, but all I can do with sed is pull out the version, maybe then split it. Shell scripts (portably) don't have arrays. So shell is out. Gawk would be OK, I'm sure I could do it with Gawk, but... some yahoo somewhere would probably use an ancient version of awk, and I'd be dead to some degree... So, C it is, unless you really, truely, deeply, yearningly, desire to have a Gawk-script. If so, let me know. I feel like a high-school kid with a hard-to-please teacher!! Hopefully, I can pass this course! By: Steve Murphy (murf) 2005-05-16 14:46:37 OOOps!  I edited the Makefile patch, but forgot to save it before uploading to Mantis. Sorry. The "real" diffu is now attached. You most likely don't want the leftovers of builtin mp3 stuff, now torn out! By: Mark Monnin (wrmem) 2005-05-16 16:38:37 First Asterisk crash in a while.  I think this one is due to this change, since it's ast_yylex that's high in the chain.  I have a "thread apply all bt" if need be. bison (GNU Bison) 1.875c flex 2.5.31 on a RHEL 3 system BTW.  I still like the new flex/bison grammer. -Mark (gdb) bt #0  0x0025fa8e in _int_malloc () from /lib/tls/libc.so.6 #1  0x0025ee9d in malloc () from /lib/tls/libc.so.6 #2  0x00266190 in strdup () from /lib/tls/libc.so.6 #3  0x080b501b in make_str (s=0x1 <Address 0x1 out of bounds>) at ast_expr.y:198 #4  0x080b160b in ast_yylex (lvalp=0x323188, ast_yylloc=0xb72b9450, karoto=0x0) at ast_expr.y:395 ASTERISK-1  0x080b151e in ast_yyparse (kota=0xb72b9490) at ast_expr.c:1072 ASTERISK-2  0x080b177d in ast_expr (arg=0xb72ba4f0 " \"40513\" = \"\" ") at ast_expr.y:4 31 ASTERISK-3  0x08089350 in pbx_substitute_variables_helper_full (c=0x952f4a8, cp1=0x9572b 18 "$[ \"${AUTH_MAILBOX}\" = \"\" ]?goodbye", cp2=0xb72c0700 "",    count=8191, headp=0x0) at pbx.c:1521 ASTERISK-4  0x0808033f in pbx_substitute_variables_helper (c=0xe, cp1=0xe <Address 0xe o ut of bounds>, cp2=0xe <Address 0xe out of bounds>, count=14)    at pbx.c:1541 ASTERISK-5  0x08089700 in pbx_extension_helper (c=0x952f4a8, con=0xe, context=0x952f5f8 "agent-login", exten=0x952f6ec "s", priority=7, label=0x0,    callerid=0x9572c93 "GotoIf", action=156221616) at pbx.c:1617 ASTERISK-6 0x08080cc4 in ast_pbx_run (c=0x952f4a8) at pbx.c:2227 ASTERISK-7 0x0808a0ff in pbx_thread (data=0xe) at pbx.c:2476 ASTERISK-8 0x00a0edec in start_thread () from /lib/tls/libpthread.so.0 ASTERISK-9 0x002caa2a in clone () from /lib/tls/libc.so.6 (gdb) Lines for extensions.conf exten => s,n(startlogin),Playback(welcome) exten => s,n(login),VMAuthenticate(@agents) exten => s,n,GotoIf($[ "${AUTH_MAILBOX}" = "" ]?goodbye) exten => s,n,Playback(welcome) By: Steve Murphy (murf) 2005-05-16 16:48:48 Something funny about the stack trace... 3 0x080b501b in make_str (s=0x1 <Address 0x1 out of bounds>) at ast_expr.y:198 4 0x080b160b in ast_yylex (lvalp=0x323188, ast_yylloc=0xb72b9450, karoto=0x0) at ast_expr.y:395 0000005 0x080b151e in ast_yyparse (kota=0xb72b9490) at ast_expr.c:1072 0000006 0x080b177d in ast_expr (arg=0xb72ba4f0 " \"40513\" = \"\" ") at ast_expr.y:4 31 It's mentioning variables and files from the old hand-written parser, ast_expr.y, and ast_expr.c; I'd expect to see the ast_expr2.y, and ast_expr2.fl mentioned instead, if the new guys actually got linked in. Go to your asterisk top level dir, and remove the ast_expr*.o files, and rebuild. There should be some messages to let you see what's happening during the build. One thing I notice about the make, is that if previous stuff By: Mark Monnin (wrmem) 2005-05-16 17:18:29 Ok. It's not the new code.  Did a removed the ast_expr*.o and forced a "make clean."  Still old code and still crashed.   Did some digging.  Makefile is still commenting out the test for flex/bison version.  So, "just for fun", hardcoded vercomp.sh to return "true" to test force new code, to see if it behaved differently.  Still crashes at (what I think is) the same dialplan line.   So, I'm thinking it maybe NOT related to the changes in the parser.   Odd though. (gdb) where #0  0x00206c77 in memset () from /lib/tls/libc.so.6 #1  0xb72b64f8 in ?? () #2  0x001fe53c in calloc () from /lib/tls/libc.so.6 #3  0x080b6514 in ast_yylex (yylval_param=0xb72b431c, yylloc_param=0xb72b5460,    yyscanner=0x979bf88) at ast_expr2.fl:70 #4  0x080b1e1e in ast_yyparse (parseio=0x978b7d0) at ast_expr2.c:1120 ASTERISK-1  0x080b71fb in ast_expr (arg=0xb72b64f0 " \"40513\"") at ast_expr2.fl:105 ASTERISK-2  0x08089990 in pbx_substitute_variables_helper_full (c=0x0, cp1=0x0,    cp2=0x0, count=0, headp=0x0) at pbx.c:1521 ASTERISK-3  0x00000000 in ?? () # /usr/sbin/asterisk -V Asterisk CVS-HEAD-05/16/05-17:00:14 By: Steve Murphy (murf) 2005-05-16 17:48:24 Here's something that may or may not help. Do a "make testexpr2" and then, if there's any way on earth at all you can determine which expr it's trying to eval, via gdb, or whatever, then you can run the testexpr2 program as follows: ./testexpr2 "expr" where "expr" is the expr you want to test. It will generate a parse error on the command line, etc... you might grep you extensions.conf for the $[ to see all the possible candidates. By: Mark Monnin (wrmem) 2005-05-16 17:55:44 Ok.  More detective work.  It's not your problem at all.  Sorry for that.  It's the app_voicemail above it.   If I do a "cvs update -r1.211 apps/app_voicemail.c" and recompile, it works, I I do a "cvs update -r1.212 apps/app_voicemail.c" it crashes.  Looks like it's clobbering things it shouldn't and then messes up the next stuff.  Off to do some more debugging on it.   Sorry for the false alarm on this one. By: Steve Murphy (murf) 2005-05-16 18:41:28 Whew! I was sweating... but still, ... I really hate it when I mess up, and I did it again. I've attached an even "more real" Makefile patch. (see *.realreal) Seems in all my eagerness to get the thing out, I forgot to turn the "vercomp.sh" commands into "vercomp" commands. And, I see that the egrep -o that oej said didn't work on his system, wasn't just some theory. I was actually using it! Duh! So, I changed that to use sed instead. Sorry. Some days I can go HOURS, being perfect. Today, tho, for some reason, it's down to minutes. Now, it's: +FLEXVER_GT_2_5_31=$(shell ./vercomp flex \>= 2.5.31) +BISONVER=$(shell bison --version | grep \^bison | sed 's/.* \([0-9]\+\.[-0-9.]\+[a-z]\?\)/\1/' ) +BISONVERGE_85=$(shell ./vercomp bison \>= 1.85 ) BISONVER is generated purely for aesthetics, it's output only appearing in an echo statement. OK. Now, everybody, give it a spin. See if it works. By: Michael Jerris (mikej) 2005-05-18 22:08:05 files this time around are: ast_expr2.fl, ast_expr2.y, README.variables.diffu, vercomp.c, Makefile.diffu-vercomp-c-exec-realreal By: Steve Murphy (murf) 2005-05-18 23:07:10 The list looks good to me. By: Kevin P. Fleming (kpfleming) 2005-05-19 00:00:06 I'm working on this one right now... I've implemented the Makefile changes in a different way, that allows make to still use its implicit rules to build the vercomp and ast_expr.o targets (and also ensures that the 'host' compiler is used to build vercomp if a cross compiler is being used for the rest of Asterisk). I'm nearly done testing it, so it will be committed shortly. Oh, also I've added a copyright/license notice to the top of vercomp.c, but I don't have your proper name and email address... if you want the header to have those details please let me know. By: Kevin P. Fleming (kpfleming) 2005-05-19 00:03:09 Committed to CVS HEAD, thanks! By: Steve Murphy (murf) 2005-05-19 09:45:36 Uh, OK. "murf" in the copyright field all by itself is pretty ugly... If you are going to assign it to me, use "Steven Michael Murphy", with email addr murf at e-tools dot com, in machine readable format, of course. (Avoiding mailbots here). If that's a bother, assign the copyright to Digium, FSF, or whatever's best. By: Kevin P. Fleming (kpfleming) 2005-05-19 10:49:39 Updated the copyright header... and also changed the Makefile a bit more, we were having some shell portability problems on FreeBSD. By: Tilghman Lesher (tilghman) 2005-06-07 11:36:11 I'm reopening this bug, as it relates to bug ASTERISK-4362 -- apparently there's a bug in this new expression parser, as it relates to string comparisons. By: Steve Murphy (murf) 2005-06-07 22:15:39 Anything I can do to help? By: Michael Jerris (mikej) 2005-06-08 07:58:36 murf is on this new issue in the 4478 bug.  No need to have 2 bugs open for the same issue. By: Digium Subversion (svnbot) 2008-01-15 15:02:34.000-0600 Repository: asterisk Revision: 3444 U   trunk/redhat/asterisk.spec ------------------------------------------------------------------------ r3444 | markster | 2008-01-15 15:02:34 -0600 (Tue, 15 Jan 2008) | 2 lines Put header files in -devel package (bug ASTERISK-2030) ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=3444 By: Digium Subversion (svnbot) 2008-01-15 15:34:57.000-0600 Repository: asterisk Revision: 5691 U   trunk/Makefile A   trunk/ast_expr2.fl A   trunk/ast_expr2.y U   trunk/doc/README.variables A   trunk/vercomp.sh ------------------------------------------------------------------------ r5691 | kpfleming | 2008-01-15 15:34:56 -0600 (Tue, 15 Jan 2008) | 2 lines add upgraded expression parser (bug ASTERISK-2030) ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=5691 By: Digium Subversion (svnbot) 2008-01-15 15:35:32.000-0600 Repository: asterisk Revision: 5731 _U  trunk/ U   trunk/.cvsignore U   trunk/Makefile A   trunk/vercomp.c D   trunk/vercomp.sh ------------------------------------------------------------------------ r5731 | kpfleming | 2008-01-15 15:35:31 -0600 (Tue, 15 Jan 2008) | 2 lines use new C-coded version comparison program for bison and flex (bug ASTERISK-2030, with different Makefile changes) ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=5731