Summary:ASTERISK-02030: [patch] Update of ast_expr to use flex scanner - spaces no longer necc.
Reporter:Steve Murphy (murf)Labels:
Date Opened:2004-07-15 23:56:30Date Closed:2011-06-07 14:10:09
Versions:Frequency of
Environment:Attachments:( 0) ast_expr2.fl
( 1) ast_expr2.y
( 2) diff.Makefile-154
( 3) Makefile.diffu-vercomp-c-exec-realreal
( 4) Makefile-152.diffu
( 5) README.variables.diffu
( 6) vercomp.c
( 7) vercomp.pl
( 8) vercomp.sh
Description:Even tho my previous patches eased some of the restrictions
with spaces between tokens, most unix users and programmers
are used to writing expressions where spaces are optional.

I've taken the plunge; I put together a flex scanner
to interleave with the bison grammar for exprs, and
spaces are not required to separate tokens.

There are pros and cons, tho!


  The lexer is now described in flex format. It will
  be easier to extend and maintain. At least, it should be!

  You won't have newbies complaining about the way it
  behaves, or asking why they are getting syntax errors,
  or asking why the thing isn't working the way they

  Legacy extensions.conf scripts 'depending' on spaces to
separate tokens in $[] expressions may break. While such
constructs may be rare, it is impossible to predict how
many such scripts will be affected.

Comments:By: Mark Spencer (markster) 2004-07-15 23:59:48

I'd like to keep this around for post 1.0

Changing the syntax for stuff really caused a lot of heartburn the last time even though the change was very minor.  We're so close to a 1.0 release now that I'd rather wait until after that before merging this.

Does this also mean we need flex instead of just bison?  Are there Makefile changes as well?

By: Steve Murphy (murf) 2004-07-16 00:33:22

The purpose of this upgrade is to make $[] as fool proof as possible.
When I first started playing with extensions.conf, and making my own
$[] expressions, I didn't know that README.variables existed and explained
everything. I didn't know spaces were a big issue. As a long time unix
user and programmer, I had a builtin expectation that spaces were
just for making things work or just making them pretty. I think enough
newbies have the same expectation, that the above will keep them from
stubbing their toes as they play.

Well, using a flex generated scanner with ast_expr.y fixes this problem,
and, I hope, makes the source easier to extend and maintain while we are at it.

However, there will be some bumps for users implementing the new code!
First of all, they need to have a cutting-edge version of flex to get a
re-entrant C scanner. This is a new thing in flex. Ya have to have
flex-2.5.31 or later! (32 isn't out yet, BTW). So, this improvement
wasn't possible a while back.

To help keep down the hundreds of queries that will most surely arise
as to why things aren't working (just like the necc. upgrade to bison-1.85
last time around), I've a patch for the Makefile that shows which version
of flex and bison you have, and tells you what you need, and where you can
get it. I hope this calms the angry waters in this respect.

The Makefile patch will add the necc. rules to have both ast_expr2.y and
ast_expr2.fl run thru their respective generators, and the resulting C
files compiled and linked into asterisk. I've made them different files,
so that "legacy" users can revert easily to their existing setup if
it breaks their setup too badly.

What's new beyond the flex scanner?  It's mostly explained in the README.variables2 file. I've reworded it about spaces separating tokens.
I've had to add the unary-minus operator, because a leading minus sign
was OK (I think) under the old regime, and just part of the number token,
as long as they didn't separate it from the number by a space.
I've made the colon operator, so it strips any double quotes from its arguments,
so this operator will work as expected, even if one or both of its operands
are wrapped in double quotes. I got rid of GE, LE, AND NE operators. They
were not documented in the README.variable file. It looks like they were
accepted by the previous lexer. The reason for wiping them: They
were not documented in the README.variable file. They are not
necessary, as there are the <=, >=, and != operators. -- So I made the code
agree with the spec...

To Build: throw ast_expr2.fl and ast_expr2.y into the asterisk/ dir.
Apply the Makefile-expr2-patch file there. Throw the README.variables2
into the doc dir. do a make, make install. restart asterisk, and see
how it works, but first you might review your extensions.conf file for
any of the issue discussed above or at the end of the README.variables2 file.

To test the parser/scanner outside of asterisk: type in "make testexpr2",
which will make a small standalone executable. it takes one arg, a
string to evaluate. you can try stuff like:

./testexpr2 ' 1+3*7+22 * 34 '
./testexpr2 ' "Five Hundred" :"Five" '

and whatever else you want. It will output the result of the evaluation.

BTW, I had the idea that adding the unary ! (logical complement) operator
might be a "GOOD THING", but haven't done it yet.

By: Steve Murphy (murf) 2004-07-16 00:37:48

Gee Wiz, you are quick. You got in a comment before I even finished my Bug Note!
Hopefully, the verbage answers your questions.

I did what I did, and I've placed it before you. It's up to you to decide
when (and even if) you want to merge it into asterisk. I won't be upset either
way. I'll try to help smooth things over when you do, like last time. Just
give me warning, so I can schedule some time to help put out fires, if you
do merge it in.

By: twisted (twisted) 2004-10-27 15:44:19

okay, digging in the way-back machine here.  This was flagged for post-1.0 review.  Seeing the state of the bugtracker now, This is a call for action.

Please check current cvs and make patches apply cleanly, and let's get some more activity going on this bug, before we have to close it for being open w/o activity for too long.


By: Steve Murphy (murf) 2004-10-27 18:03:33

I haven't changed a byte in either of the flex/bison files since I uploaded
them in, what, July.

They've been incorporated in my version of asterisk and have worked

I have noted that the Makefile has gone thru a bunch of revisions since then,
so I uploaded a new patch for the Makefile.

By: twisted (twisted) 2004-11-14 21:25:24.000-0600

Gonna poke at this again.   Note, that to keep these open, we need to keep activity going on them, otherwise, they can get closed due to lack of interest.


By: Steve Murphy (murf) 2004-11-23 08:16:30.000-0600

Just updated CVS on my machine, and no conflicts, so the patches I've turned in should still work. Let me know when you plan to push this into the tree, I'll try to help with the probs, that is, IF you are going to push this into the tree!

By: Russell Bryant (russell) 2004-12-10 22:28:26.000-0600

I'm sitting next to Mark right now.  There is still interest in your patch.  

Is there a way that we could use this while still maintaining backwards compatibility with older systems?  Could you detect what version of flex is available?


By: Steve Murphy (murf) 2004-12-11 08:53:22.000-0600

The makefile patch basically does a flex --version and tells the user that
they need to run flex-2.5.31 or greater. It doesn't detect the version and
either stop the compile or reroute the build. I hoped that this would help
reduce the questions raised when they do the builds.

What 'xactly do you want? A script to detect the version of flex, and stop
the build if they haven't upgraded (with explanation of where to pick up the
proper version,etc)?  Or, do you want to make the build conditional, and use
the current version of the expr stuff if they have an older bison/flex, and
use the newer .y/.fl if they have the newer versions of the bison/flex?

By: Steve Murphy (murf) 2004-12-11 15:52:42.000-0600

I just uploaded a new version of ast_expr2.y, because I see that some
corrections were made to routines in ast_expr.y, that I carried over. I have
made the same mods, mostly in arithmetic error condition handling, in the
new code.

By: Mark Spencer (markster) 2004-12-11 22:06:51.000-0600

I'm thinking we select the right code based on the version of flex that would be best.

By: Steve Murphy (murf) 2004-12-12 17:39:29.000-0600

OK. I've uploaded Makefile.diffunew, which contains some simple
mechanisms to detect versions of bison and flex, and route the compilation

For Flex: if version 2.5.31 is present, the new flex/bison source is used. Otherwise, the current file is used, and verbage printed informing the
user that installing a new version will upgrade the code... and where to
get the latest source. This is done using make conditional statements and variables.

For Bison: if version is less than 1.85, a NOTE is issued, informing the
user where to get newer versions, if there is trouble. I am running RH9, and
have bison 1.35 installed, and it works fine. For others with the same version,
it doesn't work at all. So for now, I just issue a NOTE, and say they "may" have problems. This is done with shell statements.

Both approaches are pretty crude, since shell scripts can't handle floating point numbers. But since we know the domain of versions of flex and bison, and
that bison versions are 1.x, and the only version of flex that will work under the new regime is 2.5.31, what I've done will work for now. I have to get hold of the scripts that can handle arbitrary version comparisons, like what the GNU configure scripts do. This can wait until the next version of flex.

What I've done works with the new version of flex, and the old version of bison. Please, see if they work OK for the other cases for you before pushing into CVS.

By: twisted (twisted) 2005-01-04 01:00:13.000-0600

I'd like to clean up the files a bit, as well as shove this back to the top of the list so it's visible again.  murf - which files can go and which need to stay?

By: Steve Murphy (murf) 2005-01-04 11:30:08.000-0600

It looks like you could remove Makefile-expr2-patch and Makefile-expr2-patch2, and the 17K ast_expr2.y file (the older of the two).

When you're done, you should have:

By: twisted (twisted) 2005-01-04 12:05:41.000-0600

Cleaned up ;)  thanks murf

By: twisted (twisted) 2005-01-28 19:46:48.000-0600

Let's take a trip on the wayback machine... markster, is this going anywhere?

By: Russell Bryant (russell) 2005-02-13 23:55:07.000-0600

We should probably merge this (if so) before 1.2 ...

We need to get some people testing it.

edited on: 02-13-05 23:56

By: Mark Spencer (markster) 2005-03-02 23:09:27.000-0600

Is it possible to document syntax which is compatible with both the old and new systems, and at a minimum to be sure we *can* have common syntax even if it can't for some reason be backwards compatible.

By: Steve Murphy (murf) 2005-03-03 14:10:23.000-0600

After sifting thru the source for the "old" and "new" parsers, I came up
with the following differences in syntax:


--- nothing ---

Added (or handled differently):

Unary Minus now handled as syntax element (-1) would be parsed as before, but not (- 1). --(or will the "old" version handle -1 at all? I haven't tested).

Unary (!) complement operator added. Seemed logical.

"=~" operator added; This operator is the same as :, but not fixed to the beginning of the string.


And thatis it. Except: README.variables2 (attached file) goes over the
incompatibilities list with the pre-flex/bison grammar, mostly involving the
use of spaces and double quotes in the new regime. I've been using the new
stuff so long now, I'm forgetting what version of stuff YOU are using...

Oh, and I saw today that README.variables has had some additions that I have
not tracked in README.variables2. I've just updated my version to include the
new variables and paragraph on removing characters (which by the way, was a
great addition, much needed, and I hadn't thought to include).

I'll attach my updated README.variables2.

By: Olle Johansson (oej) 2005-03-17 08:37:43.000-0600

We have docs and a patch - are we waiting for Markster or anything else that needs to be fixed?

By: Kevin P. Fleming (kpfleming) 2005-03-17 09:35:35.000-0600

I believe we are waiting on a corrected Makefile patch, but Mark never mentioned that here (only in the conference call). I don't know specifically what is wrong with the current one (haven't had time to look), but I think it has something to do with the flex/bison version checking snippets.

By: Michael Jerris (mikej) 2005-04-17 22:54:14

Reminder sent to murf

murf-  are you working on an updated makefile patch for this that better handles the version numbers?  The issue was with requiring a specific version vs. a version > x.y.z

By: Kevin P. Fleming (kpfleming) 2005-04-29 12:56:55

Mark, can we put this one to bed? I'll fix up the Makefile to do what you want, if you can document it here.

By: Michael Jerris (mikej) 2005-04-29 13:58:12

the issue in the makefile was for the testing of the specific version of flex ifeq (${FLEXVER},flex 2.5.31), instead of a > kind of compare that is a little more difficult due to the need to parse the 2.5.31.  The bison check is only checking the x portion of 1.x and that will break as well if bison ever goes to a 2. release.

By: Steve Murphy (murf) 2005-04-29 14:49:42

Sorry, didn't see any mail about this until today. The last I remember seeing was the comment dated 17 Mar, to which I couldn't say much.

The trouble with the more general version number comparison is that it involves some intricate SH shell coding, or a program. All that trouble isn't even guaranteed to work forever, either, because in some future version 1.x or 2.x or whatever, if they modify the version string more than just a tad, it may break all the fancy code you invent. It turns into just more code to keep maintained.

Neither Bison nor flex are being updated very often. It'd be easier to just follow the version history with individual entries in the makefile, than to invent the code to parse and compare version numbers.

That's my 2ยข. If you require it, I'll write a script or program, whichever you prefer. I may be able to hack something together from the version comparison stuff in the configure script stuff from GNU... whatever. Let me know.

By: Kevin P. Fleming (kpfleming) 2005-04-29 19:23:21

I think we can safely rely on the flex/bison version numbering scheme staying consistent, since they are both GNU projects and they tend not to change things like that very often. Given that, I would think that some bash and/or awk scripting could be done to do the proper checks; certainly it's acceptable for these checks to be in a script external to the Makefile that just produces an output file for the remainder of the Makefile to be dependent on.

By: Steve Murphy (murf) 2005-05-02 14:49:17

OK, just attached a perl script, vercomp.pl, and a new Makefile based on the latest version (1.152), with the perl script in use.

The perl script takes 3 args:  <program>  <comparison-type>  <version#>

The program is the program name or path to a program.

The comparison type is either =, >, <, <=, or >=.

The version# to compare with is a string like 1.875, or 1.5.4.  It can be of any size of version number. By this, I mean it can consist of any number of version numbers separated by dots.

It runs the indicated program with the --version option, and collects the output. It tosses all but the first line of the output. So the version number had better be in the first line of output. Bison and flex both do this, maybe a bunch of others do, also, but that's irrelevant.

It then plucks the version "word" from the rest of the output. It's expected to
be the last word in the first line. Bison and flex do this. At least, for now.
The "words" are separated by a space.

It takes both version strings and splits them into arrays at the dots.
It compares the two versions by starting comparing the first elements,
and proceeds to the right until either a difference is found, or all elements
are the same.

If either the program, or the version # supplied on the arg, have any non-numeric characters associated with any of the numbers in the set, then the comparison between numbers will be alphabetic. If both sides are pure numbers,
then the comparison is numeric. The version of bison I'm using is 1.875c, for instance.

The perl script prints a "true" or "false" to stdout depending on whether the program's version vs the command-line supplied version follow the relationship indicated by the <comparison-type>.

It's as general as I could think of at the moment. To be more powerful, all you have to do is add more code. Ha.

It isn't perfect. It's pretty dumb about non-numeric codes. I can't foresee all the possible problems. But if bison and flex stick with their current versioning  system, then there shouldn't be any problems. If you try to use this script on
more complicated version strings, be prepared to make it smarter.

By: Kevin P. Fleming (kpfleming) 2005-05-04 13:13:26

Nice work!

Just to confirm what to do for the commit:

- vercomp.pl should be added
- ast_expr2.fl should be added
- ast_expr2.y should be added
- README.variables2 is a replacement for README.variables, which can be merged
- Makefile-152.diffu is the diff for the top-level Makefile

Anything else?

By: Steve Murphy (murf) 2005-05-04 21:51:36

Dang. I really get scatter brained at times. I attached the README.privacy. Wrong bug. Tried to stop it, but it went in, and I don't have authority to remove it.

Anyway, yes, as best I can tell, you have the list correct. You have two README.variables2, you should delete the older one. (and the README.privacy, of course, also!).

Make sure the vercomp.pl has execute permission, of course.

And, you can remove the Makefile.diffunew (replaced by Makefile-152.diffu)

And away you go. The existing bison and flex versions should determine the proper ast_expr files to use.

Good Luck, and I should be around to help with any fires that may blaze.

By: Michael Jerris (mikej) 2005-05-05 07:14:26

Cleaned up files, this looks ready to go.

By: Kevin P. Fleming (kpfleming) 2005-05-05 08:50:46

Sorry for the delay in responding, but README.variables2 is missing quite a bit of changes that have happened in README.variables since 2005-03-03. We'll need a diff against the current README.variables that only adds the parts relevant to these patches...

By: Steve Murphy (murf) 2005-05-05 11:40:35

Uh, just one problem; the file README.variables2 is a separate file, justified as such by the fact that some users will still be using the "old" parsers, some users will be using the "NEW" parser code, based on their versions of flex, bison.

Now, ..., I guess it would be a hassle to keep the two versions up to date. So, I'll synthesize a single version, based on the CVS HEAD version, that includes special notes on the new code.

I hope this is  acceptable.

By: Michael Jerris (mikej) 2005-05-08 15:13:50

using vercomp.pl means we now have a perl requirement for building asterisk.  Is there a way to do this in shell scripts without adding the new dependency?

By: Michael Jerris (mikej) 2005-05-08 15:14:41

By: Steve Murphy (murf) 2005-05-08 16:05:35

OK. I'll see if I can convert the perl script to a shell script.
Sending the reminders sends me a copy of the bug notice, but so does the
status changes, like confirmed->feedback or viceversa. Seems like it used to send an update if someone attached a note. I've generated a bug about this, and nothing so far invalidates the hypothesis...

I should be done in a day or two. I've been sick with a head cold, but am recovering...

By: Steve Murphy (murf) 2005-05-11 20:08:50

OK, I've done it. a new "vercomp.sh" script is created, that follows the exact same venue as vercomp.pl, and a new Makefile diff (against now 154), have been attached.

Anything else?

Actually, the shell script is really a BASH script. Hope that isn't too strict a requirement...?

By: Kevin P. Fleming (kpfleming) 2005-05-15 20:31:29

Committed to CVS HEAD, thanks! (And major thanks for sticking with this for so long and going so through so many iterations...)

By: Michael Jerris (mikej) 2005-05-15 22:14:34

vercomp.sh is blowing up:

[root@phone asterisk]# ./vercomp.sh
./vercomp.sh: line 1: --version: command not found
./vercomp.sh: line 27: conditional binary operator expected
./vercomp.sh: line 27: syntax error near `=~'
./vercomp.sh: line 27: `[[ $progver1 =~ '([^ ]+$)'  ]]'

for several diff versions of bison.  File is also seeing this issue.  It seems this script is not well!

By: Kevin P. Fleming (kpfleming) 2005-05-15 22:25:29

That is not how it is supposed to be run... look at the Makefile, there are three arguments required for the script to do its job.

By: Michael Jerris (mikej) 2005-05-15 22:34:50

from when running make, I get the same error when pulling the arguments right from the make file, sorry, I coppied the wrong one above.

[root@phone asterisk]# make
./vercomp.sh: line 27: conditional binary operator expected
./vercomp.sh: line 27: syntax error near `=~'
./vercomp.sh: line 27: `[[ $progver1 =~ '([^ ]+$)'  ]]'

By: Steve Murphy (murf) 2005-05-15 22:38:46

OK. I'll look into it.

By: Steve Murphy (murf) 2005-05-15 22:41:01

What versions of flex/bison are you using?

By: Michael Jerris (mikej) 2005-05-15 22:42:54

[root@phone asterisk]# bison --version
bison (GNU Bison) 1.875c
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO

file had the same response on 1.35 and 2.0

By: Kevin P. Fleming (kpfleming) 2005-05-15 23:18:03

Well, you can fix the syntax error report by wrapping the variable reference in quotes so that it won't disappear if it's not set, but obviously there must be some reason why the version is not being properly extracted into the variable.

By: Steve Murphy (murf) 2005-05-15 23:32:43

Just pulled out bison versions 1.25, 1.30, 1.34, 1.50, 1.75, and 2.0,
which I have not yet played with; all work fine with vercomp.sh...
And, on top of that, the flex versions 2.5.4 and 2.5.31, I have previously

To get a --version arg not accepted, implies that the program fired up
with "bison", isn't bison (or flex, as the case may be).

The other possibility is bash. the syntax error might be a result of a bash
version not supporting the constructs I use in the script. So, also, let me know which version of bash you are using... I have GNU bash, version 3.00.14(1)-release;

GNU has source for bash versions 1.14.7, 2.0, 2.01, 2.01.1, 2.02, 2.03, 2.04,
2.05, 2.05a, and 2.05b, and 3.0... and patches for several intermediate versions.... If this ends up being a bash compatibility problem, this could get a little involved....!

By: Steve Murphy (murf) 2005-05-15 23:45:14

Or, it could be "head -1"... try running the command to extract the version
by hand, and see what you get, and share it with me.

By: Olle Johansson (oej) 2005-05-16 05:08:05

The vercomp.sh script adds a dependence to bash in /bin/bash. Do we want that to happen? I don't like accidentally adding new dependecies...

Also egrep -o doesn't work on my system... (FreeBSD 4). Can we create another solution that is a bit more portable to solve this without adding the requirement to install bash, and install it in /bin...

By: Michael Jerris (mikej) 2005-05-16 05:53:21

[root@phone asterisk]# bison --version | head -1
bison (GNU Bison) 1.875c

bash:  GNU bash, version 2.05b.0(1)-release (i386-redhat-linux-gnu)

By: Steve Murphy (murf) 2005-05-16 08:12:33

Perl isn't allowed. Bash isn't allowed. Some egrep opts not allowed... is that how bash does the =~. Can't code a C-program, this has to be avaiable during the make, not afterwards.

Maybe gawk will work? I really am having doubts about sh... the tricky part is pulling the version from the first line of the answer to the version request...
and breaking it into peices at the '.' -- and egrep -o would have been a natural fallback.

Any ideas on a completely portable attack?

By: Kevin P. Fleming (kpfleming) 2005-05-16 09:18:22

It should be possible to write this script using /bin/sh and POSIX-shell features (nothing bash-specific). That's a separate issue from why it's failing on some people's systems, though...

Can anyone having this complete failure problem get me SSH access to the failing box? I need to get that part of this issue fixed quickly.

By: Michael Jerris (mikej) 2005-05-16 09:32:38

Got kevin access to my dev box.

By: Kevin P. Fleming (kpfleming) 2005-05-16 09:48:05

OK, here's the problem: regular expression matching is a bash-3.0 feature, not available in bash-2.0x. I've suppressed the calls to vercomp.sh from Makefile for now, until we can decide what the best course of action is.

This _could_ be done using a small C program if necessary, as long as the dependencies are set up properly, I believe.

By: Olle Johansson (oej) 2005-05-16 11:20:58

Good ol' sed is on every system though, isn't it?

By: Kevin P. Fleming (kpfleming) 2005-05-16 11:42:27

Yes, sed is pretty safe to use, as is awk/gawk.

murf, whatever path you take, if you need help with Makefile changes to accomodate it let me know.

By: Mark Monnin (wrmem) 2005-05-16 12:35:52

FYI - Flex v2.4.31 is really new, and you can't get it from the Gnu website (it's now a sourceforge project).  Also, the version shipped with Fedora Core 3 and RHEL 3 & 4 are all 2.5.4.  

Expect a lot of end-user problems getting this as it's not very common right now.

By: Michael Jerris (mikej) 2005-05-16 12:45:25

wrmem, the beauty of this patch is that it runs fine if you do not have the newer versions, it just is better if you do.  The problems that people were having was with that version check.

By: Steve Murphy (murf) 2005-05-16 13:31:19


I've uploaded another attempt at portability. A simple C program to compare
the versions and return, as before "true" or "false" to the assertions.

A new diffu with Makefile to compile and use the program...

+ifneq ($(wildcard vercomp),vercomp)  ### if vercomp hasn't been compiled yet...
+zx=$(shell echo make vercomp)   ### uh, maybe you should take this out
+zx=$(shell gcc -o vercomp vercomp.c)  ## compile it silently, very silently
+FLEXVER_GT_2_5_31=$(shell ./vercomp.sh flex \>= 2.5.31)
+BISONVER=$(shell bison --version | grep \^bison | egrep -o '[0-9]+\.[-0-9.]+[a-z]?' )
+BISONVERGE_85=$(shell ./vercomp.sh bison \>= 1.85 )
+ifeq (${FLEXVER_GT_2_5_31},true)
+FLEXOBJS=ast_expr2.o ast_expr2f.o

As you can see, I buried the gcc for vercomp in the conditional statements
of the Makefile, so it gets done in-line with the tests and variable assignments, before the actual "making" is done. I've tested it on
bison 2.0, 1.875c, and flex 2.5.4 and 2.5.31. All seems well....

The program uses string.h and stdio.h. That's all, folks. So, the dependency is gcc (hardwired), but if you don't like that, you can tweek it to say cc, or $(CC) or whatever you think best...

Sed, and gawk, might be portable, but all I can do with sed is pull out the version, maybe then split it. Shell scripts (portably) don't have arrays. So shell is out. Gawk would be OK, I'm sure I could do it with Gawk, but... some yahoo somewhere would probably use an ancient version of awk, and I'd be dead to some degree... So, C it is, unless you really, truely, deeply, yearningly, desire to have a Gawk-script. If so, let me know. I feel like a high-school kid with a hard-to-please teacher!! Hopefully, I can pass this course!

By: Steve Murphy (murf) 2005-05-16 14:46:37

OOOps!  I edited the Makefile patch, but forgot to save it before
uploading to Mantis. Sorry. The "real" diffu is now attached.
You most likely don't want the leftovers of builtin mp3 stuff, now torn out!

By: Mark Monnin (wrmem) 2005-05-16 16:38:37

First Asterisk crash in a while.  I think this one is due to this change, since it's ast_yylex that's high in the chain.  I have a "thread apply all bt" if need be.

bison (GNU Bison) 1.875c
flex 2.5.31
on a RHEL 3 system

BTW.  I still like the new flex/bison grammer.


(gdb) bt
#0  0x0025fa8e in _int_malloc () from /lib/tls/libc.so.6
#1  0x0025ee9d in malloc () from /lib/tls/libc.so.6
#2  0x00266190 in strdup () from /lib/tls/libc.so.6
#3  0x080b501b in make_str (s=0x1 <Address 0x1 out of bounds>) at ast_expr.y:198
#4  0x080b160b in ast_yylex (lvalp=0x323188, ast_yylloc=0xb72b9450, karoto=0x0)
at ast_expr.y:395
ASTERISK-1  0x080b151e in ast_yyparse (kota=0xb72b9490) at ast_expr.c:1072
ASTERISK-2  0x080b177d in ast_expr (arg=0xb72ba4f0 " \"40513\" = \"\" ") at ast_expr.y:4
ASTERISK-3  0x08089350 in pbx_substitute_variables_helper_full (c=0x952f4a8, cp1=0x9572b
18 "$[ \"${AUTH_MAILBOX}\" = \"\" ]?goodbye", cp2=0xb72c0700 "",
   count=8191, headp=0x0) at pbx.c:1521
ASTERISK-4  0x0808033f in pbx_substitute_variables_helper (c=0xe, cp1=0xe <Address 0xe o
ut of bounds>, cp2=0xe <Address 0xe out of bounds>, count=14)
   at pbx.c:1541
ASTERISK-5  0x08089700 in pbx_extension_helper (c=0x952f4a8, con=0xe, context=0x952f5f8
"agent-login", exten=0x952f6ec "s", priority=7, label=0x0,
   callerid=0x9572c93 "GotoIf", action=156221616) at pbx.c:1617
ASTERISK-6 0x08080cc4 in ast_pbx_run (c=0x952f4a8) at pbx.c:2227
ASTERISK-7 0x0808a0ff in pbx_thread (data=0xe) at pbx.c:2476
ASTERISK-8 0x00a0edec in start_thread () from /lib/tls/libpthread.so.0
ASTERISK-9 0x002caa2a in clone () from /lib/tls/libc.so.6

Lines for extensions.conf
exten => s,n(startlogin),Playback(welcome)
exten => s,n(login),VMAuthenticate(@agents)
exten => s,n,GotoIf($[ "${AUTH_MAILBOX}" = "" ]?goodbye)
exten => s,n,Playback(welcome)

By: Steve Murphy (murf) 2005-05-16 16:48:48

Something funny about the stack trace...

3 0x080b501b in make_str (s=0x1 <Address 0x1 out of bounds>) at ast_expr.y:198
4 0x080b160b in ast_yylex (lvalp=0x323188, ast_yylloc=0xb72b9450, karoto=0x0)
at ast_expr.y:395
0000005 0x080b151e in ast_yyparse (kota=0xb72b9490) at ast_expr.c:1072
0000006 0x080b177d in ast_expr (arg=0xb72ba4f0 " \"40513\" = \"\" ") at ast_expr.y:4

It's mentioning variables and files from the old hand-written parser, ast_expr.y, and ast_expr.c; I'd expect to see the ast_expr2.y, and ast_expr2.fl mentioned instead, if the new guys actually got linked in.

Go to your asterisk top level dir, and remove the ast_expr*.o files,
and rebuild. There should be some messages to let you see what's happening during the build.

One thing I notice about the make, is that if previous stuff

By: Mark Monnin (wrmem) 2005-05-16 17:18:29

Ok. It's not the new code.  Did a removed the ast_expr*.o and forced a "make clean."  Still old code and still crashed.  

Did some digging.  Makefile is still commenting out the test for flex/bison version.  So, "just for fun", hardcoded vercomp.sh to return "true" to test force new code, to see if it behaved differently.  Still crashes at (what I think is) the same dialplan line.   So, I'm thinking it maybe NOT related to the changes in the parser.   Odd though.

(gdb) where
#0  0x00206c77 in memset () from /lib/tls/libc.so.6
#1  0xb72b64f8 in ?? ()
#2  0x001fe53c in calloc () from /lib/tls/libc.so.6
#3  0x080b6514 in ast_yylex (yylval_param=0xb72b431c, yylloc_param=0xb72b5460,
   yyscanner=0x979bf88) at ast_expr2.fl:70
#4  0x080b1e1e in ast_yyparse (parseio=0x978b7d0) at ast_expr2.c:1120
ASTERISK-1  0x080b71fb in ast_expr (arg=0xb72b64f0 " \"40513\"") at ast_expr2.fl:105
ASTERISK-2  0x08089990 in pbx_substitute_variables_helper_full (c=0x0, cp1=0x0,
   cp2=0x0, count=0, headp=0x0) at pbx.c:1521
ASTERISK-3  0x00000000 in ?? ()

# /usr/sbin/asterisk -V
Asterisk CVS-HEAD-05/16/05-17:00:14

By: Steve Murphy (murf) 2005-05-16 17:48:24

Here's something that may or may not help.

Do a "make testexpr2"

and then, if there's any way on earth at all you can determine which
expr it's trying to eval, via gdb, or whatever, then you can run
the testexpr2 program as follows:

./testexpr2 "expr"

where "expr" is the expr you want to test. It will generate a parse error on the command line, etc... you might grep you extensions.conf for the $[ to see all the possible candidates.

By: Mark Monnin (wrmem) 2005-05-16 17:55:44

Ok.  More detective work.  It's not your problem at all.  Sorry for that.  It's the app_voicemail above it.  

If I do a "cvs update -r1.211 apps/app_voicemail.c" and recompile, it works, I I do a "cvs update -r1.212 apps/app_voicemail.c" it crashes.  Looks like it's clobbering things it shouldn't and then messes up the next stuff.  Off to do some more debugging on it.  

Sorry for the false alarm on this one.

By: Steve Murphy (murf) 2005-05-16 18:41:28

Whew! I was sweating...

but still, ...

I really hate it when I mess up, and I did it again. I've attached an even "more real" Makefile patch. (see *.realreal) Seems in all my eagerness to get the thing out, I forgot to turn the "vercomp.sh" commands into "vercomp" commands.

And, I see that the egrep -o that oej said didn't work on his system, wasn't just some theory. I was actually using it! Duh! So, I changed that to use sed instead. Sorry. Some days I can go HOURS, being perfect. Today, tho, for some reason, it's down to minutes.

Now, it's:

+FLEXVER_GT_2_5_31=$(shell ./vercomp flex \>= 2.5.31)
+BISONVER=$(shell bison --version | grep \^bison | sed 's/.* \([0-9]\+\.[-0-9.]\+[a-z]\?\)/\1/' )
+BISONVERGE_85=$(shell ./vercomp bison \>= 1.85 )

BISONVER is generated purely for aesthetics, it's output only appearing in
an echo statement.

OK. Now, everybody, give it a spin. See if it works.

By: Michael Jerris (mikej) 2005-05-18 22:08:05

files this time around are: ast_expr2.fl, ast_expr2.y, README.variables.diffu, vercomp.c, Makefile.diffu-vercomp-c-exec-realreal

By: Steve Murphy (murf) 2005-05-18 23:07:10

The list looks good to me.

By: Kevin P. Fleming (kpfleming) 2005-05-19 00:00:06

I'm working on this one right now... I've implemented the Makefile changes in a different way, that allows make to still use its implicit rules to build the vercomp and ast_expr.o targets (and also ensures that the 'host' compiler is used to build vercomp if a cross compiler is being used for the rest of Asterisk).

I'm nearly done testing it, so it will be committed shortly.

Oh, also I've added a copyright/license notice to the top of vercomp.c, but I don't have your proper name and email address... if you want the header to have those details please let me know.

By: Kevin P. Fleming (kpfleming) 2005-05-19 00:03:09

Committed to CVS HEAD, thanks!

By: Steve Murphy (murf) 2005-05-19 09:45:36

Uh, OK. "murf" in the copyright field all by itself is pretty ugly...

If you are going to assign it to me, use "Steven Michael Murphy", with email addr murf at e-tools dot com, in machine readable format, of course. (Avoiding mailbots here).

If that's a bother, assign the copyright to Digium, FSF, or whatever's best.

By: Kevin P. Fleming (kpfleming) 2005-05-19 10:49:39

Updated the copyright header... and also changed the Makefile a bit more, we were having some shell portability problems on FreeBSD.

By: Tilghman Lesher (tilghman) 2005-06-07 11:36:11

I'm reopening this bug, as it relates to bug ASTERISK-4362 -- apparently there's a bug in this new expression parser, as it relates to string comparisons.

By: Steve Murphy (murf) 2005-06-07 22:15:39

Anything I can do to help?

By: Michael Jerris (mikej) 2005-06-08 07:58:36

murf is on this new issue in the 4478 bug.  No need to have 2 bugs open for the same issue.

By: Digium Subversion (svnbot) 2008-01-15 15:02:34.000-0600

Repository: asterisk
Revision: 3444

U   trunk/redhat/asterisk.spec

r3444 | markster | 2008-01-15 15:02:34 -0600 (Tue, 15 Jan 2008) | 2 lines

Put header files in -devel package (bug ASTERISK-2030)



By: Digium Subversion (svnbot) 2008-01-15 15:34:57.000-0600

Repository: asterisk
Revision: 5691

U   trunk/Makefile
A   trunk/ast_expr2.fl
A   trunk/ast_expr2.y
U   trunk/doc/README.variables
A   trunk/vercomp.sh

r5691 | kpfleming | 2008-01-15 15:34:56 -0600 (Tue, 15 Jan 2008) | 2 lines

add upgraded expression parser (bug ASTERISK-2030)



By: Digium Subversion (svnbot) 2008-01-15 15:35:32.000-0600

Repository: asterisk
Revision: 5731

_U  trunk/
U   trunk/.cvsignore
U   trunk/Makefile
A   trunk/vercomp.c
D   trunk/vercomp.sh

r5731 | kpfleming | 2008-01-15 15:35:31 -0600 (Tue, 15 Jan 2008) | 2 lines

use new C-coded version comparison program for bison and flex (bug ASTERISK-2030, with different Makefile changes)