| Summary: | ASTERISK-01740: [patch] Modularized external query API modules | ||
| Reporter: | Rob Gagnon (rgagnon) | Labels: | |
| Date Opened: | 2004-06-01 17:40:09 | Date Closed: | 2011-06-07 14:04:46 |
| Priority: | Major | Regression? | No |
| Status: | Closed/Complete | Components: | Core/General |
| Versions: | Frequency of Occurrence | ||
| Related Issues: | |||
| Environment: | Attachments: | ( 0) ast_data_link.html ( 1) asterisk-lt.spec ( 2) data_odbc.c.patch.txt ( 3) extensions_pg.sql ( 4) friends_postgres.sql | |
| Description: | The formal public introduction of "ast_data"!! The code in the attached tarball removes all SQL code from app_voicemail.c, chan_sip.c, chan_iax2.c, and adds the ability to do extension lookups from an external module to pbx.c. I have attached the first fully functional ast_data.tar.gz file here. Major updates will be posted here as needed, but the live changing version is available at http://svn.asteriskdocs.org/res_data/ You can also see the contents of individual files within the tarball by going to the URL above, as well as the project's changelog. ****** ADDITIONAL INFORMATION ****** Currently MySQL, PostGreSQL, and ODBC database modules are included in the tarball. The .conf file should be helpful as to how to configure things. Database connection pooling is in each data_xxx.c module to help avoid locking contentions with single-threaded database engines. Modules for RADIUS, LDAP, etc. could be written by following the example data_xxx.c modules. When installed, if you delete the /etc/asterisk/data.conf file, Asterisk will create the appropriate file for you based on information in your sip.conf, voicemail.conf, and iax.conf files if it needs to. There are CLI commands as well: show data mysql status show data odbc status show data pgsql status show data handlers data debug data no debug All .patch.txt files inside the tarball were up-to-date with CVS as of the time of this post. Again, if you want the most recent version, you can obtain it from the URL above. | ||
| Comments: | By: Filip Olsson (folsson) 2004-06-02 20:08:39 Nice going with the odbc and pgsql backends! A number of problems still exists IMHO: This isn't as flexible as flat files concerning includes and that kind of stuff, especially when handling include =>'s extensions.conf We'll just have to face the fact that no two users with a large number of extensions, sip/iax-peers, vm-users will want to have the same DB-structure. I propose we make the queries configurable. The number of variables going into the queries is always the same so it shouldn't be that hard implementing it. Concerning the field-names we expect to get from the query one could always select a field AS the-name-we-expect. Stored procedures... By: Brian West (bkw918) 2004-06-02 23:31:17 Also code duplication is bad. bkw By: Rob Gagnon (rgagnon) 2004-06-03 03:51:21 folsson.... Always on the configurable queries kick huh? (inside joke for others).... I am working on that for next release/upgrade of this initial ast_data. bkw: Code duplication? Some is necessary as they are stand-alone modules. Also, the "poolentry" structs in each are not the same, so the code looks similar, the stuff refering to poolentries cannot be un-duplicated since the definitions in each module are different. By: nix (nix) 2004-06-05 08:13:07 What is this rubbish in the Makefile? My locate command (SuSE 9.1) does not have a -n1 option.. Can we come up with a slightly more portable way of doing this? PGSQLCLIENT=$(shell echo `locate -n1 libpq.so`) By: nix (nix) 2004-06-06 17:54:15 Description: This system allows centralisation of all authentication into one "friends" table. It includes VIEWs for each different channel type so that this system can be used directly with Asterisk with no code modification (Currently ast_data does need to be patched in, of course..) I use Postgres RULEs to make sure UPDATEs to the VIEWs get to the correct table I have also added some extra columns to support app_prepaid directly from the same table with no fuss nor muss. Each channel and calling card can be independently disabled. In future I plan on using these RULEs to also keep and audit trail of account changes, but that will have to wait until another day to be implimented. If anyone has any extra features they can think of, I would be happy to impliment them also. By: nix (nix) 2004-06-06 18:01:57 folsson: I can see no pressing reason to make the queries configurable, as all this stuff can/should be handled inside VIEWs and stored procedures. The exception to this is of course braindead DBs like mysql, as well as LDAP queries (should someone get around to writing a LDAP backend). A RADIUS backend also would not need configurable queries as you handle that stuff on the RADIUS server itself (Thats the whole point of using RADIUS to abstract the network stuff from the actual DB/Flatfile that holds the info). The postgres schema that I have posted above can be pushed/pulled/squeezed to do as many different things as you need before replying to Asterisk in the format that asterisk expects. It is currently quite simple but I will add extra functionality to is as time progresses. By: nix (nix) 2004-06-06 18:39:25 Hmm... Extensions generated on the fly from stored procedures just occured to me... This could be really fun... By: Rob Gagnon (rgagnon) 2004-06-07 10:19:14 nix: Thanks for the modification to the sample data.conf file. I've added that to the SVN version. Also, I made some changes to the extensions.sql file as we had some MySQL-only SQL in there. It should be a little more generic now. However, I've found with some DB's that using a SERIAL for a primary key can lead to problems during updates. I guess PostGres doesn't have that problem? I do think that configurable queries from the data.conf file is a more generic way to go for people rather than having a multitude of .SQL files for them to pick from to create their own database. The code ast_data is built around uses the existing .SQL definitions that were already in the repository, in order to support as many existing users as possible. By: nix (nix) 2004-06-07 10:57:20 I have not had any problems with SERIAL as a primary key (thats what its designed for..) and I have some tables with tens of millions of rows in the multigigabyte range.. In my opinion, the existing non-centralised design of separate sip and iax friends tables is wrong, however it is not an issue for me any more thanks to the postgres support in ast_data. I know that some others do not agree with me, which is why I did not bother to submit a patch to the C code to change this.. Backward compatibility is a good thing(tm). I do however think that having a decent centralised example schema in svn/cvs (as an addition to a default one) is a good idea as I know I am not the only person to want this. Have only one sample schema for Postgres and making it the same as the MySQL schema is akin to ripping out the seatbelts, airbags, airconditioning and windscreen from a BMW just because you think it should feel the same as the Suzuki motorbike you learnt to ride on when you were a kid... edited on: 06-07-04 10:43 By: nix (nix) 2004-06-07 12:19:26 It appears that the iaxfriends code is not backwards compatible in any case, because it requires the "type" field which does not exist in asterisk cvs.. "SELECT * FROM iaxfriends WHERE name='%s'" " AND (type='%d' OR type='%d');", By: Rob Gagnon (rgagnon) 2004-06-07 12:41:12 OK. Patch coming for the iax-friends.sql file in a short while. By: Rob Gagnon (rgagnon) 2004-06-07 13:32:54 Link to current location of source code added to replace actual ast_data.tar.gz file... This should save some bandwidth for digium, and provide an up-to-date source for everyone By: Matthew Nicholson (mnicholson) 2004-06-09 11:37:55 When the database is changed, does asterisk automatically update it's config? Or does it check the db each time it needs a config option? By: nix (nix) 2004-06-09 15:59:37 Asterisk re-checks the DB for every connection, there is no need for a reload. By: Matthew Nicholson (mnicholson) 2004-06-10 17:49:24 I may be able to add a microsoft sql backend for this, but I'll have to look at the code some more. By: nix (nix) 2004-06-10 18:00:20 It should be fairly trivial to use the odbc driver to access MSSQL. Others are already doing it with cdr_odbc. By: nix (nix) 2004-06-10 18:22:58 I would like to see ast_data go into mainline CVS as soon as possible. It has been stable for me so far, and certainly adds much needed functionality. By: Rob Gagnon (rgagnon) 2004-06-10 22:15:27 Excellent. Thanks for the comments Nix, and all. Also, thanks for the wider testing base. By: exomorph (exomorph) 2004-06-11 12:07:21 Thanks Nix for the progress report. We need a few more like this (good or bad) so that we can add this to cvs-head.. By: srathje (srathje) 2004-06-15 10:05:44 AST_DATA version 143 with "Asterisk CVS-HEAD-06/14/04-23:04:06" resolved my problem with segfaulting thus enabling me to dig deeper.. :-) As before, I only have "VoiceMail = pgsql" defined in data.conf. Pressing "0" while leaving voicemail I can jump to "exten => o,1,[whatever]". I have one in extensions.conf that will catch this event and it works if I disable ast_data. 16-6-04 (srathje) Eh.. Withdrawn for now, watch this space for a more comprehensive test. I realised that I need to do a bit more testing to narrow this thing down.. edited on: 06-16-04 07:50 By: cypromis (cypromis) 2004-06-17 04:17:26 hmmm mostly works perfectly fine but, how do I include a context into aanother context ? doing it in extensions.conf want help so there must be a way to do it via data_pgsql.so. Also it seems that there is a bug which keeps the module from remembering iax2 registrations at least the regseconds field stays empty. Also MWI on SIP phones is not working. So much for now, more later. And ............... thanks for this fantastic module :) By: Rob Gagnon (rgagnon) 2004-06-17 14:05:03 Odd... I use the extensions.conf file for the includes, and just leave the actual extensions in the database. Will have to review. One note though, I have a couple fixed extensions in my context in extensions.conf. Maybe thats the difference. Will have to see what happens between having: [context] include => local and... [context] include => local exten => 999,1, VoicemailMain(@context) Maybe having a single extension under the context makes a difference as to whether the include has an effect or not By: cypromis (cypromis) 2004-06-18 10:07:57 ok I have it like after a couple of goto's :) it lands in a context that includes 13 - 14 other contexts it searches only through the extensions in that context nbever checks the ones from the other contexts Basically the included contexts are all in the db. he contexts that include the other contexts are in extensions.conf But asterisk simply prints on load a warning that the incldued contexts do not exist and ignores them if I put them into extensions.conf with some stupid entry, it still does not read the rest of the entries from the db. edited on: 06-18-04 10:42 By: cypromis (cypromis) 2004-06-18 11:07:21 rgagnon: I tried both of them By: Terry Wilson (twilson) 2004-06-30 10:29:09 Is this still under active development? The link no longer works, and no one has posted anything here in nearly two weeks. Was just wanting to check out the code, and couldn't get to it. Link responds with "Could not open the requested SVN filesystem" By: exomorph (exomorph) 2004-06-30 10:39:22 Yes it is still in active development. I'm having a problem with my server, and haven't had a chance to fix it... It will be fixed today tho. Thanks, --Greg By: Rob Gagnon (rgagnon) 2004-06-30 10:43:29 Yes, development is highly active. The link was down for a bit while some server issues were being worked out. The site is back online Thanks to Exomorph and a "chmod" command :-) edited on: 06-30-04 13:37 By: Rob Gagnon (rgagnon) 2004-06-30 13:53:34 Reminder sent to twilson svn.asteriskdocs.org is back online if you wish to read up some more By: rwjblue (rwjblue) 2004-06-30 14:03:03 Working flawlessly! I am using it against CVS-HEAD 2004-06-25. This is making my life much easier. I can't wait to get this into cvs. rwjblue By: Rob Gagnon (rgagnon) 2004-06-30 16:29:34 NEW FILE: app_directory.c.patch.txt check the link above for latest files. app_directory is now ast_data enabled with this patch By: rwjblue (rwjblue) 2004-07-01 09:17:01 No problems with app_directory. Working well against cvs 2004-06-30. Thanks rgagnon. By: Rob Gagnon (rgagnon) 2004-07-03 12:17:40 Thanks, rwjblue, for the comments, and the testing. News from the front: There have been some heavy modifications to all of the data_xxx.c modules as preparations for configurable queries in data.conf are being programmed. You should see #define DEFAULT_SQL_SELECT_xxxxxx statements near the top of each data_xxx.c module now. You will note that I am not using the %s, %d, etc. snprintf() formatting characters for placement of the variables, but am using more script-like "${Context}" placeholders in the queries, which allow your SQL's to not require ordered positional formatting paramters. Soon those DEFAULT_SQL_xxxx definitions will be replaced with variables read from data.conf. Keep your eyes open for those maybe by July 9th, 2004, or sooner I hope. I am also contemplating combining data_mysql.c, data_odbc.c, and data_pgsql.c into one module with #define statements as needed to branch during compile time to produce the module you need----I am running into a lot of triple modifications due to the similar-but-different code in each module. By: Terry Wilson (twilson) 2004-07-05 17:34:37 Since you are handling the database backend stuff for voicemail, it would be nice if 'show voicemail users' from the CLI worked for voicemailbox configs stored in the db back-end... I know you all have more pressing issues, but I figured since you were patching app_voicemail.c anyway... :-) I was looking at the patch for the sipfriends stuff, and it looks like it could get unwieldy having to add a column for every option you wanted to add. Would it make more since to follow a setup closer to what the retrieve_sip_conf_from_mysql.pl script does? i.e. have a table that contains a keyword/data combo so that you could store any current or future options in the database w/o having to modify the database structure? Just a thought. Anyway, testing has gone great using res_config alongside ast_data. I'm still using res_config for extensions.conf/sip.conf (I have a lot of data to migrate) and using ast_data for voicemail/directory. No problems so far. By: Terry Wilson (twilson) 2004-07-06 03:17:58 Don't know "for sure" that this is a bug with ast_data, but did notice that there was some modification to pbx.c's pbx_extension_helper function. I got two segfaults after moving a production server to ast_data to really test it out(also upgraded CVS version of *, hence the uncertainty). Anyway, here's the bt full if it helps anyone. (gdb) #0 0x0807a364 in pbx_extension_helper (c=0xbe948bb0, context=0xbe948d08 "company", exten=0x6b206c6c <Address 0x6b206c6c out of bounds>, priority=1, callerid=0x20617461 <Address 0x20617461 out of bounds>, action=-1) at pbx.c:798 e = (struct ast_exten *) 0x9727750 sw = (struct ast_switch *) 0x0 data = 0x0 newstack = 1 res = -1 status = 4 incstack = {0x9425c60 "", 0x807ac1c "�\215v", 0x43b72000 <Address 0x43b72000 out of bounds>, 0x1 <Address 0x1 out of bounds>, 0xfc2a5c "X�", 0x5 <Address 0x5 out of bounds>, 0x43b71954 "8\032�C", 0x43b7192c "|F\a\b�\213\224�\b\215\224��\215\224�\001", 0xc15591 "\203� \205��\002", 0x1 <Address 0x1 out of bounds>, 0x1a <Address 0x1a out of bounds>, 0x43b71ac4 "\\*�", 0x0, 0x43b71dd0 "�\032�C", 0x43b71954 "8\032�C", 0x43b71ac4 "\\*�", 0x2 <Address 0x2 out of bounds>, 0xffffffe4 <Address 0xffffffe4 out of bounds>, 0x43b71864 "", 0x43b71ac4 "\\* 0xfbe420 "U\211�\203�$e�\b", 0x0, 0x43b718cf "6751", 0x8082863 "\205�t\032F\017�\f>\204�\210�u�\213]��\004;", 0x80cfba7 "( )-.", 0x31 <Address 0x31 out of bounds>, 0x4 <Address 0x4 out of bounds>, 0x9526c98 "1089097042.20", 0x9526c98 "1089097042.20", 0xbe948bb0 "SIP/6751-ce6c", 0x43b71928 "\210\032�C|F\a\b�\213\224�\b\215\224��\215\224�\001", 0x8085356 "\213\233�\003", 0x9526c98 "1089097042.20", 0xbe94a364 "1089097042.20", 0x1f <Address 0x1f out of bounds>, 0x9081b4 "�\034c\t", 0x9081b8 "", 0x907898 "�G\023", 0x908180 "", 0x1 <Address 0x1 out of bounds>, 0x43b718ec "", 0x844949 "\213E�\203�H\211E�\213}�\213M�\213WT\211U�9�\017\204\221", 0x43b718c1 "Terry Wilson", 0x43b718cf "6751", 0x72655400 <Address 0x72655400 out of bounds>, 0x57207972 <Address 0x57207972 out of bounds>, 0x6f736c69 <Address 0x6f736c69 out of bounds>, 0x3600006e <Address 0x3600006e out of bounds>, 0x313537 "", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8085c87 "\213\233�\003", 0x9526c58 "RM�@�:\017", 0x0, 0x0, 0x0} passdata = "SIP/6752", '\0' <repeats 8183 times> stacklen = 1 tmp = "\033[1;36;40mDial\033[0;37;40m\000\000;37;40m\000l 6 01:10:10", '\0' <repeats 17 times>, "\004\000\000\0008\000\000\000\001\000\000\000\006\000\000" tmp2 = "\033[1;35;40mSIP/6751-ce6c\033[0;37;40m\0007;40m", '\0' <repeats 25 times>, "Jul 6 01:56:04" tmp3 = "\033[1;35;40mSIP/6752\033[0;37;40m\000uvio\033[0;37;40m\000;40m", '\0' <repeats 2364 times>, ",\017\212\000\230x\220\000D\001\204\000��\203\000\000\020\000\000\230x\220\000�|q\t\204��C\033$\203\000�|q\t\000 /�\0000/�\001\000\000\000����\000\000\000\000\002\b\000\000\000\000\000\000\000\000\000\000{\200\003\000�\201\000\000\001", '\0' <repeats 23 times>, "O\000\000\000\000\000\000\000\000\020\000\000\b\000\000\000\000\000\000\000tB�@\000\000\000\000�\207V@\000\000\000\000�\207V@\000\000\000\000{\200\003\000\000\000\000"... #1 0x0807467c in ast_pbx_run (c=0xbe948bb0) at pbx.c:1836 digit = 0 '\0' exten = '\0' <repeats 255 times> pos = 0 waittime = 1136073072 res = 0 #2 0x0807ac11 in pbx_thread (data=0xffffffff) at pbx.c:2061 No locals. #3 0x00fb9dfc in start_thread () from /lib/tls/libpthread.so.0 No symbol table info available. #4 0x008aef2a in clone () from /lib/tls/libc.so.6 No symbol table info available. By: Rob Gagnon (rgagnon) 2004-07-06 11:15:54 twilson, not sure what, if anything is related exactly at that location. From your BT, line 798 should be "(*e)->datad((*e)->data);" with pbx.c v1.135 patched with ast_data. This puts you in the middle of the "free_dynamic_ext()" function, which I do not see in your stack trace. With an un-patched v1.135 of pbx.c, line 798 is in the "pbx_substitute_variables_temp()" function. I would suggest the following: cd /usr/src/asterisk rm pbx.c cvs update pbx.c patch -p0 </what/ever/path/pbx.c.patch.txt with the current patch from ast_data for pbx.c. Also, you might want to see if the bug exists before patching as well. As well, any other information needed to replicate the issue. Thanks By: Rob Gagnon (rgagnon) 2004-07-06 22:30:40 Large code change completed today. Normally I don't post every little Subversion commit notification to this page, but I want to make sure that anyone using ast_data is aware that some new code comitted just a few minutes ago contains large changes to major parts of ast_data. So as not to repeat what the changes are, please see the changelog.txt file here: http://svn.asteriskdocs.org/res_data/changelog.txt By: Terry Wilson (twilson) 2004-07-10 19:33:53 With CVS from 7/10 and ast_data rev 196, the first few voicemail email notifications I get show a normal from field (specified in voicemail.conf as serveremail=voicemail@company.com), but after a while the from field switches to "random ascii chars"@machine-name.company.com. Occasionally, the right name comes back up for an email or two, then switches back. I have reproduced this several times on two different servers with same cvs and ast_data code. I have tried just using a static voicemail.conf entry with same cvs, but no ast_data, and cannot reproduce the problem. I am using the mysql for voicemail with ast_data. By: Rob Gagnon (rgagnon) 2004-07-11 01:58:51 OK. A couple questions: 1) Are you speaking about the global serveremail in the voicemail.conf file, or the serveremail=xxx option in the "options" field of the database? 2) Can you show an example of the "From: " string, preferably, from the source code of the email, it should look like either "From: Asterisk Pbx <email@domain.com>" or "From: Custom Name <email@domain.com>" 3) can you copy a snippet from the asterisk CLI logs while this is happening and post it. Please use at least verbose level 4. (ie: asterisk -rvvvv) The messages in the log around where you have a problem email may help. By: Terry Wilson (twilson) 2004-07-12 11:56:26 I will have to set up my test boxes again with ast_data so I can get you the info about the voicemail issue. I took ast_data out again because I still (with a complete fresh checkout from CVS and the rev 196 was still getting the core dump that I posted above (although in line 800--same line dying). It appears that pbx_extension_helper calls free_dynamic_exten and it dies on the "(*e)->datad((*e)->data);" line. Which I'm assuming would mean that there is something wrong with the checks being done before it is called, but my C is VERY rusty... Is anyone else using ast_data with res_config? I am only using it for VM/Directory, while using res_config for extensions/sip. By: Terry Wilson (twilson) 2004-07-12 12:00:51 Here is a copy of the from line: From: Voicemail System <`?x?.??@server.company.com> I am talking about the global serveremail line which has always been set to: serveremail=voicemail@company.com By: Rob Gagnon (rgagnon) 2004-07-12 17:25:55 Thanks, twilson, that will help a bit. One thing I forgot to ask, is if you can also get the output with "data debug on" when this happens. That will show where the funky characters are originating hopefully. By: twisted (twisted) 2004-07-23 20:49:16 Yee haw! I love me some ast_data, but i'm gonna have to prod at it a bit here just to (hopefully) get interest going agian. -housekeeping By: Rob Gagnon (rgagnon) 2004-07-23 23:31:33 No worries... code changes are on-going right now to get fully configurable queries into the modules. The data_pgsql.c module is almost fully complete on my development side. I will be porting those changes into the data_odbc and data_mysql modules soon, and then they will be committed. The notes here stay kinda quiet, but if you check the changelog for the project periodically at: http://svn.asteriskdocs.org/res_data/changelog.txt you will see what is going on in it. By: Rob Gagnon (rgagnon) 2004-07-26 15:30:44 Just finished up adding the ability to configure the SQL statements with variables in all data_xxx.c modules. Documentation on exactly how to use them is coming in the next few days, but for now, please see the changelog at: http://svn.asteriskdocs.org/res_data/changelog.txt By: kbauer (kbauer) 2004-08-18 02:41:58 Hi, I hope this is the right place: When I try to compile * with ast_data on FreeBSD 5.2.1 I get the following error: gcc -g -o asterisk -Wl,-E io.o sched.o logger.o frame.o loader.o config.o channel.o translate.o file.o say.o pbx.o cli.o md5.o term.o ulaw.o alaw.o callerid.o fskmodem.o image.o app.o cdr.o tdd.o acl.o rtp.o manager.o asterisk.o ast_expr.o dsp.o chanvars.o indications.o autoservice.o db.o privacy.o astmm.o enum.o srv.o dns.o aescrypt.o aestab.o aeskey.o data.o utils.o editline/libedit.a db1-ast/libdb1.a stdtime/libtime.a -lc_r -lncurses -lm -lcrypto -lssl data.o: In function `ast_data_make_reservations': /opt/src/asterisk/asterisk/data.c:889: undefined reference to `strdupa' gmake: *** [asterisk] Error 1 Now I googled around alot and found a hint, that seems to work, although I haven't checked if everything works fine: Someone ( I can't remember his name) has already made a patch for * and 'strdupa' called 'ast_strdupa' in '<asterisk/channel.h>'. Now if I include this file in 'data.c' and change 'strdupa' in 'ast_strdupa' it compiles just fine (tested just today with latest versions). The question now is, is this OK, should this work or am I breaking smth. I'm not a C-programmer, so I don't have the know-how to check if the code is feasible. If this is a working patch, it maybe could be included with the obligate 'if FreeBSD ...' Thanks a lot, best regards, Kurt By: kbauer (kbauer) 2004-08-18 04:20:14 Another issue regarding ast_data and FreeBSD -> the Makefile in data-subdirectory need some minor changes: 1. 'locate -n1' should be 'locate -l1' 2. env: should be '/usr/bin/env' br, Kurt By: nix (nix) 2004-08-18 05:18:05 Both 'locate -n1' AND 'locate -l1' do NOT work on any version of SuSE Linux up to and including SuSE 9.1 (The latest version). I suggest we come up with some more portable way of doing this... > cat /etc/SuSE-release SuSE Linux 9.1 (i586) VERSION = 9.1 > rpm -q findutils-locate findutils-locate-4.1.7-860 By: Rob Gagnon (rgagnon) 2004-08-18 10:55:44 kbauer: Thanks for the find on BSD. I have taken that, and added it to the codebase for ast_data. It appears anyone that has needed a portable strdupa() function has done exactly that in other parts of the code. Re: the Makefile: Thats just something I got lazy on when I first created it, but I do need to make it smarter. By: Rob Gagnon (rgagnon) 2004-08-18 12:13:36 I have changed the Makefile for the "data/" directory to eliminate the use of "locate" I have tested under Fedora Core 1 just fine, and borrowed the Makefile examples for the BSD stuff. It will be helpful if someone can test it under BSD :-) Anyone using Darwin? I did not set the directory locations up for that at all at this time. New Makefile: http://svn.asteriskdocs.org/res_data/ast_data/data_Makefile By: kbauer (kbauer) 2004-08-19 02:34:40 Hi, I tested it with revision 228 and it works like a charm :-)) on FreebSD 5.2.1 The only thing that probably is still breaking things, is that '/bin/env' is not valid in freeBSD, as it has to be 'usr/bin/env' nevertheless, a _BIG_ thankyou for your work and quick response. By: Rob Gagnon (rgagnon) 2004-08-19 09:22:55 I'm actually not sure if the env: tag is needed in that Makefile, but I used another Makefile as example to get the one for this project. I notice now that the other Makefiles in the asterisk repository just use "env" without the full path, so I modified the Makefile for ast_data to match. That should be the last thing there to help you. By: Rob Gagnon (rgagnon) 2004-08-30 11:11:59 Just an update. Rev 232 was just comitted and includes code to support the addition of the new "extraparams" item in the sip_peer struct. This affected all data_xxx.c modules, the data.c and data.h files were also modified slightly as well to support a new attribute to handle extraparams. Have fun! By: Terry Wilson (twilson) 2004-08-30 16:34:22 Is the end goal going to be to include *complete* support for all SIP user/peer options (qualify, etc.)? Are there any plans for implementing a scheme so that all extensions information is made available dynamically through the db (i.e. includes, etc.)? Just curious to see exactly where the project is heading. Thanks! By: Rob Gagnon (rgagnon) 2004-09-30 12:24:38 Recent news: To keep a version of ast_data in-line with asterisk release 1.0, I have created ast_data-1.0.0.tar.gz on the main ast_data site. The above tarball file will remain unchanged as opposed to the rest of the SVN repository which remain in active development. By: Rob Gagnon (rgagnon) 2004-09-30 12:25:26 Also created ast_data-1.0.1.tar.gz today to go with the new Asterisk Stable release By: Brian West (bkw918) 2004-10-08 19:05:53 Mark wants to know if you want to give input on the new realtime stuff that he and anthm are working on? OEJ opened a bug on it http://bugs.digium.com/bug_view_page.php?bug_id=0002606 Post comments if you can. Thanks, Brian By: Rob Gagnon (rgagnon) 2004-10-11 09:38:17 OK. I have obviously noticed the project (as in the conflicts it created with ast_data in CVS :-) ). I have checked it out a bit. When I get more time, I would like to look into it more in depth. By: Mark Spencer (markster) 2004-10-16 15:52:23 Any comments you can supply on the new realtime engine? By: John Corde (corde) 2004-10-19 11:24:14 found a minor bug in data_odbc.c (from ast_data-1.0.1.tar.gz). It is a missing comma in DEFAULT_SQL_UPDATE_SIP_FRIEND. Attached the patch data_odbc.c.patch.txt By: nix (nix) 2004-10-20 15:33:35 We are using this system with postgresql in production on a number of servers. ALL data on our servers including extensions is drawn from and submitted too Postgres stored procedures, and ast_data has stood up to the job admirably. Good work everyone! When can we expect to see this in mainline cvs? By: Rob Gagnon (rgagnon) 2004-10-20 15:53:02 corde: Thanks for the bug fix. That is now part of the mainline ast_data code and the 1.0.1.tarball nix: Thanks for the support. Glad things are working for you. With the new realtime projects going, it could be the way things will go with ast_data. Currently the realtime queries support Voicemail, IAX, and SIP. ast_data still provides realtime lookups for extensions, and the voicemail directory which realtime() does not do just yet. By: cypromis (cypromis) 2004-11-02 06:01:51.000-0600 ast_data seems to bugger the dundi stuff a bit in 1.0.2 and for example the map interaction stops working. Any hints ? My c is heavily rusted ... By: drumpie (drumpie) 2004-11-10 10:45:37.000-0600 I guess we found some bugs: -If we use ast_data (with asterisk 1.0.2), voicemails sent to email doesn't have the message attached. Without ast_data, it works perfectly fine. -"The "language" key in sip.conf/[general] isn't working when using ast_data. Language setting always defaults to "language=en" (we would like to have language=nl ;)) -Feature request: Add per-user "language" setting to database, or even better add all settings. By: Rob Gagnon (rgagnon) 2004-11-16 11:32:55.000-0600 Code changed to support addition of 'language' field as requested. I should finally get to the include feature soon. By: j (j) 2004-11-24 12:22:09.000-0600 Ast Data Rocks rob! Couple things: revision 277 doesn't seem to patch correctly with cvs -r 1-0-2 nor the 1.0.2 tar file from asterisk. It has several patch failures. Isn't it possible to consolidate the iax and sip account tables into one? They share the same fields with minor exceptions that can easily be filtered out using appropriate select commands. This would make administration and high-end development MUCH easier, at least from my point of view. Is ast_data planned to be incorporated into production release in the near future? It definitely should! If I can help out in any way, please let me know. Thanks! j By: Mark Spencer (markster) 2004-11-24 12:24:01.000-0600 I did add support for extensions and hopefully directory will be possible soon too. Do you have any more comments regarding the "realtime" functionality that is in Asterisk now? By: nix (nix) 2004-11-24 12:48:08.000-0600 j: Please take a look at "friends_postgres.sql" attached to this bug. It does EXAACTLY what you ask for! It may need updated to match extra feilds that have been added recently though. I will take a look at this tomorrow. We are using it sucessfully in production on a number of "high end" servers. By: Rob Gagnon (rgagnon) 2004-11-24 13:11:41.000-0600 j: Thanks for the words of encouragement. Funny thing is I had just updated the source for ast_data prior to your comments, but had not committed them to the SVN repository yet. They should apply clean now. Also j, You can change the SQL statements in the data.conf file if you want to, so that your IAX users would be selected from the same table as your SIP users. The reason this is not like that by default, is so that people CAN run systems where you are allowed to login as a SIP user, but NOT as an IAX user, or vice-versa. Mark: I have not had time to fully review the extensions/realtime stuff yet, but I would imagine it would work OK, or you would not commit it right? :-) By: j (j) 2004-11-24 13:13:04.000-0600 Oh ok great. problem is I'm using mysql, which I should have mentioned. :( What I meant by that was less of a sql hack like using a view, and more a C coding revision... a change the #definitions which define the sql used in the C code to call up the sql data could be updated instead of using views which is obviously postgres only .. Are you able to patch 277 against the latest asterisk version as well? Thanks for the quick response too :) j By: Rob Gagnon (rgagnon) 2004-11-24 13:21:28.000-0600 we are now at v278 which should patch clean as of this message time. You may note that the #define'd SQL's have the word "DEFAULT" in them. Every one of them may be changed dynamically by putting an SQL into the data.conf file to replace it. example: (in data.conf) ExtensionLookupSQL = SELECT * FROM some_other_table WHERE field_name='${Context}'....... etc... etc... To see all the variable names that are used to replace default SQL, check around lines 2371-2409 of the data_mysql.c source file. By: j (j) 2004-11-24 13:30:18.000-0600 Thanks the sql configuration helps a lot! I still seem to be having a problem patching it though. I downloaded the newest -r 1-0-2 revision from CVS and did a little digging.. there seems to be changes in some of the asterisk code, such as app_voicemai, etc. Did you try applying the patch to the latest cvs? or am I barking ip the wrong tree? I also tried applying the patch to the 1.0.2 tarball from the asterisk ftp. They both give me failed chunks .. :( j By: j (j) 2004-11-24 14:19:06.000-0600 Ok. It patches perfectly against the cvs tagged head (which is what ya download when you don't put any -r tags). Thanks again for all your help. Oh. How were you planning on doing the extensions.conf include=>context stuff? I patched one of your c files in order to allow extensions to be looked up with an sql "where context in (context1, context2)" clause which worked recursively .. I'll send it along for your perusal if you are interested. Anyway, again thanks for the help! Much appreciated! j By: nix (nix) 2004-11-24 15:01:29.000-0600 rgagnon: Please take another look at the sql schema attached to the bug. Having a centralised table for friends does not mean that you cannot still selectively enable/disable certain channel types, and indeed my schema allows exactly that. It is a little complicated because I made it backwards compatible with the old style sip and iax friends (ie. without ast_data). Given that ast_data allows configurable queries, it can be simplified alot (the views and triggers can be removed) however this doesn't modify the idea or the function of a centralised authentication table. By: Rob Gagnon (rgagnon) 2004-11-27 02:52:05.000-0600 OK. I have been very busy with paying work for a while to get to the "Include=> " code.... As well, you cannot use the live SVN ast_data against anything but CVS HEAD of asterisk. The ast_data_102 package is meant for that, but it might be out of date a bit. I would expect possible patch failures there. Anyone know why this was closed? By: j (j) 2004-11-29 16:37:39.000-0600 actually, it's pretty simple to add context inclusions right into the sql.. Here's how I did it: Separate table for a simple list of contexts to include: CREATE TABLE extensions_include_context( id int unsigned PRIMARY KEY AUTO_INCREMENT, context VARCHAR(40) NOT NULL, context_include VARCHAR(40) NOT NULL, INDEX (id), INDEX (context, context_include), UNIQUE (context, context_include) ) TYPE=INNODB; then in the data.conf file update the extensions lookup sql to: SELECT extension, application, args FROM extensions WHERE ((context = '${Context}') OR (context in (select distinct context_include from extensions_include_context where context = '${Context}'))) AND (extension = '${Extension}' OR extension LIKE '\\_%%') AND priority = '${Priority}' ORDER BY LEFT(extension, 1) ASC, CHAR_LENGTH(extension) DESC; [ === ] in summary: where ((context = '${Context}') OR (context in (select distinct context_include from extensions_include_context where context = '${Context}'))) [ === ] Already tested (a bit) and seems to work great! Lemme know if I've just done something stupid :) Am confused about the patch thing though.. the only place 279 would patch cleanly is against whatever the default tag is using cvs to download asterisk. I read on a board a while ago that if you don't put any revision arguments on the cvs command, that it does in fact download the HEAD branch... Of course, I could be wrong ... By: j (j) 2004-11-29 16:43:36.000-0600 Anything But the cvs HEAD tag .. Anything But .. oops.. read it wrong .. <reminds himself to try not to respond or post when he's tired> Sorry. By: Rob Gagnon (rgagnon) 2004-11-30 11:40:07.000-0600 not sure why this keeps getting closed. By: Brian West (bkw918) 2004-11-30 12:00:26.000-0600 Its getting closed because its offically a third party application. It doesn't belong here anymore. bkw By: Rob Gagnon (rgagnon) 2004-11-30 12:23:40.000-0600 OK. Sorry to re-open, but seems only way to comment. The original idea was to get this actually added in as part of Asterisk. It seemed, a few months ago, that that was going to happen, but no one ever communicated otherwise. I do remember at one time a "Post 1.0" comment added to the title. By: Brian West (bkw918) 2004-11-30 12:28:11.000-0600 Last I heard it was not, and realtime was the direction mark was going. bkw By: j (j) 2004-12-09 13:39:25.000-0600 Greetings again! I've got a pretty big problem that I can't seem to get around and thought you might like to know about: With the bleeding edge code from cvs and the svn (as well as every time I've tried this with cvs downloads) I get a coredump. When I move the configuration back to simple files, the * machine works fine, but off sql it core dumps. The only error I get is "Ouch ... error while writing audio data: : Broken pipe" , which a little research on the net will dictate a zaptel card problem, but I can disprove that in this particular instance because we have NO zaptel cards in the machine... It's possible (and probable) that my config somewhere in the database is incorrect, but the system still shouldn't core dump.. .. right? hehe... Is there anything I can do to help figure this out? Any advice? Thanks again guys! j By: j (j) 2004-12-09 16:12:11.000-0600 After a full day of looking through stuff, and playing with a lot of it, I think I have solved the problem. I want to make sure I'm not creating major memory leaks here: In pbx.c somewhere around lines 850(something) in the function pbx_find_extension where it tries to strncpy the extension and application to the ast_exten struct asterisk was core dumping. The definition of the ast_exten struct says that the two members are char pointers, but no where does it allocate space for them, so before the two lines I simply put a calloc, (malloc was creating weird results) and it seems to work ok. Of course, I have very little overall knowlege of how * uses all of this, so I don't know if it frees the ast_exten struct, let alone the two members I allocated memory for. Is this correct? Or did I make a boo boo? Here are the changed lines: e->exten = calloc(strlen(extension) + 1, sizeof(extension)); strncpy(e->exten, extension, strlen(extension)); e->app = calloc(strlen(extension) + 1, sizeof(extension)); strncpy(e->app, application, strlen(application)); e->data = strdup(arguments); e->datad = free; e->priority = priority; *status = STATUS_SUCCESS; Any thoughts on this? Thanks! j By: Brian West (bkw918) 2004-12-09 16:15:06.000-0600 Ouch ... error while writing audio data: : Broken pipe Thats an mpg123 error. bkw By: Anthony Minessale (anthm) 2004-12-15 21:18:55.000-0600 j, that code is *WAY* wrong, don't do that...... That function is only meant to set e to point at some predefined extension in the dialplan not allocate anything. if you are getting a segfault the issue is most likely in the patch you are adding not in the core code since that code is executed every time you make a call somebody would surely know if it had a bug. By: j (j) 2004-12-16 07:21:31.000-0600 um... ok This is where the segfault is happening though. I did a heck of a lot of debugging and all roads lead to that point in code. Perhaps someone could help me out then? The modification I made seems to work, although probably causes horrible memory leaks ... but if that modifaction is wrong I'm at a loss. I understand that the "ouch" error may be directly from mp3 libraries or progs, but I think that's due to * segfaulting and cutting off the mp3 stuff, not actually from the mp3 stuff directly. I'm trying to roll this out in large scale production use very very shortly, and I like this solution better than all the others I've seen for dynamic configuration of *, I've even already built a large database architecture around it, but if I can't get it working properly I'm going to need suggestions on another solution. Thanks j By: Rob Gagnon (rgagnon) 2004-12-16 08:25:33.000-0600 I am getting reports from other users that it may be something new added to pbx_find_extension() that the ast_data patch does not know about. When I get to my office, I will check it out. By: Rob Gagnon (rgagnon) 2004-12-16 11:57:08.000-0600 segfault in pbx.c is fixed. Some time ago in pbx.c v1.155 or 1.156, the ast_exten struct dropped the fixed length character strings "exten, app, and cidmatch"... I didn't notice that, and was strncpy()ing into those. I fixed the patch, and the memory is now dynamically allocated and free'd as needed By: j (j) 2004-12-16 13:30:33.000-0600 Thanks a bunch! j By: Rob Gagnon (rgagnon) 2004-12-16 13:46:25.000-0600 j: No problem. You were right on the money with where the problem was originating, but I went with a strdup() call instead of the combo: calloc() strncpy()... As well, Code was added to the "free_dynamic_extensio()" function to watch for and free those dynamically added pieces of information. By: j (j) 2004-12-16 13:56:37.000-0600 Doh. hehe. For some really stupid reason, I was under a (false) impression that the memory would be freed somewhere later in the code .. Anyway, glad I could be of any use at all :) j By: kasoft (kasoft) 2004-12-27 06:00:55.000-0600 If I use ast_data i can't call with my Iaxy Hardware any numbers. I get a busy tone after dialed one number. I thought that it is a Asterisk bug and tried same configuration with same asterisk release (v1.02) and it worked. So I think the bug is on ast_data side. By: Rob Gagnon (rgagnon) 2004-12-28 10:39:25.000-0600 kasoft: You will need to configure your equipment to not send any digits until all of them are typed in by the user. Due to the nature of SQL, there will be no match on a single digit entered By: j (j) 2004-12-29 11:24:17.000-0600 I'm sure you guys are on top of this, but.. What is the SIP_TEMPONLY flag supposed to be defined as? I can't find it in any of the * src files, and seeing as it's undefined, it's causing compile errors. Btw: there was a cvs update last night .. :) Thanks j By: Brian West (bkw918) 2004-12-29 20:21:49.000-0600 This will need to be moved off to its own project page. It can't linger here in the bug tracker till the end of time. Its not a bug so we must narrow or focus down to just bugs since this tracker is getting out of hand as it is... this only helps mark and us bug marshals focus on stuff that is truly a bug. Thanks, Brian | ||