| Summary: | ASTERISK-14438: cdr_custom produces incorrect csv format for clid | ||
| Reporter: | dkerr (dkerr) | Labels: | |
| Date Opened: | 2009-07-08 20:27:57 | Date Closed: | 2009-09-30 15:19:21 |
| Priority: | Minor | Regression? | No |
| Status: | Closed/Complete | Components: | CDR/cdr_custom |
| Versions: | Frequency of Occurrence | ||
| Related Issues: | |||
| Environment: | Attachments: | ( 0) csv_quote_14.txt | |
| Description: | The csv file produced when cdr_custom is enabled is incorrect and cannot be parsed by any csv parser (including excel, open office, php, perl, etc. etc.. The problem occurs in the handling of caller ID strings. Take this example template for cdr_custom.conf. [mappings] Master.csv => "${CDR(clid)}","${CDR(src)}","${CDR(dst)}" This specifies quoted CSV. The problem is that the format of a caller ID is '"lastname, firstname" <123456>' Which contains quoted text. The cdr_custom file produced would look like this... ""lastname, firstname" <123456>","123456","100" The correct format (which is produced by Asterisk default logging) is... """lastname, firstname"" <123456>","123456","100" Note that quotes (") are translated into two consecutive quotes (""). ****** ADDITIONAL INFORMATION ****** Examining the source to cdr_csv.c I notice that there is special handling to turn quotes into two consecutive quotes. However, the source to cdr_custom.c has no such special handling. Rather the whole job of parsing the custom template and substituting values for the ${CDR()} variables is handed off to pbx_substitute_variables_helper() function which is in pbx.c. This function does not (and probably should not) do anything special with quotes. The only fix I can think of is that cdr_custom.c needs to parse out the variable names from the template, substitute the values one-by-one, then handle the quotes before building the complete cdr csv record for saving to the log file. | ||
| Comments: | By: dkerr (dkerr) 2009-07-08 20:36:16 POTENTIAL WORK AROUND... If the cdr_custom.conf file is changed from... [mappings] Master.csv => "${CDR(clid)}","${CDR(src)}","${CDR(dst)}", etc etc to... [mappings] Master.csv => ${CDR(clid)},"${CDR(src)}","${CDR(dst)}", etc etc Then most csv parsers will correctly handle the resulting cdr record. Note the removal of the quotes around the caller ID variable. This causes asterisk to log, for example... "lastname, firstname" <123456>,"123456","100" so the whole CDR string is not bounded by quotes, but the caller ID name field is (which may legally contain a comma). Excel and other csv parsers can handle this, while they cannot handle... ""lastname, firstname" <123456>","123456","100" which is what asterisk produced today with the sample cdr_custom.conf template. David By: Matthias Nick (mnick) 2009-09-30 10:14:09 added a patch including a new dialplan function 'CSV_QUOTE' By: dkerr (dkerr) 2009-09-30 10:39:09 Thanks, I'll test out this patch and let you know how it goes. By: Digium Subversion (svnbot) 2009-09-30 11:23:10 Repository: asterisk Revision: 221157 U branches/1.4/configs/cdr_custom.conf.sample U branches/1.4/funcs/func_strings.c ------------------------------------------------------------------------ r221157 | mnick | 2009-09-30 10:41:46 -0500 (Wed, 30 Sep 2009) | 8 lines added a new dialplan function 'CSV_QUOTE' and changed the cdr_custom.sample.conf (closes issue ASTERISK-14438) Reported by: dkerr Patches: csv_quote_14.txt uploaded by mnick (license ) Tested by: mnick ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=221157 By: Digium Subversion (svnbot) 2009-09-30 14:45:23 Repository: asterisk Revision: 221368 _U trunk/ U trunk/configs/cdr_custom.conf.sample U trunk/funcs/func_strings.c ------------------------------------------------------------------------ r221368 | mnick | 2009-09-30 14:45:23 -0500 (Wed, 30 Sep 2009) | 23 lines Merged revisions 221153,221157,221303 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r221153 | mnick | 2009-09-30 10:37:39 -0500 (Wed, 30 Sep 2009) | 2 lines check bounds - prevents for buffer overflow ........ r221157 | mnick | 2009-09-30 10:41:46 -0500 (Wed, 30 Sep 2009) | 8 lines added a new dialplan function 'CSV_QUOTE' and changed the cdr_custom.sample.conf (closes issue ASTERISK-14438) Reported by: dkerr Patches: csv_quote_14.txt uploaded by mnick (license ) Tested by: mnick ........ r221303 | mnick | 2009-09-30 14:02:00 -0500 (Wed, 30 Sep 2009) | 2 lines changed the prototype definition of csv_quote ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=221368 By: Digium Subversion (svnbot) 2009-09-30 15:05:02 Repository: asterisk Revision: 221369 _U branches/1.6.0/ U branches/1.6.0/configs/cdr_custom.conf.sample U branches/1.6.0/funcs/func_strings.c ------------------------------------------------------------------------ r221369 | mnick | 2009-09-30 15:05:01 -0500 (Wed, 30 Sep 2009) | 30 lines Merged revisions 221368 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r221368 | mnick | 2009-09-30 14:42:36 -0500 (Wed, 30 Sep 2009) | 23 lines Merged revisions 221153,221157,221303 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r221153 | mnick | 2009-09-30 10:37:39 -0500 (Wed, 30 Sep 2009) | 2 lines check bounds - prevents for buffer overflow ........ r221157 | mnick | 2009-09-30 10:41:46 -0500 (Wed, 30 Sep 2009) | 8 lines added a new dialplan function 'CSV_QUOTE' and changed the cdr_custom.sample.conf (closes issue ASTERISK-14438) Reported by: dkerr Patches: csv_quote_14.txt uploaded by mnick (license ) Tested by: mnick ........ r221303 | mnick | 2009-09-30 14:02:00 -0500 (Wed, 30 Sep 2009) | 2 lines changed the prototype definition of csv_quote ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=221369 By: Digium Subversion (svnbot) 2009-09-30 15:12:49 Repository: asterisk Revision: 221370 _U branches/1.6.1/ U branches/1.6.1/configs/cdr_custom.conf.sample U branches/1.6.1/funcs/func_strings.c ------------------------------------------------------------------------ r221370 | mnick | 2009-09-30 15:12:49 -0500 (Wed, 30 Sep 2009) | 30 lines Merged revisions 221368 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r221368 | mnick | 2009-09-30 14:42:36 -0500 (Wed, 30 Sep 2009) | 23 lines Merged revisions 221153,221157,221303 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r221153 | mnick | 2009-09-30 10:37:39 -0500 (Wed, 30 Sep 2009) | 2 lines check bounds - prevents for buffer overflow ........ r221157 | mnick | 2009-09-30 10:41:46 -0500 (Wed, 30 Sep 2009) | 8 lines added a new dialplan function 'CSV_QUOTE' and changed the cdr_custom.sample.conf (closes issue ASTERISK-14438) Reported by: dkerr Patches: csv_quote_14.txt uploaded by mnick (license ) Tested by: mnick ........ r221303 | mnick | 2009-09-30 14:02:00 -0500 (Wed, 30 Sep 2009) | 2 lines changed the prototype definition of csv_quote ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=221370 By: Digium Subversion (svnbot) 2009-09-30 15:19:21 Repository: asterisk Revision: 221371 _U branches/1.6.2/ U branches/1.6.2/configs/cdr_custom.conf.sample U branches/1.6.2/funcs/func_strings.c ------------------------------------------------------------------------ r221371 | mnick | 2009-09-30 15:19:21 -0500 (Wed, 30 Sep 2009) | 30 lines Merged revisions 221368 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r221368 | mnick | 2009-09-30 14:42:36 -0500 (Wed, 30 Sep 2009) | 23 lines Merged revisions 221153,221157,221303 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r221153 | mnick | 2009-09-30 10:37:39 -0500 (Wed, 30 Sep 2009) | 2 lines check bounds - prevents for buffer overflow ........ r221157 | mnick | 2009-09-30 10:41:46 -0500 (Wed, 30 Sep 2009) | 8 lines added a new dialplan function 'CSV_QUOTE' and changed the cdr_custom.sample.conf (closes issue ASTERISK-14438) Reported by: dkerr Patches: csv_quote_14.txt uploaded by mnick (license ) Tested by: mnick ........ r221303 | mnick | 2009-09-30 14:02:00 -0500 (Wed, 30 Sep 2009) | 2 lines changed the prototype definition of csv_quote ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=221371 | ||