Perl Guru Forums: Need a Custom or Prewritten Perl Program?: I need a program that...: Generate a C file from CSV file using perl

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Need a Custom or Prewritten Perl Program?: I need a program that...:
Generate a C file from CSV file using perl

gopsi1234
Novice

Jun 19, 2014, 6:21 AM

Post #1 of 22 (57686 views)

Generate a C file from CSV file using perl	Can't Post

Hi monk,

I need to write a perl script which will generate a *.c file from a *.csv file. The input csv file will contain information of certain chip in the following format:

CHIP,xxxx
vendorid,0xBF
memid,0x2501
size,256
devid,yyy
cmd,1
addr,3
mode,0
dummy,0
conf_wid,0
rate, 20
END

And the output c file will contain the above information as an array element in a CONST array in the following format:

{0x2501, FLASH_CONFIGURATION_DEVICE( 256,yyy,1,3,0,0,0, 20, 0xBF)}

There will be multiple chip information in the csv file and each chip information has to be stored in the c file.

I am new to perl scripting. Can you please provide a sample perl script file for one entry of chip so that i can carry on with the rest of the chip.

Zhris
Enthusiast

Jun 19, 2014, 2:28 PM

Post #2 of 22 (57631 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi,

Here is possibly something to get you going. You describe your input as a CSV but your example and description of the data does not correlate with this format, I have assumed your input is chunks of key value pairs. It reads each CHIP .. END chunk of your input data, parses it into a hash, then outputs it in your desired format. I'm sure you will figure out how to read from and write to actual files.

Code

#!/usr/bin/perl 
use strict; 
use warnings; 
 
my $hash = { }; 
 
while ( <DATA> ) 
{ 
    if ( /CHIP/ .. /END/ ) 
    { 
        #chomp; 
         
        if ( /END/ ) 
        { 
            printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s, %s)}\n",  
                   @{$hash}{qw/memid size devid cmd addr mode dummy conf_wid rate vendorid/};  
             
            $hash = { }; 
        } 
        else 
        { 
            my ( $key, $val ) = $_ =~ /(\S+)\s*,\s*(\S+)/; 
             
            $hash->{$key} = $val; 
        } 
    } 
} 
 
__DATA__ 
CHIP,xxxx  
vendorid,0xBF  
memid,0x2501  
size,256  
devid,yyy  
cmd,1  
addr,3  
mode,0  
dummy,0  
conf_wid,0  
rate, 20  
END  
 
CHIP,xxxx  
vendorid,0xBF  
memid,0x2501  
size,2564 
devid,yyy  
cmd,1  
addr,3  
mode,0  
dummy,0  
conf_wid,0  
rate, 20  
END

Hope this helps,

Chris

(This post was edited by Zhris on Jun 19, 2014, 2:46 PM)

gopsi1234
Novice

Jun 20, 2014, 7:55 AM

Post #3 of 22 (57260 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Zhris

Thanks for the code. I could convert the output in my desire format. And regarding the input CSV file, yes the format is in this way:

vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate
CHIP XXX
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20
END
CHIP YYY
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20
END

And the output will be in the format as previously mentioned. Can you please suggest what changes need to be done in the code in order to produce the same output based on this new input?

Zhris
Enthusiast

Jun 20, 2014, 6:42 PM

Post #4 of 22 (56520 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Heres one of many ways, keeping to the flip-flop/range approach:

Code

#!/usr/bin/perl  
use strict;  
use warnings;  
  
while ( <DATA> )  
{  
    if ( /CHIP/ .. /END/ and not /CHIP|END/ )  
    { 
        s/\s+$//; 
         
        printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s, %s)}\n",   
               (split /, /)[1..9,0]; 
    }  
} 
  
__DATA__  
vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate  
CHIP XXX  
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20  
END  
CHIP YYY  
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20  
END

Chris

(This post was edited by Zhris on Jun 20, 2014, 6:49 PM)

gopsi1234
Novice

Jun 21, 2014, 1:48 AM

Post #5 of 22 (56243 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

In Reply To

Hi Zhris,

Thanks for the code. Its working as per the format I wanted. In the csv file there is a cmd column, whose information shouldn't be there in the output file as because this information is fixed and the rest of the information might change, so what modification has to be done inside the while loop so that the cmd column is not printed and the rest of the information is printed in the output file as below?

{0x2501, FLASH_CONFIGURATION_DEVICE(256, yyy, 3, 0, 0, 0, 20, 0xBF)}

Also if I dont include the END statement at the last and the rest of the information is there in the csv file as the following format

vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate

CHIP XXX
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20

CHIP YYY
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20

What changes has to be done inside the while loop so that it produces the same output as before( with no cmd column as mentioned above)? Actually the END statement was used for easy parsing, but it seems it is not required in the csv file. I tried editing your code but couldn't reach the desired output. Your help will be really appreciated.

Zhris
Enthusiast

Jun 21, 2014, 2:46 AM

Post #6 of 22 (56202 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi,

Quote

what modification has to be done inside the while loop so that the cmd column is not printed and the rest of the information is printed in the output file as below

Just remove cmd's index from the slice, and also a %s from the format pattern.

Code

printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",    
       (split /, /)[1..3,5..9,0];

Quote

Actually the END statement was used for easy parsing, but it seems it is not required in the csv file.

A possibility could be to simply re-adjust the condition to match the appropriate rows. In the example below, I have made the assumption that rows we need begin with "0x".

Code

#!/usr/bin/perl   
use strict;   
use warnings;   
 
while ( <DATA> ) 
{ 
    if ( /^0x/ ) 
    { 
        s/\s+$//; 
         
        printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",    
               (split /, /)[1..3,5..9,0];  
    } 
} 
 
__DATA__   
vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate  
 
CHIP XXX  
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20  
 
CHIP YYY  
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20  
 
 
 
CHIP WWW 
 
0xBF, 0x2501, 1024, yyy, 1, 3, 0, 0, 0, 20

If the input format is guaranteed regular then an alternative might be to set the input record separator to double newline and process each chunk:

Code

#!/usr/bin/perl   
use strict;   
use warnings;   
 
$/ = "\n\n"; 
 
while ( <DATA> ) 
{ 
    next if $. == 1; # strip off head. 
     
    s/\s+$//; 
    
    printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",    
           (split /\n|, /)[2..4,6..10,1];  
} 
 
__DATA__   
vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate  
 
CHIP XXX  
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20  
 
CHIP YYY  
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20  
 
CHIP WWW 
0xBF, 0x2501, 1024, yyy, 1, 3, 0, 0, 0, 20

Or if there are no double newlines, read two rows at a time:

Code

#!/usr/bin/perl   
use strict;   
use warnings;   
 
while ( <DATA> ) 
{ 
    next if $. == 1; # strip off head. 
     
    my $next_row = <DATA>; 
       $next_row =~ s/\s+$//; 
    
    printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",    
           (split /, /, $next_row)[1..3,5..9,0];  
} 
 
__DATA__   
vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate  
CHIP XXX  
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20  
CHIP YYY  
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20  
CHIP WWW 
0xBF, 0x2501, 1024, yyy, 1, 3, 0, 0, 0, 20

Or even:

Code

#!/usr/bin/perl    
use strict;    
use warnings;    
  
while ( <DATA> )  
{  
    next if $. == 1 or not $. % 2; 
     
    s/\s+$//; 
     
    printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",     
           (split /, /)[1..3,5..9,0];   
}  
  
__DATA__    
vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate   
CHIP XXX   
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20   
CHIP YYY   
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20   
CHIP WWW  
0xBF, 0x2501, 1024, yyy, 1, 3, 0, 0, 0, 20

etc etc etc...

Since you don't use the CHIP *** rows, perhaps they could also be removed, producing a much more regular CSV file to have to deal with:

Code

__DATA__   
vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate  
0xBF, 0x2501, 256, yyy, 1, 3, 0, 0, 0, 20  
0xBF, 0x2501, 512, yyy, 1, 3, 0, 0, 0, 20  
0xBF, 0x2501, 1024, yyy, 1, 3, 0, 0, 0, 20

Chris

(This post was edited by Zhris on Jun 21, 2014, 4:08 AM)

gopsi1234
Novice

Jun 22, 2014, 3:23 AM

Post #7 of 22 (55257 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Zhris

Once again, heartfelt thanks for the perl code. What I observed that starting from

"If the input format is guaranteed regular then an alternative might be to set the input record separator to double newline and process each chunk:"

Whatever perl script is written below as per the input, it is not generating the desired output. Could you please check in your system once , because after i ran the code written by you in my system, it is giving some "Missing arguments in printf message.." and is not producing the correct output.

Zhris
Enthusiast

Jun 22, 2014, 12:09 PM

Post #8 of 22 (54963 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi,

Running that piece of code I don't receive the error that you do using http://www.compileonline.com/execute_perl_online.php. In fact the output is blank because upon copying and pasting the code it appears to add extra whitespace on the end of blank rows of data, not sure why, but I temporarily resolved using $/ = "\n \n"; .

If you want an alternative that processes the same structure of data, use the 4th example above but adjust the modulus condition to next if $. == 1 or $. % 3 != 1;.

Otherwise could you please post a) the exact code you are executing b) the data you are running against c) the output/error, so that I can replicate your issue.

Chris

(This post was edited by Zhris on Jun 22, 2014, 2:03 PM)

gopsi1234
Novice

Jun 23, 2014, 3:10 AM

Post #9 of 22 (54367 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Zhris

I have run your perl code in http://www.compileonline.com/execute_perl_online.php and i observed that it is running fine in that website. So, what I figured out is that the version of perl is different for me than the one used in the website. The website is using v5.10.1 and i have perl5.14.2.exe version. So as you requested, here is the code and the output associated with it

Code

#!/usr/bin/perl  
  
use strict;   
use warnings;  
open (DATA, "<input.txt") or die "Can't open file $!"; 
while ( <DATA> )  
{  
    next if $. == 1 or not $. % 2;  
      
    s/\s+$//;  
      
    printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",      
           (split /, /)[1..3,5..9,0]; 
}  
close(DATA);

and the output

Code

 {0x2501, FLASH_CONFIGURATION_DEVICE( 32, xxx, 3, 0, 0, 0, 20, 0xBF)} 
 {0x2501, FLASH_CONFIGURATION_DEVICE( 64, yyy, 3, 0, 0, 0, 20, 0xBF)} 
 {0x2501, FLASH_CONFIGURATION_DEVICE(128, zzz, 3, 0, 0, 0, 20, 0xBF)} 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
Missing argument in printf at flash_table_5.pl line 14, <DATA> line 9. 
 {, FLASH_CONFIGURATION_DEVICE(, , , , , , , )} 
} 
Press any key to continue . . .

and the data

Code

/* vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate */ 
CHIP xxx 
0xBF, 0x2501,  32, xxx, 1, 3, 0, 0, 0, 20 
CHIP YYY 
0xBF, 0x2501,  64, yyy, 1, 3, 0, 0, 0, 20 
CHIP zzz 
0xBF, 0x2501, 128, zzz, 1, 3, 0, 0, 0, 20

Now can you please tell me why is this happening when the version is changed? Also can you please explain me the condition you have used in the last 3 example and suppose if someone has wrongly entered insufficient information in the data, how to catch that and print error in the output file.

BillKSmith
Veteran

Jun 23, 2014, 4:18 AM

Post #10 of 22 (54320 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

The error messages strongly suggest that you have blank lines at the end of your data file. Perhaps they are not sent to the website.
Good Luck,
Bill

Zhris
Enthusiast

Jun 23, 2014, 9:31 AM

Post #11 of 22 (54112 views)

Re: [BillKSmith] Generate a C file from CSV file using perl [In reply to]	Can't Post

I concur with Bill. Remove the blank lines or adjust the code i.e. include "/^\s*$/ or" in the next condition or etc.

Heres two last examples which are much more forgiving with regards to the data format, but by no means flawless. In the first, I have broken up the core processes into separate functions, including basic validation. The second takes a regexp approach. There should be enough examples by now for you to fine tune to suit your particular dataset format.
_______________________________________________________________

Code

#!/usr/bin/perl   
use strict;    
use warnings;   
 
while ( <DATA> ) 
{   
    if ( /^CHIP/ ) 
    {         
        my @cols = row_to_cols( scalar <DATA> ); # fetch columns for row beneath CHIP row 
         
        print "invalid columns on row $.\n" and next unless validate_cols( @cols ); # if columns do not validate print error then move onto next row 
        
        printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n", transform_cols( @cols ); # print formatted row using transformed columns 
    } 
} 
 
sub row_to_cols 
{ 
    my $row = shift; 
    return split /\s*,\s*/, $row; # split row up in columns using comma separator 
} 
 
sub validate_cols 
{ 
    my @cols = @_; 
    return @cols == 10 ? 1 : 0; # if number of columns equals 10, return true, otherwise return false 
} 
 
sub transform_cols 
{ 
    my @cols = @_; 
    $cols[-1] =~ s/\s+$//;     # remove whitespace from end of last column. 
    return @cols[1..3,5..9,0]; # slice required columns and reorder 
} 
 
__DATA__ 
/* vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate */  
 
 
blah 
CHIP xxx  
0xBF,            0x2501,  32, xxx, 1,    3, 0, 0, 0,         20     
 
 
CHIP            YYY  
0xBF, 0x2501,            64, yyy, 1, 3, 0         , 0, 20  
 
 
 
 
foo 
 
CHIP            YYY  
0xBF, 0x2501,            64, yyy, 1, 3, 0,         0, 0, 20  
bar 
 
 
CHIP zzz  
0xBF,         0x2501, 128, zzz, 1,        3, 0, 0, 0, 20

Code

{0x2501, FLASH_CONFIGURATION_DEVICE(32, xxx, 3, 0, 0, 0, 20, 0xBF)} 
invalid columns on row 10 
{0x2501, FLASH_CONFIGURATION_DEVICE(64, yyy, 3, 0, 0, 0, 20, 0xBF)} 
{0x2501, FLASH_CONFIGURATION_DEVICE(128, zzz, 3, 0, 0, 0, 20, 0xBF)}

_______________________________________________________________

Code

#!/usr/bin/perl   
use strict;    
use warnings;  
 
$/ = 'CHIP'; 
 
while ( <DATA> ) 
{   
    next if $. == 1 or $_ !~ /^.+(?:\n\s*)+(.+?)\s*\n/;  
     
    printf "{%s, FLASH_CONFIGURATION_DEVICE(%s, %s, %s, %s, %s, %s, %s, %s)}\n",       
           (split /\s*,\s*/, $1)[1..3,5..9,0];  
} 
 
__DATA__ 
/* vendorid, memid, size, devid, cmd, addr, mode, dummy, conf_wid, rate */  
 
 
blah 
CHIP xxx  
0xBF,            0x2501,  32, xxx, 1,    3, 0, 0, 0,         20     
 
 
CHIP            YYY  
 
 
0xBF, 0x2501,            64, yyy, 1, 3, 0     ,0,0    , 0, 20  
 
 
 
 
foo 
 
CHIP            YYY  
0xBF, 0x2501,            77, yyy, 1, 3, 0,         0, 0, 20  
bar 
 
 
CHIP zzz  
0xBF,         0x2501, 128, zzz, 1,        3, 0, 0, 0, 20

Code

{0x2501, FLASH_CONFIGURATION_DEVICE(32, xxx, 3, 0, 0, 0, 20, 0xBF)} 
{0x2501, FLASH_CONFIGURATION_DEVICE(64, yyy, 3, 0, 0, 0, 0, 0xBF)} 
{0x2501, FLASH_CONFIGURATION_DEVICE(77, yyy, 3, 0, 0, 0, 20, 0xBF)} 
{0x2501, FLASH_CONFIGURATION_DEVICE(128, zzz, 3, 0, 0, 0, 20, 0xBF)}

_______________________________________________________________

Chris

(This post was edited by Zhris on Jun 23, 2014, 9:54 AM)

gopsi1234
Novice

Jun 23, 2014, 11:14 PM

Post #12 of 22 (53610 views)

Re: [BillKSmith] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Bill

Yes you are right. There were blank lines at the end of the data files. It's working fine now. Thanks for informing that.

Hi Zhris

Glad to see such a flawless perl code from you. Will implement that and let you know about the progress. Thanks again.

gopsi1234
Novice

Jul 1, 2014, 2:16 AM

Post #13 of 22 (45154 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Zhris

I need to keep the input csv file information in a hash array and then print it to a c file, so that in future, if someone add a new column in between the parsing of data should not change. In case of hash array, the column header can be used as a hash keys to get the value from the respective column and the input file is now a proper csv file. Below is the input file.

Code

/*Below is the input file */ 
vendorid, memid, size, devid, cmd, addr, conf_width, dummy, mode, rate, devicename 
 
0xAF, 0x5501,   32,  est, 1, 3, 0, 0, 0, 20, xxx 
0xAF, 0x5502,   64,  est, 1, 3, 0, 0, 0, 20, yyy 
0xAF, 0x5503,  128, est, 1, 3, 0, 0, 0, 20, zzz

Can you please help me in parsing the data from the input file and store it a hash array and finally printing it in the usual format i.e

Code

{0x5501, FLASH_CONFIGURATION_DEVICE(32, xxx, 3, 0, 0, 0, 20, 0xAF)}

(This post was edited by gopsi1234 on Jul 1, 2014, 2:29 AM)

Zhris
Enthusiast

Jul 1, 2014, 3:38 AM

Post #14 of 22 (45114 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Gopsi,

Perhaps you could try and write the script yourself based on what you have learnt so far and if you get stuck post the code / problem. I'm sure you will agree that this is the best approach if you want to develop your Perl skills.

With regards to using the header to fetch the associated values, you will probably want to use hash slices http://perldoc.perl.org/perldata.html#Slices.

Code

my @head_list_print = qw( vendorid conf_width ); 
 
... 
 
my %row_hash;  
@row_hash{@head_list} = @row_list; # head_list and row_list being the corresponding rows split into lists. 
printf "%s, %s\n", @row_hash{@head_list_print};

Even better though, use a CSV parsing module such as Text::CSV (https://metacpan.org/pod/Text::CSV) to do the hard work for you, Note the getline_hr method to achieve the above.

Regards,

Chris

(This post was edited by Zhris on Jul 1, 2014, 3:51 AM)

gopsi1234
Novice

Jul 1, 2014, 7:43 AM

Post #15 of 22 (45003 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi Zhris here is my perl code

Code

open (DATA, "<input.csv") or die "Can't open  $!"; 
my %data; 
my @names; 
while(<DATA>){ 
 my @list=split(/,/); 
 for(my $i=0; $i<=$#list;$i++){ 
  if($.==1){ 
   $name[$i]=$list[$i]; 
  } 
   
  else{ 
   push @{$data{$name[$i]}},$list[$i]; 
    
  } 
 } 
} 
foreach(keys %data){ 
 printf "{%s, %s\n}", @{$data{vendorid}}, @{$data{memid}}; 
}

Code

And i am facing trouble printing two values currently from the csv files. The output is printing as 
{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF 
}{0xBF, 0xBF

where as it should be

Code

0xBF,0x5501 
0xBF,0x5502

Moreover

1. I could do it with Text::CSV_XV but I cannot use Text::CSV_XV as because the module is not installed in our system and i cannot change the perl version installed.
2. Now the input file is as below

Code

vendorid,memid,size,devid,cmd,addr,conf_width,dummy,mode,rate,devicename 
0xBF,0x5501,32,est,1,3,0,0,0,20,xxx  
0xBF,0x5502,64,est,1,3,0,0,0,20,yyy  
0xBF,0x5503,128,est,1,3,0,0,0,20,zzz

As you can see there is no gaps/space in the input file between each column, as because whenever i was putting space, the parsing is not happening and an error is throwing. I am unable to tackle this issue. I want flexibility in terms of spaces i give between column, but not finding the right perl code to tackle that.

So, these are issues, can u please go through the code once, and tell me the changes i need to do to get the below output

Code

{0x5501, FLASH_CONFIGURATION_DEVICE(32, est, 3, 0, 0, 0, 20, 0xBF)} 
{0x5502, FLASH_CONFIGURATION_DEVICE(64, est, 3, 0, 0, 0, 20, 0xBF)}

(This post was edited by gopsi1234 on Jul 1, 2014, 7:47 AM)

Zhris
Enthusiast

Jul 1, 2014, 8:32 AM

Post #16 of 22 (44971 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

Hi gopsi,

Firstly, it looks as though you are making good progress, so well done for that!

Rather than give you a working version right away lets see if we can address some of your issues and perhaps you can fix your code yourself.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

At the top of every script you write you should always use strict and warnings:

Code

use strict; 
use warnings;

These will report on various issues during compilation and run time. You'll notice the following:

Code

Global symbol "@name" requires explicit package name at line 13. 
Global symbol "@name" requires explicit package name at line 17. 
Execution aborted due to compilation errors.

You declare the variable @names at the top, but actually use @name.

Upon rerunning the following error is reported:

Code

Can't use an undefined value as an ARRAY reference at line 26, <DATA> chunk 4.

This is because you have tried to use a non existent hash entry 'memid' as an array reference. When you split the line, you didn't incorporate the space after each comma and the key is actually ' memid'.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

When you enter the while loop, the first thing you will need to do is remove the newline character on the end of each line. You would usually use the chomp function to do this. However you have additional whitespace, therefore I suggest using a regexp. You also mentioned you have removed blank lines, but alternatively you can just skip them using another regexp:

Code

while (<DATA>) 
{ 
    next if m/^\s*$/; # skip blank lines. 
    s/\s+$//; # remove any whitespace from the end of the line. 
    ...

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

You can then go ahead and split the line as you have done, however incorporate the space after the comma, and just to be on the safe side consider incorporating potential space before the comma:

Code

my @list = split(/\s*,\s*/);

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Now remember, you already have the row as a list in @list. When you need to store the header, just copy this list:

Code

@names = @list;

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

You have decided upon storing your data into a hash of arrays, then iterating through it at the end to actually print out the data. Its perfectly possible to use a hash of arrays, but doesn't make as much sense as using an array of hashes, since each row is unique/individual. I would use the hash slice technique I showed you in the previous post, and push the hash onto the array:

Code

my %hash;   
@hash{@names} = @list; 
push @data, \%hash;

This will ensure if a new column is added, you don't have to make adjustments to the code. An alternative is to use an array of arrays, then worry about matching up each value to its column name later on:

Code

push @data, \@list; 
 
... 
 
my @head_b = qw(memid vendorid); 
my @indexes; 
 
for my $key (@head_b) 
{ 
    my ($index) = grep { $key eq $head[$_] } 0 .. $#head; 
    push @indexes, $index; 
} 
 
for (@data) 
{ 
    printf "%s, %s\n", @{$_}[@indexes]; 
}

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Before attempting to print out the data, another useful module is Data::Dumper. You can dump your data structure to stdout and inspect it to check its how you expected:

Code

use strict; 
use warnings; 
use Data::Dumper; 
 
my @data; 
 
while (<DATA>) 
{ 
    ... 
} 
 
print Dumper \@data;

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Use the technique I showed you in the previous post in a loop to print the data:

Code

for (@data) 
{ 
    printf "%s, %s\n", @{$_}{qw/vendorid memid/}; 
}

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I've actually used some bad practices, its better to use named variables whenever possible, it is easier to understand code like this, and you won't have problems if $_ is reassigned i.e. a loop within a loop:

Code

... 
while (my $row = <DATA>) { 
... 
for my $row (@data) { 
...

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I hope you find these pointers useful.

Regards,

Chris

(This post was edited by Zhris on Jul 2, 2014, 7:44 AM)

Zhris
Enthusiast

Jul 2, 2014, 7:44 AM

Post #17 of 22 (44370 views)

Re: [gopsi1234] Generate a C file from CSV file using perl [In reply to]	Can't Post

From private message:

Quote

Hey Chris

Its working fine now. Below is the code. Can it be better than this. Here i have separated the header first and then parsed the other values. Is it possible to seperate header also and putting the entire thing under 1 while loop. Right now, m using 2. Also what else can be improved?

Code

  
#!/usr/bin/perl -w  
 
# use strict;  
# use warnings;  
open (DATA, "<input.csv") or die "Can't open flash config repository: $!";  
 
my @headers;  
while(<DATA>){  
next if m/^\s*$/ or /\A#/; # skip blank lines or comments  
s/\s+$//; # remove any whitespace from the end of the line.  
@headers=split(/\s*,\s*/); # split the header names.  
last; # break out of the loop: the header was found  
}  
my @records;  
while(<DATA>){  
next if m/^\s*$/ or /\A#/; # skip blank lines or comments  
my @fields = split(/\s*,\s*/);  
my %hash;  
@hash{@headers} = @fields; # use hash slice to assign fields to headers  
push @records, \%hash; # add this hashref to records  
}  
for my $row (@records)  
{  
printf "%s, %s\n", @{$row}{qw/vendorid memid/};  
}

________________________________________________________________

1) Don't comment out using strict and warnings. As stated previously, if they raise any issues, you should fix rather than ignore. http://www.perlmonks.org/?node=use%20warnings%20vs.%20perl%20-w

2) The modern approach to opening a file is to use the 3 way form with a lexical variable filehandle, over a bareword filehandle.

3) In your previous code snippet, you used the 'if ( $. == 1 ) {' condition to handle the header, is there any reason you can't do this, i.e. it assumes the header is on line 1. If you aren't already aware, $. contains the line number you are currently reading from the filehandle, therefore the above condition fundamentally checks if we are on line 1. In my version below I assume the first valid line reached is the header line. I simply check if @headers contains any values, if it doesn't then we must be on the header line, we assign it some values, then other iterations @headers will contain values, therefore assumed records.

4) Although not necessary, its good practice to close the filehandle once you have finished with it.

5) Although not implemented in my version below, you don't actually have to store the records in @records, you might as well print directly i.e.:

Code

push @records, \%hash; 
 
printf "%s, %s\n", @hash{qw/vendorid memid/};

Here is my version of your code:

Code

#!/usr/bin/perl 
use strict;  
use warnings; 
 
my @headers; 
my @records; 
 
open my $fh, '<', 'input.csv' or die "Can't open flash config repository: $!";  
 
while (my $line = <$fh>) 
{ 
    next if ( $line =~ m/^\s*$/ or $line =~ m/\A#/ ); # skip blank lines or comments  
     
    $line =~ s/\s+$//; # remove any whitespace from the end of the line. 
 
    my @fields = split /\s*,\s*/, $line; # split line up into individual fields. 
 
    if (@headers) # headers have already been assigned. 
    { 
        my %hash;  
        @hash{@headers} = @fields; # use hash slice to assign fields to headers  
        push @records, \%hash; # add this hashref to records  
    } 
    else # headers have not yet been assigned. 
    { 
        @headers = @fields; # assign fields to headers. 
    } 
} 
 
close $fh; 
 
for my $entry (@records)  
{  
    printf "%s, %s\n", @$entry{qw/vendorid memid/};  
}

input.csv:

Code

 
 
vendorid,memid,size,devid,cmd,addr,conf_width,dummy,mode,rate,devicename  
 
 
 
 
0xBF,0x5501,32,est,1,3,0,0,0,20,xxx   
#comment 
0xBF,0x5502,64,est,1,3,0,0,0,20,yyy   
0xBF,0x5503,128,est,1,3,0,0,0,20,zzz

Regards,

Chris

(This post was edited by Zhris on Jul 2, 2014, 8:07 AM)

FishMonger
Veteran / Moderator

Jul 2, 2014, 9:09 AM

Post #18 of 22 (44324 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Assuming the header line is line 1 in the file, I'd extract that array data prior to the loop so that we don't need to test it with an if/else block for each line in the file.

Code

my @headers = split /\s*,\s*/, scalar <$fh>; 
chomp @headers;

Zhris
Enthusiast

Jul 2, 2014, 9:36 AM

Post #19 of 22 (44308 views)

Re: [FishMonger] Generate a C file from CSV file using perl [In reply to]	Can't Post

Good point. Note though that chomp @headers is an overkill since only the last element contains a newline, perhaps chomp $headers[-1] is better.

If the CSV is particularly large and the header isn't on line 1, then his original double while loop doesn't seem like a bad approach either. Or perhaps there is a better way to remove that condition?

Chris

(This post was edited by Zhris on Jul 2, 2014, 9:39 AM)

FishMonger
Veteran / Moderator

Jul 2, 2014, 9:48 AM

Post #20 of 22 (44295 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Quote

perhaps chomp $headers[-1] is better

I thought of doing that, but since we are Perl programmers, which typically strive to be lazy, I didn't want to waste the time typing those 4 extra characters to save that nano or pico second in cpu time. :-)

Regarding your last comment, I haven't followed the thread in enough detail to say if there is or isn't a better approach to removing that condition.

FishMonger
Veteran / Moderator

Jul 2, 2014, 10:06 AM

Post #21 of 22 (44284 views)

Re: [Zhris] Generate a C file from CSV file using perl [In reply to]	Can't Post

Maybe a do {...} until (...) block at the top of the while loop to assign @headers would be "better" than the dual while loops.

Benchmark tests would need to be run to see which is faster.

Zhris
Enthusiast

Jul 2, 2014, 12:44 PM

Post #22 of 22 (44207 views)

Re: [FishMonger] Generate a C file from CSV file using perl [In reply to]	Can't Post

Just to put code to your alternative suggestion (I have never used a do until before):

Code

my $header; 
do { $header = <DATA>; } until( $header !~ /^(\s*$|#)/ ); 
my @headers = split /\s*,\s*/, $header;

Chris

(This post was edited by Zhris on Jul 2, 2014, 12:47 PM)

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives