CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
Grep list of keywords

 



nothing3
New User

Apr 3, 2016, 7:52 PM

Post #1 of 7 (2224 views)
Grep list of keywords Can't Post

I am trying to open a log file, search it against a list of keywords, print out the entire line of any line that is found with one of those keywords and output it to another log file. When I try the code below I get a syntax error on line 12 with my grep command. I don't know what I'm doing wrong.


Code
  

#!/usr/bin/perl

open(FILE, "</data/bro/scripts/Keywords.txt");
my @keywords = <FILE>;
open(FILE, "</data/bro/logs/%f/http.22:10:00-01:20:00.log.gz");
my @logfile = <FILE>;
close(FILE);
open(FILE, ">>Results.log.gz");
my @results = <FILE>;

foreach $_ (@logfile) {
@results = grep -f @keywords (@logfile);
print FILE "@results";
close(FILE);
exit;
}



Laurent_R
Veteran / Moderator

Apr 3, 2016, 11:45 PM

Post #2 of 7 (2217 views)
Re: [nothing3] Grep list of keywords [In reply to] Can't Post

This:


Code
@results = grep -f @keywords (@logfile);


may be correct grep syntax under bash or ksh, but bot for Perl's grep.

Among the other problems:
- You read zipped files as if they were plain text, this isn't going to work. Same thing for appending to a zip file.

- You are reading @result from <FILE>, but the file handle has been open in append mode.

The logic in this part of the code:


Code
foreach $_ (@logfile) {  
@results = grep -f @keywords (@logfile);
print FILE "@results";
close(FILE);
exit;
}


is totally wrong. It reads lines from @logfile into $_, within the for loop you need yo work on $_, not on the array. Also don't close your file iwithin the loop.

Finally, your keywrods would be best stored into a hash, not an array.


Laurent_R
Veteran / Moderator

Apr 4, 2016, 1:53 AM

Post #3 of 7 (2208 views)
Re: [nothing3] Grep list of keywords [In reply to] Can't Post

I did not have time to complete when I wrote my earlier message, but the exit in the foreach loop is another thing that does not make real sense to me.


nothing3
New User

Apr 4, 2016, 2:25 AM

Post #4 of 7 (2204 views)
Re: [Laurent_R] Grep list of keywords [In reply to] Can't Post

First off, thank you very much for taking the time to respond. As I am sure you can tell, I am very new to Perl and have not had any formal education in it. I revised the code using some methods I found online and your response as guidance, is it closer to what I am trying to accomplish?


Code
 
#!/usr/bin/perl

use IO::Uncompress::Gzip qw($GzipError);
use IO::Compress::Gzip qw(gzip $GzipError) ;

open(FILE, "</data/bro/scripts/Keywords.txt");
my %keywords = <FILE>;
close(FILE);

my $logfile = IO::Uncompress::Gzip->new( /data/bro/logs/%f/http.22:10:00-01:20:00.log.gz )
or die "IO::Uncompress::Gzip failed: $GzipError\n";

open(FILE, ">Results.txt");
my @results = <FILE>;

foreach $_ ($logfile)
{
if (grep{%keywords}($_))
{
print $0;
}

else
{
next;
}
}

my @results = IO::compress::Gzip( results.log.gz )
or die "IO::compress::Gzip failed: $GzipError\n";
close(FILE);



(This post was edited by nothing3 on Apr 4, 2016, 10:25 PM)


Laurent_R
Veteran / Moderator

Apr 4, 2016, 6:22 AM

Post #5 of 7 (2190 views)
Re: [nothing3] Grep list of keywords [In reply to] Can't Post

Leaving aside the fact that the files are compressed, since I do not know by heart the syntax of modules specialized for that, I would amend your logic as follows (untested).

For populating the hash:

Code
my %keywords = map { chomp $_; $_, 1 } <FILE>;


The foreach loop:


Code
foreach my $line ($logfile)                                                                       
{
my $flag_found = grep { exists $keyword{$_} } split /\s+/, $line;
print $line if $flag_found;
}



nothing3
New User

Apr 4, 2016, 9:11 PM

Post #6 of 7 (2178 views)
Re: [Laurent_R] Grep list of keywords [In reply to] Can't Post

I have made some changes again following your advice and am receiving the following error:

Undefined subroutine &IO::Compress::Gzip called at script.pl line 25.


Code
 
#!/usr/bin/perl

use IO::Uncompress::Gunzip qw($GunzipError);
use IO::Compress::Gzip qw(gzip $GzipError) ;
use diagnostics;
use strict;
use warnings;

open(FILE, "</data/bro/scripts/Keywords.txt");
my %keywords = map { chomp $_; $_, 1 } <FILE>;
close(FILE);

my $logfile = IO::Uncompress::Gunzip->new( "/data/bro/logs/2016-04-04/http.01:20:00-03:10:00.log.gz" )
or die "IO::Uncompress::Gunzip failed: $GunzipError\n";

open(FILE, "+>Results.txt");
my @results = <FILE>;

foreach my $line ($logfile)
{
my $flag_found = grep {exists $keywords{$_} } split /\s+/, $line;
print $line if $flag_found;
}

IO::Compress::Gzip("results.gz")
or die "IO::Compress::Gunzip failed: $GzipError\n";
close(FILE);



BillKSmith
Veteran

Apr 5, 2016, 3:54 PM

Post #7 of 7 (2161 views)
Re: [nothing3] Grep list of keywords [In reply to] Can't Post

I recommend that you write and debug a version of your program that works with uncompressed files. (You will probably also have to make up a small set of test data.) This will allow you to concentrate on perl without the complication of the modules and their OO-interface. Without the module usage errors, Perl will be able to give you more useful error messages for your main task.

When you have finished that, read the documentation for both modules (especially the OO-interface sections). It appears that all you will actually have to do is replace the IO HANDLES with IO::COMPRESS/UNCOMPRESS objects.
Good Luck,
Bill

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives