CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Error reading files from dir

 



joeferns79
Novice

Jan 17, 2014, 1:55 PM

Post #1 of 25 (5650 views)
Error reading files from dir Can't Post

I've got a script that's supposed to read files from a directory, and do pattern match on each file in the dir. If I run this script from within the dir that contains the files, it works fine but it fails to find any file if I run it from the parent dir.
  1. #!/usr/bin/perl
  2. use strict;
  3. use warnings;
  4. opendir my $DIR, '/home/tester1/logs/' or die "opendir .: $!\n";
  5. print "$DIR\n";
  6. my @files = grep /\.log$/i, readdir $DIR;
  7. closedir $DIR;
  8. # print "Got ", scalar @files, " files\n";
  9. open my $out,'>',"report.txt";
  10. foreach my $file (@files) {
  11. open my $FILE, '<', $file or die "$file: $!\n";
  12. while (<$FILE>) {
  13. print $out $_ if /^<string to search>/;
  14. }
  15. close $FILE;
  16. }
  17. close $out;




This is the output I get ...

bash-4.1$ ./test.pl
GLOB(0x200161f8)
test1.log: A file or directory in the path name does not exist.
bash-4.1$


FishMonger
Veteran / Moderator

Jan 17, 2014, 2:11 PM

Post #2 of 25 (5648 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

readdir only returns the filename without its path. If you're not in the same directory as the path given to opendir, then you'll need to prepend that path to the filename in your open call.

Please read the documentation for the readdir function.
perldoc readdir
http://perldoc.perl.org/functions/readdir.html


Laurent_R
Veteran / Moderator

Jan 18, 2014, 3:14 PM

Post #3 of 25 (5589 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

Fishmonger explained what the problem is and how to solve it with readdir (i.e. prepending the file name with the path). But, instead of readdir, you might consider using the glob function, which returns the filenames with their relative or absolute path (depending on what you passed to glob).

As an example:

Code
my @file_list = glob "./Old_PL_programs/*.pl";


gives me the following list:

Code
./Old_PL_programs/24.pl 
./Old_PL_programs/2tab.pl
./Old_PL_programs/aa.pl
...


As you can see, I passed a relative path and the function returns a list of files with their relative path.

In most cases, this is easier to use than the opendir/readdir combination.


joeferns79
Novice

Jan 21, 2014, 12:31 PM

Post #4 of 25 (5228 views)
Re: [Laurent_R] Error reading files from dir [In reply to] Can't Post

Thanks, Fishmonger/Laurent_R!

So, I wanted to take this a little further and read multiple directories from the parent dir, and read multiple files from within each dir. In addition, I wanted to output a file for each of these directories based on a search pattern.

This is what I have so far but it doesn't seem to output anything except it prints the directory names...

Code
#!/usr/bin/perl  

use strict;
use warnings;


my $dir = '/home/tester1/logs/';

opendir(DIR, $dir) or die $!;

while (my $file = readdir(DIR)) {

next unless (-d "$dir/$file");
print "$file\n";
chdir '$dir/$file'; #go into dirX inside of /home/tester1/logs/
opendir my $DIR2, '.' or die "opendir .: $!\n";
my @files2 = grep /\.log$/i, readdir $DIR2;

open my $out,'>',"../report_$file.txt";

foreach my $file1 (@files2) {
open my $FILE, '<', $file1 or die "$dir/$file: $!\n";
while (<$FILE>) {
print $out $_ if /^<string to search>/;
}
close $FILE;
}
close $out;
closedir $DIR2;
}
closedir(DIR);
exit 0;



(This post was edited by FishMonger on Jan 21, 2014, 1:15 PM)


FishMonger
Veteran / Moderator

Jan 21, 2014, 1:37 PM

Post #5 of 25 (5217 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

The first problem I see is with this line.

Code
chdir '$dir/$file';

The single quotes will prevent variable interpolation. Change them to double quotes.


FishMonger
Veteran / Moderator

Jan 21, 2014, 1:55 PM

Post #6 of 25 (5214 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

You might find it easier to assign the dirs to an array and then loop over the array.


Code
my @dirs = grep { -d } <$dir/*>; 
foreach my $dir (@dirs) {

or

Code
foreach my $dir ( grep { -d } <$dir/*> ) {



Laurent_R
Veteran / Moderator

Jan 21, 2014, 3:07 PM

Post #7 of 25 (5202 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

This simple recursive function, taken from a Perl tutorial I published on line quite recently (in French), might give you some ideas:


Code
sub browse_dir { 
my ($path) = @_;
my @dir_entries = glob("$path/*");
foreach my $entry (@dir_entries) {
do_something_with_file ($entry) if -f $entry;
browse_dir($entry) if -d $entry;
}
}


Please ask if you need explanations.

Having said that, the code above was only a starting point for much more powerful constructs. You might also want to take a look at the File::Find module, which does something similar to this and many other useful things.


joeferns79
Novice

Jan 31, 2014, 9:38 AM

Post #8 of 25 (4061 views)
Re: [Laurent_R] Error reading files from dir [In reply to] Can't Post

Thanks again, guys! Sorry for the delay but I was going by FishMonger's solution. It is now creating the file in each dir but the name of the file is weird. Here's the code...


Code
  

#!/usr/bin/perl

use strict;
use warnings;


my $dir = '/home/tester1/logs/';

opendir(DIR, $dir) or die $!;

my @dirs = grep { -d } <$dir/*>;

foreach my $dir2 (@dirs) {

print "$dir2\n";
chdir "$dir2";
opendir my $dir2, '.' or die "opendir .: $!\n";
my @files2 = grep /\.log$/i, readdir $dir2;

open my $out,'>',"report_'$dir2'.txt";

foreach my $file1 (@files2) {
open my $FILE, '<', $file1 or die "$file1: $!\n";
while (<$FILE>) {
print $out $_ if /^<string to search>/;
}
close $FILE;
}
close $out;
closedir $dir2;

}

closedir(DIR);
exit 0;



If I execute this script, this is what I see printed, which is fine.

bash-4.1$ ./Summary.pl
/home/tester1/logs/prod1
/home/tester1/logs/prod2
/home/tester1/logs/prod3
/home/tester1/logs/prod4
/home/tester1/logs/prod5



Now, if I go into each of the directories above, I see a file in each of them, but they're named similar to this .... report_'GLOB(0x2007e920)'.txt.

For whatever reason, it's printing the directory name as a GLOB, instead of "report_prod1.txt", etc.

Do you think it is trying to name the file "report_/home/tester1/logs/prod1.txt" instead of "report_prod1.txt" and that's what the issue is?


FishMonger
Veteran / Moderator

Jan 31, 2014, 9:57 AM

Post #9 of 25 (4058 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

You're using the same var name within the same scope for both the loop var and the dir handle. Don't do that!!


(This post was edited by FishMonger on Jan 31, 2014, 9:57 AM)


Laurent_R
Veteran / Moderator

Feb 1, 2014, 10:43 AM

Post #10 of 25 (3968 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

Besides the error pointed out by FishMonger (the use of $dir2 for refering to two different types of objects), I would suggest that you make the effort to format and especially to indent your code correctly and consistently, that will save you a lot of debugging time. Perhaps something like this (untested):


Code
#!/usr/bin/perl   

use strict;
use warnings;

my $dir = '/home/tester1/logs/';
# opendir(DIR, $dir) or die $!; not needed, next line does it better
my @dirs = grep { -d } <$dir/*>;

foreach my $dir2 (@dirs) {
print "$dir2\n";
chdir "$dir2";
# rather than the two lines below, why don't you use the same syntax as above:
# my @files2 = grep /\.log$/i, <$dir2/*.*>;
# BTW, that would be sufficient to solve your dir handle bug
opendir my $DIR, '.' or die "opendir .: $!\n";
my @files2 = grep /\.log$/i, readdir $DIR;

open my $out,'>',"report_${dir2}.txt";
foreach my $file1 (@files2) {
open my $FILE, '<', $file1 or die "$file1: $!\n";
while (<$FILE>) {
print $out $_ if /^<string to search>/;
}
close $FILE;
}
close $out;
closedir $DIR;
}
# closedir(DIR); no longer needed
exit 0;


Please note that I made some other small editions to your code and added some comments here and there.

Now, after this reformating, the indentation show clearly where the flow control start and end instructions are and the program is much clearer to read and understand. You don't need to use the same indenting conventions I used above (such as 4 spaces for each indentation level), but you really need to be consistent.

Every good programmer that I know or have known is serious about code formating in general and indentation in particular. I would actually never hire a programmer writing source code with such sloppy formatting as the code in your post above. Wink


Kenosis
User

Feb 1, 2014, 12:58 PM

Post #11 of 25 (3955 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

The assistance you've been given is excellent. But in case you may be interested, the following provides an option using the module File::Find::Rule for descending into directories:

Code
use strict; 
use warnings;
use File::Find::Rule;
use File::Basename;

my $dir = '/home/tester1/logs/';
my @files = File::Find::Rule->file()->name('*.log')->maxdepth(2)->in($dir);

for my $path (@files) {
my ( $filename, $suffix ) = ( fileparse($path) )[ 0, 1 ];
my $out;

open my $FILE, '<', $path or die "$path: $!\n";

while (<$FILE>) {
if (/^\Q<string to search>\E/) {
open $out, '>', $suffix . "report_$filename.txt" unless defined $out;
print $out $_;
}
}
close $out if defined $out;
close $FILE;
}

maxdepth() is set to 2, so it'll descend just below the top level. Without this parameter, it'll go as deep as the existing directory structure.

Another change is that a report file is created only if a match is found within the file it's examining. Without doing this, the script would create empty report files for files with no matches.

Lastly, note that \Q ... \E is used in the matching regex. This quotes (escapes) any meta-characters which may be present in the search string.

Hope this helps!


(This post was edited by Kenosis on Feb 1, 2014, 5:59 PM)


joeferns79
Novice

Feb 4, 2014, 1:49 PM

Post #12 of 25 (3599 views)
Re: [Laurent_R] Error reading files from dir [In reply to] Can't Post

Laurent_R,

I tried it by making changes as per your response.

It returns...

Couldn't open: A file or directory in the path name does not exist. at ./Summary.pl line 22.

Line 22 is ...

open my $out,'>',"report_${dir2}.txt" or die "Couldn't open: $!";


joeferns79
Novice

Feb 4, 2014, 1:53 PM

Post #13 of 25 (3597 views)
Re: [Kenosis] Error reading files from dir [In reply to] Can't Post

Kenosis,

Unfortunately, it looks like it can't find the module "File::Find::Rule"



bash-4.1$ ./Summary3.pl
Can't locate File/Find/Rule.pm in @INC (@INC contains: /usr/opt/perl5/lib/5.8.8/aix-thread-multi /usr/opt/perl5/lib/5.8.8 /usr/opt/perl5/lib/site_perl/5.8.8/aix-thread-multi /usr/opt/perl5/lib/site_perl/5.8.8 /usr/opt/perl5/lib/site_perl .) at ./Summary3.pl line 5.
BEGIN failed--compilation aborted at ./Summary3.pl line 5.


Kenosis
User

Feb 4, 2014, 2:48 PM

Post #14 of 25 (3592 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

Have you tried installing it?


Laurent_R
Veteran / Moderator

Feb 5, 2014, 10:43 AM

Post #15 of 25 (3508 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post


In Reply To
Couldn't open: A file or directory in the path name does not exist. at ./Summary.pl line 22.

Line 22 is ...

open my $out,'>',"report_${dir2}.txt" or die "Couldn't open: $!";


Well, it depends what the $dir2 variable contains. If it is a path containing backslashes such as, e.g. "/usr/bin", they you end up trying to create a file named "report_/usr/bin.txt" and the OS is unlikely to let you open such a file (that is unless your current directory has a "report_" subdirectory having itself a "usr" sub-subdirectory, in which case it will happily create a "bin.txt" file in that sub-subdirectory provided you have sufficient privileges to to it).

Print to the screen the name of the file you are trying create right before your open command or in the message passed to the die function, the origin of the problem will probably be obvious.

This is a general rule of debuging (in most programming languages): when some command fails, add a print statement before it with all the parameters used and run again (or you might use the Perl debugger).

In addition, you might have to filter out the "." and ".." directories at the beginning of your program, you certainly don't want to cd to them nor to open files with these names.


(This post was edited by Laurent_R on Feb 5, 2014, 10:54 AM)


joeferns79
Novice

Feb 5, 2014, 3:51 PM

Post #16 of 25 (3486 views)
Re: [Kenosis] Error reading files from dir [In reply to] Can't Post

Kenosis,

Installing it is a lengthy process right now. I'll try an alternate solution for now.


joeferns79
Novice

Feb 5, 2014, 4:24 PM

Post #17 of 25 (3483 views)
Re: [Laurent_R] Error reading files from dir [In reply to] Can't Post

Laurent_R,

That's true. It's trying to name the file with the absolute path.

bash-4.1$ ./UAESummary.pl
Couldn't open: A file or directory in the path name does not exist.:/home/tester1/logs/prod1 at ./UAESummary.pl line 22.


So, I am actually trying to include only the last directory name in the output filename to differentiate between the files I get from the other directories.

Eg. These are my directories...

/home/tester1/logs/prod1
/home/tester1/logs/prod2
/home/tester1/logs/prod3
/home/tester1/logs/prod4
/home/tester1/logs/prod5

Once I cd to "/home/tester1/logs/prod1", which is referenced by "$dir2", I want to create an output file in there named "report_prod1.txt". Then I cd to ""/home/tester1/logs/prod2", and create a file in there named "report_prod2.txt", etc.

Is the Basename module useful in this situation?


joeferns79
Novice

Feb 5, 2014, 6:01 PM

Post #18 of 25 (3473 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

Laurent_R,

So, I finally got it to work my adding the following....


Code
  

my @suffixlist;
my $bname = basename($dir2,@suffixlist);

my @files2 = grep /\.log$/i, <$dir2/*.*>;
open my $out,'>',"report_$bname.txt" or die "Couldn't open: $!:$bname";



Thanks for everything guys, I really appreciate it!


Kenosis
User

Feb 5, 2014, 8:20 PM

Post #19 of 25 (3463 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

Looks like you're getting a great handle on this! In case you're interested--and since installing the module's a bit too much, at this time--the following is functionally equivalent to the earlier post, sans File::Find::Rule:

Code
use strict; 
use warnings;
use File::Basename;

my $dir = '/home/tester1/logs';
my ( @paths, %FHs );

for my $dir2 ( grep -d, <$dir/*> ) {
for my $path (<$dir2/*.log>) {
push @paths, $path;
}
}

for my $path (@paths) {
my ( $filename, $suffix ) = ( fileparse($path) )[ 0, 1 ];
my $currDir = basename $suffix;

open my $FILE, '<', $path or die "$path: $!\n";

while (<$FILE>) {
if (/^\Q<string to search>\E/) {
open $FHs{$currDir}, '>', $suffix . "report_$currDir.txt" unless defined $FHs{$currDir};
print { $FHs{$currDir} } $_;
}
}
close $FILE;
}

close $FHs{$_} for keys %FHs;



(This post was edited by Kenosis on Feb 6, 2014, 9:32 AM)


joeferns79
Novice

Feb 6, 2014, 7:32 AM

Post #20 of 25 (3369 views)
Re: [Kenosis] Error reading files from dir [In reply to] Can't Post

Kenosis,

It looks like your script creates output files for each input file that contains the search pattern, and then prepends each of the files with "report_" and appends it with ".txt"

Eg. It goes into each dir under "/home/tester1/logs" like prod1,prod2, etc. Then searches all files in prod1,prod2, etc for the string and creates output files for those. So, if prod1 contains files log1,log2,log3, but only log1 & logs2 has the string, it creates output files named report_log1.txt, report_log2.txt.

I was looking for something that creates only 1 file that contains the results from all the logs in prod1, and that file should be named "report_prod1.txt"


Kenosis
User

Feb 6, 2014, 8:25 AM

Post #21 of 25 (3356 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

You're correct. The script above has been corrected to create only one file in a dir.


joeferns79
Novice

Feb 6, 2014, 9:09 AM

Post #22 of 25 (3341 views)
Re: [Kenosis] Error reading files from dir [In reply to] Can't Post

So, now the name of the output file is good but it looks like this file is being overwritten for every input file read. The list of files should probably be placed in an array, and then used in a for loop to parse through the array and print the output.


Kenosis
User

Feb 6, 2014, 9:28 AM

Post #23 of 25 (3329 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

I think it's one of those mornings. It's not necessary to use an array. A hash of file handles handles this case.

Fixed.

The close at the end of the script is unnecessary, since all opened files will be closed when the script's done. Included for good form.


(This post was edited by Kenosis on Feb 6, 2014, 9:30 AM)


joeferns79
Novice

Feb 6, 2014, 10:03 AM

Post #24 of 25 (3320 views)
Re: [Kenosis] Error reading files from dir [In reply to] Can't Post

Perfect! Thank you, Sir!


Kenosis
User

Feb 6, 2014, 10:22 AM

Post #25 of 25 (3317 views)
Re: [joeferns79] Error reading files from dir [In reply to] Can't Post

You're most welcome!

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives