Perl Guru Forums: Perl Programming Help: Advanced: FTP to multiple directories and finding specific files.

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Advanced:
FTP to multiple directories and finding specific files.

cuboidgraphix
User

Dec 24, 2008, 6:56 AM

Post #1 of 22 (19299 views)

FTP to multiple directories and finding specific files.	Can't Post

Hello guys,
I've dealt with perl for some time but I've never FTP'd before.
I have a project whereby I need to FTP into a server, and look into multiple directories and find certain files and copy those files into a directory in a different server where I'm running the script from.

the Directories are as follows:

D010AMAB
D000AMAA
D010AMA9
D000AMA8
D010AMA5
D000AMA6
D010AMA7
D000AMA4
D010AMA3
D000AMA2
D010AMA1
D000AMA0

The files I'm looking for are files that look like..
R081223* (where * is other date time.)

So far what I've read has really confused me. I was wondering if someone can direct me to some tutorials, or even work done similar to this.

Thanks.

FishMonger
Veteran / Moderator

Dec 24, 2008, 7:35 AM

Post #2 of 22 (19297 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

What have you tried?

You might want to look at Net::FTP::Recursive
http://search.cpan.org/~jdlee/Net-FTP-Recursive-2.02/Recursive.pm

cuboidgraphix
User

Dec 24, 2008, 9:13 AM

Post #3 of 22 (19294 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Well, I read that the simplest way to do it is by having a basic ftp session and then looping it for each directory, so i came up with this:

Code

#!/usr/bin/perl 
my $epochYesterday = time - 24 * 60 * 60; 
my ($year, $month, $day) = (localtime($epochYesterday))[5,4,3]; 
$year += 1900; 
$y = substr($year, 2,2); 
$month++; 
 
$Yday = "R$y$month$day"; 
 
use Net::FTP; 
chdir "home/AMA"; 
@myDIR =('D010AMAB', 'D000AMAA', 'D010AMA9', 'D000AMA8', 'D010AMA5', 'D000AMA6', 'D010AMA7', 'D000AMA4', 'D010AMA3', 'D000AMA2', 'D010AMA1', 'D000AMA0'); 
foreach $DIR(@myDIR){ 
$ftp = Net::FTP->new("##.##.##.##", Debug => 0); 
$ftp->login("user",'pass'); 
$ftp->cwd("/$DIR"); 
print "Accessed $DIR\n"; 
$ftp->get("{$Yday}.*"); 
$ftp->quit; 
}

The problem is that id does not find the files simliar to ..

R081223* ... rather it only looks for files named R081223.

An example of a file is : R081223200083AMA

Oh, btw.. I'll take a look at the link you sent me.

Any ideas what's wrong with my script?

Thanks.

FishMonger
Veteran / Moderator

Dec 24, 2008, 10:26 AM

Post #4 of 22 (19285 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Lets start at the beginning.

Every Perl script you write should include the warnings and strict pragma which will help you write better quality code and point out common mistakes that can be difficult to track down.

Use strict;
use warnings;

Your date calculation can be simplified with the use of the strftime function from the POSIX module.

Use POSIX qw/strftime/;
my $Yday = strftime("R%y%m%d", localtime(time - 86400));

Purely as matter of style and following �Perl Best Practices� it's best to keep your line lengths below 80 chars. Which in this case means to split the @myDIR assignment across multiple lines and put it in column format.

Code

my @myDIR =('D010AMAB', 'D000AMAA', 'D010AMA9', 'D000AMA8', 
            'D010AMA5', 'D000AMA6', 'D010AMA7', 'D000AMA4', 
            'D010AMA3', 'D000AMA2', 'D010AMA1', 'D000AMA0' 
);

It doesn't make sense to open/close the ftp session for each and every directory. The foreach loop should only deal with retrieving the files and outputting status messages as needed.

It's been awhile since I use the Net::FTP module so, I'd need to run a test but I think the get call should be:
$ftp->get(�${Yday}*�);

However. It might be better to use the Net::FTP::Simple module which supports regex filtering.
http://search.cpan.org/~wilco/Net-FTP-Simple-0.0007/lib/Net/FTP/Simple.pm

cuboidgraphix
User

Dec 24, 2008, 11:01 AM

Post #5 of 22 (19281 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Thanks for all the advice FishMonger. It has really helped me a lot understanding the importance of having a clean practice of perl.

The reason why I don't use 'Use Strict and Warnings and POSIX' is that on the server I'm working on it says..

Can't locate object method "Use" via package "strict" at ./getama.pl line 3.

and so on.

In reference to the $ftp->get , it give me this error.

File unavailable. Requested action not taken.

I don't know why I am not getting the files from the server. It really sucks.

KevinR
Veteran

Dec 24, 2008, 11:05 AM

Post #6 of 22 (19279 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

thats because it should be all lower-case "use":

use strict;
use warnings;

all of perls builtin functions are all lower-case and all pragams are lower-case. Perl is case-sensitive so you need to know where you can use upper and lower case characters in perl code.
-------------------------------------------------

FishMonger
Veteran / Moderator

Dec 24, 2008, 11:39 AM

Post #7 of 22 (19278 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

My fault, I had a typo. As Kevin pointed out, it should have been in lowercase.

cuboidgraphix
User

Dec 29, 2008, 2:00 PM

Post #8 of 22 (19235 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Hi guys, well I'm still working on my script but I don't know why I can't get it to work.

My code is below:

Code

#!/usr/bin/perl 
use strict; 
use warnings; 
my $epochYesterday = time - 24 * 60 * 60; 
my ($year, $month, $day) = (localtime($epochYesterday))[5,4,3]; 
$year += 1900; 
my $y = substr($year, 2,2); 
$month++; 
 
my $Yday = "R$y$month$day"; 
 
use Net::FTP; 
chdir "/home/AMA"; 
my @myDIR =('D010AMAB', 'D000AMAA', 'D010AMA9', 'D000AMA8', 
            'D010AMA5', 'D000AMA6', 'D010AMA7', 'D000AMA4', 
            'D010AMA3', 'D000AMA2', 'D010AMA1', 'D000AMA0' 
); 
 
my $ftp = Net::FTP->new("host", Debug => 0); 
$ftp->login("user",'pass'); 
print $ftp->message(), "\n"; 
$ftp->binary; 
foreach my $DIR(@myDIR){ 
        $ftp->cwd("/$DIR"); 
        print $ftp->message(), "\n"; 
        my @file = grep { /^([^.].*)?$Yday/ } $ftp->ls(); 
        foreach my $file(@file){ 
                $ftp->get($file); 
                print $ftp->message(), "\n"; 
        } 
        print $ftp->message(), "\n"; 
} 
$ftp->quit;

The script works when I remove the.. $ftp->binary;
but when I add it in.. I get this message:

Directory changed.
TYPE must be ASCII

I need the files to be transferred Binary and not ASCII. Can you help Fishmonger?

Please and thanks!

FishMonger
Veteran / Moderator

Dec 30, 2008, 7:09 AM

Post #9 of 22 (19225 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

What type of files are you retrieving?

Saying that it doesn't work is too vague. Are you saying that the files fail to transfer when using binary mode?

Have you tried enabling debug to get more details of the exchange? And, it might be good to adjust the error handling.

I don't see any reason why are you doing chdir "/home/AMA";.

On a side note, your method of calculating the date is kludgy. It would be much cleaner and more efficient to use the strftime function from the POSIX module.

Code

#!/usr/bin/perl 
 
use strict; 
use warnings; 
use Net::FTP; 
use POSIX qw/strftime/; 
 
my $Yday = strftime("R%y%m%d", localtime(time - 86400)); 
  
my @myDIR =('D010AMAB', 'D000AMAA', 'D010AMA9', 'D000AMA8',  
            'D010AMA5', 'D000AMA6', 'D010AMA7', 'D000AMA4',  
            'D010AMA3', 'D000AMA2', 'D010AMA1', 'D000AMA0'  
);  
  
my $ftp = Net::FTP->new("host", Debug => 1) 
          or die "Failed to connect to ftp server ", ftp->message(); 
 
print ftp->message(), "\n"; 
 
$ftp->login("user",'pass') 
      or die "Failed to login to ftp server ", ftp->message(); 
 
print ftp->message(), "\n"; 
 
$ftp->binary or warn "Unable to switch to binary mode "; 
print ftp->message(), "\n"; 
 
foreach my $DIR(@myDIR){ 
    print "changing cwd to $DIR\n"; 
    $ftp->cwd("/$DIR") 
          or warn "failed to cd to /$DIR ", ftp->message() and next; 
    print $ftp->message(), "\n"; 
 
    my @file = grep { /^([^.].*)?$Yday/ } $ftp->ls(); 
    foreach my $file(@file){ 
        print "retriving file '$file'\n"; 
        $ftp->get($file) 
              or warn "failed to retrieve '$file' ", ftp->message() and next; 
        print $ftp->message(), "\n"; 
    } 
}

cuboidgraphix
User

Dec 30, 2008, 7:32 AM

Post #10 of 22 (19223 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

I'm sorry I'm not being forthwith with my explanations.
The files I am trying to copy are CDRs or call logs for a telecommunications company. These CDRs/AMA files do not have a file extension. The content are simply in binary code, thus I do not want to transfer them in ASCII mode. The problem is that whenever I remove the $ftp->binary, the files are found and transferred successfully in the ASCII mode, which are useless to me at that point. The reason why I have the chdir "/home/AMA" ... I don't really know why I have it, but I saw it on a sample code, but basically it is the repository where I want the files copied to.

*Note: The script is ran from the backup server. It's suppose to log into our 'SWITCH' and retrieve the files that are found and are then copied to the same backup server.

The code I'm trying to achieve is one that would log into our 'SWITCH' and check the directories for yesterday's CDRs and copy them into a backup server.

If I haven't stressed it enough, I really need them to be copied/transferred in binary mode, since ASCII would corrupt the file.

I hope I'm being more direct with my purpose, and I really appreciate you taking time to help me with this problem.

Thanks.

Oh btw ... The posix code that you had given me before did not work. I had the same error I had stated before, and I had 'use'. That's why I am still using my original code. Maybe if you repost the code, I can copy and paste it and see if it works.

Thanks again.

(This post was edited by cuboidgraphix on Dec 30, 2008, 7:39 AM)

FishMonger
Veteran / Moderator

Dec 30, 2008, 8:11 AM

Post #11 of 22 (19219 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

Oh btw ... The posix code that you had given me before did not work. I had the same error I had stated before, and I had 'use'. That's why I am still using my original code. Maybe if you repost the code, I can copy and paste it and see if it works.

Thanks again.

Quote

[rkb@mybox ~]$ cat date.pl
#!/usr/bin/perl

use strict;
use warnings;
use POSIX qw/strftime/;

my $Yday = strftime("R%y%m%d", localtime(time - 86400));
print "$Yday\n";

[rkb@mybox ~]$ ./date.pl
R081229

If you receive an error related to the use POSIX line, then your perl installation is corrupt. POSIX is a core module.

You may want to consider switching to one of the SFTP modules so you won't have to worry about setting the binary mode.

http://search.cpan.org/~dbrobins/Net-SFTP-0.10/lib/Net/SFTP.pm
http://search.cpan.org/~dbrobins/Net-SSH2-0.18/lib/Net/SSH2/SFTP.pm

Another option would be to use Net::SCP
http://search.cpan.org/~ivan/Net-SCP-0.08/SCP.pm
http://search.cpan.org/~rybskej/Net-SCP-Expect-0.14/Expect.pm

cuboidgraphix
User

Dec 30, 2008, 9:45 AM

Post #12 of 22 (19217 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Hi again Fishmonger,
I tried the code you last wrote and it worked clean for the most part. But I still get the same error I get with my script.

Code

Net::FTP=GLOB(0x9d6210)>>> user USERNAME 
Net::FTP=GLOB(0x9d6210)<<< 331 Password required. 
Net::FTP=GLOB(0x9d6210)>>> PASS .... 
Net::FTP=GLOB(0x9d6210)<<< 230 User logged in, proceed 
Net::FTP=GLOB(0x9d6210)>>> TYPE I 
Net::FTP=GLOB(0x9d6210)<<< 200 Command okay. 
changing cwd to D010AMAB 
Net::FTP=GLOB(0x9d6210)>>> CWD /D010AMAB 
Net::FTP=GLOB(0x9d6210)<<< 200 Directory changed. 
Directory changed. 
 
Net::FTP=GLOB(0x9d6210)>>> PORT 10,24,1,51,85,224 
Net::FTP=GLOB(0x9d6210)<<< 200 PORT Command okay 
Net::FTP=GLOB(0x9d6210)>>> NLST 
Net::FTP=GLOB(0x9d6210)<<< 503 TYPE must be ASCII 
changing cwd to D000AMAA 
Net::FTP=GLOB(0x9d6210)>>> CWD /D000AMAA 
Net::FTP=GLOB(0x9d6210)<<< 200 Directory changed. 
Directory changed.

It gives that error for every directory. < 503 TYPE must be ASCII.

Well I'm glad I've learned plenty from your help, but it sucks I can't get this script to work. I'll take your advice and look at the SFTP option. I don't know how long it will take me to write it, but I hope you're still around to guide me.

For all the rest of the forum, I hope this script helps them if they want it to work in ASCII code.

Code

#!/usr/bin/perl 
 
use strict; 
use warnings; 
use Net::FTP; 
use POSIX qw/strftime/; 
 
my $Yday = strftime("R%y%m%d", localtime(time - 86400)); 
 
my @myDIR =('D010AMAB', 'D000AMAA', 'D010AMA9', 'D000AMA8', 
            'D010AMA5', 'D000AMA6', 'D010AMA7', 'D000AMA4', 
            'D010AMA3', 'D000AMA2', 'D010AMA1', 'D000AMA0' 
); 
 
my $ftp = Net::FTP->new("HOST", Debug => 1) 
          or die "Failed to connect to FTP server "; 
 
print $ftp->message(), "\n"; 
 
$ftp->login("USER",'PASS') 
      or die "Failed to login to FTP server ", print $ftp->message(); 
 
#$ftp->binary or warn "Unable to switch to binary mode "; 
 
foreach my $DIR(@myDIR){ 
    print "changing cwd to $DIR\n"; 
    $ftp->cwd("/$DIR") 
          or warn "Failed to cd to /$DIR ", print $ftp->message() and next; 
    print $ftp->message(), "\n"; 
 
    my @file = grep { /^([^.].*)?$Yday/ } $ftp->ls(); 
    foreach my $file(@file){ 
        print "retriving file '$file'\n"; 
        $ftp->get($file) 
              or warn "Failed to retrieve '$file' ", print $ftp->message() and next; 
        print $ftp->message(), "\n"; 
    } 
}

FishMonger
Veteran / Moderator

Dec 30, 2008, 9:59 AM

Post #13 of 22 (19215 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

The problem is coming from this line:
my @file = grep { /^([^.].*)?$Yday/ } $ftp->ls();

The ls command needs to be in ascii mode. Do the setting of binary mode after that line. And since you're executing the ls command in each dir, you'll need to set it to ascii prior to calling $ftp>ls

Code

$ftp->ascii; 
my @file = grep { /^([^.].*)?$Yday/ } $ftp->ls(); 
$ftp->binary;

(This post was edited by FishMonger on Dec 30, 2008, 10:02 AM)

FishMonger
Veteran / Moderator

Dec 30, 2008, 10:08 AM

Post #14 of 22 (19213 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Actually, since ascii is the default mode, it might be more appropriate to but the $ftp->ascii after the inner foreach loop

cuboidgraphix
User

Dec 30, 2008, 11:14 AM

Post #15 of 22 (19210 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

Hey FishMonger... It worked..

Thanks a lot.

Your modifications worked.

God Bless You Always..

I believe this case is closed. :)

cuboidgraphix
User

Dec 30, 2008, 11:21 AM

Post #16 of 22 (19209 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

And here's the final code. All credits go to FishMonger.

Code

#!/usr/bin/perl 
 
use strict; 
use warnings; 
use Net::FTP; 
use POSIX qw/strftime/; 
 
my $Yday = strftime("R%y%m%d", localtime(time - 86400)); 
 
chdir "/home/AMA"; 
my @myDIR =('D010AMAB', 'D000AMAA', 'D010AMA9', 'D000AMA8', 
            'D010AMA5', 'D000AMA6', 'D010AMA7', 'D000AMA4', 
            'D010AMA3', 'D000AMA2', 'D010AMA1', 'D000AMA0' 
); 
 
my $ftp = Net::FTP->new("HOST", Debug => 1) 
          or die "Failed to connect to FTP server "; 
print $ftp->message(), "\n"; 
 
$ftp->login("USER",'PASS') 
      or die "Failed to login to FTP server "; 
print $ftp->message(), "\n"; 
 
foreach my $DIR(@myDIR){ 
    print "changing cwd to $DIR\n\n"; 
    $ftp->cwd("/$DIR") 
          or warn "Failed to cd to /$DIR ", print $ftp->message() and next; 
    print $ftp->message(); 
 
    $ftp->ascii or warn "Unable to switch to ascii mode "; 
    my @file = grep { /^([^.].*)?$Yday/ } $ftp->ls(); 
    $ftp->binary or warn "Unable to switch to binary mode "; 
    foreach my $file(@file){ 
        print "retriving file '$file'\n"; 
        $ftp->get($file) 
              or warn "Failed to retrieve '$file' ", print $ftp->message() and next; 
        print $ftp->message(); 
    } 
}

KevinR
Veteran

Dec 30, 2008, 11:28 AM

Post #17 of 22 (19206 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

Hey FishMonger... It worked..

Thanks a lot.

Your modifications worked.

God Bless You Always..

I believe this case is closed. :)

Good work Fish. Cool

-------------------------------------------------

FishMonger
Veteran / Moderator

Dec 30, 2008, 11:34 AM

Post #18 of 22 (19203 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

Hey FishMonger... It worked..

Thanks a lot.

Your modifications worked.

God Bless You Always..

I believe this case is closed. :)

You're welcome, glad I was able to help.

FishMonger
Veteran / Moderator

Dec 30, 2008, 11:40 AM

Post #19 of 22 (19201 views)

Re: [KevinR] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

Good work Fish. Cool

Thanks

Hey, are you still monitoring the CF Perl fourm? It sure has slowed down A LOT since I became the mod. Unsure

Should I take that personally? Unsure

KevinR
Veteran

Dec 30, 2008, 11:50 AM

Post #20 of 22 (19198 views)

Re: [FishMonger] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

Good work Fish. Cool

Thanks

Hey, are you still monitoring the CF Perl fourm? It sure has slowed down A LOT since I became the mod. Unsure

Should I take that personally? Unsure

I didn't notice that it slowed down after you were made the moderator but now that you mention it...... Wink

Yea, I still monitor it, I posted a few times recently but it is slow, all the perl forums seem slow right now.
-------------------------------------------------

cuboidgraphix
User

Dec 30, 2008, 12:11 PM

Post #21 of 22 (19195 views)

Re: [KevinR] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

I'd like to help out with posting some scripts guys, but I'm afraid my scripts are really nasty/amateur looking and not as clean as FishMonger has written this script.

I've had little scripts I've been diddling with over the past month.. I'll gladly post them on this forum if you'd like.

FishMonger
Veteran / Moderator

Dec 30, 2008, 12:34 PM

Post #22 of 22 (19190 views)

Re: [cuboidgraphix] FTP to multiple directories and finding specific files. [In reply to]	Can't Post

In Reply To

I'd like to help out with posting some scripts guys, but I'm afraid my scripts are really nasty/amateur looking and not as clean as FishMonger has written this script.

Don't be too concerned about that at this point; it takes time to learn. When I first started out helping on the Perl forums, my code was functional, but pretty poor quality.

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives