CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
issues with Mozrepl running wiht WWW::Mechanize::Firefox

 



dilbert
User

Oct 25, 2012, 8:51 AM

Post #1 of 6 (1780 views)
issues with Mozrepl running wiht WWW::Mechanize::Firefox Can't Post

good evening dear perlgurus


first of all i hope you are all right and everything is well on your site...

I have some troubles with a perl-script that turns out to be not 100% optimal.

now i am tryin to find a better solution either in Ruby / Python or PHP - but if you have ideas to re-work the perl-script. i would be glad too.

The question: Is there a way to specify Net::Telnet timeout with WWW::Mechanize::Firefox?
At the moment my internet connection [a quite fast dsl one] is very slow and sometimes I get error

Code
with $mech->get(): 
command timed-out at /usr/local/share/perl/5.12.3/MozRepl/Client.pm line 186

SEE THIS ONE: $mech->repl->repl->timeout(100000);


Unfortunatly it does not work: Can't locate object method "timeout" via package "MozRepl"

Documentation says this should:

Code
$mech->repl->repl->setup_client( { extra_client_args => { timeout => 1 +80 } } );


problem: I have a list of 2500 websites and need to grab a thumbnail screenshot (!) of them. How do I do that?
I could try to parse the sites either with Perl.- Mechanize would be a good thing. Note: i only need the results as a thumbnails that are a maximum 240 pixels in the long dimension. At the moment i have a solution which is slow and does not give back thumbnails: How to make the script running faster with less overhead - spiting out the thumbnails


My prerequisites: addon/mozrepl/
the module WWW::Mechanize::Firefox;
the module imager


This is my source ... see a snippet [example]of the sites i have in the url-list.

urls.txt [the list of sources in a file]


Code
www.google.com 
www.cnn.com
www.msnbc.com
news.bbc.co.uk
www.bing.com
www.yahoo.com - and so on and so forth...:


What i have tried allready; here it is:


Code
 
#!/usr/bin/perl

use strict;
use warnings;
use WWW::Mechanize::Firefox;

my $mech = new WWW::Mechanize::Firefox();

open(INPUT, "<urls.txt") or die $!;

while (<INPUT>) {
chomp;
print "$_\n";
$mech->get($_);
my $png = $mech->content_as_png();
my $name = "$_";
$name =~s/^www\.//;
$name .= ".png";
open(OUTPUT, ">$name");
print OUTPUT $png;
sleep (5);
}


Code
 
Well this does not care about the size:

See the output commandline:


Code
linux-vi17:/home/martin/perl # perl mecha_test_1.pl 
www.google.com
www.cnn.com
www.msnbc.com
command timed-out at /usr/lib/perl5/site_perl/5.12.3/MozRepl/Client.pm line 186
linux-vi17:/home/martin/perl #


Question: how to extend the solution either to make sure that it does not stop in a time out.
Note again: i only need the results as a thumbnails that are a maximum 240 pixels in the long dimension.
As a prerequisites, i allready have installed the module imager.
How to make the script running faster with less overhead - spiting out the thumbnails

Update: in addition to the mothere is a Monksthread perlmonks.org/?node_id=901572


i also tried out this one here:


Code
$mech->repl->repl->setup_client( { extra_client_args => { timeout => 5*60 } } ); 

putting links to @list and use eval

while (scalar(@list)) {
my $link = pop(@list);
print "trying $link\n";
eval{
$mech->get($link);
sleep (5);
my $png = $mech->content_as_png();
my $name = "$_";
$name =~s/^www\.//;
$name .= ".png";
open(OUTPUT, ">$name");
print OUTPUT $png;
close(OUTPUT);
}
if ($@){
print "link: $link failed\n";
push(@list,$link);#put the end of the list
next;
}
print "$link is done!\n";

}


Question: is there a Ruby / Python /PHP-Solution that runs more efficient - or can you suggest a Perl-solution that is more stable..


Look forward to hear from you

Thx for any and all help in advance

have a great day

greetings


wickedxter
User

Oct 25, 2012, 1:15 PM

Post #2 of 6 (1775 views)
Re: [dilbert] issues with Mozrepl running wiht WWW::Mechanize::Firefox [In reply to] Can't Post

This works for me and is fast I also added Parallel::ForkManager if you need to use it uncomment the things out. This used about 18mb of memeory and 40% over 4 cores useing ubuntu 12.04 and perl 5.14.2


Code
#!/usr/bin/perl 

use WWW::Mechanize::Firefox;
use strict;
use warnings;
#use Parallel::ForkManager;



#my $fork = Parallel::ForkManager->new(2);

#sites
my @urls = qw(http://www.google.com http://www.yahoo.com http://www.cnn.com http://www.bing.com http://www.nbcnews.com/);
#temp base dir
my $temp = '/home/aaron/cgi-bin/';

for my $each (@urls){
#$fork->start and next;
my $mech = WWW::Mechanize::Firefox->new(launch => 'firefox',create => 1,);

$each =~ /www\.(\w+)\.com/;
my $name = $1;
print "creating $name.png\n";


$mech->get($each);
my $png = $mech->content_as_png(undef, undef, {width => 240,height => 240});

my $dir_name = "$temp/$name".".png";
open my $file, ">", "$dir_name" or die "couldnt create $temp/$1.png";
binmode $file;
print {$file} $png;

close $file;
sleep 5; # sleep some to give it a little time to make sure things compleated..... you'll need this more using fork

#$fork->finish;
}
print "Well All done!\n";
#$fork->wait_all_children;



(This post was edited by wickedxter on Oct 25, 2012, 1:57 PM)


dilbert
User

Oct 25, 2012, 2:59 PM

Post #3 of 6 (1768 views)
Re: [wickedxter] issues with Mozrepl running wiht WWW::Mechanize::Firefox [In reply to] Can't Post

hello dear wickedxter


many many thanks - great to hear from you....

all looks great and i cannot wait to test it.

many many greetings

dilbert


dilbert
User

Oct 25, 2012, 4:39 PM

Post #4 of 6 (1762 views)
Re: [dilbert] issues with Mozrepl running wiht WWW::Mechanize::Firefox [In reply to] Can't Post

hello have running mozrepl
but i ve gotten this error.

Code
 
martin@linux-wyee:~/perl> perl moz_test1.pl
Can't locate WWW/Mechanize/Firefox.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.16.0/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.16.0 /usr/lib/perl5/vendor_perl/5.16.0/i586-linux-thread-multi /usr/lib/perl5/vendor_perl/5.16.0 /usr/lib/perl5/5.16.0/i586-linux-thread-multi /usr/lib/perl5/5.16.0 /usr/lib/perl5/site_perl/5.16.0/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.16.0 /usr/lib/perl5/site_perl .) at moz_test1.pl line 4.
BEGIN failed--compilation aborted at moz_test1.pl line 4.
martin@linux-wyee:~/perl> ^C
martin@linux-
wyee:~/perl>


hmm - donno whats wrong here... try to fgure it out


wickedxter
User

Oct 25, 2012, 6:54 PM

Post #5 of 6 (1758 views)
Re: [dilbert] issues with Mozrepl running wiht WWW::Mechanize::Firefox [In reply to] Can't Post

thats saying it cant locate the module....

what linux r u using? is this on a home computer or server?

if its on ur own computer ull have to look threw the dir to find where the module was installed to.... on mine it went to the /home/aaron/perl5/lib/perl5/ is the base to the module installion for cpan and might match yours or something close.

add this module to the script at the top before the WWW::Mechanize::Firefox

Code
#add the dir to where you found the installion of the modules 
use lib ("/home/aaron/perl5/lib/perl5/");


this adds the directory to the @INC array for locations to search for the modules....


(This post was edited by wickedxter on Oct 25, 2012, 7:06 PM)


dilbert
User

Oct 25, 2012, 9:08 PM

Post #6 of 6 (1753 views)
Re: [wickedxter] issues with Mozrepl running wiht WWW::Mechanize::Firefox [In reply to] Can't Post

hello

this is a notebook (my own ) running opensuse 12.2

i try to add the module

i will come back later the weekend

[sorry i am very busy at teh moment]

many thanks
greetings
dilbert

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives