CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX 		Search Posts SEARCH
POSTS 		Who's Online WHO'S
ONLINE 		Log in LOG
IN

Home: Perl Programming Help: Intermediate:
How to check an URL instead of using HEAD

 


angeloulivieri
Novice

Mar 21, 2014, 5:45 AM

Post #1 of 11 (6030 views)
How to check an URL instead of using HEAD Can't Post
Hi all, this code on my computer is not working

Code
  use LWP::UserAgent; 
 use HTTP::Headers; 
 
my $link = "ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/reldate.txt"; 
 
simple_get($link); 
 
sub simple_get { 
        my $url = shift; 
 
        use HTTP::Request; 
        use LWP::UserAgent; 
        my $ua = new LWP::UserAgent; 
        my $request = HTTP::Request->new(HEAD => $url); 
        my $response = $ua->request($request); 
        my $error = 1 unless ($response->is_success); 
 
        if ($error){ 
                print "There is an error\n"; 
        }else{ 
                print "OK!\n"; 
        } 
}
It returns: There is an error Even if I know that the link is correct, I told to the mantainers of Uniprot.org but they said that for them it's fine.

I waswondering if there is a different way from HEAD to check if a URL is working. Because actually I want to check the link and, after a while, download it. For the download I find out the FILE::FETCH module that helped me. For the URL check I can't find anything else...

Thank you all

Angelo


(This post was edited by FishMonger on Mar 21, 2014, 6:01 AM)


FishMonger
Veteran / Moderator

Mar 21, 2014, 6:26 AM

Post #2 of 11 (6024 views)
Re: [angeloulivieri] How to check an URL instead of using HEAD [In reply to] Can't Post
Your code works for me.

I added a couple additional lines to download and print the file.

Code
c:\test>type angeloulivieri.pl 
Code
use strict; 
use warnings; 
use LWP::UserAgent; 
use HTTP::Headers; 
use LWP::Simple; 
 
my $link = "ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/reldate.txt"; 
 
simple_get($link); 
 
sub simple_get { 
        my $url = shift; 
 
        use HTTP::Request; 
        use LWP::UserAgent; 
        my $ua = new LWP::UserAgent; 
        my $request = HTTP::Request->new(HEAD => $url); 
        my $response = $ua->request($request); 
        my $error = 1 unless ($response->is_success); 
 
        if ($error){ 
                print "There is an error\n"; 
        }else{ 
            getprint($url); 
                #print "OK!\n"; 
        } 
}
 
c:\test>angeloulivieri.pl 
UniProt Knowledgebase Release 2014_03 consists of: 
UniProtKB/Swiss-Prot Release 2014_03 of 19-Mar-2014 
UniProtKB/TrEMBL Release 2014_03 of 19-Mar-2014 


	

(This post was edited by FishMonger on Mar 21, 2014, 6:27 AM)


angeloulivieri
Novice

Mar 21, 2014, 6:42 AM

Post #3 of 11 (6020 views)
Re: [FishMonger] How to check an URL instead of using HEAD [In reply to] Can't Post
Sorry, I've not been clear. I know that this code is not working only for me so I wanted to know if there's another way to check if a link is working.


FishMonger
Veteran / Moderator

Mar 21, 2014, 7:09 AM

Post #4 of 11 (6016 views)
Re: [angeloulivieri] How to check an URL instead of using HEAD [In reply to] Can't Post
The first step would be to verify that the url works in the browser, which is what I did before testing your script. I suspect that it will fail for you possibly due to firewall or proxy settings.

If it works in the browser, but not in the script, then you need to check the return code and status message to find out why it failed.

HTTP::Response - HTTP style response message https://metacpan.org/pod/HTTP::Response
HTTP::Status - HTTP Status code processing https://metacpan.org/pod/HTTP::Status
HTTP::Message - HTTP style message (base class) https://metacpan.org/pod/HTTP::Message


angeloulivieri
Novice

Mar 21, 2014, 7:21 AM

Post #5 of 11 (6014 views)
Re: [FishMonger] How to check an URL instead of using HEAD [In reply to] Can't Post
I tried with this code
sub simple_get {
my $url = shift;
my $ua = LWP::UserAgent->new;
my $request = HTTP::Request->new(GET => $url);
my $response = $ua->request($request);
warn "ERROR!" unless $response->is_success;
print $response->as_string;
}
And I got the output:

404 File 'reldate.txt' not found
Client-Date: Thu, 20 Mar 2014 16:50:25 GMT

This means that the resource is not there. But it is there!
So what there can be? Can it be possible that my DNS is not working?


FishMonger
Veteran / Moderator

Mar 21, 2014, 7:25 AM

Post #6 of 11 (6012 views)
Re: [angeloulivieri] How to check an URL instead of using HEAD [In reply to] Can't Post
Have your tried doing nslookup, ping and tracert tests to ftp.uniprot.org?

If those are successful, then run an nmap scan to see what ports are open.

Quote
c:\test>nmap ftp.uniprot.org

Starting Nmap 6.25 ( http://nmap.org ) at 2014-03-21 07:26 Pacific Daylight Time
Nmap scan report for ftp.uniprot.org (141.161.180.197)
Host is up (0.051s latency).
Not shown: 996 filtered ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
80/tcp open http
443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 8.11 seconds
nmap download link: http://nmap.org/download


(This post was edited by FishMonger on Mar 21, 2014, 7:29 AM)


angeloulivieri
Novice

Mar 21, 2014, 8:34 AM

Post #7 of 11 (6007 views)
Re: [FishMonger] How to check an URL instead of using HEAD [In reply to] Can't Post
I did it and I got the same result you had...

The website is working!

The most important question now:
Is there a way in PERL, different from using HEAD function, to look if a URL is working?


FishMonger
Veteran / Moderator

Mar 21, 2014, 8:50 AM

Post #8 of 11 (6004 views)
Re: [angeloulivieri] How to check an URL instead of using HEAD [In reply to] Can't Post
What's wrong with using HEAD?

Download the head section is the most efficient way to find out if you can access the url. But if you want another option you could download the entire content.

In either case, I'd normally use LWP::Simple because of it's cleaner/shorter syntax.


angeloulivieri
Novice

Mar 21, 2014, 9:11 AM

Post #9 of 11 (6002 views)
Re: [FishMonger] How to check an URL instead of using HEAD [In reply to] Can't Post
Yes I was also using that but it is not working anymore.
I solved the problem of download by using FILE::FETCH.

I don't know why LWP::Simple is not working. Can it be a problem with my DNS? Can I manually change the DNS used by LWP with a PERL native function?


FishMonger
Veteran / Moderator

Mar 21, 2014, 9:25 AM

Post #10 of 11 (5996 views)
Re: [angeloulivieri] How to check an URL instead of using HEAD [In reply to] Can't Post
There is no FILE::FETCH module, but there is a File::Fetch module. Case is important!!

I can't say why you're having inconsistent results with the LWP:: modules, especially considering that File::Fetch uses the LWP::UserAgent module to retrieve the url.


angeloulivieri
Novice

Mar 21, 2014, 10:06 AM

Post #11 of 11 (5994 views)
Re: [FishMonger] How to check an URL instead of using HEAD [In reply to] Can't Post
Ok... thank you

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives