CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
LWP Browser->Get Challenge

 

First page Previous page 1 2 Next page Last page  View All


rkellerjr
Novice

Aug 29, 2013, 5:14 AM

Post #26 of 34 (858 views)
Re: [FishMonger] LWP Browser->Get Challenge [In reply to] Can't Post

Each file has different content so they are not the same.


FishMonger
Veteran / Moderator

Aug 29, 2013, 6:34 AM

Post #27 of 34 (854 views)
Re: [rkellerjr] LWP Browser->Get Challenge [In reply to] Can't Post

Is the contents of each file what you expect upto that point in the file, or do they contain data that you didn't expect?

Can you give me the link so that I can run some tests?

This response header sent up a red flag for me.

Quote
X-Died: read failed: Inappropriate I/O control operation at C:/perl5_16/site/lib/LWP/Protocol/http.pm line 414.


First, it's telling you that there was an I/O error and that error is most likely the reason your data was truncated.

Second, it's coming from http.pm and since you're accessing an https page, I'd expect the error to come from https.pm.

Try adding protocols_allowed([ 'https' ]) to the constructor to see if that makes any difference.


rkellerjr
Novice

Aug 29, 2013, 6:46 AM

Post #28 of 34 (852 views)
Re: [FishMonger] LWP Browser->Get Challenge [In reply to] Can't Post

I wish I could give you the link Ron but our agreement with the customer doesn't allow me to.

This morning I decided to modify the program and download all of the XML files to see what would happen. There are five data groups I download and we've only been discussing one. What I found was that not everything is 16k. I get some at 27k, 18k, 19k, etc. and some are complete files while most are not.

It does look like they are cutting off my download stream. I'll research how to add the protocol and let you know the results.

By the way, a big thanks for sticking with me on this.


rkellerjr
Novice

Aug 29, 2013, 6:59 AM

Post #29 of 34 (850 views)
Re: [rkellerjr] LWP Browser->Get Challenge [In reply to] Can't Post

OK, that didn't change anything. I do have the https.pm in the LWP\protocol folder. Got the same "X-Died: read failed" in http.pm. The line of code I added was...

Code
$browser->protocols_allowed(['https']);



FishMonger
Veteran / Moderator

Aug 29, 2013, 7:02 AM

Post #30 of 34 (848 views)
Re: [rkellerjr] LWP Browser->Get Challenge [In reply to] Can't Post

You may want to add one or more callback handlers that monitor different parts of the transmission.

http://search.cpan.org/~gaas/libwww-perl-6.05/lib/LWP/UserAgent.pm#Handlers


rkellerjr
Novice

Aug 29, 2013, 7:32 AM

Post #31 of 34 (845 views)
Re: [FishMonger] LWP Browser->Get Challenge [In reply to] Can't Post

Oh boy, something else to learn :)


rkellerjr
Novice

Aug 29, 2013, 8:21 AM

Post #32 of 34 (841 views)
Re: [rkellerjr] LWP Browser->Get Challenge [In reply to] Can't Post

Here's something interesting. I turned on the show_progress and notice the first two lines of the first call to download the first page. Ring any bells? I have an e-mail to the client to verify our login/password information but, if I do not have that right, why would it download partial pages? The client indicates they have changed nothing on their end.

** GET https://dataserve.epitomy.com/users/ari/models/HAY/?page=1 ==> 401 Unauthorized (3s)
** GET https://dataserve.epitomy.com/users/ari/models/HAY/?page=1 ==> 200 OK (1s)
Cache-Control: private
Date: Thu, 29 Aug 2013 15:06:39 GMT
Server: Microsoft-IIS/7.0
Content-Length: 40674
Content-Type: application/xml; charset=utf-8
Client-Aborted: die
Client-Date: Thu, 29 Aug 2013 15:06:39 GMT
Client-Peer: 195.10.226.140:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /CN=Emerald
Client-SSL-Cert-Subject: /CN=Emerald
Client-SSL-Cipher: AES256-SHA
Client-SSL-Socket-Class: IO::Socket::SSL
Client-SSL-Warning: Peer hostname match with certificate not verified
Set-Cookie: ASP.NET_SessionId=csegdurfxyeolbvxfcbxwodf; path=/; HttpOnly
X-AspNet-Version: 4.0.30319
X-Died: read failed: Inappropriate I/O control operation at C:/perl5_16/site/lib/LWP/Protocol/http.pm line 414.
X-Powered-By: ASP.NET


FishMonger
Veteran / Moderator

Aug 29, 2013, 9:11 AM

Post #33 of 34 (835 views)
Re: [rkellerjr] LWP Browser->Get Challenge [In reply to] Can't Post

It appears that it's having an initial problem with the authentication and does a redo and the second attempt succeeds.

Then while reading the data it encounters an I/O error and dies with the "read failed" error.

The relevant code is:

Code
    my $complete; 
$response = $self->collect($arg, $response, sub {
my $buf = ""; #prevent use of uninitialized value in SSLeay.xs
my $n;
READ:
{
$n = $socket->read_entity_body($buf, $size);
unless (defined $n) {
redo READ if $!{EINTR} || $!{EAGAIN};
die "read failed: $!"; # <-- line 414
}
redo READ if $n == -1;
}
$complete++ if !$n;
return \$buf;


Since I can't do any direct testing, I'm not sure what code changes to suggest.

My best suggestion is to post a question on perlmonks. A good number of them are well known module authors and maintainers.

The only other suggestion would be to use a higher level module that subclasses these LWP modules. Specifically, I'm referring to WWW::Mechanize which is the most commonly used module for navigating web pages.

http://search.cpan.org/~ether/WWW-Mechanize-1.73/lib/WWW/Mechanize.pm


rkellerjr
Novice

Aug 29, 2013, 9:25 AM

Post #34 of 34 (833 views)
Re: [FishMonger] LWP Browser->Get Challenge [In reply to] Can't Post

Thanks Ron. It was also suggested that I attempt to use cURL instead of LWP and I have gotten it to work with that so I'll use cURL for now.

Thanks for your patience and your help.

First page Previous page 1 2 Next page Last page  View All
 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives