CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
LWP::Useragent max_size doesn't work

 



xchg
Novice

Jan 31, 2010, 7:50 AM

Post #1 of 2 (1012 views)
LWP::Useragent max_size doesn't work Can't Post

Hi, I have this program


Code
#!/usr/bin/perl 

use LWP::UserAgent;

my $ua = LWP::UserAgent->new();
$ua->timeout(10);
$ua->env_proxy;
$ua->max_size(1024);

my $request = new HTTP::Request('GET', 'http://google.sk');

$response = $ua->request($request);

if ($response->is_success) {
print $response->content; # or whatever
}
else {
die $response->status_line;
}


When I run it, I get google page source, but no 1024 bytes, as I expect, but

Code
# perl dtest.pl |wc -c 
2225


Where's problem?


7stud
Enthusiast

Feb 1, 2010, 9:35 PM

Post #2 of 2 (991 views)
Re: [xchg] LWP::Useragent max_size doesn't work [In reply to] Can't Post

Might be this:


Quote
$ua->max_size( $bytes )
Get/set the size limit for response content. The default is undef, which means that there is no limit. If the returned response content is only partial, because the size limit was exceeded, then a "Client-Aborted" header will be added to the response. The content might end up longer than max_size....


Data sent over a network(e.g. by a server) is broken up into chunks of unknown size by the operating system. So for instance if the server sends 10,000 bytes of data to the client, that may get broken up into chunks of 1,000 bytes; 5,000 bytes; and 4,000 bytes. It sounds like the way things work is that the ua reads a chunk of data, then checks whether the max_size has been surpassed. If so, the ua stops reading data. Therefore, if the max_size is 1,024 bytes, and the ua reads the first chunk it receives over the network, and the first chunk is 1,000 bytes, the ua will keep reading data. If the next chunk that comes across the network is 5,000 bytes, then it sounds like the ua will read the whole chunk, and then the ua will record that it has read a total of 6,000 bytes. Next, the ua will check the total read so for against max_size, and because max_size is 1,024 bytes, the ua will stop reading. However, the content length will be 6,000 bytes--not max_size bytes.

If all you want is 1,024 bytes, then you should do something like this:


Code
my $content = substr $response->content(), 0, 1024;


It seems kind of stupid that the ua doesn't just add that line of code itself--but there must be some good reason the designer chose not to do that.


(This post was edited by 7stud on Feb 2, 2010, 3:26 AM)

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives