CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
Search Posts SEARCH
Who's Online WHO'S
Log in LOG

Home: Perl Programming Help: Intermediate:
LWP::Useragent max_size doesn't work



Jan 31, 2010, 7:50 AM

Post #1 of 2 (1012 views)
LWP::Useragent max_size doesn't work Can't Post

Hi, I have this program


use LWP::UserAgent;

my $ua = LWP::UserAgent->new();

my $request = new HTTP::Request('GET', '');

$response = $ua->request($request);

if ($response->is_success) {
print $response->content; # or whatever
else {
die $response->status_line;

When I run it, I get google page source, but no 1024 bytes, as I expect, but

# perl |wc -c 

Where's problem?


Feb 1, 2010, 9:35 PM

Post #2 of 2 (991 views)
Re: [xchg] LWP::Useragent max_size doesn't work [In reply to] Can't Post

Might be this:

$ua->max_size( $bytes )
Get/set the size limit for response content. The default is undef, which means that there is no limit. If the returned response content is only partial, because the size limit was exceeded, then a "Client-Aborted" header will be added to the response. The content might end up longer than max_size....

Data sent over a network(e.g. by a server) is broken up into chunks of unknown size by the operating system. So for instance if the server sends 10,000 bytes of data to the client, that may get broken up into chunks of 1,000 bytes; 5,000 bytes; and 4,000 bytes. It sounds like the way things work is that the ua reads a chunk of data, then checks whether the max_size has been surpassed. If so, the ua stops reading data. Therefore, if the max_size is 1,024 bytes, and the ua reads the first chunk it receives over the network, and the first chunk is 1,000 bytes, the ua will keep reading data. If the next chunk that comes across the network is 5,000 bytes, then it sounds like the ua will read the whole chunk, and then the ua will record that it has read a total of 6,000 bytes. Next, the ua will check the total read so for against max_size, and because max_size is 1,024 bytes, the ua will stop reading. However, the content length will be 6,000 bytes--not max_size bytes.

If all you want is 1,024 bytes, then you should do something like this:

my $content = substr $response->content(), 0, 1024;

It seems kind of stupid that the ua doesn't just add that line of code itself--but there must be some good reason the designer chose not to do that.

(This post was edited by 7stud on Feb 2, 2010, 3:26 AM)


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives