ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.9 by root, Wed Jun 4 13:51:53 2008 UTC vs.
Revision 1.16 by root, Fri Jun 6 12:57:48 2008 UTC

… …
8 8
9=head1 DESCRIPTION 9=head1 DESCRIPTION
10 10
11This module is an L<AnyEvent> user, you need to make sure that you use and 11This module is an L<AnyEvent> user, you need to make sure that you use and
12run a supported event loop. 12run a supported event loop.
13
14This module implements a simple, stateless and non-blocking HTTP
15client. It supports GET, POST and other request methods, cookies and more,
16all on a very low level. It can follow redirects supports proxies and
17automatically limits the number of connections to the values specified in
18the RFC.
19
20It should generally be a "good client" that is enough for most HTTP
21tasks. Simple tasks should be simple, but complex tasks should still be
22possible as the user retains control over request and response headers.
23
24The caller is responsible for authentication management, cookies (if
25the simplistic implementation in this module doesn't suffice), referer
26and other high-level protocol details for which this module offers only
27limited support.
13 28
14=head2 METHODS 29=head2 METHODS
15 30
16=over 4 31=over 4
17 32
… …
29use AnyEvent::Socket (); 44use AnyEvent::Socket ();
30use AnyEvent::Handle (); 45use AnyEvent::Handle ();
31 46
32use base Exporter::; 47use base Exporter::;
33 48
34our $VERSION = '1.0'; 49our $VERSION = '1.01';
35 50
36our @EXPORT = qw(http_get http_request); 51our @EXPORT = qw(http_get http_request);
37 52
38our $USERAGENT = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 53our $USERAGENT = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
39our $MAX_RECURSE = 10; 54our $MAX_RECURSE = 10;
… …
41our $PERSISTENT_TIMEOUT = 2; 56our $PERSISTENT_TIMEOUT = 2;
42our $TIMEOUT = 300; 57our $TIMEOUT = 300;
43 58
44# changing these is evil 59# changing these is evil
45our $MAX_PERSISTENT_PER_HOST = 2; 60our $MAX_PERSISTENT_PER_HOST = 2;
46our $MAX_PER_HOST = 4; # not respected yet :( 61our $MAX_PER_HOST = 4;
47 62
48our $PROXY; 63our $PROXY;
64our $ACTIVE = 0;
49 65
50my %KA_COUNT; # number of open keep-alive connections per host 66my %KA_COUNT; # number of open keep-alive connections per host
67my %CO_SLOT; # number of open connections, and wait queue, per host
51 68
52=item http_get $url, key => value..., $cb->($data, $headers) 69=item http_get $url, key => value..., $cb->($data, $headers)
53 70
54Executes an HTTP-GET request. See the http_request function for details on 71Executes an HTTP-GET request. See the http_request function for details on
55additional parameters. 72additional parameters.
… …
74response headers as second argument. 91response headers as second argument.
75 92
76All the headers in that hash are lowercased. In addition to the response 93All the headers in that hash are lowercased. In addition to the response
77headers, the three "pseudo-headers" C<HTTPVersion>, C<Status> and 94headers, the three "pseudo-headers" C<HTTPVersion>, C<Status> and
78C<Reason> contain the three parts of the HTTP Status-Line of the same 95C<Reason> contain the three parts of the HTTP Status-Line of the same
79name. 96name. If the server sends a header multiple lines, then their contents
97will be joined together with C<\x00>.
80 98
81If an internal error occurs, such as not being able to resolve a hostname, 99If an internal error occurs, such as not being able to resolve a hostname,
82then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599> 100then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599>
83and the C<Reason> pseudo-header will contain an error message. 101and the C<Reason> pseudo-header will contain an error message.
84 102
… …
104Whether to recurse requests or not, e.g. on redirects, authentication 122Whether to recurse requests or not, e.g. on redirects, authentication
105retries and so on, and how often to do so. 123retries and so on, and how often to do so.
106 124
107=item headers => hashref 125=item headers => hashref
108 126
109The request headers to use. 127The request headers to use. Currently, C<http_request> may provide its
128own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
129and will provide defaults for C<User-Agent:> and C<Referer:>.
110 130
111=item timeout => $seconds 131=item timeout => $seconds
112 132
113The time-out to use for various stages - each connect attempt will reset 133The time-out to use for various stages - each connect attempt will reset
114the timeout, as will read or write activity. Default timeout is 5 minutes. 134the timeout, as will read or write activity. Default timeout is 5 minutes.
… …
123 143
124=item body => $string 144=item body => $string
125 145
126The request body, usually empty. Will be-sent as-is (future versions of 146The request body, usually empty. Will be-sent as-is (future versions of
127this module might offer more options). 147this module might offer more options).
148
149=item cookie_jar => $hash_ref
150
151Passing this parameter enables (simplified) cookie-processing, loosely
152based on the original netscape specification.
153
154The C<$hash_ref> must be an (initially empty) hash reference which will
155get updated automatically. It is possible to save the cookie_jar to
156persistent storage with something like JSON or Storable, but this is not
157recommended, as expire times are currently being ignored.
158
159Note that this cookie implementation is not of very high quality, nor
160meant to be complete. If you want complete cookie management you have to
161do that on your own. C<cookie_jar> is meant as a quick fix to get some
162cookie-using sites working. Cookies are a privacy disaster, do not use
163them unless required to.
128 164
129=back 165=back
130 166
131Example: make a simple HTTP GET request for http://www.nethype.de/ 167Example: make a simple HTTP GET request for http://www.nethype.de/
132 168
… …
148 } 184 }
149 ; 185 ;
150 186
151=cut 187=cut
152 188
189sub _slot_schedule;
190sub _slot_schedule($) {
191 my $host = shift;
192
193 while ($CO_SLOT{$host}[0] < $MAX_PER_HOST) {
194 if (my $cb = shift @{ $CO_SLOT{$host}[1] }) {
195 # somebody wants that slot
196 ++$CO_SLOT{$host}[0];
197 ++$ACTIVE;
198
199 $cb->(AnyEvent::Util::guard {
200 --$ACTIVE;
201 --$CO_SLOT{$host}[0];
202 _slot_schedule $host;
203 });
204 } else {
205 # nobody wants the slot, maybe we can forget about it
206 delete $CO_SLOT{$host} unless $CO_SLOT{$host}[0];
207 last;
208 }
209 }
210}
211
212# wait for a free slot on host, call callback
213sub _get_slot($$) {
214 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
215
216 _slot_schedule $_[0];
217}
218
153sub http_request($$$;@) { 219sub http_request($$@) {
154 my $cb = pop; 220 my $cb = pop;
155 my ($method, $url, %arg) = @_; 221 my ($method, $url, %arg) = @_;
156 222
157 my %hdr; 223 my %hdr;
158 224
… …
172 my $proxy = $arg{proxy} || $PROXY; 238 my $proxy = $arg{proxy} || $PROXY;
173 my $timeout = $arg{timeout} || $TIMEOUT; 239 my $timeout = $arg{timeout} || $TIMEOUT;
174 240
175 $hdr{"user-agent"} ||= $USERAGENT; 241 $hdr{"user-agent"} ||= $USERAGENT;
176 242
177 my ($host, $port, $path, $scheme); 243 my ($scheme, $authority, $upath, $query, $fragment) =
244 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|;
245
246 $scheme = lc $scheme;
247
248 my $uport = $scheme eq "http" ? 80
249 : $scheme eq "https" ? 443
250 : return $cb->(undef, { Status => 599, Reason => "only http and https URL schemes supported" });
251
252 $hdr{referer} ||= "$scheme://$authority$upath"; # leave out fragment and query string, just a heuristic
253
254 $authority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
255 or return $cb->(undef, { Status => 599, Reason => "unparsable URL" });
256
257 my $uhost = $1;
258 $uport = $2 if defined $2;
259
260 $uhost =~ s/^\[(.*)\]$/$1/;
261 $upath .= "?$query" if length $query;
262
263 $upath =~ s%^/?%/%;
264
265 # cookie processing
266 if (my $jar = $arg{cookie_jar}) {
267 %$jar = () if $jar->{version} < 1;
268
269 my @cookie;
270
271 while (my ($chost, $v) = each %$jar) {
272 next unless $chost eq substr $uhost, -length $chost;
273 next unless $chost =~ /^\./;
274
275 while (my ($cpath, $v) = each %$v) {
276 next unless $cpath eq substr $upath, 0, length $cpath;
277
278 while (my ($k, $v) = each %$v) {
279 next if $scheme ne "https" && exists $v->{secure};
280 push @cookie, "$k=$v->{value}";
281 }
282 }
283 }
284
285 $hdr{cookie} = join "; ", @cookie
286 if @cookie;
287 }
288
289 my ($rhost, $rport, $rpath); # request host, port, path
178 290
179 if ($proxy) { 291 if ($proxy) {
180 ($host, $port, $scheme) = @$proxy; 292 ($rhost, $rport, $scheme) = @$proxy;
181 $path = $url; 293 $rpath = $url;
182 } else { 294 } else {
183 ($scheme, my $authority, $path, my $query, my $fragment) = 295 ($rhost, $rport, $rpath) = ($uhost, $uport, $upath);
184 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|;
185
186 $port = $scheme eq "http" ? 80
187 : $scheme eq "https" ? 443
188 : return $cb->(undef, { Status => 599, Reason => "$url: only http and https URLs supported" });
189
190 $authority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
191 or return $cb->(undef, { Status => 599, Reason => "$url: unparsable URL" });
192
193 $host = $1;
194 $port = $2 if defined $2;
195
196 $host =~ s/^\[(.*)\]$/$1/;
197 $path .= "?$query" if length $query;
198
199 $path = "/" unless $path;
200
201 $hdr{host} = $host = lc $host; 296 $hdr{host} = $uhost;
202 } 297 }
203 298
204 $scheme = lc $scheme;
205
206 my %state;
207
208 $hdr{"content-length"} = length $arg{body}; 299 $hdr{"content-length"} = length $arg{body};
209 300
301 my %state = (connect_guard => 1);
302
303 _get_slot $uhost, sub {
304 $state{slot_guard} = shift;
305
306 return unless $state{connect_guard};
307
210 $state{connect_guard} = AnyEvent::Socket::tcp_connect $host, $port, sub { 308 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
211 $state{fh} = shift 309 $state{fh} = shift
212 or return $cb->(undef, { Status => 599, Reason => "$!" }); 310 or return $cb->(undef, { Status => 599, Reason => "$!" });
213 311
214 delete $state{connect_guard}; # reduce memory usage, save a tree 312 delete $state{connect_guard}; # reduce memory usage, save a tree
215 313
216 # get handle 314 # get handle
217 $state{handle} = new AnyEvent::Handle 315 $state{handle} = new AnyEvent::Handle
218 fh => $state{fh}, 316 fh => $state{fh},
219 ($scheme eq "https" ? (tls => "connect") : ()); 317 ($scheme eq "https" ? (tls => "connect") : ());
220 318
221 # limit the number of persistent connections 319 # limit the number of persistent connections
222 if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) { 320 if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
223 ++$KA_COUNT{$_[1]}; 321 ++$KA_COUNT{$_[1]};
224 $state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} }; 322 $state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} };
225 $hdr{connection} = "keep-alive"; 323 $hdr{connection} = "keep-alive";
226 delete $hdr{connection}; # keep-alive not yet supported 324 delete $hdr{connection}; # keep-alive not yet supported
227 } else { 325 } else {
228 delete $hdr{connection}; 326 delete $hdr{connection};
229 } 327 }
230 328
231 # (re-)configure handle 329 # (re-)configure handle
232 $state{handle}->timeout ($timeout); 330 $state{handle}->timeout ($timeout);
233 $state{handle}->on_error (sub { 331 $state{handle}->on_error (sub {
332 my $errno = "$!";
234 %state = (); 333 %state = ();
235 $cb->(undef, { Status => 599, Reason => "$!" }); 334 $cb->(undef, { Status => 599, Reason => $errno });
236 }); 335 });
237 $state{handle}->on_eof (sub { 336 $state{handle}->on_eof (sub {
238 %state = (); 337 %state = ();
239 $cb->(undef, { Status => 599, Reason => "unexpected end-of-file" }); 338 $cb->(undef, { Status => 599, Reason => "unexpected end-of-file" });
240 }); 339 });
241 340
242 # send request 341 # send request
243 $state{handle}->push_write ( 342 $state{handle}->push_write (
244 "$method $path HTTP/1.0\015\012" 343 "$method $rpath HTTP/1.0\015\012"
245 . (join "", map "$_: $hdr{$_}\015\012", keys %hdr) 344 . (join "", map "$_: $hdr{$_}\015\012", keys %hdr)
246 . "\015\012" 345 . "\015\012"
247 . (delete $arg{body}) 346 . (delete $arg{body})
248 );
249
250 %hdr = (); # reduce memory usage, save a kitten
251
252 # status line
253 $state{handle}->push_read (line => qr/\015?\012/, sub {
254 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) \s+ ([^\015\012]+)/ix
255 or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])" }));
256
257 my %hdr = ( # response headers
258 HTTPVersion => ",$1",
259 Status => ",$2",
260 Reason => ",$3",
261 ); 347 );
262 348
349 %hdr = (); # reduce memory usage, save a kitten
350
351 # status line
352 $state{handle}->push_read (line => qr/\015?\012/, sub {
353 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) \s+ ([^\015\012]+)/ix
354 or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])" }));
355
356 my %hdr = ( # response headers
357 HTTPVersion => "\x00$1",
358 Status => "\x00$2",
359 Reason => "\x00$3",
360 );
361
263 # headers, could be optimized a bit 362 # headers, could be optimized a bit
264 $state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub { 363 $state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub {
265 for ("$_[1]\012") { 364 for ("$_[1]\012") {
266 # we support spaces in field names, as lotus domino 365 # we support spaces in field names, as lotus domino
267 # creates them. 366 # creates them.
268 $hdr{lc $1} .= ",$2" 367 $hdr{lc $1} .= "\x00$2"
269 while /\G 368 while /\G
270 ([^:\000-\037]+): 369 ([^:\000-\037]+):
271 [\011\040]* 370 [\011\040]*
272 ((?: [^\015\012]+ | \015?\012[\011\040] )*) 371 ((?: [^\015\012]+ | \015?\012[\011\040] )*)
273 \015?\012 372 \015?\012
274 /gxc; 373 /gxc;
275 374
276 /\G$/ 375 /\G$/
277 or return $cb->(undef, { Status => 599, Reason => "garbled response headers" }); 376 or return (%state = (), $cb->(undef, { Status => 599, Reason => "garbled response headers" }));
278 } 377 }
279 378
280 substr $_, 0, 1, "" 379 substr $_, 0, 1, ""
281 for values %hdr; 380 for values %hdr;
282 381
283 my $finish = sub { 382 my $finish = sub {
383 %state = ();
384
385 # set-cookie processing
386 if ($arg{cookie_jar} && exists $hdr{"set-cookie"}) {
387 for (split /\x00/, $hdr{"set-cookie"}) {
388 my ($cookie, @arg) = split /;\s*/;
389 my ($name, $value) = split /=/, $cookie, 2;
390 my %kv = (value => $value, map { split /=/, $_, 2 } @arg);
391
392 my $cdom = (delete $kv{domain}) || $uhost;
393 my $cpath = (delete $kv{path}) || "/";
394
395 $cdom =~ s/^.?/./; # make sure it starts with a "."
396
397 next if $cdom =~ /\.$/;
398
399 # this is not rfc-like and not netscape-like. go figure.
400 my $ndots = $cdom =~ y/.//;
401 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
402
403 # store it
404 $arg{cookie_jar}{version} = 1;
405 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
406 }
407 }
408
284 if ($_[1]{Status} =~ /^30[12]$/ && $recurse) { 409 if ($_[1]{Status} =~ /^30[12]$/ && $recurse) {
410 # microsoft and other assholes don't give a shit for following standards,
411 # try to support a common form of broken Location header.
412 $_[1]{location} =~ s%^/%$scheme://$uhost:$uport/%;
413
285 http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb); 414 http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
415 } else {
416 $cb->($_[0], $_[1]);
417 }
418 };
419
420 if ($hdr{Status} =~ /^(?:1..|204|304)$/ or $method eq "HEAD") {
421 $finish->(undef, \%hdr);
286 } else { 422 } else {
287 $cb->($_[0], $_[1]); 423 if (exists $hdr{"content-length"}) {
424 $_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
425 # could cache persistent connection now
426 if ($hdr{connection} =~ /\bkeep-alive\b/i) {
427 # but we don't, due to misdesigns, this is annoyingly complex
428 };
429
430 $finish->($_[1], \%hdr);
431 });
432 } else {
433 # too bad, need to read until we get an error or EOF,
434 # no way to detect winged data.
435 $_[0]->on_error (sub {
436 $finish->($_[0]{rbuf}, \%hdr);
437 });
438 $_[0]->on_eof (undef);
439 $_[0]->on_read (sub { });
440 }
288 } 441 }
289 }; 442 });
290
291 if ($hdr{Status} =~ /^(?:1..|204|304)$/ or $method eq "HEAD") {
292 %state = ();
293 $finish->(undef, \%hdr);
294 } else {
295 if (exists $hdr{"content-length"}) {
296 $_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
297 # could cache persistent connection now
298 if ($hdr{connection} =~ /\bkeep-alive\b/i) {
299 # but we don't, due to misdesigns, this is annoyingly complex
300 };
301
302 %state = ();
303 $finish->($_[1], \%hdr);
304 });
305 } else {
306 # too bad, need to read until we get an error or EOF,
307 # no way to detect winged data.
308 $_[0]->on_error (sub {
309 %state = ();
310 $finish->($_[0]{rbuf}, \%hdr);
311 });
312 $_[0]->on_eof (undef);
313 $_[0]->on_read (sub { });
314 }
315 }
316 }); 443 });
444 }, sub {
445 $timeout
317 }); 446 };
318 }, sub {
319 $timeout
320 }; 447 };
321 448
322 defined wantarray && AnyEvent::Util::guard { %state = () } 449 defined wantarray && AnyEvent::Util::guard { %state = () }
323} 450}
324 451
325sub http_get($$;@) { 452sub http_get($@) {
326 unshift @_, "GET"; 453 unshift @_, "GET";
327 &http_request 454 &http_request
328} 455}
329 456
330sub http_head($$;@) { 457sub http_head($@) {
331 unshift @_, "HEAD"; 458 unshift @_, "HEAD";
332 &http_request 459 &http_request
333} 460}
334 461
335sub http_post($$$;@) { 462sub http_post($$@) {
336 unshift @_, "POST", "body"; 463 unshift @_, "POST", "body";
337 &http_request 464 &http_request
338} 465}
339 466
340=back 467=back
… …
367 494
368The maximum time to cache a persistent connection, in seconds (default: 2). 495The maximum time to cache a persistent connection, in seconds (default: 2).
369 496
370Not implemented currently. 497Not implemented currently.
371 498
499=item $AnyEvent::HTTP::ACTIVE
500
501The number of active connections. This is not the number of currently
502running requests, but the number of currently open and non-idle TCP
503connections. This number of can be useful for load-leveling.
504
372=back 505=back
373 506
374=cut 507=cut
375 508
376sub set_proxy($) { 509sub set_proxy($) {

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines