ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.42 by root, Mon Jul 6 00:08:16 2009 UTC vs.
Revision 1.50 by root, Tue Aug 11 13:20:42 2009 UTC

… …
41use strict; 41use strict;
42no warnings; 42no warnings;
43 43
44use Errno (); 44use Errno ();
45 45
46use AnyEvent 4.452 (); 46use AnyEvent 4.8 ();
47use AnyEvent::Util (); 47use AnyEvent::Util ();
48use AnyEvent::Socket (); 48use AnyEvent::Socket ();
49use AnyEvent::Handle (); 49use AnyEvent::Handle ();
50 50
51use base Exporter::; 51use base Exporter::;
52 52
53our $VERSION = '1.12'; 53our $VERSION = '1.42';
54 54
55our @EXPORT = qw(http_get http_post http_head http_request); 55our @EXPORT = qw(http_get http_post http_head http_request);
56 56
57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58our $MAX_RECURSE = 10; 58our $MAX_RECURSE = 10;
59our $MAX_PERSISTENT = 8; 59our $MAX_PERSISTENT = 8;
60our $PERSISTENT_TIMEOUT = 2; 60our $PERSISTENT_TIMEOUT = 2;
61our $TIMEOUT = 300; 61our $TIMEOUT = 300;
62 62
63# changing these is evil 63# changing these is evil
64our $MAX_PERSISTENT_PER_HOST = 2; 64our $MAX_PERSISTENT_PER_HOST = 0;
65our $MAX_PER_HOST = 4; 65our $MAX_PER_HOST = 4;
66 66
67our $PROXY; 67our $PROXY;
68our $ACTIVE = 0; 68our $ACTIVE = 0;
69 69
… …
138 138
139=item headers => hashref 139=item headers => hashref
140 140
141The request headers to use. Currently, C<http_request> may provide its 141The request headers to use. Currently, C<http_request> may provide its
142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers 142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
143and will provide defaults for C<User-Agent:> and C<Referer:>. 143and will provide defaults for C<User-Agent:> and C<Referer:> (this can be
144suppressed by using C<undef> for these headers in which case they won't be
145sent at all).
144 146
145=item timeout => $seconds 147=item timeout => $seconds
146 148
147The time-out to use for various stages - each connect attempt will reset 149The time-out to use for various stages - each connect attempt will reset
148the timeout, as will read or write activity. Default timeout is 5 minutes. 150the timeout, as will read or write activity. Default timeout is 5 minutes.
… …
150=item proxy => [$host, $port[, $scheme]] or undef 152=item proxy => [$host, $port[, $scheme]] or undef
151 153
152Use the given http proxy for all requests. If not specified, then the 154Use the given http proxy for all requests. If not specified, then the
153default proxy (as specified by C<$ENV{http_proxy}>) is used. 155default proxy (as specified by C<$ENV{http_proxy}>) is used.
154 156
155C<$scheme> must be either missing or C<http> for HTTP, or C<https> for 157C<$scheme> must be either missing, C<http> for HTTP or C<https> for
156HTTPS. 158HTTPS.
157 159
158=item body => $string 160=item body => $string
159 161
160The request body, usually empty. Will be-sent as-is (future versions of 162The request body, usually empty. Will be-sent as-is (future versions of
… …
221This callback is useful when the data is too large to be held in memory 223This callback is useful when the data is too large to be held in memory
222(so the callback writes it to a file) or when only some information should 224(so the callback writes it to a file) or when only some information should
223be extracted, or when the body should be processed incrementally. 225be extracted, or when the body should be processed incrementally.
224 226
225It is usually preferred over doing your own body handling via 227It is usually preferred over doing your own body handling via
226C<want_body_handle>. 228C<want_body_handle>, but in case of streaming APIs, where HTTP is
229only used to create a connection, C<want_body_handle> is the better
230alternative, as it allows you to install your own event handler, reducing
231resource usage.
227 232
228=item want_body_handle => $enable 233=item want_body_handle => $enable
229 234
230When enabled (default is disabled), the behaviour of AnyEvent::HTTP 235When enabled (default is disabled), the behaviour of AnyEvent::HTTP
231changes considerably: after parsing the headers, and instead of 236changes considerably: after parsing the headers, and instead of
… …
243This is useful with some push-type services, where, after the initial 248This is useful with some push-type services, where, after the initial
244headers, an interactive protocol is used (typical example would be the 249headers, an interactive protocol is used (typical example would be the
245push-style twitter API which starts a JSON/XML stream). 250push-style twitter API which starts a JSON/XML stream).
246 251
247If you think you need this, first have a look at C<on_body>, to see if 252If you think you need this, first have a look at C<on_body>, to see if
248that doesn'T solve your problem in a better way. 253that doesn't solve your problem in a better way.
249 254
250=back 255=back
251 256
252Example: make a simple HTTP GET request for http://www.nethype.de/ 257Example: make a simple HTTP GET request for http://www.nethype.de/
253 258
… …
309 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 314 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
310 315
311 _slot_schedule $_[0]; 316 _slot_schedule $_[0];
312} 317}
313 318
314our $qr_nl = qr<\015?\012>; 319our $qr_nl = qr{\015?\012};
315our $qr_nlnl = qr<\015?\012\015?\012>; 320our $qr_nlnl = qr{(?<![^\012])\015?\012};
316 321
317our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 }; 322our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
318our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" }; 323our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
319 324
320sub http_request($$@) { 325sub http_request($$@) {
… …
354 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 359 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
355 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url }); 360 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url });
356 361
357 my $uhost = $1; 362 my $uhost = $1;
358 $uport = $2 if defined $2; 363 $uport = $2 if defined $2;
364
365 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost";
359 366
360 $uhost =~ s/^\[(.*)\]$/$1/; 367 $uhost =~ s/^\[(.*)\]$/$1/;
361 $upath .= "?$query" if length $query; 368 $upath .= "?$query" if length $query;
362 369
363 $upath =~ s%^/?%/%; 370 $upath =~ s%^/?%/%;
… …
396 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path 403 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
397 404
398 if ($proxy) { 405 if ($proxy) {
399 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy); 406 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy);
400 407
408 $rscheme = "http" unless defined $rscheme;
409
401 # don't support https requests over https-proxy transport, 410 # don't support https requests over https-proxy transport,
402 # can't be done with tls as spec'ed, unless you double-encrypt. 411 # can't be done with tls as spec'ed, unless you double-encrypt.
403 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https"; 412 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
404 } else { 413 } else {
405 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath); 414 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
406 } 415 }
407 416
408 $hdr{"user-agent"} ||= $USERAGENT; 417 # leave out fragment and query string, just a heuristic
409 $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic 418 $hdr{referer} ||= "$uscheme://$uauthority$upath" unless exists $hdr{referer};
419 $hdr{"user-agent"} ||= $USERAGENT unless exists $hdr{"user-agent"};
410 420
411 $hdr{host} = "$uhost:$uport";
412 $hdr{"content-length"} = length $arg{body}; 421 $hdr{"content-length"} = length $arg{body};
413 422
414 my %state = (connect_guard => 1); 423 my %state = (connect_guard => 1);
415 424
416 _get_slot $uhost, sub { 425 _get_slot $uhost, sub {
… …
418 427
419 return unless $state{connect_guard}; 428 return unless $state{connect_guard};
420 429
421 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub { 430 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
422 $state{fh} = shift 431 $state{fh} = shift
432 or do {
433 my $err = "$!";
434 %state = ();
423 or return (%state = (), $cb->(undef, { Status => 599, Reason => "$!", URL => $url })); 435 return $cb->(undef, { Status => 599, Reason => $err, URL => $url });
436 };
437
424 pop; # free memory, save a tree 438 pop; # free memory, save a tree
425 439
426 return unless delete $state{connect_guard}; 440 return unless delete $state{connect_guard};
427 441
428 # get handle 442 # get handle
… …
461 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls}; 475 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};
462 476
463 # send request 477 # send request
464 $state{handle}->push_write ( 478 $state{handle}->push_write (
465 "$method $rpath HTTP/1.0\015\012" 479 "$method $rpath HTTP/1.0\015\012"
466 . (join "", map "\u$_: $hdr{$_}\015\012", keys %hdr) 480 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
467 . "\015\012" 481 . "\015\012"
468 . (delete $arg{body}) 482 . (delete $arg{body})
469 ); 483 );
470 484
471 %hdr = (); # reduce memory usage, save a kitten 485 %hdr = (); # reduce memory usage, save a kitten
… …
482 URL => ",$url" 496 URL => ",$url"
483 ); 497 );
484 498
485 # headers, could be optimized a bit 499 # headers, could be optimized a bit
486 $state{handle}->unshift_read (line => $qr_nlnl, sub { 500 $state{handle}->unshift_read (line => $qr_nlnl, sub {
487 for ("$_[1]\012") { 501 for ("$_[1]") {
488 y/\015//d; # weed out any \015, as they show up in the weirdest of places. 502 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
489 503
490 # things seen, not parsed: 504 # things seen, not parsed:
491 # p3pP="NON CUR OTPi OUR NOR UNI" 505 # p3pP="NON CUR OTPi OUR NOR UNI"
492 506
493 $hdr{lc $1} .= ",$2" 507 $hdr{lc $1} .= ",$2"
494 while /\G 508 while /\G
495 ([^:\000-\037]+): 509 ([^:\000-\037]*):
496 [\011\040]* 510 [\011\040]*
497 ((?: [^\012]+ | \012[\011\040] )*) 511 ((?: [^\012]+ | \012[\011\040] )*)
498 \012 512 \012
499 /gxc; 513 /gxc;
500 514
… …
589 603
590 redo if /\G\s*,/gc; 604 redo if /\G\s*,/gc;
591 } 605 }
592 } 606 }
593 607
594 if ($redirect) { 608 if ($redirect && exists $hdr{location}) {
595 # we ignore any errors, as it is very common to receive 609 # we ignore any errors, as it is very common to receive
596 # Content-Length != 0 but no actual body 610 # Content-Length != 0 but no actual body
597 # we also access %hdr, as $_[1] might be an erro 611 # we also access %hdr, as $_[1] might be an erro
598 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb); 612 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb);
599 } else { 613 } else {
… …
654 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr) 668 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr)
655 if $len <= length $_[0]{rbuf}; 669 if $len <= length $_[0]{rbuf};
656 }); 670 });
657 } else { 671 } else {
658 $_[0]->on_error (sub { 672 $_[0]->on_error (sub {
659 $! == Errno::EPIPE 673 $! == Errno::EPIPE || !$!
660 ? $finish->(delete $_[0]{rbuf}, \%hdr) 674 ? $finish->(delete $_[0]{rbuf}, \%hdr)
661 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }); 675 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url });
662 }); 676 });
663 $_[0]->on_read (sub { }); 677 $_[0]->on_read (sub { });
664 } 678 }
… …
688 }); 702 });
689 } else { 703 } else {
690 &$handle_actual_request; 704 &$handle_actual_request;
691 } 705 }
692 706
693 }, sub { 707 }, $arg{on_prepare} || sub { $timeout };
694 $timeout
695 };
696 }; 708 };
697 709
698 defined wantarray && AnyEvent::Util::guard { %state = () } 710 defined wantarray && AnyEvent::Util::guard { %state = () }
699} 711}
700 712
… …
732=item $AnyEvent::HTTP::USERAGENT 744=item $AnyEvent::HTTP::USERAGENT
733 745
734The default value for the C<User-Agent> header (the default is 746The default value for the C<User-Agent> header (the default is
735C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>). 747C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
736 748
737=item $AnyEvent::HTTP::MAX_PERSISTENT 749=item $AnyEvent::HTTP::MAX_PER_HOST
738 750
739The maximum number of persistent connections to keep open (default: 8). 751The maximum number of concurrent connections to the same host (identified
752by the hostname). If the limit is exceeded, then the additional requests
753are queued until previous connections are closed.
740 754
741Not implemented currently. 755The default value for this is C<4>, and it is highly advisable to not
742 756increase it.
743=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
744
745The maximum time to cache a persistent connection, in seconds (default: 2).
746
747Not implemented currently.
748 757
749=item $AnyEvent::HTTP::ACTIVE 758=item $AnyEvent::HTTP::ACTIVE
750 759
751The number of active connections. This is not the number of currently 760The number of active connections. This is not the number of currently
752running requests, but the number of currently open and non-idle TCP 761running requests, but the number of currently open and non-idle TCP

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines