ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.42 by root, Mon Jul 6 00:08:16 2009 UTC vs.
Revision 1.54 by root, Wed Jun 16 18:09:52 2010 UTC

… …
41use strict; 41use strict;
42no warnings; 42no warnings;
43 43
44use Errno (); 44use Errno ();
45 45
46use AnyEvent 4.452 (); 46use AnyEvent 5.0 ();
47use AnyEvent::Util (); 47use AnyEvent::Util ();
48use AnyEvent::Socket (); 48use AnyEvent::Socket ();
49use AnyEvent::Handle (); 49use AnyEvent::Handle ();
50 50
51use base Exporter::; 51use base Exporter::;
52 52
53our $VERSION = '1.12'; 53our $VERSION = '1.44';
54 54
55our @EXPORT = qw(http_get http_post http_head http_request); 55our @EXPORT = qw(http_get http_post http_head http_request);
56 56
57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58our $MAX_RECURSE = 10; 58our $MAX_RECURSE = 10;
59our $MAX_PERSISTENT = 8; 59our $MAX_PERSISTENT = 8;
60our $PERSISTENT_TIMEOUT = 2; 60our $PERSISTENT_TIMEOUT = 2;
61our $TIMEOUT = 300; 61our $TIMEOUT = 300;
62 62
63# changing these is evil 63# changing these is evil
64our $MAX_PERSISTENT_PER_HOST = 2; 64our $MAX_PERSISTENT_PER_HOST = 0;
65our $MAX_PER_HOST = 4; 65our $MAX_PER_HOST = 4;
66 66
67our $PROXY; 67our $PROXY;
68our $ACTIVE = 0; 68our $ACTIVE = 0;
69 69
… …
138 138
139=item headers => hashref 139=item headers => hashref
140 140
141The request headers to use. Currently, C<http_request> may provide its 141The request headers to use. Currently, C<http_request> may provide its
142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers 142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
143and will provide defaults for C<User-Agent:> and C<Referer:>. 143and will provide defaults for C<User-Agent:> and C<Referer:> (this can be
144suppressed by using C<undef> for these headers in which case they won't be
145sent at all).
144 146
145=item timeout => $seconds 147=item timeout => $seconds
146 148
147The time-out to use for various stages - each connect attempt will reset 149The time-out to use for various stages - each connect attempt will reset
148the timeout, as will read or write activity. Default timeout is 5 minutes. 150the timeout, as will read or write activity, i.e. this is not an overall
151timeout.
152
153Default timeout is 5 minutes.
149 154
150=item proxy => [$host, $port[, $scheme]] or undef 155=item proxy => [$host, $port[, $scheme]] or undef
151 156
152Use the given http proxy for all requests. If not specified, then the 157Use the given http proxy for all requests. If not specified, then the
153default proxy (as specified by C<$ENV{http_proxy}>) is used. 158default proxy (as specified by C<$ENV{http_proxy}>) is used.
154 159
155C<$scheme> must be either missing or C<http> for HTTP, or C<https> for 160C<$scheme> must be either missing, C<http> for HTTP or C<https> for
156HTTPS. 161HTTPS.
157 162
158=item body => $string 163=item body => $string
159 164
160The request body, usually empty. Will be-sent as-is (future versions of 165The request body, usually empty. Will be-sent as-is (future versions of
… …
186verification) TLS context. 191verification) TLS context.
187 192
188The default for this option is C<low>, which could be interpreted as "give 193The default for this option is C<low>, which could be interpreted as "give
189me the page, no matter what". 194me the page, no matter what".
190 195
196=item on_prepare => $callback->($fh)
197
198In rare cases you need to "tune" the socket before it is used to
199connect (for exmaple, to bind it on a given IP address). This parameter
200overrides the prepare callback passed to C<AnyEvent::Socket::tcp_connect>
201and behaves exactly the same way (e.g. it has to provide a
202timeout). See the description for the C<$prepare_cb> argument of
203C<AnyEvent::Socket::tcp_connect> for details.
204
191=item on_header => $callback->($headers) 205=item on_header => $callback->($headers)
192 206
193When specified, this callback will be called with the header hash as soon 207When specified, this callback will be called with the header hash as soon
194as headers have been successfully received from the remote server (not on 208as headers have been successfully received from the remote server (not on
195locally-generated errors). 209locally-generated errors).
… …
221This callback is useful when the data is too large to be held in memory 235This callback is useful when the data is too large to be held in memory
222(so the callback writes it to a file) or when only some information should 236(so the callback writes it to a file) or when only some information should
223be extracted, or when the body should be processed incrementally. 237be extracted, or when the body should be processed incrementally.
224 238
225It is usually preferred over doing your own body handling via 239It is usually preferred over doing your own body handling via
226C<want_body_handle>. 240C<want_body_handle>, but in case of streaming APIs, where HTTP is
241only used to create a connection, C<want_body_handle> is the better
242alternative, as it allows you to install your own event handler, reducing
243resource usage.
227 244
228=item want_body_handle => $enable 245=item want_body_handle => $enable
229 246
230When enabled (default is disabled), the behaviour of AnyEvent::HTTP 247When enabled (default is disabled), the behaviour of AnyEvent::HTTP
231changes considerably: after parsing the headers, and instead of 248changes considerably: after parsing the headers, and instead of
… …
243This is useful with some push-type services, where, after the initial 260This is useful with some push-type services, where, after the initial
244headers, an interactive protocol is used (typical example would be the 261headers, an interactive protocol is used (typical example would be the
245push-style twitter API which starts a JSON/XML stream). 262push-style twitter API which starts a JSON/XML stream).
246 263
247If you think you need this, first have a look at C<on_body>, to see if 264If you think you need this, first have a look at C<on_body>, to see if
248that doesn'T solve your problem in a better way. 265that doesn't solve your problem in a better way.
249 266
250=back 267=back
251 268
252Example: make a simple HTTP GET request for http://www.nethype.de/ 269Example: make a simple HTTP GET request for http://www.nethype.de/
253 270
… …
309 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 326 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
310 327
311 _slot_schedule $_[0]; 328 _slot_schedule $_[0];
312} 329}
313 330
314our $qr_nl = qr<\015?\012>; 331our $qr_nl = qr{\015?\012};
315our $qr_nlnl = qr<\015?\012\015?\012>; 332our $qr_nlnl = qr{(?<![^\012])\015?\012};
316 333
317our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 }; 334our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
318our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" }; 335our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
319 336
320sub http_request($$@) { 337sub http_request($$@) {
… …
354 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 371 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
355 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url }); 372 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url });
356 373
357 my $uhost = $1; 374 my $uhost = $1;
358 $uport = $2 if defined $2; 375 $uport = $2 if defined $2;
376
377 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost"
378 unless exists $hdr{host};
359 379
360 $uhost =~ s/^\[(.*)\]$/$1/; 380 $uhost =~ s/^\[(.*)\]$/$1/;
361 $upath .= "?$query" if length $query; 381 $upath .= "?$query" if length $query;
362 382
363 $upath =~ s%^/?%/%; 383 $upath =~ s%^/?%/%;
… …
396 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path 416 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
397 417
398 if ($proxy) { 418 if ($proxy) {
399 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy); 419 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy);
400 420
421 $rscheme = "http" unless defined $rscheme;
422
401 # don't support https requests over https-proxy transport, 423 # don't support https requests over https-proxy transport,
402 # can't be done with tls as spec'ed, unless you double-encrypt. 424 # can't be done with tls as spec'ed, unless you double-encrypt.
403 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https"; 425 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
404 } else { 426 } else {
405 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath); 427 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
406 } 428 }
407 429
408 $hdr{"user-agent"} ||= $USERAGENT; 430 # leave out fragment and query string, just a heuristic
409 $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic 431 $hdr{referer} ||= "$uscheme://$uauthority$upath" unless exists $hdr{referer};
432 $hdr{"user-agent"} ||= $USERAGENT unless exists $hdr{"user-agent"};
410 433
411 $hdr{host} = "$uhost:$uport";
412 $hdr{"content-length"} = length $arg{body}; 434 $hdr{"content-length"} = length $arg{body}
435 if length $arg{body} || $method ne "GET";
413 436
414 my %state = (connect_guard => 1); 437 my %state = (connect_guard => 1);
415 438
416 _get_slot $uhost, sub { 439 _get_slot $uhost, sub {
417 $state{slot_guard} = shift; 440 $state{slot_guard} = shift;
418 441
419 return unless $state{connect_guard}; 442 return unless $state{connect_guard};
420 443
421 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub { 444 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
422 $state{fh} = shift 445 $state{fh} = shift
446 or do {
447 my $err = "$!";
448 %state = ();
423 or return (%state = (), $cb->(undef, { Status => 599, Reason => "$!", URL => $url })); 449 return $cb->(undef, { Status => 599, Reason => $err, URL => $url });
450 };
451
424 pop; # free memory, save a tree 452 pop; # free memory, save a tree
425 453
426 return unless delete $state{connect_guard}; 454 return unless delete $state{connect_guard};
427 455
428 # get handle 456 # get handle
… …
461 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls}; 489 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};
462 490
463 # send request 491 # send request
464 $state{handle}->push_write ( 492 $state{handle}->push_write (
465 "$method $rpath HTTP/1.0\015\012" 493 "$method $rpath HTTP/1.0\015\012"
466 . (join "", map "\u$_: $hdr{$_}\015\012", keys %hdr) 494 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
467 . "\015\012" 495 . "\015\012"
468 . (delete $arg{body}) 496 . (delete $arg{body})
469 ); 497 );
498
499 # return if error occured during push_write()
500 return unless %state;
470 501
471 %hdr = (); # reduce memory usage, save a kitten 502 %hdr = (); # reduce memory usage, save a kitten
472 503
473 # status line 504 # status line
474 $state{handle}->push_read (line => $qr_nl, sub { 505 $state{handle}->push_read (line => $qr_nl, sub {
… …
482 URL => ",$url" 513 URL => ",$url"
483 ); 514 );
484 515
485 # headers, could be optimized a bit 516 # headers, could be optimized a bit
486 $state{handle}->unshift_read (line => $qr_nlnl, sub { 517 $state{handle}->unshift_read (line => $qr_nlnl, sub {
487 for ("$_[1]\012") { 518 for ("$_[1]") {
488 y/\015//d; # weed out any \015, as they show up in the weirdest of places. 519 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
489 520
490 # things seen, not parsed: 521 # things seen, not parsed:
491 # p3pP="NON CUR OTPi OUR NOR UNI" 522 # p3pP="NON CUR OTPi OUR NOR UNI"
492 523
493 $hdr{lc $1} .= ",$2" 524 $hdr{lc $1} .= ",$2"
494 while /\G 525 while /\G
495 ([^:\000-\037]+): 526 ([^:\000-\037]*):
496 [\011\040]* 527 [\011\040]*
497 ((?: [^\012]+ | \012[\011\040] )*) 528 ((?: [^\012]+ | \012[\011\040] )*)
498 \012 529 \012
499 /gxc; 530 /gxc;
500 531
… …
589 620
590 redo if /\G\s*,/gc; 621 redo if /\G\s*,/gc;
591 } 622 }
592 } 623 }
593 624
594 if ($redirect) { 625 if ($redirect && exists $hdr{location}) {
595 # we ignore any errors, as it is very common to receive 626 # we ignore any errors, as it is very common to receive
596 # Content-Length != 0 but no actual body 627 # Content-Length != 0 but no actual body
597 # we also access %hdr, as $_[1] might be an erro 628 # we also access %hdr, as $_[1] might be an erro
598 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb); 629 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb);
599 } else { 630 } else {
… …
654 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr) 685 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr)
655 if $len <= length $_[0]{rbuf}; 686 if $len <= length $_[0]{rbuf};
656 }); 687 });
657 } else { 688 } else {
658 $_[0]->on_error (sub { 689 $_[0]->on_error (sub {
659 $! == Errno::EPIPE 690 $! == Errno::EPIPE || !$!
660 ? $finish->(delete $_[0]{rbuf}, \%hdr) 691 ? $finish->(delete $_[0]{rbuf}, \%hdr)
661 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }); 692 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url });
662 }); 693 });
663 $_[0]->on_read (sub { }); 694 $_[0]->on_read (sub { });
664 } 695 }
… …
688 }); 719 });
689 } else { 720 } else {
690 &$handle_actual_request; 721 &$handle_actual_request;
691 } 722 }
692 723
693 }, sub { 724 }, $arg{on_prepare} || sub { $timeout };
694 $timeout
695 };
696 }; 725 };
697 726
698 defined wantarray && AnyEvent::Util::guard { %state = () } 727 defined wantarray && AnyEvent::Util::guard { %state = () }
699} 728}
700 729
… …
721=over 4 750=over 4
722 751
723=item AnyEvent::HTTP::set_proxy "proxy-url" 752=item AnyEvent::HTTP::set_proxy "proxy-url"
724 753
725Sets the default proxy server to use. The proxy-url must begin with a 754Sets the default proxy server to use. The proxy-url must begin with a
726string of the form C<http://host:port> (optionally C<https:...>). 755string of the form C<http://host:port> (optionally C<https:...>), croaks
756otherwise.
757
758To clear an already-set proxy, use C<undef>.
727 759
728=item $AnyEvent::HTTP::MAX_RECURSE 760=item $AnyEvent::HTTP::MAX_RECURSE
729 761
730The default value for the C<recurse> request parameter (default: C<10>). 762The default value for the C<recurse> request parameter (default: C<10>).
731 763
732=item $AnyEvent::HTTP::USERAGENT 764=item $AnyEvent::HTTP::USERAGENT
733 765
734The default value for the C<User-Agent> header (the default is 766The default value for the C<User-Agent> header (the default is
735C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>). 767C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
736 768
737=item $AnyEvent::HTTP::MAX_PERSISTENT 769=item $AnyEvent::HTTP::MAX_PER_HOST
738 770
739The maximum number of persistent connections to keep open (default: 8). 771The maximum number of concurrent connections to the same host (identified
772by the hostname). If the limit is exceeded, then the additional requests
773are queued until previous connections are closed.
740 774
741Not implemented currently. 775The default value for this is C<4>, and it is highly advisable to not
742 776increase it.
743=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
744
745The maximum time to cache a persistent connection, in seconds (default: 2).
746
747Not implemented currently.
748 777
749=item $AnyEvent::HTTP::ACTIVE 778=item $AnyEvent::HTTP::ACTIVE
750 779
751The number of active connections. This is not the number of currently 780The number of active connections. This is not the number of currently
752running requests, but the number of currently open and non-idle TCP 781running requests, but the number of currently open and non-idle TCP
… …
755=back 784=back
756 785
757=cut 786=cut
758 787
759sub set_proxy($) { 788sub set_proxy($) {
789 if (length $_[0]) {
760 $PROXY = [$2, $3 || 3128, $1] if $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix; 790 $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix
791 or Carp::croak "$_[0]: invalid proxy URL";
792 $PROXY = [$2, $3 || 3128, $1]
793 } else {
794 undef $PROXY;
795 }
761} 796}
762 797
763# initialise proxy from environment 798# initialise proxy from environment
799eval {
764set_proxy $ENV{http_proxy}; 800 set_proxy $ENV{http_proxy};
801};
765 802
766=head1 SEE ALSO 803=head1 SEE ALSO
767 804
768L<AnyEvent>. 805L<AnyEvent>.
769 806

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines