ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.25 by root, Mon Jul 21 05:42:07 2008 UTC vs.
Revision 1.119 by root, Sun Jun 8 23:33:28 2014 UTC

15This module is an L<AnyEvent> user, you need to make sure that you use and 15This module is an L<AnyEvent> user, you need to make sure that you use and
16run a supported event loop. 16run a supported event loop.
17 17
18This module implements a simple, stateless and non-blocking HTTP 18This module implements a simple, stateless and non-blocking HTTP
19client. It supports GET, POST and other request methods, cookies and more, 19client. It supports GET, POST and other request methods, cookies and more,
20all on a very low level. It can follow redirects supports proxies and 20all on a very low level. It can follow redirects, supports proxies, and
21automatically limits the number of connections to the values specified in 21automatically limits the number of connections to the values specified in
22the RFC. 22the RFC.
23 23
24It should generally be a "good client" that is enough for most HTTP 24It should generally be a "good client" that is enough for most HTTP
25tasks. Simple tasks should be simple, but complex tasks should still be 25tasks. Simple tasks should be simple, but complex tasks should still be
36 36
37=cut 37=cut
38 38
39package AnyEvent::HTTP; 39package AnyEvent::HTTP;
40 40
41use strict; 41use common::sense;
42no warnings;
43 42
44use Carp; 43use Errno ();
45 44
46use AnyEvent (); 45use AnyEvent 5.0 ();
47use AnyEvent::Util (); 46use AnyEvent::Util ();
48use AnyEvent::Socket ();
49use AnyEvent::Handle (); 47use AnyEvent::Handle ();
50 48
51use base Exporter::; 49use base Exporter::;
52 50
53our $VERSION = '1.03'; 51our $VERSION = 2.2;
54 52
55our @EXPORT = qw(http_get http_post http_head http_request); 53our @EXPORT = qw(http_get http_post http_head http_request);
56 54
57our $USERAGENT = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 55our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58our $MAX_RECURSE = 10; 56our $MAX_RECURSE = 10;
59our $MAX_PERSISTENT = 8;
60our $PERSISTENT_TIMEOUT = 2; 57our $PERSISTENT_TIMEOUT = 3;
61our $TIMEOUT = 300; 58our $TIMEOUT = 300;
62 59our $MAX_PER_HOST = 4; # changing this is evil
63# changing these is evil
64our $MAX_PERSISTENT_PER_HOST = 2;
65our $MAX_PER_HOST = 4;
66 60
67our $PROXY; 61our $PROXY;
68our $ACTIVE = 0; 62our $ACTIVE = 0;
69 63
70my %KA_COUNT; # number of open keep-alive connections per host 64my %KA_CACHE; # indexed by uhost currently, points to [$handle...] array
71my %CO_SLOT; # number of open connections, and wait queue, per host 65my %CO_SLOT; # number of open connections, and wait queue, per host
72 66
73=item http_get $url, key => value..., $cb->($data, $headers) 67=item http_get $url, key => value..., $cb->($data, $headers)
74 68
75Executes an HTTP-GET request. See the http_request function for details on 69Executes an HTTP-GET request. See the http_request function for details on
76additional parameters. 70additional parameters and the return value.
77 71
78=item http_head $url, key => value..., $cb->($data, $headers) 72=item http_head $url, key => value..., $cb->($data, $headers)
79 73
80Executes an HTTP-HEAD request. See the http_request function for details on 74Executes an HTTP-HEAD request. See the http_request function for details
81additional parameters. 75on additional parameters and the return value.
82 76
83=item http_post $url, $body, key => value..., $cb->($data, $headers) 77=item http_post $url, $body, key => value..., $cb->($data, $headers)
84 78
85Executes an HTTP-POST request with a request body of C<$bod>. See the 79Executes an HTTP-POST request with a request body of C<$body>. See the
86http_request function for details on additional parameters. 80http_request function for details on additional parameters and the return
81value.
87 82
88=item http_request $method => $url, key => value..., $cb->($data, $headers) 83=item http_request $method => $url, key => value..., $cb->($data, $headers)
89 84
90Executes a HTTP request of type C<$method> (e.g. C<GET>, C<POST>). The URL 85Executes a HTTP request of type C<$method> (e.g. C<GET>, C<POST>). The URL
91must be an absolute http or https URL. 86must be an absolute http or https URL.
92 87
88When called in void context, nothing is returned. In other contexts,
89C<http_request> returns a "cancellation guard" - you have to keep the
90object at least alive until the callback get called. If the object gets
91destroyed before the callback is called, the request will be cancelled.
92
93The callback will be called with the response data as first argument 93The callback will be called with the response body data as first argument
94(or C<undef> if it wasn't available due to errors), and a hash-ref with 94(or C<undef> if an error occurred), and a hash-ref with response headers
95response headers as second argument. 95(and trailers) as second argument.
96 96
97All the headers in that hash are lowercased. In addition to the response 97All the headers in that hash are lowercased. In addition to the response
98headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason> 98headers, the "pseudo-headers" (uppercase to avoid clashing with possible
99response headers) C<HTTPVersion>, C<Status> and C<Reason> contain the
99contain the three parts of the HTTP Status-Line of the same name. The 100three parts of the HTTP Status-Line of the same name. If an error occurs
101during the body phase of a request, then the original C<Status> and
102C<Reason> values from the header are available as C<OrigStatus> and
103C<OrigReason>.
104
100pseudo-header C<URL> contains the original URL (which can differ from the 105The pseudo-header C<URL> contains the actual URL (which can differ from
101requested URL when following redirects). 106the requested URL when following redirects - for example, you might get
107an error that your URL scheme is not supported even though your URL is a
108valid http URL because it redirected to an ftp URL, in which case you can
109look at the URL pseudo header).
102 110
111The pseudo-header C<Redirect> only exists when the request was a result
112of an internal redirect. In that case it is an array reference with
113the C<($data, $headers)> from the redirect response. Note that this
114response could in turn be the result of a redirect itself, and C<<
115$headers->{Redirect}[1]{Redirect} >> will then contain the original
116response, and so on.
117
103If the server sends a header multiple lines, then their contents will be 118If the server sends a header multiple times, then their contents will be
104joined together with C<\x00>. 119joined together with a comma (C<,>), as per the HTTP spec.
105 120
106If an internal error occurs, such as not being able to resolve a hostname, 121If an internal error occurs, such as not being able to resolve a hostname,
107then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599> 122then C<$data> will be C<undef>, C<< $headers->{Status} >> will be
108and the C<Reason> pseudo-header will contain an error message. 123C<590>-C<599> and the C<Reason> pseudo-header will contain an error
124message. Currently the following status codes are used:
125
126=over 4
127
128=item 595 - errors during connection establishment, proxy handshake.
129
130=item 596 - errors during TLS negotiation, request sending and header processing.
131
132=item 597 - errors during body receiving or processing.
133
134=item 598 - user aborted request via C<on_header> or C<on_body>.
135
136=item 599 - other, usually nonretryable, errors (garbled URL etc.).
137
138=back
109 139
110A typical callback might look like this: 140A typical callback might look like this:
111 141
112 sub { 142 sub {
113 my ($body, $hdr) = @_; 143 my ($body, $hdr) = @_;
124 154
125=over 4 155=over 4
126 156
127=item recurse => $count (default: $MAX_RECURSE) 157=item recurse => $count (default: $MAX_RECURSE)
128 158
129Whether to recurse requests or not, e.g. on redirects, authentication 159Whether to recurse requests or not, e.g. on redirects, authentication and
130retries and so on, and how often to do so. 160other retries and so on, and how often to do so.
161
162Only redirects to http and https URLs are supported. While most common
163redirection forms are handled entirely within this module, some require
164the use of the optional L<URI> module. If it is required but missing, then
165the request will fail with an error.
131 166
132=item headers => hashref 167=item headers => hashref
133 168
134The request headers to use. Currently, C<http_request> may provide its 169The request headers to use. Currently, C<http_request> may provide its own
135own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers 170C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers and
136and will provide defaults for C<User-Agent:> and C<Referer:>. 171will provide defaults at least for C<TE:>, C<Referer:> and C<User-Agent:>
172(this can be suppressed by using C<undef> for these headers in which case
173they won't be sent at all).
174
175You really should provide your own C<User-Agent:> header value that is
176appropriate for your program - I wouldn't be surprised if the default
177AnyEvent string gets blocked by webservers sooner or later.
178
179Also, make sure that your headers names and values do not contain any
180embedded newlines.
137 181
138=item timeout => $seconds 182=item timeout => $seconds
139 183
140The time-out to use for various stages - each connect attempt will reset 184The time-out to use for various stages - each connect attempt will reset
141the timeout, as will read or write activity. Default timeout is 5 minutes. 185the timeout, as will read or write activity, i.e. this is not an overall
186timeout.
187
188Default timeout is 5 minutes.
142 189
143=item proxy => [$host, $port[, $scheme]] or undef 190=item proxy => [$host, $port[, $scheme]] or undef
144 191
145Use the given http proxy for all requests. If not specified, then the 192Use the given http proxy for all requests, or no proxy if C<undef> is
146default proxy (as specified by C<$ENV{http_proxy}>) is used. 193used.
147 194
148C<$scheme> must be either missing or C<http> for HTTP, or C<https> for 195C<$scheme> must be either missing or must be C<http> for HTTP.
149HTTPS. 196
197If not specified, then the default proxy is used (see
198C<AnyEvent::HTTP::set_proxy>).
150 199
151=item body => $string 200=item body => $string
152 201
153The request body, usually empty. Will be-sent as-is (future versions of 202The request body, usually empty. Will be sent as-is (future versions of
154this module might offer more options). 203this module might offer more options).
155 204
156=item cookie_jar => $hash_ref 205=item cookie_jar => $hash_ref
157 206
158Passing this parameter enables (simplified) cookie-processing, loosely 207Passing this parameter enables (simplified) cookie-processing, loosely
159based on the original netscape specification. 208based on the original netscape specification.
160 209
161The C<$hash_ref> must be an (initially empty) hash reference which will 210The C<$hash_ref> must be an (initially empty) hash reference which
162get updated automatically. It is possible to save the cookie_jar to 211will get updated automatically. It is possible to save the cookie jar
163persistent storage with something like JSON or Storable, but this is not 212to persistent storage with something like JSON or Storable - see the
164recommended, as expire times are currently being ignored. 213C<AnyEvent::HTTP::cookie_jar_expire> function if you wish to remove
214expired or session-only cookies, and also for documentation on the format
215of the cookie jar.
165 216
166Note that this cookie implementation is not of very high quality, nor 217Note that this cookie implementation is not meant to be complete. If
167meant to be complete. If you want complete cookie management you have to 218you want complete cookie management you have to do that on your
168do that on your own. C<cookie_jar> is meant as a quick fix to get some 219own. C<cookie_jar> is meant as a quick fix to get most cookie-using sites
169cookie-using sites working. Cookies are a privacy disaster, do not use 220working. Cookies are a privacy disaster, do not use them unless required
170them unless required to. 221to.
222
223When cookie processing is enabled, the C<Cookie:> and C<Set-Cookie:>
224headers will be set and handled by this module, otherwise they will be
225left untouched.
226
227=item tls_ctx => $scheme | $tls_ctx
228
229Specifies the AnyEvent::TLS context to be used for https connections. This
230parameter follows the same rules as the C<tls_ctx> parameter to
231L<AnyEvent::Handle>, but additionally, the two strings C<low> or
232C<high> can be specified, which give you a predefined low-security (no
233verification, highest compatibility) and high-security (CA and common-name
234verification) TLS context.
235
236The default for this option is C<low>, which could be interpreted as "give
237me the page, no matter what".
238
239See also the C<sessionid> parameter.
240
241=item session => $string
242
243The module might reuse connections to the same host internally. Sometimes
244(e.g. when using TLS), you do not want to reuse connections from other
245sessions. This can be achieved by setting this parameter to some unique
246ID (such as the address of an object storing your state data, or the TLS
247context) - only connections using the same unique ID will be reused.
248
249=item on_prepare => $callback->($fh)
250
251In rare cases you need to "tune" the socket before it is used to
252connect (for example, to bind it on a given IP address). This parameter
253overrides the prepare callback passed to C<AnyEvent::Socket::tcp_connect>
254and behaves exactly the same way (e.g. it has to provide a
255timeout). See the description for the C<$prepare_cb> argument of
256C<AnyEvent::Socket::tcp_connect> for details.
257
258=item tcp_connect => $callback->($host, $service, $connect_cb, $prepare_cb)
259
260In even rarer cases you want total control over how AnyEvent::HTTP
261establishes connections. Normally it uses L<AnyEvent::Socket::tcp_connect>
262to do this, but you can provide your own C<tcp_connect> function -
263obviously, it has to follow the same calling conventions, except that it
264may always return a connection guard object.
265
266There are probably lots of weird uses for this function, starting from
267tracing the hosts C<http_request> actually tries to connect, to (inexact
268but fast) host => IP address caching or even socks protocol support.
269
270=item on_header => $callback->($headers)
271
272When specified, this callback will be called with the header hash as soon
273as headers have been successfully received from the remote server (not on
274locally-generated errors).
275
276It has to return either true (in which case AnyEvent::HTTP will continue),
277or false, in which case AnyEvent::HTTP will cancel the download (and call
278the finish callback with an error code of C<598>).
279
280This callback is useful, among other things, to quickly reject unwanted
281content, which, if it is supposed to be rare, can be faster than first
282doing a C<HEAD> request.
283
284The downside is that cancelling the request makes it impossible to re-use
285the connection. Also, the C<on_header> callback will not receive any
286trailer (headers sent after the response body).
287
288Example: cancel the request unless the content-type is "text/html".
289
290 on_header => sub {
291 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
292 },
293
294=item on_body => $callback->($partial_body, $headers)
295
296When specified, all body data will be passed to this callback instead of
297to the completion callback. The completion callback will get the empty
298string instead of the body data.
299
300It has to return either true (in which case AnyEvent::HTTP will continue),
301or false, in which case AnyEvent::HTTP will cancel the download (and call
302the completion callback with an error code of C<598>).
303
304The downside to cancelling the request is that it makes it impossible to
305re-use the connection.
306
307This callback is useful when the data is too large to be held in memory
308(so the callback writes it to a file) or when only some information should
309be extracted, or when the body should be processed incrementally.
310
311It is usually preferred over doing your own body handling via
312C<want_body_handle>, but in case of streaming APIs, where HTTP is
313only used to create a connection, C<want_body_handle> is the better
314alternative, as it allows you to install your own event handler, reducing
315resource usage.
316
317=item want_body_handle => $enable
318
319When enabled (default is disabled), the behaviour of AnyEvent::HTTP
320changes considerably: after parsing the headers, and instead of
321downloading the body (if any), the completion callback will be
322called. Instead of the C<$body> argument containing the body data, the
323callback will receive the L<AnyEvent::Handle> object associated with the
324connection. In error cases, C<undef> will be passed. When there is no body
325(e.g. status C<304>), the empty string will be passed.
326
327The handle object might or might not be in TLS mode, might be connected
328to a proxy, be a persistent connection, use chunked transfer encoding
329etc., and configured in unspecified ways. The user is responsible for this
330handle (it will not be used by this module anymore).
331
332This is useful with some push-type services, where, after the initial
333headers, an interactive protocol is used (typical example would be the
334push-style twitter API which starts a JSON/XML stream).
335
336If you think you need this, first have a look at C<on_body>, to see if
337that doesn't solve your problem in a better way.
338
339=item persistent => $boolean
340
341Try to create/reuse a persistent connection. When this flag is set
342(default: true for idempotent requests, false for all others), then
343C<http_request> tries to re-use an existing (previously-created)
344persistent connection to the host and, failing that, tries to create a new
345one.
346
347Requests failing in certain ways will be automatically retried once, which
348is dangerous for non-idempotent requests, which is why it defaults to off
349for them. The reason for this is because the bozos who designed HTTP/1.1
350made it impossible to distinguish between a fatal error and a normal
351connection timeout, so you never know whether there was a problem with
352your request or not.
353
354When reusing an existent connection, many parameters (such as TLS context)
355will be ignored. See the C<session> parameter for a workaround.
356
357=item keepalive => $boolean
358
359Only used when C<persistent> is also true. This parameter decides whether
360C<http_request> tries to handshake a HTTP/1.0-style keep-alive connection
361(as opposed to only a HTTP/1.1 persistent connection).
362
363The default is true, except when using a proxy, in which case it defaults
364to false, as HTTP/1.0 proxies cannot support this in a meaningful way.
365
366=item handle_params => { key => value ... }
367
368The key-value pairs in this hash will be passed to any L<AnyEvent::Handle>
369constructor that is called - not all requests will create a handle, and
370sometimes more than one is created, so this parameter is only good for
371setting hints.
372
373Example: set the maximum read size to 4096, to potentially conserve memory
374at the cost of speed.
375
376 handle_params => {
377 max_read_size => 4096,
378 },
171 379
172=back 380=back
173 381
174Example: make a simple HTTP GET request for http://www.nethype.de/ 382Example: do a simple HTTP GET request for http://www.nethype.de/ and print
383the response body.
175 384
176 http_request GET => "http://www.nethype.de/", sub { 385 http_request GET => "http://www.nethype.de/", sub {
177 my ($body, $hdr) = @_; 386 my ($body, $hdr) = @_;
178 print "$body\n"; 387 print "$body\n";
179 }; 388 };
180 389
181Example: make a HTTP HEAD request on https://www.google.com/, use a 390Example: do a HTTP HEAD request on https://www.google.com/, use a
182timeout of 30 seconds. 391timeout of 30 seconds.
183 392
184 http_request 393 http_request
185 GET => "https://www.google.com", 394 HEAD => "https://www.google.com",
395 headers => { "user-agent" => "MySearchClient 1.0" },
186 timeout => 30, 396 timeout => 30,
187 sub { 397 sub {
188 my ($body, $hdr) = @_; 398 my ($body, $hdr) = @_;
189 use Data::Dumper; 399 use Data::Dumper;
190 print Dumper $hdr; 400 print Dumper $hdr;
191 } 401 }
192 ; 402 ;
193 403
404Example: do another simple HTTP GET request, but immediately try to
405cancel it.
406
407 my $request = http_request GET => "http://www.nethype.de/", sub {
408 my ($body, $hdr) = @_;
409 print "$body\n";
410 };
411
412 undef $request;
413
194=cut 414=cut
415
416#############################################################################
417# wait queue/slots
195 418
196sub _slot_schedule; 419sub _slot_schedule;
197sub _slot_schedule($) { 420sub _slot_schedule($) {
198 my $host = shift; 421 my $host = shift;
199 422
221 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 444 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
222 445
223 _slot_schedule $_[0]; 446 _slot_schedule $_[0];
224} 447}
225 448
449#############################################################################
450# cookie handling
451
452# expire cookies
453sub cookie_jar_expire($;$) {
454 my ($jar, $session_end) = @_;
455
456 %$jar = () if $jar->{version} != 1;
457
458 my $anow = AE::now;
459
460 while (my ($chost, $paths) = each %$jar) {
461 next unless ref $paths;
462
463 while (my ($cpath, $cookies) = each %$paths) {
464 while (my ($cookie, $kv) = each %$cookies) {
465 if (exists $kv->{_expires}) {
466 delete $cookies->{$cookie}
467 if $anow > $kv->{_expires};
468 } elsif ($session_end) {
469 delete $cookies->{$cookie};
470 }
471 }
472
473 delete $paths->{$cpath}
474 unless %$cookies;
475 }
476
477 delete $jar->{$chost}
478 unless %$paths;
479 }
480}
481
482# extract cookies from jar
483sub cookie_jar_extract($$$$) {
484 my ($jar, $scheme, $host, $path) = @_;
485
486 %$jar = () if $jar->{version} != 1;
487
488 my @cookies;
489
490 while (my ($chost, $paths) = each %$jar) {
491 next unless ref $paths;
492
493 if ($chost =~ /^\./) {
494 next unless $chost eq substr $host, -length $chost;
495 } elsif ($chost =~ /\./) {
496 next unless $chost eq $host;
497 } else {
498 next;
499 }
500
501 while (my ($cpath, $cookies) = each %$paths) {
502 next unless $cpath eq substr $path, 0, length $cpath;
503
504 while (my ($cookie, $kv) = each %$cookies) {
505 next if $scheme ne "https" && exists $kv->{secure};
506
507 if (exists $kv->{_expires} and AE::now > $kv->{_expires}) {
508 delete $cookies->{$cookie};
509 next;
510 }
511
512 my $value = $kv->{value};
513
514 if ($value =~ /[=;,[:space:]]/) {
515 $value =~ s/([\\"])/\\$1/g;
516 $value = "\"$value\"";
517 }
518
519 push @cookies, "$cookie=$value";
520 }
521 }
522 }
523
524 \@cookies
525}
526
527# parse set_cookie header into jar
528sub cookie_jar_set_cookie($$$$) {
529 my ($jar, $set_cookie, $host, $date) = @_;
530
531 my $anow = int AE::now;
532 my $snow; # server-now
533
534 for ($set_cookie) {
535 # parse NAME=VALUE
536 my @kv;
537
538 # expires is not http-compliant in the original cookie-spec,
539 # we support the official date format and some extensions
540 while (
541 m{
542 \G\s*
543 (?:
544 expires \s*=\s* ([A-Z][a-z][a-z]+,\ [^,;]+)
545 | ([^=;,[:space:]]+) (?: \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^;,[:space:]]*) ) )?
546 )
547 }gcxsi
548 ) {
549 my $name = $2;
550 my $value = $4;
551
552 if (defined $1) {
553 # expires
554 $name = "expires";
555 $value = $1;
556 } elsif (defined $3) {
557 # quoted
558 $value = $3;
559 $value =~ s/\\(.)/$1/gs;
560 }
561
562 push @kv, @kv ? lc $name : $name, $value;
563
564 last unless /\G\s*;/gc;
565 }
566
567 last unless @kv;
568
569 my $name = shift @kv;
570 my %kv = (value => shift @kv, @kv);
571
572 if (exists $kv{"max-age"}) {
573 $kv{_expires} = $anow + delete $kv{"max-age"};
574 } elsif (exists $kv{expires}) {
575 $snow ||= parse_date ($date) || $anow;
576 $kv{_expires} = $anow + (parse_date (delete $kv{expires}) - $snow);
577 } else {
578 delete $kv{_expires};
579 }
580
581 my $cdom;
582 my $cpath = (delete $kv{path}) || "/";
583
584 if (exists $kv{domain}) {
585 $cdom = delete $kv{domain};
586
587 $cdom =~ s/^\.?/./; # make sure it starts with a "."
588
589 next if $cdom =~ /\.$/;
590
591 # this is not rfc-like and not netscape-like. go figure.
592 my $ndots = $cdom =~ y/.//;
593 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
594 } else {
595 $cdom = $host;
596 }
597
598 # store it
599 $jar->{version} = 1;
600 $jar->{lc $cdom}{$cpath}{$name} = \%kv;
601
602 redo if /\G\s*,/gc;
603 }
604}
605
606#############################################################################
607# keepalive/persistent connection cache
608
609# fetch a connection from the keepalive cache
610sub ka_fetch($) {
611 my $ka_key = shift;
612
613 my $hdl = pop @{ $KA_CACHE{$ka_key} }; # currently we reuse the MOST RECENTLY USED connection
614 delete $KA_CACHE{$ka_key}
615 unless @{ $KA_CACHE{$ka_key} };
616
617 $hdl
618}
619
620sub ka_store($$) {
621 my ($ka_key, $hdl) = @_;
622
623 my $kaa = $KA_CACHE{$ka_key} ||= [];
624
625 my $destroy = sub {
626 my @ka = grep $_ != $hdl, @{ $KA_CACHE{$ka_key} };
627
628 $hdl->destroy;
629
630 @ka
631 ? $KA_CACHE{$ka_key} = \@ka
632 : delete $KA_CACHE{$ka_key};
633 };
634
635 # on error etc., destroy
636 $hdl->on_error ($destroy);
637 $hdl->on_eof ($destroy);
638 $hdl->on_read ($destroy);
639 $hdl->timeout ($PERSISTENT_TIMEOUT);
640
641 push @$kaa, $hdl;
642 shift @$kaa while @$kaa > $MAX_PER_HOST;
643}
644
645#############################################################################
646# utilities
647
648# continue to parse $_ for headers and place them into the arg
649sub _parse_hdr() {
650 my %hdr;
651
652 # things seen, not parsed:
653 # p3pP="NON CUR OTPi OUR NOR UNI"
654
655 $hdr{lc $1} .= ",$2"
656 while /\G
657 ([^:\000-\037]*):
658 [\011\040]*
659 ((?: [^\012]+ | \012[\011\040] )*)
660 \012
661 /gxc;
662
663 /\G$/
664 or return;
665
666 # remove the "," prefix we added to all headers above
667 substr $_, 0, 1, ""
668 for values %hdr;
669
670 \%hdr
671}
672
673#############################################################################
674# http_get
675
676our $qr_nlnl = qr{(?<![^\012])\015?\012};
677
678our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
679our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
680
681# maybe it should just become a normal object :/
682
683sub _destroy_state(\%) {
684 my ($state) = @_;
685
686 $state->{handle}->destroy if $state->{handle};
687 %$state = ();
688}
689
690sub _error(\%$$) {
691 my ($state, $cb, $hdr) = @_;
692
693 &_destroy_state ($state);
694
695 $cb->(undef, $hdr);
696 ()
697}
698
699our %IDEMPOTENT = (
700 DELETE => 1,
701 GET => 1,
702 HEAD => 1,
703 OPTIONS => 1,
704 PUT => 1,
705 TRACE => 1,
706
707 ACL => 1,
708 "BASELINE-CONTROL" => 1,
709 BIND => 1,
710 CHECKIN => 1,
711 CHECKOUT => 1,
712 COPY => 1,
713 LABEL => 1,
714 LINK => 1,
715 MERGE => 1,
716 MKACTIVITY => 1,
717 MKCALENDAR => 1,
718 MKCOL => 1,
719 MKREDIRECTREF => 1,
720 MKWORKSPACE => 1,
721 MOVE => 1,
722 ORDERPATCH => 1,
723 PROPFIND => 1,
724 PROPPATCH => 1,
725 REBIND => 1,
726 REPORT => 1,
727 SEARCH => 1,
728 UNBIND => 1,
729 UNCHECKOUT => 1,
730 UNLINK => 1,
731 UNLOCK => 1,
732 UPDATE => 1,
733 UPDATEREDIRECTREF => 1,
734 "VERSION-CONTROL" => 1,
735);
736
226sub http_request($$@) { 737sub http_request($$@) {
227 my $cb = pop; 738 my $cb = pop;
228 my ($method, $url, %arg) = @_; 739 my ($method, $url, %arg) = @_;
229 740
230 my %hdr; 741 my %hdr;
742
743 $arg{tls_ctx} = $TLS_CTX_LOW if $arg{tls_ctx} eq "low" || !exists $arg{tls_ctx};
744 $arg{tls_ctx} = $TLS_CTX_HIGH if $arg{tls_ctx} eq "high";
231 745
232 $method = uc $method; 746 $method = uc $method;
233 747
234 if (my $hdr = $arg{headers}) { 748 if (my $hdr = $arg{headers}) {
235 while (my ($k, $v) = each %$hdr) { 749 while (my ($k, $v) = each %$hdr) {
236 $hdr{lc $k} = $v; 750 $hdr{lc $k} = $v;
237 } 751 }
238 } 752 }
239 753
754 # pseudo headers for all subsequent responses
755 my @pseudo = (URL => $url);
756 push @pseudo, Redirect => delete $arg{Redirect} if exists $arg{Redirect};
757
240 my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE; 758 my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;
241 759
242 return $cb->(undef, { Status => 599, Reason => "recursion limit reached", URL => $url }) 760 return $cb->(undef, { @pseudo, Status => 599, Reason => "Too many redirections" })
243 if $recurse < 0; 761 if $recurse < 0;
244 762
245 my $proxy = $arg{proxy} || $PROXY; 763 my $proxy = exists $arg{proxy} ? $arg{proxy} : $PROXY;
246 my $timeout = $arg{timeout} || $TIMEOUT; 764 my $timeout = $arg{timeout} || $TIMEOUT;
247 765
248 $hdr{"user-agent"} ||= $USERAGENT;
249
250 my ($scheme, $authority, $upath, $query, $fragment) = 766 my ($uscheme, $uauthority, $upath, $query, undef) = # ignore fragment
251 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|; 767 $url =~ m|^([^:]+):(?://([^/?#]*))?([^?#]*)(?:(\?[^#]*))?(?:#(.*))?$|;
252 768
253 $scheme = lc $scheme; 769 $uscheme = lc $uscheme;
254 770
255 my $uport = $scheme eq "http" ? 80 771 my $uport = $uscheme eq "http" ? 80
256 : $scheme eq "https" ? 443 772 : $uscheme eq "https" ? 443
257 : return $cb->(undef, { Status => 599, Reason => "only http and https URL schemes supported", URL => $url }); 773 : return $cb->(undef, { @pseudo, Status => 599, Reason => "Only http and https URL schemes supported" });
258 774
259 $hdr{referer} ||= "$scheme://$authority$upath"; # leave out fragment and query string, just a heuristic
260
261 $authority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 775 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
262 or return $cb->(undef, { Status => 599, Reason => "unparsable URL", URL => $url }); 776 or return $cb->(undef, { @pseudo, Status => 599, Reason => "Unparsable URL" });
263 777
264 my $uhost = $1; 778 my $uhost = lc $1;
265 $uport = $2 if defined $2; 779 $uport = $2 if defined $2;
266 780
781 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost"
782 unless exists $hdr{host};
783
267 $uhost =~ s/^\[(.*)\]$/$1/; 784 $uhost =~ s/^\[(.*)\]$/$1/;
268 $upath .= "?$query" if length $query; 785 $upath .= $query if length $query;
269 786
270 $upath =~ s%^/?%/%; 787 $upath =~ s%^/?%/%;
271 788
272 # cookie processing 789 # cookie processing
273 if (my $jar = $arg{cookie_jar}) { 790 if (my $jar = $arg{cookie_jar}) {
274 %$jar = () if $jar->{version} < 1; 791 my $cookies = cookie_jar_extract $jar, $uscheme, $uhost, $upath;
275 792
793 $hdr{cookie} = join "; ", @$cookies
276 my @cookie; 794 if @$cookies;
277 795 }
278 while (my ($chost, $v) = each %$jar) { 796
279 next unless $chost eq substr $uhost, -length $chost; 797 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
280 next unless $chost =~ /^\./; 798
281 799 if ($proxy) {
282 while (my ($cpath, $v) = each %$v) { 800 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy);
283 next unless $cpath eq substr $upath, 0, length $cpath; 801
284 802 $rscheme = "http" unless defined $rscheme;
285 while (my ($k, $v) = each %$v) { 803
286 next if $scheme ne "https" && exists $v->{secure}; 804 # don't support https requests over https-proxy transport,
287 push @cookie, "$k=$v->{value}"; 805 # can't be done with tls as spec'ed, unless you double-encrypt.
806 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
807
808 $rhost = lc $rhost;
809 $rscheme = lc $rscheme;
810 } else {
811 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
812 }
813
814 # leave out fragment and query string, just a heuristic
815 $hdr{referer} = "$uscheme://$uauthority$upath" unless exists $hdr{referer};
816 $hdr{"user-agent"} = $USERAGENT unless exists $hdr{"user-agent"};
817
818 $hdr{"content-length"} = length $arg{body}
819 if length $arg{body} || $method ne "GET";
820
821 my $idempotent = $IDEMPOTENT{$method};
822
823 # default value for keepalive is true iff the request is for an idempotent method
824 my $persistent = exists $arg{persistent} ? !!$arg{persistent} : $idempotent;
825 my $keepalive = exists $arg{keepalive} ? !!$arg{keepalive} : !$proxy;
826 my $was_persistent; # true if this is actually a recycled connection
827
828 # the key to use in the keepalive cache
829 my $ka_key = "$uscheme\x00$uhost\x00$uport\x00$arg{sessionid}";
830
831 $hdr{connection} = ($persistent ? $keepalive ? "keep-alive, " : "" : "close, ") . "Te"; #1.1
832 $hdr{te} = "trailers" unless exists $hdr{te}; #1.1
833
834 my %state = (connect_guard => 1);
835
836 my $ae_error = 595; # connecting
837
838 # handle actual, non-tunneled, request
839 my $handle_actual_request = sub {
840 $ae_error = 596; # request phase
841
842 my $hdl = $state{handle};
843
844 $hdl->starttls ("connect") if $uscheme eq "https" && !exists $hdl->{tls};
845
846 # send request
847 $hdl->push_write (
848 "$method $rpath HTTP/1.1\015\012"
849 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
850 . "\015\012"
851 . (delete $arg{body})
852 );
853
854 # return if error occurred during push_write()
855 return unless %state;
856
857 # reduce memory usage, save a kitten, also re-use it for the response headers.
858 %hdr = ();
859
860 # status line and headers
861 $state{read_response} = sub {
862 return unless %state;
863
864 for ("$_[1]") {
865 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
866
867 /^HTTP\/0*([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\012]*) )? \012/gxci
868 or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Invalid server response" };
869
870 # 100 Continue handling
871 # should not happen as we don't send expect: 100-continue,
872 # but we handle it just in case.
873 # since we send the request body regardless, if we get an error
874 # we are out of-sync, which we currently do NOT handle correctly.
875 return $state{handle}->push_read (line => $qr_nlnl, $state{read_response})
876 if $2 eq 100;
877
878 push @pseudo,
879 HTTPVersion => $1,
880 Status => $2,
881 Reason => $3,
882 ;
883
884 my $hdr = _parse_hdr
885 or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Garbled response headers" };
886
887 %hdr = (%$hdr, @pseudo);
888 }
889
890 # redirect handling
891 # relative uri handling forced by microsoft and other shitheads.
892 # we give our best and fall back to URI if available.
893 if (exists $hdr{location}) {
894 my $loc = $hdr{location};
895
896 if ($loc =~ m%^//%) { # //
897 $loc = "$rscheme:$loc";
898
899 } elsif ($loc eq "") {
900 $loc = $url;
901
902 } elsif ($loc !~ /^(?: $ | [^:\/?\#]+ : )/x) { # anything "simple"
903 $loc =~ s/^\.\/+//;
904
905 if ($loc !~ m%^[.?#]%) {
906 my $prefix = "$rscheme://$uhost:$uport";
907
908 unless ($loc =~ s/^\///) {
909 $prefix .= $upath;
910 $prefix =~ s/\/[^\/]*$//;
911 }
912
913 $loc = "$prefix/$loc";
914
915 } elsif (eval { require URI }) { # uri
916 $loc = URI->new_abs ($loc, $url)->as_string;
917
918 } else {
919 return _error %state, $cb, { @pseudo, Status => 599, Reason => "Cannot parse Location (URI module missing)" };
920 #$hdr{Status} = 599;
921 #$hdr{Reason} = "Unparsable Redirect (URI module missing)";
922 #$recurse = 0;
923 }
924 }
925
926 $hdr{location} = $loc;
927 }
928
929 my $redirect;
930
931 if ($recurse) {
932 my $status = $hdr{Status};
933
934 # industry standard is to redirect POST as GET for
935 # 301, 302 and 303, in contrast to HTTP/1.0 and 1.1.
936 # also, the UA should ask the user for 301 and 307 and POST,
937 # industry standard seems to be to simply follow.
938 # we go with the industry standard. 308 is defined
939 # by rfc7238
940 if ($status == 301 or $status == 302 or $status == 303) {
941 # HTTP/1.1 is unclear on how to mutate the method
942 $method = "GET" unless $method eq "HEAD";
943 $redirect = 1;
944 } elsif ($status == 307 or $status == 308) {
945 $redirect = 1;
288 } 946 }
289 } 947 }
948
949 my $finish = sub { # ($data, $err_status, $err_reason[, $persistent])
950 if ($state{handle}) {
951 # handle keepalive
952 if (
953 $persistent
954 && $_[3]
955 && ($hdr{HTTPVersion} < 1.1
956 ? $hdr{connection} =~ /\bkeep-?alive\b/i
957 : $hdr{connection} !~ /\bclose\b/i)
958 ) {
959 ka_store $ka_key, delete $state{handle};
960 } else {
961 # no keepalive, destroy the handle
962 $state{handle}->destroy;
963 }
964 }
965
966 %state = ();
967
968 if (defined $_[1]) {
969 $hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1];
970 $hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2];
971 }
972
973 # set-cookie processing
974 if ($arg{cookie_jar}) {
975 cookie_jar_set_cookie $arg{cookie_jar}, $hdr{"set-cookie"}, $uhost, $hdr{date};
976 }
977
978 if ($redirect && exists $hdr{location}) {
979 # we ignore any errors, as it is very common to receive
980 # Content-Length != 0 but no actual body
981 # we also access %hdr, as $_[1] might be an erro
982 $state{recurse} =
983 http_request (
984 $method => $hdr{location},
985 %arg,
986 recurse => $recurse - 1,
987 Redirect => [$_[0], \%hdr],
988 sub {
989 %state = ();
990 &$cb
991 },
992 );
993 } else {
994 $cb->($_[0], \%hdr);
995 }
996 };
997
998 $ae_error = 597; # body phase
999
1000 my $chunked = $hdr{"transfer-encoding"} =~ /\bchunked\b/i; # not quite correct...
1001
1002 my $len = $chunked ? undef : $hdr{"content-length"};
1003
1004 # body handling, many different code paths
1005 # - no body expected
1006 # - want_body_handle
1007 # - te chunked
1008 # - 2x length known (with or without on_body)
1009 # - 2x length not known (with or without on_body)
1010 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
1011 $finish->(undef, 598 => "Request cancelled by on_header");
1012 } elsif (
1013 $hdr{Status} =~ /^(?:1..|204|205|304)$/
1014 or $method eq "HEAD"
1015 or (defined $len && $len == 0) # == 0, not !, because "0 " is true
1016 ) {
1017 # no body
1018 $finish->("", undef, undef, 1);
1019
1020 } elsif (!$redirect && $arg{want_body_handle}) {
1021 $_[0]->on_eof (undef);
1022 $_[0]->on_error (undef);
1023 $_[0]->on_read (undef);
1024
1025 $finish->(delete $state{handle});
1026
1027 } elsif ($chunked) {
1028 my $cl = 0;
1029 my $body = "";
1030 my $on_body = $arg{on_body} || sub { $body .= shift; 1 };
1031
1032 $state{read_chunk} = sub {
1033 $_[1] =~ /^([0-9a-fA-F]+)/
1034 or return $finish->(undef, $ae_error => "Garbled chunked transfer encoding");
1035
1036 my $len = hex $1;
1037
1038 if ($len) {
1039 $cl += $len;
1040
1041 $_[0]->push_read (chunk => $len, sub {
1042 $on_body->($_[1], \%hdr)
1043 or return $finish->(undef, 598 => "Request cancelled by on_body");
1044
1045 $_[0]->push_read (line => sub {
1046 length $_[1]
1047 and return $finish->(undef, $ae_error => "Garbled chunked transfer encoding");
1048 $_[0]->push_read (line => $state{read_chunk});
1049 });
1050 });
1051 } else {
1052 $hdr{"content-length"} ||= $cl;
1053
1054 $_[0]->push_read (line => $qr_nlnl, sub {
1055 if (length $_[1]) {
1056 for ("$_[1]") {
1057 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
1058
1059 my $hdr = _parse_hdr
1060 or return $finish->(undef, $ae_error => "Garbled response trailers");
1061
1062 %hdr = (%hdr, %$hdr);
1063 }
1064 }
1065
1066 $finish->($body, undef, undef, 1);
1067 });
1068 }
1069 };
1070
1071 $_[0]->push_read (line => $state{read_chunk});
1072
1073 } elsif ($arg{on_body}) {
1074 if (defined $len) {
1075 $_[0]->on_read (sub {
1076 $len -= length $_[0]{rbuf};
1077
1078 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
1079 or return $finish->(undef, 598 => "Request cancelled by on_body");
1080
1081 $len > 0
1082 or $finish->("", undef, undef, 1);
1083 });
1084 } else {
1085 $_[0]->on_eof (sub {
1086 $finish->("");
1087 });
1088 $_[0]->on_read (sub {
1089 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
1090 or $finish->(undef, 598 => "Request cancelled by on_body");
1091 });
1092 }
1093 } else {
1094 $_[0]->on_eof (undef);
1095
1096 if (defined $len) {
1097 $_[0]->on_read (sub {
1098 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), undef, undef, 1)
1099 if $len <= length $_[0]{rbuf};
1100 });
1101 } else {
1102 $_[0]->on_error (sub {
1103 ($! == Errno::EPIPE || !$!)
1104 ? $finish->(delete $_[0]{rbuf})
1105 : $finish->(undef, $ae_error => $_[2]);
1106 });
1107 $_[0]->on_read (sub { });
1108 }
1109 }
1110 };
1111
1112 # if keepalive is enabled, then the server closing the connection
1113 # before a response can happen legally - we retry on idempotent methods.
1114 if ($was_persistent && $idempotent) {
1115 my $old_eof = $hdl->{on_eof};
1116 $hdl->{on_eof} = sub {
1117 _destroy_state %state;
1118
1119 %state = ();
1120 $state{recurse} =
1121 http_request (
1122 $method => $url,
1123 %arg,
1124 recurse => $recurse - 1,
1125 keepalive => 0,
1126 sub {
1127 %state = ();
1128 &$cb
1129 }
1130 );
1131 };
1132 $hdl->on_read (sub {
1133 return unless %state;
1134
1135 # as soon as we receive something, a connection close
1136 # once more becomes a hard error
1137 $hdl->{on_eof} = $old_eof;
1138 $hdl->push_read (line => $qr_nlnl, $state{read_response});
1139 });
1140 } else {
1141 $hdl->push_read (line => $qr_nlnl, $state{read_response});
290 } 1142 }
291
292 $hdr{cookie} = join "; ", @cookie
293 if @cookie;
294 } 1143 };
295 1144
296 my ($rhost, $rport, $rpath); # request host, port, path 1145 my $prepare_handle = sub {
1146 my ($hdl) = $state{handle};
297 1147
298 if ($proxy) { 1148 $hdl->on_error (sub {
299 ($rhost, $rport, $scheme) = @$proxy; 1149 _error %state, $cb, { @pseudo, Status => $ae_error, Reason => $_[2] };
300 $rpath = $url; 1150 });
301 } else { 1151 $hdl->on_eof (sub {
302 ($rhost, $rport, $rpath) = ($uhost, $uport, $upath); 1152 _error %state, $cb, { @pseudo, Status => $ae_error, Reason => "Unexpected end-of-file" };
303 $hdr{host} = $uhost; 1153 });
1154 $hdl->timeout_reset;
1155 $hdl->timeout ($timeout);
304 } 1156 };
305 1157
306 $hdr{"content-length"} = length $arg{body}; 1158 # connected to proxy (or origin server)
1159 my $connect_cb = sub {
1160 my $fh = shift
1161 or return _error %state, $cb, { @pseudo, Status => $ae_error, Reason => "$!" };
307 1162
308 my %state = (connect_guard => 1); 1163 return unless delete $state{connect_guard};
1164
1165 # get handle
1166 $state{handle} = new AnyEvent::Handle
1167 %{ $arg{handle_params} },
1168 fh => $fh,
1169 peername => $uhost,
1170 tls_ctx => $arg{tls_ctx},
1171 ;
1172
1173 $prepare_handle->();
1174
1175 #$state{handle}->starttls ("connect") if $rscheme eq "https";
1176
1177 # now handle proxy-CONNECT method
1178 if ($proxy && $uscheme eq "https") {
1179 # oh dear, we have to wrap it into a connect request
1180
1181 # maybe re-use $uauthority with patched port?
1182 $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012\015\012");
1183 $state{handle}->push_read (line => $qr_nlnl, sub {
1184 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
1185 or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Invalid proxy connect response ($_[1])" };
1186
1187 if ($2 == 200) {
1188 $rpath = $upath;
1189 $handle_actual_request->();
1190 } else {
1191 _error %state, $cb, { @pseudo, Status => $2, Reason => $3 };
1192 }
1193 });
1194 } else {
1195 $handle_actual_request->();
1196 }
1197 };
309 1198
310 _get_slot $uhost, sub { 1199 _get_slot $uhost, sub {
311 $state{slot_guard} = shift; 1200 $state{slot_guard} = shift;
312 1201
313 return unless $state{connect_guard}; 1202 return unless $state{connect_guard};
314 1203
315 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub { 1204 # try to use an existing keepalive connection, but only if we, ourselves, plan
316 $state{fh} = shift 1205 # on a keepalive request (in theory, this should be a separate config option).
317 or return $cb->(undef, { Status => 599, Reason => "$!", URL => $url }); 1206 if ($persistent && $KA_CACHE{$ka_key}) {
1207 $was_persistent = 1;
318 1208
319 delete $state{connect_guard}; # reduce memory usage, save a tree 1209 $state{handle} = ka_fetch $ka_key;
1210 $state{handle}->destroyed
1211 and die "AnyEvent::HTTP: unexpectedly got a destructed handle (1), please report.";#d#
1212 $prepare_handle->();
1213 $state{handle}->destroyed
1214 and die "AnyEvent::HTTP: unexpectedly got a destructed handle (2), please report.";#d#
1215 $handle_actual_request->();
320 1216
321 # get handle
322 $state{handle} = new AnyEvent::Handle
323 fh => $state{fh},
324 ($scheme eq "https" ? (tls => "connect") : ());
325
326 # limit the number of persistent connections
327 if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
328 ++$KA_COUNT{$_[1]};
329 $state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} };
330 $hdr{connection} = "keep-alive";
331 delete $hdr{connection}; # keep-alive not yet supported
332 } else { 1217 } else {
333 delete $hdr{connection}; 1218 my $tcp_connect = $arg{tcp_connect}
334 } 1219 || do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };
335 1220
336 # (re-)configure handle 1221 $state{connect_guard} = $tcp_connect->($rhost, $rport, $connect_cb, $arg{on_prepare} || sub { $timeout });
337 $state{handle}->timeout ($timeout);
338 $state{handle}->on_error (sub {
339 my $errno = "$!";
340 %state = ();
341 $cb->(undef, { Status => 599, Reason => $errno, URL => $url });
342 });
343 $state{handle}->on_eof (sub {
344 %state = ();
345 $cb->(undef, { Status => 599, Reason => "unexpected end-of-file", URL => $url });
346 });
347
348 # send request
349 $state{handle}->push_write (
350 "$method $rpath HTTP/1.0\015\012"
351 . (join "", map "$_: $hdr{$_}\015\012", keys %hdr)
352 . "\015\012"
353 . (delete $arg{body})
354 );
355
356 %hdr = (); # reduce memory usage, save a kitten
357
358 # status line
359 $state{handle}->push_read (line => qr/\015?\012/, sub {
360 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
361 or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])", URL => $url }));
362
363 my %hdr = ( # response headers
364 HTTPVersion => "\x00$1",
365 Status => "\x00$2",
366 Reason => "\x00$3",
367 URL => "\x00$url"
368 );
369
370 # headers, could be optimized a bit
371 $state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub {
372 for ("$_[1]\012") {
373 # we support spaces in field names, as lotus domino
374 # creates them.
375 $hdr{lc $1} .= "\x00$2"
376 while /\G
377 ([^:\000-\037]+):
378 [\011\040]*
379 ((?: [^\015\012]+ | \015?\012[\011\040] )*)
380 \015?\012
381 /gxc;
382
383 /\G$/
384 or return (%state = (), $cb->(undef, { Status => 599, Reason => "garbled response headers", URL => $url }));
385 }
386
387 substr $_, 0, 1, ""
388 for values %hdr;
389
390 my $finish = sub {
391 %state = ();
392
393 # set-cookie processing
394 if ($arg{cookie_jar} && exists $hdr{"set-cookie"}) {
395 for (split /\x00/, $hdr{"set-cookie"}) {
396 my ($cookie, @arg) = split /;\s*/;
397 my ($name, $value) = split /=/, $cookie, 2;
398 my %kv = (value => $value, map { split /=/, $_, 2 } @arg);
399
400 my $cdom = (delete $kv{domain}) || $uhost;
401 my $cpath = (delete $kv{path}) || "/";
402
403 $cdom =~ s/^.?/./; # make sure it starts with a "."
404
405 next if $cdom =~ /\.$/;
406
407 # this is not rfc-like and not netscape-like. go figure.
408 my $ndots = $cdom =~ y/.//;
409 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
410
411 # store it
412 $arg{cookie_jar}{version} = 1;
413 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
414 }
415 }
416
417 # microsoft and other shitheads don't give a shit for following standards,
418 # try to support some common forms of broken Location headers.
419 if ($_[1]{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
420 $_[1]{location} =~ s/^\.\/+//;
421
422 my $url = "$scheme://$uhost:$uport";
423
424 unless ($_[1]{location} =~ s/^\///) {
425 $url .= $upath;
426 $url =~ s/\/[^\/]*$//;
427 }
428
429 $_[1]{location} = "$url/$_[1]{location}";
430 }
431
432 if ($_[1]{Status} =~ /^30[12]$/ && $recurse && $method ne "POST") {
433 # apparently, mozilla et al. just change POST to GET here
434 # more research is needed before we do the same
435 http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
436 } elsif ($_[1]{Status} == 303 && $recurse) {
437 # even http/1.1 is unlear on how to mutate the method
438 $method = "GET" unless $method eq "HEAD";
439 http_request ($method => $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
440 } elsif ($_[1]{Status} == 307 && $recurse && $method =~ /^(?:GET|HEAD)$/) {
441 http_request ($method => $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
442 } else {
443 $cb->($_[0], $_[1]);
444 }
445 };
446
447 if ($hdr{Status} =~ /^(?:1..|204|304)$/ or $method eq "HEAD") {
448 $finish->(undef, \%hdr);
449 } else {
450 if (exists $hdr{"content-length"}) {
451 $_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
452 # could cache persistent connection now
453 if ($hdr{connection} =~ /\bkeep-alive\b/i) {
454 # but we don't, due to misdesigns, this is annoyingly complex
455 };
456
457 $finish->($_[1], \%hdr);
458 });
459 } else {
460 # too bad, need to read until we get an error or EOF,
461 # no way to detect winged data.
462 $_[0]->on_error (sub {
463 $finish->($_[0]{rbuf}, \%hdr);
464 });
465 $_[0]->on_eof (undef);
466 $_[0]->on_read (sub { });
467 }
468 }
469 });
470 });
471 }, sub {
472 $timeout
473 }; 1222 }
474 }; 1223 };
475 1224
476 defined wantarray && AnyEvent::Util::guard { %state = () } 1225 defined wantarray && AnyEvent::Util::guard { _destroy_state %state }
477} 1226}
478 1227
479sub http_get($@) { 1228sub http_get($@) {
480 unshift @_, "GET"; 1229 unshift @_, "GET";
481 &http_request 1230 &http_request
492 &http_request 1241 &http_request
493} 1242}
494 1243
495=back 1244=back
496 1245
1246=head2 DNS CACHING
1247
1248AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for
1249the actual connection, which in turn uses AnyEvent::DNS to resolve
1250hostnames. The latter is a simple stub resolver and does no caching
1251on its own. If you want DNS caching, you currently have to provide
1252your own default resolver (by storing a suitable resolver object in
1253C<$AnyEvent::DNS::RESOLVER>) or your own C<tcp_connect> callback.
1254
497=head2 GLOBAL FUNCTIONS AND VARIABLES 1255=head2 GLOBAL FUNCTIONS AND VARIABLES
498 1256
499=over 4 1257=over 4
500 1258
501=item AnyEvent::HTTP::set_proxy "proxy-url" 1259=item AnyEvent::HTTP::set_proxy "proxy-url"
502 1260
503Sets the default proxy server to use. The proxy-url must begin with a 1261Sets the default proxy server to use. The proxy-url must begin with a
504string of the form C<http://host:port> (optionally C<https:...>). 1262string of the form C<http://host:port>, croaks otherwise.
1263
1264To clear an already-set proxy, use C<undef>.
1265
1266When AnyEvent::HTTP is loaded for the first time it will query the
1267default proxy from the operating system, currently by looking at
1268C<$ENV{http_proxy>}.
1269
1270=item AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]
1271
1272Remove all cookies from the cookie jar that have been expired. If
1273C<$session_end> is given and true, then additionally remove all session
1274cookies.
1275
1276You should call this function (with a true C<$session_end>) before you
1277save cookies to disk, and you should call this function after loading them
1278again. If you have a long-running program you can additionally call this
1279function from time to time.
1280
1281A cookie jar is initially an empty hash-reference that is managed by this
1282module. It's format is subject to change, but currently it is like this:
1283
1284The key C<version> has to contain C<1>, otherwise the hash gets
1285emptied. All other keys are hostnames or IP addresses pointing to
1286hash-references. The key for these inner hash references is the
1287server path for which this cookie is meant, and the values are again
1288hash-references. Each key of those hash-references is a cookie name, and
1289the value, you guessed it, is another hash-reference, this time with the
1290key-value pairs from the cookie, except for C<expires> and C<max-age>,
1291which have been replaced by a C<_expires> key that contains the cookie
1292expiry timestamp. Session cookies are indicated by not having an
1293C<_expires> key.
1294
1295Here is an example of a cookie jar with a single cookie, so you have a
1296chance of understanding the above paragraph:
1297
1298 {
1299 version => 1,
1300 "10.0.0.1" => {
1301 "/" => {
1302 "mythweb_id" => {
1303 _expires => 1293917923,
1304 value => "ooRung9dThee3ooyXooM1Ohm",
1305 },
1306 },
1307 },
1308 }
1309
1310=item $date = AnyEvent::HTTP::format_date $timestamp
1311
1312Takes a POSIX timestamp (seconds since the epoch) and formats it as a HTTP
1313Date (RFC 2616).
1314
1315=item $timestamp = AnyEvent::HTTP::parse_date $date
1316
1317Takes a HTTP Date (RFC 2616) or a Cookie date (netscape cookie spec) or a
1318bunch of minor variations of those, and returns the corresponding POSIX
1319timestamp, or C<undef> if the date cannot be parsed.
505 1320
506=item $AnyEvent::HTTP::MAX_RECURSE 1321=item $AnyEvent::HTTP::MAX_RECURSE
507 1322
508The default value for the C<recurse> request parameter (default: C<10>). 1323The default value for the C<recurse> request parameter (default: C<10>).
509 1324
1325=item $AnyEvent::HTTP::TIMEOUT
1326
1327The default timeout for connection operations (default: C<300>).
1328
510=item $AnyEvent::HTTP::USERAGENT 1329=item $AnyEvent::HTTP::USERAGENT
511 1330
512The default value for the C<User-Agent> header (the default is 1331The default value for the C<User-Agent> header (the default is
513C<Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>). 1332C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
514 1333
515=item $AnyEvent::HTTP::MAX_PERSISTENT 1334=item $AnyEvent::HTTP::MAX_PER_HOST
516 1335
517The maximum number of persistent connections to keep open (default: 8). 1336The maximum number of concurrent connections to the same host (identified
1337by the hostname). If the limit is exceeded, then the additional requests
1338are queued until previous connections are closed. Both persistent and
1339non-persistent connections are counted in this limit.
518 1340
519Not implemented currently. 1341The default value for this is C<4>, and it is highly advisable to not
1342increase it much.
1343
1344For comparison: the RFC's recommend 4 non-persistent or 2 persistent
1345connections, older browsers used 2, newer ones (such as firefox 3)
1346typically use 6, and Opera uses 8 because like, they have the fastest
1347browser and give a shit for everybody else on the planet.
520 1348
521=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT 1349=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
522 1350
523The maximum time to cache a persistent connection, in seconds (default: 2). 1351The time after which idle persistent connections get closed by
524 1352AnyEvent::HTTP (default: C<3>).
525Not implemented currently.
526 1353
527=item $AnyEvent::HTTP::ACTIVE 1354=item $AnyEvent::HTTP::ACTIVE
528 1355
529The number of active connections. This is not the number of currently 1356The number of active connections. This is not the number of currently
530running requests, but the number of currently open and non-idle TCP 1357running requests, but the number of currently open and non-idle TCP
531connections. This number of can be useful for load-leveling. 1358connections. This number can be useful for load-leveling.
532 1359
533=back 1360=back
534 1361
535=cut 1362=cut
536 1363
1364our @month = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
1365our @weekday = qw(Sun Mon Tue Wed Thu Fri Sat);
1366
1367sub format_date($) {
1368 my ($time) = @_;
1369
1370 # RFC 822/1123 format
1371 my ($S, $M, $H, $mday, $mon, $year, $wday, $yday, undef) = gmtime $time;
1372
1373 sprintf "%s, %02d %s %04d %02d:%02d:%02d GMT",
1374 $weekday[$wday], $mday, $month[$mon], $year + 1900,
1375 $H, $M, $S;
1376}
1377
1378sub parse_date($) {
1379 my ($date) = @_;
1380
1381 my ($d, $m, $y, $H, $M, $S);
1382
1383 if ($date =~ /^[A-Z][a-z][a-z]+, ([0-9][0-9]?)[\- ]([A-Z][a-z][a-z])[\- ]([0-9][0-9][0-9][0-9]) ([0-9][0-9]?):([0-9][0-9]?):([0-9][0-9]?) GMT$/) {
1384 # RFC 822/1123, required by RFC 2616 (with " ")
1385 # cookie dates (with "-")
1386
1387 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3, $4, $5, $6);
1388
1389 } elsif ($date =~ /^[A-Z][a-z][a-z]+, ([0-9][0-9]?)-([A-Z][a-z][a-z])-([0-9][0-9]) ([0-9][0-9]?):([0-9][0-9]?):([0-9][0-9]?) GMT$/) {
1390 # RFC 850
1391 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3 < 69 ? $3 + 2000 : $3 + 1900, $4, $5, $6);
1392
1393 } elsif ($date =~ /^[A-Z][a-z][a-z]+ ([A-Z][a-z][a-z]) ([0-9 ]?[0-9]) ([0-9][0-9]?):([0-9][0-9]?):([0-9][0-9]?) ([0-9][0-9][0-9][0-9])$/) {
1394 # ISO C's asctime
1395 ($d, $m, $y, $H, $M, $S) = ($2, $1, $6, $3, $4, $5);
1396 }
1397 # other formats fail in the loop below
1398
1399 for (0..11) {
1400 if ($m eq $month[$_]) {
1401 require Time::Local;
1402 return eval { Time::Local::timegm ($S, $M, $H, $d, $_, $y) };
1403 }
1404 }
1405
1406 undef
1407}
1408
537sub set_proxy($) { 1409sub set_proxy($) {
538 $PROXY = [$2, $3 || 3128, $1] if $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix; 1410 if (length $_[0]) {
1411 $_[0] =~ m%^(http):// ([^:/]+) (?: : (\d*) )?%ix
1412 or Carp::croak "$_[0]: invalid proxy URL";
1413 $PROXY = [$2, $3 || 3128, $1]
1414 } else {
1415 undef $PROXY;
1416 }
539} 1417}
540 1418
541# initialise proxy from environment 1419# initialise proxy from environment
1420eval {
542set_proxy $ENV{http_proxy}; 1421 set_proxy $ENV{http_proxy};
1422};
1423
1424=head2 SHOWCASE
1425
1426This section contains some more elaborate "real-world" examples or code
1427snippets.
1428
1429=head2 HTTP/1.1 FILE DOWNLOAD
1430
1431Downloading files with HTTP can be quite tricky, especially when something
1432goes wrong and you want to resume.
1433
1434Here is a function that initiates and resumes a download. It uses the
1435last modified time to check for file content changes, and works with many
1436HTTP/1.0 servers as well, and usually falls back to a complete re-download
1437on older servers.
1438
1439It calls the completion callback with either C<undef>, which means a
1440nonretryable error occurred, C<0> when the download was partial and should
1441be retried, and C<1> if it was successful.
1442
1443 use AnyEvent::HTTP;
1444
1445 sub download($$$) {
1446 my ($url, $file, $cb) = @_;
1447
1448 open my $fh, "+<", $file
1449 or die "$file: $!";
1450
1451 my %hdr;
1452 my $ofs = 0;
1453
1454 warn stat $fh;
1455 warn -s _;
1456 if (stat $fh and -s _) {
1457 $ofs = -s _;
1458 warn "-s is ", $ofs;
1459 $hdr{"if-unmodified-since"} = AnyEvent::HTTP::format_date +(stat _)[9];
1460 $hdr{"range"} = "bytes=$ofs-";
1461 }
1462
1463 http_get $url,
1464 headers => \%hdr,
1465 on_header => sub {
1466 my ($hdr) = @_;
1467
1468 if ($hdr->{Status} == 200 && $ofs) {
1469 # resume failed
1470 truncate $fh, $ofs = 0;
1471 }
1472
1473 sysseek $fh, $ofs, 0;
1474
1475 1
1476 },
1477 on_body => sub {
1478 my ($data, $hdr) = @_;
1479
1480 if ($hdr->{Status} =~ /^2/) {
1481 length $data == syswrite $fh, $data
1482 or return; # abort on write errors
1483 }
1484
1485 1
1486 },
1487 sub {
1488 my (undef, $hdr) = @_;
1489
1490 my $status = $hdr->{Status};
1491
1492 if (my $time = AnyEvent::HTTP::parse_date $hdr->{"last-modified"}) {
1493 utime $fh, $time, $time;
1494 }
1495
1496 if ($status == 200 || $status == 206 || $status == 416) {
1497 # download ok || resume ok || file already fully downloaded
1498 $cb->(1, $hdr);
1499
1500 } elsif ($status == 412) {
1501 # file has changed while resuming, delete and retry
1502 unlink $file;
1503 $cb->(0, $hdr);
1504
1505 } elsif ($status == 500 or $status == 503 or $status =~ /^59/) {
1506 # retry later
1507 $cb->(0, $hdr);
1508
1509 } else {
1510 $cb->(undef, $hdr);
1511 }
1512 }
1513 ;
1514 }
1515
1516 download "http://server/somelargefile", "/tmp/somelargefile", sub {
1517 if ($_[0]) {
1518 print "OK!\n";
1519 } elsif (defined $_[0]) {
1520 print "please retry later\n";
1521 } else {
1522 print "ERROR\n";
1523 }
1524 };
1525
1526=head3 SOCKS PROXIES
1527
1528Socks proxies are not directly supported by AnyEvent::HTTP. You can
1529compile your perl to support socks, or use an external program such as
1530F<socksify> (dante) or F<tsocks> to make your program use a socks proxy
1531transparently.
1532
1533Alternatively, for AnyEvent::HTTP only, you can use your own
1534C<tcp_connect> function that does the proxy handshake - here is an example
1535that works with socks4a proxies:
1536
1537 use Errno;
1538 use AnyEvent::Util;
1539 use AnyEvent::Socket;
1540 use AnyEvent::Handle;
1541
1542 # host, port and username of/for your socks4a proxy
1543 my $socks_host = "10.0.0.23";
1544 my $socks_port = 9050;
1545 my $socks_user = "";
1546
1547 sub socks4a_connect {
1548 my ($host, $port, $connect_cb, $prepare_cb) = @_;
1549
1550 my $hdl = new AnyEvent::Handle
1551 connect => [$socks_host, $socks_port],
1552 on_prepare => sub { $prepare_cb->($_[0]{fh}) },
1553 on_error => sub { $connect_cb->() },
1554 ;
1555
1556 $hdl->push_write (pack "CCnNZ*Z*", 4, 1, $port, 1, $socks_user, $host);
1557
1558 $hdl->push_read (chunk => 8, sub {
1559 my ($hdl, $chunk) = @_;
1560 my ($status, $port, $ipn) = unpack "xCna4", $chunk;
1561
1562 if ($status == 0x5a) {
1563 $connect_cb->($hdl->{fh}, (format_address $ipn) . ":$port");
1564 } else {
1565 $! = Errno::ENXIO; $connect_cb->();
1566 }
1567 });
1568
1569 $hdl
1570 }
1571
1572Use C<socks4a_connect> instead of C<tcp_connect> when doing C<http_request>s,
1573possibly after switching off other proxy types:
1574
1575 AnyEvent::HTTP::set_proxy undef; # usually you do not want other proxies
1576
1577 http_get 'http://www.google.com', tcp_connect => \&socks4a_connect, sub {
1578 my ($data, $headers) = @_;
1579 ...
1580 };
543 1581
544=head1 SEE ALSO 1582=head1 SEE ALSO
545 1583
546L<AnyEvent>. 1584L<AnyEvent>.
547 1585
548=head1 AUTHOR 1586=head1 AUTHOR
549 1587
550 Marc Lehmann <schmorp@schmorp.de> 1588 Marc Lehmann <schmorp@schmorp.de>
551 http://home.schmorp.de/ 1589 http://home.schmorp.de/
552 1590
1591With many thanks to Дмитрий Шалашов, who provided countless
1592testcases and bugreports.
1593
553=cut 1594=cut
554 1595
5551 15961
556 1597

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines