ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.8 by root, Wed Jun 4 12:32:30 2008 UTC vs.
Revision 1.16 by root, Fri Jun 6 12:57:48 2008 UTC

… …
8 8
9=head1 DESCRIPTION 9=head1 DESCRIPTION
10 10
11This module is an L<AnyEvent> user, you need to make sure that you use and 11This module is an L<AnyEvent> user, you need to make sure that you use and
12run a supported event loop. 12run a supported event loop.
13
14This module implements a simple, stateless and non-blocking HTTP
15client. It supports GET, POST and other request methods, cookies and more,
16all on a very low level. It can follow redirects supports proxies and
17automatically limits the number of connections to the values specified in
18the RFC.
19
20It should generally be a "good client" that is enough for most HTTP
21tasks. Simple tasks should be simple, but complex tasks should still be
22possible as the user retains control over request and response headers.
23
24The caller is responsible for authentication management, cookies (if
25the simplistic implementation in this module doesn't suffice), referer
26and other high-level protocol details for which this module offers only
27limited support.
13 28
14=head2 METHODS 29=head2 METHODS
15 30
16=over 4 31=over 4
17 32
… …
29use AnyEvent::Socket (); 44use AnyEvent::Socket ();
30use AnyEvent::Handle (); 45use AnyEvent::Handle ();
31 46
32use base Exporter::; 47use base Exporter::;
33 48
34our $VERSION = '1.0'; 49our $VERSION = '1.01';
35 50
36our @EXPORT = qw(http_get http_request); 51our @EXPORT = qw(http_get http_request);
37 52
38our $USERAGENT = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 53our $USERAGENT = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
39our $MAX_RECURSE = 10; 54our $MAX_RECURSE = 10;
… …
41our $PERSISTENT_TIMEOUT = 2; 56our $PERSISTENT_TIMEOUT = 2;
42our $TIMEOUT = 300; 57our $TIMEOUT = 300;
43 58
44# changing these is evil 59# changing these is evil
45our $MAX_PERSISTENT_PER_HOST = 2; 60our $MAX_PERSISTENT_PER_HOST = 2;
46our $MAX_PER_HOST = 4; # not respected yet :( 61our $MAX_PER_HOST = 4;
47 62
48our $PROXY; 63our $PROXY;
64our $ACTIVE = 0;
49 65
50my %KA_COUNT; # number of open keep-alive connections per host 66my %KA_COUNT; # number of open keep-alive connections per host
67my %CO_SLOT; # number of open connections, and wait queue, per host
51 68
52=item http_get $url, key => value..., $cb->($data, $headers) 69=item http_get $url, key => value..., $cb->($data, $headers)
53 70
54Executes an HTTP-GET request. See the http_request function for details on 71Executes an HTTP-GET request. See the http_request function for details on
55additional parameters. 72additional parameters.
… …
74response headers as second argument. 91response headers as second argument.
75 92
76All the headers in that hash are lowercased. In addition to the response 93All the headers in that hash are lowercased. In addition to the response
77headers, the three "pseudo-headers" C<HTTPVersion>, C<Status> and 94headers, the three "pseudo-headers" C<HTTPVersion>, C<Status> and
78C<Reason> contain the three parts of the HTTP Status-Line of the same 95C<Reason> contain the three parts of the HTTP Status-Line of the same
79name. 96name. If the server sends a header multiple lines, then their contents
97will be joined together with C<\x00>.
80 98
81If an internal error occurs, such as not being able to resolve a hostname, 99If an internal error occurs, such as not being able to resolve a hostname,
82then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599> 100then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599>
83and the C<Reason> pseudo-header will contain an error message. 101and the C<Reason> pseudo-header will contain an error message.
84 102
… …
104Whether to recurse requests or not, e.g. on redirects, authentication 122Whether to recurse requests or not, e.g. on redirects, authentication
105retries and so on, and how often to do so. 123retries and so on, and how often to do so.
106 124
107=item headers => hashref 125=item headers => hashref
108 126
109The request headers to use. 127The request headers to use. Currently, C<http_request> may provide its
128own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
129and will provide defaults for C<User-Agent:> and C<Referer:>.
110 130
111=item timeout => $seconds 131=item timeout => $seconds
112 132
113The time-out to use for various stages - each connect attempt will reset 133The time-out to use for various stages - each connect attempt will reset
114the timeout, as will read or write activity. Default timeout is 5 minutes. 134the timeout, as will read or write activity. Default timeout is 5 minutes.
… …
124=item body => $string 144=item body => $string
125 145
126The request body, usually empty. Will be-sent as-is (future versions of 146The request body, usually empty. Will be-sent as-is (future versions of
127this module might offer more options). 147this module might offer more options).
128 148
149=item cookie_jar => $hash_ref
150
151Passing this parameter enables (simplified) cookie-processing, loosely
152based on the original netscape specification.
153
154The C<$hash_ref> must be an (initially empty) hash reference which will
155get updated automatically. It is possible to save the cookie_jar to
156persistent storage with something like JSON or Storable, but this is not
157recommended, as expire times are currently being ignored.
158
159Note that this cookie implementation is not of very high quality, nor
160meant to be complete. If you want complete cookie management you have to
161do that on your own. C<cookie_jar> is meant as a quick fix to get some
162cookie-using sites working. Cookies are a privacy disaster, do not use
163them unless required to.
164
129=back 165=back
130 166
131=back 167Example: make a simple HTTP GET request for http://www.nethype.de/
168
169 http_request GET => "http://www.nethype.de/", sub {
170 my ($body, $hdr) = @_;
171 print "$body\n";
172 };
173
174Example: make a HTTP HEAD request on https://www.google.com/, use a
175timeout of 30 seconds.
176
177 http_request
178 GET => "https://www.google.com",
179 timeout => 30,
180 sub {
181 my ($body, $hdr) = @_;
182 use Data::Dumper;
183 print Dumper $hdr;
184 }
185 ;
132 186
133=cut 187=cut
134 188
189sub _slot_schedule;
190sub _slot_schedule($) {
191 my $host = shift;
192
193 while ($CO_SLOT{$host}[0] < $MAX_PER_HOST) {
194 if (my $cb = shift @{ $CO_SLOT{$host}[1] }) {
195 # somebody wants that slot
196 ++$CO_SLOT{$host}[0];
197 ++$ACTIVE;
198
199 $cb->(AnyEvent::Util::guard {
200 --$ACTIVE;
201 --$CO_SLOT{$host}[0];
202 _slot_schedule $host;
203 });
204 } else {
205 # nobody wants the slot, maybe we can forget about it
206 delete $CO_SLOT{$host} unless $CO_SLOT{$host}[0];
207 last;
208 }
209 }
210}
211
212# wait for a free slot on host, call callback
213sub _get_slot($$) {
214 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
215
216 _slot_schedule $_[0];
217}
218
135sub http_request($$$;@) { 219sub http_request($$@) {
136 my $cb = pop; 220 my $cb = pop;
137 my ($method, $url, %arg) = @_; 221 my ($method, $url, %arg) = @_;
138 222
139 my %hdr; 223 my %hdr;
140 224
… …
154 my $proxy = $arg{proxy} || $PROXY; 238 my $proxy = $arg{proxy} || $PROXY;
155 my $timeout = $arg{timeout} || $TIMEOUT; 239 my $timeout = $arg{timeout} || $TIMEOUT;
156 240
157 $hdr{"user-agent"} ||= $USERAGENT; 241 $hdr{"user-agent"} ||= $USERAGENT;
158 242
159 my ($host, $port, $path, $scheme); 243 my ($scheme, $authority, $upath, $query, $fragment) =
244 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|;
245
246 $scheme = lc $scheme;
247
248 my $uport = $scheme eq "http" ? 80
249 : $scheme eq "https" ? 443
250 : return $cb->(undef, { Status => 599, Reason => "only http and https URL schemes supported" });
251
252 $hdr{referer} ||= "$scheme://$authority$upath"; # leave out fragment and query string, just a heuristic
253
254 $authority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
255 or return $cb->(undef, { Status => 599, Reason => "unparsable URL" });
256
257 my $uhost = $1;
258 $uport = $2 if defined $2;
259
260 $uhost =~ s/^\[(.*)\]$/$1/;
261 $upath .= "?$query" if length $query;
262
263 $upath =~ s%^/?%/%;
264
265 # cookie processing
266 if (my $jar = $arg{cookie_jar}) {
267 %$jar = () if $jar->{version} < 1;
268
269 my @cookie;
270
271 while (my ($chost, $v) = each %$jar) {
272 next unless $chost eq substr $uhost, -length $chost;
273 next unless $chost =~ /^\./;
274
275 while (my ($cpath, $v) = each %$v) {
276 next unless $cpath eq substr $upath, 0, length $cpath;
277
278 while (my ($k, $v) = each %$v) {
279 next if $scheme ne "https" && exists $v->{secure};
280 push @cookie, "$k=$v->{value}";
281 }
282 }
283 }
284
285 $hdr{cookie} = join "; ", @cookie
286 if @cookie;
287 }
288
289 my ($rhost, $rport, $rpath); # request host, port, path
160 290
161 if ($proxy) { 291 if ($proxy) {
162 ($host, $port, $scheme) = @$proxy; 292 ($rhost, $rport, $scheme) = @$proxy;
163 $path = $url; 293 $rpath = $url;
164 } else { 294 } else {
165 ($scheme, my $authority, $path, my $query, my $fragment) = 295 ($rhost, $rport, $rpath) = ($uhost, $uport, $upath);
166 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|;
167
168 $port = $scheme eq "http" ? 80
169 : $scheme eq "https" ? 443
170 : return $cb->(undef, { Status => 599, Reason => "$url: only http and https URLs supported" });
171
172 $authority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
173 or return $cb->(undef, { Status => 599, Reason => "$url: unparsable URL" });
174
175 $host = $1;
176 $port = $2 if defined $2;
177
178 $host =~ s/^\[(.*)\]$/$1/;
179 $path .= "?$query" if length $query;
180
181 $path = "/" unless $path;
182
183 $hdr{host} = $host = lc $host; 296 $hdr{host} = $uhost;
184 } 297 }
185 298
186 $scheme = lc $scheme;
187
188 my %state;
189
190 $hdr{"content-length"} = length $arg{body}; 299 $hdr{"content-length"} = length $arg{body};
191 300
301 my %state = (connect_guard => 1);
302
303 _get_slot $uhost, sub {
304 $state{slot_guard} = shift;
305
306 return unless $state{connect_guard};
307
192 $state{connect_guard} = AnyEvent::Socket::tcp_connect $host, $port, sub { 308 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
193 $state{fh} = shift 309 $state{fh} = shift
194 or return $cb->(undef, { Status => 599, Reason => "$!" }); 310 or return $cb->(undef, { Status => 599, Reason => "$!" });
195 311
196 delete $state{connect_guard}; # reduce memory usage, save a tree 312 delete $state{connect_guard}; # reduce memory usage, save a tree
197 313
198 # get handle 314 # get handle
199 $state{handle} = new AnyEvent::Handle 315 $state{handle} = new AnyEvent::Handle
200 fh => $state{fh}, 316 fh => $state{fh},
201 ($scheme eq "https" ? (tls => "connect") : ()); 317 ($scheme eq "https" ? (tls => "connect") : ());
202 318
203 # limit the number of persistent connections 319 # limit the number of persistent connections
204 if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) { 320 if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
205 ++$KA_COUNT{$_[1]}; 321 ++$KA_COUNT{$_[1]};
206 $state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} }; 322 $state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} };
207 $hdr{connection} = "keep-alive"; 323 $hdr{connection} = "keep-alive";
208 delete $hdr{connection}; # keep-alive not yet supported 324 delete $hdr{connection}; # keep-alive not yet supported
209 } else { 325 } else {
210 delete $hdr{connection}; 326 delete $hdr{connection};
211 } 327 }
212 328
213 # (re-)configure handle 329 # (re-)configure handle
214 $state{handle}->timeout ($timeout); 330 $state{handle}->timeout ($timeout);
215 $state{handle}->on_error (sub { 331 $state{handle}->on_error (sub {
332 my $errno = "$!";
216 %state = (); 333 %state = ();
217 $cb->(undef, { Status => 599, Reason => "$!" }); 334 $cb->(undef, { Status => 599, Reason => $errno });
218 }); 335 });
219 $state{handle}->on_eof (sub { 336 $state{handle}->on_eof (sub {
220 %state = (); 337 %state = ();
221 $cb->(undef, { Status => 599, Reason => "unexpected end-of-file" }); 338 $cb->(undef, { Status => 599, Reason => "unexpected end-of-file" });
222 }); 339 });
223 340
224 # send request 341 # send request
225 $state{handle}->push_write ( 342 $state{handle}->push_write (
226 "$method $path HTTP/1.0\015\012" 343 "$method $rpath HTTP/1.0\015\012"
227 . (join "", map "$_: $hdr{$_}\015\012", keys %hdr) 344 . (join "", map "$_: $hdr{$_}\015\012", keys %hdr)
228 . "\015\012" 345 . "\015\012"
229 . (delete $arg{body}) 346 . (delete $arg{body})
230 );
231
232 %hdr = (); # reduce memory usage, save a kitten
233
234 # status line
235 $state{handle}->push_read (line => qr/\015?\012/, sub {
236 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) \s+ ([^\015\012]+)/ix
237 or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])" }));
238
239 my %hdr = ( # response headers
240 HTTPVersion => ",$1",
241 Status => ",$2",
242 Reason => ",$3",
243 ); 347 );
244 348
349 %hdr = (); # reduce memory usage, save a kitten
350
351 # status line
352 $state{handle}->push_read (line => qr/\015?\012/, sub {
353 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) \s+ ([^\015\012]+)/ix
354 or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])" }));
355
356 my %hdr = ( # response headers
357 HTTPVersion => "\x00$1",
358 Status => "\x00$2",
359 Reason => "\x00$3",
360 );
361
245 # headers, could be optimized a bit 362 # headers, could be optimized a bit
246 $state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub { 363 $state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub {
247 for ("$_[1]\012") { 364 for ("$_[1]\012") {
248 # we support spaces in field names, as lotus domino 365 # we support spaces in field names, as lotus domino
249 # creates them. 366 # creates them.
250 $hdr{lc $1} .= ",$2" 367 $hdr{lc $1} .= "\x00$2"
251 while /\G 368 while /\G
252 ([^:\000-\037]+): 369 ([^:\000-\037]+):
253 [\011\040]* 370 [\011\040]*
254 ((?: [^\015\012]+ | \015?\012[\011\040] )*) 371 ((?: [^\015\012]+ | \015?\012[\011\040] )*)
255 \015?\012 372 \015?\012
256 /gxc; 373 /gxc;
257 374
258 /\G$/ 375 /\G$/
259 or return $cb->(undef, { Status => 599, Reason => "garbled response headers" }); 376 or return (%state = (), $cb->(undef, { Status => 599, Reason => "garbled response headers" }));
260 } 377 }
261 378
262 substr $_, 0, 1, "" 379 substr $_, 0, 1, ""
263 for values %hdr; 380 for values %hdr;
264 381
265 my $finish = sub { 382 my $finish = sub {
383 %state = ();
384
385 # set-cookie processing
386 if ($arg{cookie_jar} && exists $hdr{"set-cookie"}) {
387 for (split /\x00/, $hdr{"set-cookie"}) {
388 my ($cookie, @arg) = split /;\s*/;
389 my ($name, $value) = split /=/, $cookie, 2;
390 my %kv = (value => $value, map { split /=/, $_, 2 } @arg);
391
392 my $cdom = (delete $kv{domain}) || $uhost;
393 my $cpath = (delete $kv{path}) || "/";
394
395 $cdom =~ s/^.?/./; # make sure it starts with a "."
396
397 next if $cdom =~ /\.$/;
398
399 # this is not rfc-like and not netscape-like. go figure.
400 my $ndots = $cdom =~ y/.//;
401 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
402
403 # store it
404 $arg{cookie_jar}{version} = 1;
405 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
406 }
407 }
408
266 if ($_[1]{Status} =~ /^30[12]$/ && $recurse) { 409 if ($_[1]{Status} =~ /^30[12]$/ && $recurse) {
410 # microsoft and other assholes don't give a shit for following standards,
411 # try to support a common form of broken Location header.
412 $_[1]{location} =~ s%^/%$scheme://$uhost:$uport/%;
413
267 http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb); 414 http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
415 } else {
416 $cb->($_[0], $_[1]);
417 }
418 };
419
420 if ($hdr{Status} =~ /^(?:1..|204|304)$/ or $method eq "HEAD") {
421 $finish->(undef, \%hdr);
268 } else { 422 } else {
269 $cb->($_[0], $_[1]); 423 if (exists $hdr{"content-length"}) {
424 $_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
425 # could cache persistent connection now
426 if ($hdr{connection} =~ /\bkeep-alive\b/i) {
427 # but we don't, due to misdesigns, this is annoyingly complex
428 };
429
430 $finish->($_[1], \%hdr);
431 });
432 } else {
433 # too bad, need to read until we get an error or EOF,
434 # no way to detect winged data.
435 $_[0]->on_error (sub {
436 $finish->($_[0]{rbuf}, \%hdr);
437 });
438 $_[0]->on_eof (undef);
439 $_[0]->on_read (sub { });
440 }
270 } 441 }
271 }; 442 });
272
273 if ($hdr{Status} =~ /^(?:1..|204|304)$/ or $method eq "HEAD") {
274 %state = ();
275 $finish->(undef, \%hdr);
276 } else {
277 if (exists $hdr{"content-length"}) {
278 $_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
279 # could cache persistent connection now
280 if ($hdr{connection} =~ /\bkeep-alive\b/i) {
281 # but we don't, due to misdesigns, this is annoyingly complex
282 };
283
284 %state = ();
285 $finish->($_[1], \%hdr);
286 });
287 } else {
288 # too bad, need to read until we get an error or EOF,
289 # no way to detect winged data.
290 $_[0]->on_error (sub {
291 %state = ();
292 $finish->($_[0]{rbuf}, \%hdr);
293 });
294 $_[0]->on_eof (undef);
295 $_[0]->on_read (sub { });
296 }
297 }
298 }); 443 });
444 }, sub {
445 $timeout
299 }); 446 };
300 }, sub {
301 $timeout
302 }; 447 };
303 448
304 defined wantarray && AnyEvent::Util::guard { %state = () } 449 defined wantarray && AnyEvent::Util::guard { %state = () }
305} 450}
306 451
307sub http_get($$;@) { 452sub http_get($@) {
308 unshift @_, "GET"; 453 unshift @_, "GET";
309 &http_request 454 &http_request
310} 455}
311 456
312sub http_head($$;@) { 457sub http_head($@) {
313 unshift @_, "HEAD"; 458 unshift @_, "HEAD";
314 &http_request 459 &http_request
315} 460}
316 461
317sub http_post($$$;@) { 462sub http_post($$@) {
318 unshift @_, "POST", "body"; 463 unshift @_, "POST", "body";
319 &http_request 464 &http_request
320} 465}
321 466
467=back
468
322=head2 GLOBAL FUNCTIONS AND VARIABLES 469=head2 GLOBAL FUNCTIONS AND VARIABLES
323 470
324=over 4 471=over 4
325 472
326=item AnyEvent::HTTP::set_proxy "proxy-url" 473=item AnyEvent::HTTP::set_proxy "proxy-url"
… …
347 494
348The maximum time to cache a persistent connection, in seconds (default: 2). 495The maximum time to cache a persistent connection, in seconds (default: 2).
349 496
350Not implemented currently. 497Not implemented currently.
351 498
499=item $AnyEvent::HTTP::ACTIVE
500
501The number of active connections. This is not the number of currently
502running requests, but the number of currently open and non-idle TCP
503connections. This number of can be useful for load-leveling.
504
352=back 505=back
353 506
354=cut 507=cut
355 508
356sub set_proxy($) { 509sub set_proxy($) {

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines