ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/README
(Generate patch)

Comparing AnyEvent-HTTP/README (file contents):
Revision 1.14 by root, Fri Dec 31 03:47:32 2010 UTC vs.
Revision 1.21 by root, Tue Jun 14 05:23:12 2011 UTC

… …
12 This module is an AnyEvent user, you need to make sure that you use and 12 This module is an AnyEvent user, you need to make sure that you use and
13 run a supported event loop. 13 run a supported event loop.
14 14
15 This module implements a simple, stateless and non-blocking HTTP client. 15 This module implements a simple, stateless and non-blocking HTTP client.
16 It supports GET, POST and other request methods, cookies and more, all 16 It supports GET, POST and other request methods, cookies and more, all
17 on a very low level. It can follow redirects supports proxies and 17 on a very low level. It can follow redirects, supports proxies, and
18 automatically limits the number of connections to the values specified 18 automatically limits the number of connections to the values specified
19 in the RFC. 19 in the RFC.
20 20
21 It should generally be a "good client" that is enough for most HTTP 21 It should generally be a "good client" that is enough for most HTTP
22 tasks. Simple tasks should be simple, but complex tasks should still be 22 tasks. Simple tasks should be simple, but complex tasks should still be
… …
51 gets destroyed before the callback is called, the request will be 51 gets destroyed before the callback is called, the request will be
52 cancelled. 52 cancelled.
53 53
54 The callback will be called with the response body data as first 54 The callback will be called with the response body data as first
55 argument (or "undef" if an error occured), and a hash-ref with 55 argument (or "undef" if an error occured), and a hash-ref with
56 response headers as second argument. 56 response headers (and trailers) as second argument.
57 57
58 All the headers in that hash are lowercased. In addition to the 58 All the headers in that hash are lowercased. In addition to the
59 response headers, the "pseudo-headers" (uppercase to avoid clashing 59 response headers, the "pseudo-headers" (uppercase to avoid clashing
60 with possible response headers) "HTTPVersion", "Status" and "Reason" 60 with possible response headers) "HTTPVersion", "Status" and "Reason"
61 contain the three parts of the HTTP Status-Line of the same name. If 61 contain the three parts of the HTTP Status-Line of the same name. If
… …
79 If the server sends a header multiple times, then their contents 79 If the server sends a header multiple times, then their contents
80 will be joined together with a comma (","), as per the HTTP spec. 80 will be joined together with a comma (","), as per the HTTP spec.
81 81
82 If an internal error occurs, such as not being able to resolve a 82 If an internal error occurs, such as not being able to resolve a
83 hostname, then $data will be "undef", "$headers->{Status}" will be 83 hostname, then $data will be "undef", "$headers->{Status}" will be
84 "59x" (usually 599) and the "Reason" pseudo-header will contain an 84 590-599 and the "Reason" pseudo-header will contain an error
85 error message. 85 message. Currently the following status codes are used:
86
87 595 - errors during connection etsbalishment, proxy handshake.
88 596 - errors during TLS negotiation, request sending and header
89 processing.
90 597 - errors during body receiving or processing.
91 598 - user aborted request via "on_header" or "on_body".
92 599 - other, usually nonretryable, errors (garbled URL etc.).
86 93
87 A typical callback might look like this: 94 A typical callback might look like this:
88 95
89 sub { 96 sub {
90 my ($body, $hdr) = @_; 97 my ($body, $hdr) = @_;
… …
104 authentication retries and so on, and how often to do so. 111 authentication retries and so on, and how often to do so.
105 112
106 headers => hashref 113 headers => hashref
107 The request headers to use. Currently, "http_request" may 114 The request headers to use. Currently, "http_request" may
108 provide its own "Host:", "Content-Length:", "Connection:" and 115 provide its own "Host:", "Content-Length:", "Connection:" and
109 "Cookie:" headers and will provide defaults for "User-Agent:" 116 "Cookie:" headers and will provide defaults at least for "TE:",
110 and "Referer:" (this can be suppressed by using "undef" for 117 "Referer:" and "User-Agent:" (this can be suppressed by using
111 these headers in which case they won't be sent at all). 118 "undef" for these headers in which case they won't be sent at
119 all).
120
121 You really should provide your own "User-Agent:" header value
122 that is appropriate for your program - I wouldn't be surprised
123 if the default AnyEvent string gets blocked by webservers sooner
124 or later.
125
126 Also, make sure that your headers names and values do not
127 contain any embedded newlines.
112 128
113 timeout => $seconds 129 timeout => $seconds
114 The time-out to use for various stages - each connect attempt 130 The time-out to use for various stages - each connect attempt
115 will reset the timeout, as will read or write activity, i.e. 131 will reset the timeout, as will read or write activity, i.e.
116 this is not an overall timeout. 132 this is not an overall timeout.
117 133
118 Default timeout is 5 minutes. 134 Default timeout is 5 minutes.
119 135
120 proxy => [$host, $port[, $scheme]] or undef 136 proxy => [$host, $port[, $scheme]] or undef
121 Use the given http proxy for all requests. If not specified, 137 Use the given http proxy for all requests, or no proxy if
122 then the default proxy (as specified by $ENV{http_proxy}) is
123 used. 138 "undef" is used.
124 139
125 $scheme must be either missing, "http" for HTTP or "https" for 140 $scheme must be either missing or must be "http" for HTTP.
126 HTTPS. 141
142 If not specified, then the default proxy is used (see
143 "AnyEvent::HTTP::set_proxy").
127 144
128 body => $string 145 body => $string
129 The request body, usually empty. Will be-sent as-is (future 146 The request body, usually empty. Will be sent as-is (future
130 versions of this module might offer more options). 147 versions of this module might offer more options).
131 148
132 cookie_jar => $hash_ref 149 cookie_jar => $hash_ref
133 Passing this parameter enables (simplified) cookie-processing, 150 Passing this parameter enables (simplified) cookie-processing,
134 loosely based on the original netscape specification. 151 loosely based on the original netscape specification.
135 152
136 The $hash_ref must be an (initially empty) hash reference which 153 The $hash_ref must be an (initially empty) hash reference which
137 will get updated automatically. It is possible to save the 154 will get updated automatically. It is possible to save the
138 cookie_jar to persistent storage with something like JSON or 155 cookie jar to persistent storage with something like JSON or
139 Storable, but this is not recommended, as expiry times are 156 Storable - see the "AnyEvent::HTTP::cookie_jar_expire" function
140 currently being ignored. 157 if you wish to remove expired or session-only cookies, and also
158 for documentation on the format of the cookie jar.
141 159
142 Note that this cookie implementation is not of very high 160 Note that this cookie implementation is not meant to be
143 quality, nor meant to be complete. If you want complete cookie 161 complete. If you want complete cookie management you have to do
144 management you have to do that on your own. "cookie_jar" is 162 that on your own. "cookie_jar" is meant as a quick fix to get
145 meant as a quick fix to get some cookie-using sites working. 163 most cookie-using sites working. Cookies are a privacy disaster,
146 Cookies are a privacy disaster, do not use them unless required 164 do not use them unless required to.
147 to. 165
166 When cookie processing is enabled, the "Cookie:" and
167 "Set-Cookie:" headers will be set and handled by this module,
168 otherwise they will be left untouched.
148 169
149 tls_ctx => $scheme | $tls_ctx 170 tls_ctx => $scheme | $tls_ctx
150 Specifies the AnyEvent::TLS context to be used for https 171 Specifies the AnyEvent::TLS context to be used for https
151 connections. This parameter follows the same rules as the 172 connections. This parameter follows the same rules as the
152 "tls_ctx" parameter to AnyEvent::Handle, but additionally, the 173 "tls_ctx" parameter to AnyEvent::Handle, but additionally, the
… …
154 predefined low-security (no verification, highest compatibility) 175 predefined low-security (no verification, highest compatibility)
155 and high-security (CA and common-name verification) TLS context. 176 and high-security (CA and common-name verification) TLS context.
156 177
157 The default for this option is "low", which could be interpreted 178 The default for this option is "low", which could be interpreted
158 as "give me the page, no matter what". 179 as "give me the page, no matter what".
180
181 See also the "sessionid" parameter.
182
183 session => $string
184 The module might reuse connections to the same host internally.
185 Sometimes (e.g. when using TLS), you do not want to reuse
186 connections from other sessions. This can be achieved by setting
187 this parameter to some unique ID (such as the address of an
188 object storing your state data, or the TLS context) - only
189 connections using the same unique ID will be reused.
159 190
160 on_prepare => $callback->($fh) 191 on_prepare => $callback->($fh)
161 In rare cases you need to "tune" the socket before it is used to 192 In rare cases you need to "tune" the socket before it is used to
162 connect (for exmaple, to bind it on a given IP address). This 193 connect (for exmaple, to bind it on a given IP address). This
163 parameter overrides the prepare callback passed to 194 parameter overrides the prepare callback passed to
… …
192 223
193 This callback is useful, among other things, to quickly reject 224 This callback is useful, among other things, to quickly reject
194 unwanted content, which, if it is supposed to be rare, can be 225 unwanted content, which, if it is supposed to be rare, can be
195 faster than first doing a "HEAD" request. 226 faster than first doing a "HEAD" request.
196 227
228 The downside is that cancelling the request makes it impossible
229 to re-use the connection. Also, the "on_header" callback will
230 not receive any trailer (headers sent after the response body).
231
197 Example: cancel the request unless the content-type is 232 Example: cancel the request unless the content-type is
198 "text/html". 233 "text/html".
199 234
200 on_header => sub { 235 on_header => sub {
201 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/ 236 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
… …
208 243
209 It has to return either true (in which case AnyEvent::HTTP will 244 It has to return either true (in which case AnyEvent::HTTP will
210 continue), or false, in which case AnyEvent::HTTP will cancel 245 continue), or false, in which case AnyEvent::HTTP will cancel
211 the download (and call the completion callback with an error 246 the download (and call the completion callback with an error
212 code of 598). 247 code of 598).
248
249 The downside to cancelling the request is that it makes it
250 impossible to re-use the connection.
213 251
214 This callback is useful when the data is too large to be held in 252 This callback is useful when the data is too large to be held in
215 memory (so the callback writes it to a file) or when only some 253 memory (so the callback writes it to a file) or when only some
216 information should be extracted, or when the body should be 254 information should be extracted, or when the body should be
217 processed incrementally. 255 processed incrementally.
… …
231 AnyEvent::Handle object associated with the connection. In error 269 AnyEvent::Handle object associated with the connection. In error
232 cases, "undef" will be passed. When there is no body (e.g. 270 cases, "undef" will be passed. When there is no body (e.g.
233 status 304), the empty string will be passed. 271 status 304), the empty string will be passed.
234 272
235 The handle object might or might not be in TLS mode, might be 273 The handle object might or might not be in TLS mode, might be
236 connected to a proxy, be a persistent connection etc., and 274 connected to a proxy, be a persistent connection, use chunked
237 configured in unspecified ways. The user is responsible for this 275 transfer encoding etc., and configured in unspecified ways. The
238 handle (it will not be used by this module anymore). 276 user is responsible for this handle (it will not be used by this
277 module anymore).
239 278
240 This is useful with some push-type services, where, after the 279 This is useful with some push-type services, where, after the
241 initial headers, an interactive protocol is used (typical 280 initial headers, an interactive protocol is used (typical
242 example would be the push-style twitter API which starts a 281 example would be the push-style twitter API which starts a
243 JSON/XML stream). 282 JSON/XML stream).
244 283
245 If you think you need this, first have a look at "on_body", to 284 If you think you need this, first have a look at "on_body", to
246 see if that doesn't solve your problem in a better way. 285 see if that doesn't solve your problem in a better way.
247 286
287 persistent => $boolean
288 Try to create/reuse a persistent connection. When this flag is
289 set (default: true for idempotent requests, false for all
290 others), then "http_request" tries to re-use an existing
291 (previously-created) persistent connection to the host and,
292 failing that, tries to create a new one.
293
294 Requests failing in certain ways will be automatically retried
295 once, which is dangerous for non-idempotent requests, which is
296 why it defaults to off for them. The reason for this is because
297 the bozos who designed HTTP/1.1 made it impossible to
298 distinguish between a fatal error and a normal connection
299 timeout, so you never know whether there was a problem with your
300 request or not.
301
302 When reusing an existent connection, many parameters (such as
303 TLS context) will be ignored. See the "session" parameter for a
304 workaround.
305
306 keepalive => $boolean
307 Only used when "persistent" is also true. This parameter decides
308 whether "http_request" tries to handshake a HTTP/1.0-style
309 keep-alive connection (as opposed to only a HTTP/1.1 persistent
310 connection).
311
312 The default is true, except when using a proxy, in which case it
313 defaults to false, as HTTP/1.0 proxies cannot support this in a
314 meaningful way.
315
316 handle_params => { key => value ... }
317 The key-value pairs in this hash will be passed to any
318 AnyEvent::Handle constructor that is called - not all requests
319 will create a handle, and sometimes more than one is created, so
320 this parameter is only good for setting hints.
321
322 Example: set the maximum read size to 4096, to potentially
323 conserve memory at the cost of speed.
324
325 handle_params => {
326 max_read_size => 4096,
327 },
328
248 Example: make a simple HTTP GET request for http://www.nethype.de/ 329 Example: do a simple HTTP GET request for http://www.nethype.de/ and
330 print the response body.
249 331
250 http_request GET => "http://www.nethype.de/", sub { 332 http_request GET => "http://www.nethype.de/", sub {
251 my ($body, $hdr) = @_; 333 my ($body, $hdr) = @_;
252 print "$body\n"; 334 print "$body\n";
253 }; 335 };
254 336
255 Example: make a HTTP HEAD request on https://www.google.com/, use a 337 Example: do a HTTP HEAD request on https://www.google.com/, use a
256 timeout of 30 seconds. 338 timeout of 30 seconds.
257 339
258 http_request 340 http_request
259 GET => "https://www.google.com", 341 GET => "https://www.google.com",
342 headers => { "user-agent" => "MySearchClient 1.0" },
260 timeout => 30, 343 timeout => 30,
261 sub { 344 sub {
262 my ($body, $hdr) = @_; 345 my ($body, $hdr) = @_;
263 use Data::Dumper; 346 use Data::Dumper;
264 print Dumper $hdr; 347 print Dumper $hdr;
265 } 348 }
266 ; 349 ;
267 350
268 Example: make another simple HTTP GET request, but immediately try 351 Example: do another simple HTTP GET request, but immediately try to
269 to cancel it. 352 cancel it.
270 353
271 my $request = http_request GET => "http://www.nethype.de/", sub { 354 my $request = http_request GET => "http://www.nethype.de/", sub {
272 my ($body, $hdr) = @_; 355 my ($body, $hdr) = @_;
273 print "$body\n"; 356 print "$body\n";
274 }; 357 };
… …
279 AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for the 362 AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for the
280 actual connection, which in turn uses AnyEvent::DNS to resolve 363 actual connection, which in turn uses AnyEvent::DNS to resolve
281 hostnames. The latter is a simple stub resolver and does no caching on 364 hostnames. The latter is a simple stub resolver and does no caching on
282 its own. If you want DNS caching, you currently have to provide your own 365 its own. If you want DNS caching, you currently have to provide your own
283 default resolver (by storing a suitable resolver object in 366 default resolver (by storing a suitable resolver object in
284 $AnyEvent::DNS::RESOLVER). 367 $AnyEvent::DNS::RESOLVER) or your own "tcp_connect" callback.
285 368
286 GLOBAL FUNCTIONS AND VARIABLES 369 GLOBAL FUNCTIONS AND VARIABLES
287 AnyEvent::HTTP::set_proxy "proxy-url" 370 AnyEvent::HTTP::set_proxy "proxy-url"
288 Sets the default proxy server to use. The proxy-url must begin with 371 Sets the default proxy server to use. The proxy-url must begin with
289 a string of the form "http://host:port" (optionally "https:..."), 372 a string of the form "http://host:port", croaks otherwise.
290 croaks otherwise.
291 373
292 To clear an already-set proxy, use "undef". 374 To clear an already-set proxy, use "undef".
375
376 When AnyEvent::HTTP is laoded for the first time it will query the
377 default proxy from the operating system, currently by looking at
378 "$ENV{http_proxy"}.
379
380 AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]
381 Remove all cookies from the cookie jar that have been expired. If
382 $session_end is given and true, then additionally remove all session
383 cookies.
384
385 You should call this function (with a true $session_end) before you
386 save cookies to disk, and you should call this function after
387 loading them again. If you have a long-running program you can
388 additonally call this function from time to time.
389
390 A cookie jar is initially an empty hash-reference that is managed by
391 this module. It's format is subject to change, but currently it is
392 like this:
393
394 The key "version" has to contain 1, otherwise the hash gets emptied.
395 All other keys are hostnames or IP addresses pointing to
396 hash-references. The key for these inner hash references is the
397 server path for which this cookie is meant, and the values are again
398 hash-references. The keys of those hash-references is the cookie
399 name, and the value, you guessed it, is another hash-reference, this
400 time with the key-value pairs from the cookie, except for "expires"
401 and "max-age", which have been replaced by a "_expires" key that
402 contains the cookie expiry timestamp.
403
404 Here is an example of a cookie jar with a single cookie, so you have
405 a chance of understanding the above paragraph:
406
407 {
408 version => 1,
409 "10.0.0.1" => {
410 "/" => {
411 "mythweb_id" => {
412 _expires => 1293917923,
413 value => "ooRung9dThee3ooyXooM1Ohm",
414 },
415 },
416 },
417 }
293 418
294 $date = AnyEvent::HTTP::format_date $timestamp 419 $date = AnyEvent::HTTP::format_date $timestamp
295 Takes a POSIX timestamp (seconds since the epoch) and formats it as 420 Takes a POSIX timestamp (seconds since the epoch) and formats it as
296 a HTTP Date (RFC 2616). 421 a HTTP Date (RFC 2616).
297 422
298 $timestamp = AnyEvent::HTTP::parse_date $date 423 $timestamp = AnyEvent::HTTP::parse_date $date
299 Takes a HTTP Date (RFC 2616) and returns the corresponding POSIX 424 Takes a HTTP Date (RFC 2616) or a Cookie date (netscape cookie spec)
425 or a bunch of minor variations of those, and returns the
300 timestamp, or "undef" if the date cannot be parsed. 426 corresponding POSIX timestamp, or "undef" if the date cannot be
427 parsed.
301 428
302 $AnyEvent::HTTP::MAX_RECURSE 429 $AnyEvent::HTTP::MAX_RECURSE
303 The default value for the "recurse" request parameter (default: 10). 430 The default value for the "recurse" request parameter (default: 10).
431
432 $AnyEvent::HTTP::TIMEOUT
433 The default timeout for conenction operations (default: 300).
304 434
305 $AnyEvent::HTTP::USERAGENT 435 $AnyEvent::HTTP::USERAGENT
306 The default value for the "User-Agent" header (the default is 436 The default value for the "User-Agent" header (the default is
307 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; 437 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
308 +http://software.schmorp.de/pkg/AnyEvent)"). 438 +http://software.schmorp.de/pkg/AnyEvent)").
309 439
310 $AnyEvent::HTTP::MAX_PER_HOST 440 $AnyEvent::HTTP::MAX_PER_HOST
311 The maximum number of concurrent connections to the same host 441 The maximum number of concurrent connections to the same host
312 (identified by the hostname). If the limit is exceeded, then the 442 (identified by the hostname). If the limit is exceeded, then the
313 additional requests are queued until previous connections are 443 additional requests are queued until previous connections are
314 closed. 444 closed. Both persistent and non-persistent connections are counted
445 in this limit.
315 446
316 The default value for this is 4, and it is highly advisable to not 447 The default value for this is 4, and it is highly advisable to not
317 increase it. 448 increase it much.
449
450 For comparison: the RFC's recommend 4 non-persistent or 2 persistent
451 connections, older browsers used 2, newers (such as firefox 3)
452 typically use 6, and Opera uses 8 because like, they have the
453 fastest browser and give a shit for everybody else on the planet.
454
455 $AnyEvent::HTTP::PERSISTENT_TIMEOUT
456 The time after which idle persistent conenctions get closed by
457 AnyEvent::HTTP (default: 3).
318 458
319 $AnyEvent::HTTP::ACTIVE 459 $AnyEvent::HTTP::ACTIVE
320 The number of active connections. This is not the number of 460 The number of active connections. This is not the number of
321 currently running requests, but the number of currently open and 461 currently running requests, but the number of currently open and
322 non-idle TCP connections. This number of can be useful for 462 non-idle TCP connections. This number can be useful for
323 load-leveling. 463 load-leveling.
324 464
465 SHOWCASE
466 This section contaisn some more elaborate "real-world" examples or code
467 snippets.
468
469 HTTP/1.1 FILE DOWNLOAD
470 Downloading files with HTTP can be quite tricky, especially when
471 something goes wrong and you want to resume.
472
473 Here is a function that initiates and resumes a download. It uses the
474 last modified time to check for file content changes, and works with
475 many HTTP/1.0 servers as well, and usually falls back to a complete
476 re-download on older servers.
477
478 It calls the completion callback with either "undef", which means a
479 nonretryable error occured, 0 when the download was partial and should
480 be retried, and 1 if it was successful.
481
482 use AnyEvent::HTTP;
483
484 sub download($$$) {
485 my ($url, $file, $cb) = @_;
486
487 open my $fh, "+<", $file
488 or die "$file: $!";
489
490 my %hdr;
491 my $ofs = 0;
492
493 warn stat $fh;
494 warn -s _;
495 if (stat $fh and -s _) {
496 $ofs = -s _;
497 warn "-s is ", $ofs;
498 $hdr{"if-unmodified-since"} = AnyEvent::HTTP::format_date +(stat _)[9];
499 $hdr{"range"} = "bytes=$ofs-";
500 }
501
502 http_get $url,
503 headers => \%hdr,
504 on_header => sub {
505 my ($hdr) = @_;
506
507 if ($hdr->{Status} == 200 && $ofs) {
508 # resume failed
509 truncate $fh, $ofs = 0;
510 }
511
512 sysseek $fh, $ofs, 0;
513
514 1
515 },
516 on_body => sub {
517 my ($data, $hdr) = @_;
518
519 if ($hdr->{Status} =~ /^2/) {
520 length $data == syswrite $fh, $data
521 or return; # abort on write errors
522 }
523
524 1
525 },
526 sub {
527 my (undef, $hdr) = @_;
528
529 my $status = $hdr->{Status};
530
531 if (my $time = AnyEvent::HTTP::parse_date $hdr->{"last-modified"}) {
532 utime $fh, $time, $time;
533 }
534
535 if ($status == 200 || $status == 206 || $status == 416) {
536 # download ok || resume ok || file already fully downloaded
537 $cb->(1, $hdr);
538
539 } elsif ($status == 412) {
540 # file has changed while resuming, delete and retry
541 unlink $file;
542 $cb->(0, $hdr);
543
544 } elsif ($status == 500 or $status == 503 or $status =~ /^59/) {
545 # retry later
546 $cb->(0, $hdr);
547
548 } else {
549 $cb->(undef, $hdr);
550 }
551 }
552 ;
553 }
554
555 download "http://server/somelargefile", "/tmp/somelargefile", sub {
556 if ($_[0]) {
557 print "OK!\n";
558 } elsif (defined $_[0]) {
559 print "please retry later\n";
560 } else {
561 print "ERROR\n";
562 }
563 };
564
325 SOCKS PROXIES 565 SOCKS PROXIES
326 Socks proxies are not directly supported by AnyEvent::HTTP. You can 566 Socks proxies are not directly supported by AnyEvent::HTTP. You can
327 compile your perl to support socks, or use an external program such as 567 compile your perl to support socks, or use an external program such as
328 socksify (dante) or tsocks to make your program use a socks proxy 568 socksify (dante) or tsocks to make your program use a socks proxy
329 transparently. 569 transparently.
330 570

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines