ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/README
(Generate patch)

Comparing AnyEvent-HTTP/README (file contents):
Revision 1.23 by root, Sun Apr 22 12:58:38 2012 UTC vs.
Revision 1.28 by root, Mon Apr 27 12:14:12 2020 UTC

… …
50 object at least alive until the callback get called. If the object 50 object at least alive until the callback get called. If the object
51 gets destroyed before the callback is called, the request will be 51 gets destroyed before the callback is called, the request will be
52 cancelled. 52 cancelled.
53 53
54 The callback will be called with the response body data as first 54 The callback will be called with the response body data as first
55 argument (or "undef" if an error occured), and a hash-ref with 55 argument (or "undef" if an error occurred), and a hash-ref with
56 response headers (and trailers) as second argument. 56 response headers (and trailers) as second argument.
57 57
58 All the headers in that hash are lowercased. In addition to the 58 All the headers in that hash are lowercased. In addition to the
59 response headers, the "pseudo-headers" (uppercase to avoid clashing 59 response headers, the "pseudo-headers" (uppercase to avoid clashing
60 with possible response headers) "HTTPVersion", "Status" and "Reason" 60 with possible response headers) "HTTPVersion", "Status" and "Reason"
… …
82 If an internal error occurs, such as not being able to resolve a 82 If an internal error occurs, such as not being able to resolve a
83 hostname, then $data will be "undef", "$headers->{Status}" will be 83 hostname, then $data will be "undef", "$headers->{Status}" will be
84 590-599 and the "Reason" pseudo-header will contain an error 84 590-599 and the "Reason" pseudo-header will contain an error
85 message. Currently the following status codes are used: 85 message. Currently the following status codes are used:
86 86
87 595 - errors during connection etsbalishment, proxy handshake. 87 595 - errors during connection establishment, proxy handshake.
88 596 - errors during TLS negotiation, request sending and header 88 596 - errors during TLS negotiation, request sending and header
89 processing. 89 processing.
90 597 - errors during body receiving or processing. 90 597 - errors during body receiving or processing.
91 598 - user aborted request via "on_header" or "on_body". 91 598 - user aborted request via "on_header" or "on_body".
92 599 - other, usually nonretryable, errors (garbled URL etc.). 92 599 - other, usually nonretryable, errors (garbled URL etc.).
… …
106 Additional parameters are key-value pairs, and are fully optional. 106 Additional parameters are key-value pairs, and are fully optional.
107 They include: 107 They include:
108 108
109 recurse => $count (default: $MAX_RECURSE) 109 recurse => $count (default: $MAX_RECURSE)
110 Whether to recurse requests or not, e.g. on redirects, 110 Whether to recurse requests or not, e.g. on redirects,
111 authentication retries and so on, and how often to do so. 111 authentication and other retries and so on, and how often to do
112 so.
113
114 Only redirects to http and https URLs are supported. While most
115 common redirection forms are handled entirely within this
116 module, some require the use of the optional URI module. If it
117 is required but missing, then the request will fail with an
118 error.
112 119
113 headers => hashref 120 headers => hashref
114 The request headers to use. Currently, "http_request" may 121 The request headers to use. Currently, "http_request" may
115 provide its own "Host:", "Content-Length:", "Connection:" and 122 provide its own "Host:", "Content-Length:", "Connection:" and
116 "Cookie:" headers and will provide defaults at least for "TE:", 123 "Cookie:" headers and will provide defaults at least for "TE:",
… …
140 $scheme must be either missing or must be "http" for HTTP. 147 $scheme must be either missing or must be "http" for HTTP.
141 148
142 If not specified, then the default proxy is used (see 149 If not specified, then the default proxy is used (see
143 "AnyEvent::HTTP::set_proxy"). 150 "AnyEvent::HTTP::set_proxy").
144 151
152 Currently, if your proxy requires authorization, you have to
153 specify an appropriate "Proxy-Authorization" header in every
154 request.
155
156 Note that this module will prefer an existing persistent
157 connection, even if that connection was made using another
158 proxy. If you need to ensure that a new connection is made in
159 this case, you can either force "persistent" to false or e.g.
160 use the proxy address in your "sessionid".
161
145 body => $string 162 body => $string
146 The request body, usually empty. Will be sent as-is (future 163 The request body, usually empty. Will be sent as-is (future
147 versions of this module might offer more options). 164 versions of this module might offer more options).
148 165
149 cookie_jar => $hash_ref 166 cookie_jar => $hash_ref
… …
178 The default for this option is "low", which could be interpreted 195 The default for this option is "low", which could be interpreted
179 as "give me the page, no matter what". 196 as "give me the page, no matter what".
180 197
181 See also the "sessionid" parameter. 198 See also the "sessionid" parameter.
182 199
183 session => $string 200 sessionid => $string
184 The module might reuse connections to the same host internally. 201 The module might reuse connections to the same host internally
185 Sometimes (e.g. when using TLS), you do not want to reuse 202 (regardless of other settings, such as "tcp_connect" or
186 connections from other sessions. This can be achieved by setting 203 "proxy"). Sometimes (e.g. when using TLS or a specfic proxy),
187 this parameter to some unique ID (such as the address of an 204 you do not want to reuse connections from other sessions. This
188 object storing your state data, or the TLS context) - only 205 can be achieved by setting this parameter to some unique ID
189 connections using the same unique ID will be reused. 206 (such as the address of an object storing your state data or the
207 TLS context, or the proxy IP) - only connections using the same
208 unique ID will be reused.
190 209
191 on_prepare => $callback->($fh) 210 on_prepare => $callback->($fh)
192 In rare cases you need to "tune" the socket before it is used to 211 In rare cases you need to "tune" the socket before it is used to
193 connect (for exmaple, to bind it on a given IP address). This 212 connect (for example, to bind it on a given IP address). This
194 parameter overrides the prepare callback passed to 213 parameter overrides the prepare callback passed to
195 "AnyEvent::Socket::tcp_connect" and behaves exactly the same way 214 "AnyEvent::Socket::tcp_connect" and behaves exactly the same way
196 (e.g. it has to provide a timeout). See the description for the 215 (e.g. it has to provide a timeout). See the description for the
197 $prepare_cb argument of "AnyEvent::Socket::tcp_connect" for 216 $prepare_cb argument of "AnyEvent::Socket::tcp_connect" for
198 details. 217 details.
… …
203 AnyEvent::HTTP establishes connections. Normally it uses 222 AnyEvent::HTTP establishes connections. Normally it uses
204 AnyEvent::Socket::tcp_connect to do this, but you can provide 223 AnyEvent::Socket::tcp_connect to do this, but you can provide
205 your own "tcp_connect" function - obviously, it has to follow 224 your own "tcp_connect" function - obviously, it has to follow
206 the same calling conventions, except that it may always return a 225 the same calling conventions, except that it may always return a
207 connection guard object. 226 connection guard object.
227
228 The connections made by this hook will be treated as equivalent
229 to connections made the built-in way, specifically, they will be
230 put into and taken from the persistent connection cache. If your
231 $tcp_connect function is incompatible with this kind of re-use,
232 consider switching off "persistent" connections and/or providing
233 a "sessionid" identifier.
208 234
209 There are probably lots of weird uses for this function, 235 There are probably lots of weird uses for this function,
210 starting from tracing the hosts "http_request" actually tries to 236 starting from tracing the hosts "http_request" actually tries to
211 connect, to (inexact but fast) host => IP address caching or 237 connect, to (inexact but fast) host => IP address caching or
212 even socks protocol support. 238 even socks protocol support.
… …
286 312
287 persistent => $boolean 313 persistent => $boolean
288 Try to create/reuse a persistent connection. When this flag is 314 Try to create/reuse a persistent connection. When this flag is
289 set (default: true for idempotent requests, false for all 315 set (default: true for idempotent requests, false for all
290 others), then "http_request" tries to re-use an existing 316 others), then "http_request" tries to re-use an existing
291 (previously-created) persistent connection to the host and, 317 (previously-created) persistent connection to same host (i.e.
318 identical URL scheme, hostname, port and sessionid) and, failing
292 failing that, tries to create a new one. 319 that, tries to create a new one.
293 320
294 Requests failing in certain ways will be automatically retried 321 Requests failing in certain ways will be automatically retried
295 once, which is dangerous for non-idempotent requests, which is 322 once, which is dangerous for non-idempotent requests, which is
296 why it defaults to off for them. The reason for this is because 323 why it defaults to off for them. The reason for this is because
297 the bozos who designed HTTP/1.1 made it impossible to 324 the bozos who designed HTTP/1.1 made it impossible to
298 distinguish between a fatal error and a normal connection 325 distinguish between a fatal error and a normal connection
299 timeout, so you never know whether there was a problem with your 326 timeout, so you never know whether there was a problem with your
300 request or not. 327 request or not.
301 328
302 When reusing an existent connection, many parameters (such as 329 When reusing an existent connection, many parameters (such as
303 TLS context) will be ignored. See the "session" parameter for a 330 TLS context) will be ignored. See the "sessionid" parameter for
304 workaround. 331 a workaround.
305 332
306 keepalive => $boolean 333 keepalive => $boolean
307 Only used when "persistent" is also true. This parameter decides 334 Only used when "persistent" is also true. This parameter decides
308 whether "http_request" tries to handshake a HTTP/1.0-style 335 whether "http_request" tries to handshake a HTTP/1.0-style
309 keep-alive connection (as opposed to only a HTTP/1.1 persistent 336 keep-alive connection (as opposed to only a HTTP/1.1 persistent
… …
371 Sets the default proxy server to use. The proxy-url must begin with 398 Sets the default proxy server to use. The proxy-url must begin with
372 a string of the form "http://host:port", croaks otherwise. 399 a string of the form "http://host:port", croaks otherwise.
373 400
374 To clear an already-set proxy, use "undef". 401 To clear an already-set proxy, use "undef".
375 402
376 When AnyEvent::HTTP is laoded for the first time it will query the 403 When AnyEvent::HTTP is loaded for the first time it will query the
377 default proxy from the operating system, currently by looking at 404 default proxy from the operating system, currently by looking at
378 "$ENV{http_proxy"}. 405 "$ENV{http_proxy"}.
379 406
380 AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end] 407 AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]
381 Remove all cookies from the cookie jar that have been expired. If 408 Remove all cookies from the cookie jar that have been expired. If
… …
383 cookies. 410 cookies.
384 411
385 You should call this function (with a true $session_end) before you 412 You should call this function (with a true $session_end) before you
386 save cookies to disk, and you should call this function after 413 save cookies to disk, and you should call this function after
387 loading them again. If you have a long-running program you can 414 loading them again. If you have a long-running program you can
388 additonally call this function from time to time. 415 additionally call this function from time to time.
389 416
390 A cookie jar is initially an empty hash-reference that is managed by 417 A cookie jar is initially an empty hash-reference that is managed by
391 this module. It's format is subject to change, but currently it is 418 this module. Its format is subject to change, but currently it is as
392 like this: 419 follows:
393 420
394 The key "version" has to contain 1, otherwise the hash gets emptied. 421 The key "version" has to contain 2, otherwise the hash gets cleared.
395 All other keys are hostnames or IP addresses pointing to 422 All other keys are hostnames or IP addresses pointing to
396 hash-references. The key for these inner hash references is the 423 hash-references. The key for these inner hash references is the
397 server path for which this cookie is meant, and the values are again 424 server path for which this cookie is meant, and the values are again
398 hash-references. The keys of those hash-references is the cookie 425 hash-references. Each key of those hash-references is a cookie name,
399 name, and the value, you guessed it, is another hash-reference, this 426 and the value, you guessed it, is another hash-reference, this time
400 time with the key-value pairs from the cookie, except for "expires" 427 with the key-value pairs from the cookie, except for "expires" and
401 and "max-age", which have been replaced by a "_expires" key that 428 "max-age", which have been replaced by a "_expires" key that
402 contains the cookie expiry timestamp. 429 contains the cookie expiry timestamp. Session cookies are indicated
430 by not having an "_expires" key.
403 431
404 Here is an example of a cookie jar with a single cookie, so you have 432 Here is an example of a cookie jar with a single cookie, so you have
405 a chance of understanding the above paragraph: 433 a chance of understanding the above paragraph:
406 434
407 { 435 {
408 version => 1, 436 version => 2,
409 "10.0.0.1" => { 437 "10.0.0.1" => {
410 "/" => { 438 "/" => {
411 "mythweb_id" => { 439 "mythweb_id" => {
412 _expires => 1293917923, 440 _expires => 1293917923,
413 value => "ooRung9dThee3ooyXooM1Ohm", 441 value => "ooRung9dThee3ooyXooM1Ohm",
… …
437 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; 465 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
438 +http://software.schmorp.de/pkg/AnyEvent)"). 466 +http://software.schmorp.de/pkg/AnyEvent)").
439 467
440 $AnyEvent::HTTP::MAX_PER_HOST 468 $AnyEvent::HTTP::MAX_PER_HOST
441 The maximum number of concurrent connections to the same host 469 The maximum number of concurrent connections to the same host
442 (identified by the hostname). If the limit is exceeded, then the 470 (identified by the hostname). If the limit is exceeded, then
443 additional requests are queued until previous connections are 471 additional requests are queued until previous connections are
444 closed. Both persistent and non-persistent connections are counted 472 closed. Both persistent and non-persistent connections are counted
445 in this limit. 473 in this limit.
446 474
447 The default value for this is 4, and it is highly advisable to not 475 The default value for this is 4, and it is highly advisable to not
448 increase it much. 476 increase it much.
449 477
450 For comparison: the RFC's recommend 4 non-persistent or 2 persistent 478 For comparison: the RFC's recommend 4 non-persistent or 2 persistent
451 connections, older browsers used 2, newers (such as firefox 3) 479 connections, older browsers used 2, newer ones (such as firefox 3)
452 typically use 6, and Opera uses 8 because like, they have the 480 typically use 6, and Opera uses 8 because like, they have the
453 fastest browser and give a shit for everybody else on the planet. 481 fastest browser and give a shit for everybody else on the planet.
454 482
455 $AnyEvent::HTTP::PERSISTENT_TIMEOUT 483 $AnyEvent::HTTP::PERSISTENT_TIMEOUT
456 The time after which idle persistent conenctions get closed by 484 The time after which idle persistent connections get closed by
457 AnyEvent::HTTP (default: 3). 485 AnyEvent::HTTP (default: 3).
458 486
459 $AnyEvent::HTTP::ACTIVE 487 $AnyEvent::HTTP::ACTIVE
460 The number of active connections. This is not the number of 488 The number of active connections. This is not the number of
461 currently running requests, but the number of currently open and 489 currently running requests, but the number of currently open and
462 non-idle TCP connections. This number can be useful for 490 non-idle TCP connections. This number can be useful for
463 load-leveling. 491 load-leveling.
464 492
465 SHOWCASE 493 SHOWCASE
466 This section contaisn some more elaborate "real-world" examples or code 494 This section contains some more elaborate "real-world" examples or code
467 snippets. 495 snippets.
468 496
469 HTTP/1.1 FILE DOWNLOAD 497 HTTP/1.1 FILE DOWNLOAD
470 Downloading files with HTTP can be quite tricky, especially when 498 Downloading files with HTTP can be quite tricky, especially when
471 something goes wrong and you want to resume. 499 something goes wrong and you want to resume.
… …
474 last modified time to check for file content changes, and works with 502 last modified time to check for file content changes, and works with
475 many HTTP/1.0 servers as well, and usually falls back to a complete 503 many HTTP/1.0 servers as well, and usually falls back to a complete
476 re-download on older servers. 504 re-download on older servers.
477 505
478 It calls the completion callback with either "undef", which means a 506 It calls the completion callback with either "undef", which means a
479 nonretryable error occured, 0 when the download was partial and should 507 nonretryable error occurred, 0 when the download was partial and should
480 be retried, and 1 if it was successful. 508 be retried, and 1 if it was successful.
481 509
482 use AnyEvent::HTTP; 510 use AnyEvent::HTTP;
483 511
484 sub download($$$) { 512 sub download($$$) {
… …
488 or die "$file: $!"; 516 or die "$file: $!";
489 517
490 my %hdr; 518 my %hdr;
491 my $ofs = 0; 519 my $ofs = 0;
492 520
493 warn stat $fh;
494 warn -s _;
495 if (stat $fh and -s _) { 521 if (stat $fh and -s _) {
496 $ofs = -s _; 522 $ofs = -s _;
497 warn "-s is ", $ofs; 523 warn "-s is ", $ofs;
498 $hdr{"if-unmodified-since"} = AnyEvent::HTTP::format_date +(stat _)[9]; 524 $hdr{"if-unmodified-since"} = AnyEvent::HTTP::format_date +(stat _)[9];
499 $hdr{"range"} = "bytes=$ofs-"; 525 $hdr{"range"} = "bytes=$ofs-";
… …
527 my (undef, $hdr) = @_; 553 my (undef, $hdr) = @_;
528 554
529 my $status = $hdr->{Status}; 555 my $status = $hdr->{Status};
530 556
531 if (my $time = AnyEvent::HTTP::parse_date $hdr->{"last-modified"}) { 557 if (my $time = AnyEvent::HTTP::parse_date $hdr->{"last-modified"}) {
532 utime $fh, $time, $time; 558 utime $time, $time, $fh;
533 } 559 }
534 560
535 if ($status == 200 || $status == 206 || $status == 416) { 561 if ($status == 200 || $status == 206 || $status == 416) {
536 # download ok || resume ok || file already fully downloaded 562 # download ok || resume ok || file already fully downloaded
537 $cb->(1, $hdr); 563 $cb->(1, $hdr);
… …
622 648
623AUTHOR 649AUTHOR
624 Marc Lehmann <schmorp@schmorp.de> 650 Marc Lehmann <schmorp@schmorp.de>
625 http://home.schmorp.de/ 651 http://home.schmorp.de/
626 652
627 With many thanks to Дмитрий Шалашов, who provided 653 With many thanks to Дмитрий Шалашов, who provided countless testcases
628 countless testcases and bugreports. 654 and bugreports.
629 655

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines