ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/README
(Generate patch)

Comparing AnyEvent-HTTP/README (file contents):
Revision 1.13 by root, Wed Jun 16 19:17:30 2010 UTC vs.
Revision 1.15 by root, Tue Jan 4 08:29:28 2011 UTC

46 URL must be an absolute http or https URL. 46 URL must be an absolute http or https URL.
47 47
48 When called in void context, nothing is returned. In other contexts, 48 When called in void context, nothing is returned. In other contexts,
49 "http_request" returns a "cancellation guard" - you have to keep the 49 "http_request" returns a "cancellation guard" - you have to keep the
50 object at least alive until the callback get called. If the object 50 object at least alive until the callback get called. If the object
51 gets destroyed before the callbakc is called, the request will be 51 gets destroyed before the callback is called, the request will be
52 cancelled. 52 cancelled.
53 53
54 The callback will be called with the response body data as first 54 The callback will be called with the response body data as first
55 argument (or "undef" if an error occured), and a hash-ref with 55 argument (or "undef" if an error occured), and a hash-ref with
56 response headers as second argument. 56 response headers (and trailers) as second argument.
57 57
58 All the headers in that hash are lowercased. In addition to the 58 All the headers in that hash are lowercased. In addition to the
59 response headers, the "pseudo-headers" (uppercase to avoid clashing 59 response headers, the "pseudo-headers" (uppercase to avoid clashing
60 with possible response headers) "HTTPVersion", "Status" and "Reason" 60 with possible response headers) "HTTPVersion", "Status" and "Reason"
61 contain the three parts of the HTTP Status-Line of the same name. 61 contain the three parts of the HTTP Status-Line of the same name. If
62 an error occurs during the body phase of a request, then the
63 original "Status" and "Reason" values from the header are available
64 as "OrigStatus" and "OrigReason".
62 65
63 The pseudo-header "URL" contains the actual URL (which can differ 66 The pseudo-header "URL" contains the actual URL (which can differ
64 from the requested URL when following redirects - for example, you 67 from the requested URL when following redirects - for example, you
65 might get an error that your URL scheme is not supported even though 68 might get an error that your URL scheme is not supported even though
66 your URL is a valid http URL because it redirected to an ftp URL, in 69 your URL is a valid http URL because it redirected to an ftp URL, in
76 If the server sends a header multiple times, then their contents 79 If the server sends a header multiple times, then their contents
77 will be joined together with a comma (","), as per the HTTP spec. 80 will be joined together with a comma (","), as per the HTTP spec.
78 81
79 If an internal error occurs, such as not being able to resolve a 82 If an internal error occurs, such as not being able to resolve a
80 hostname, then $data will be "undef", "$headers->{Status}" will be 83 hostname, then $data will be "undef", "$headers->{Status}" will be
81 "59x" (usually 599) and the "Reason" pseudo-header will contain an 84 590-599 and the "Reason" pseudo-header will contain an error
82 error message. 85 message. Currently the following status codes are used:
86
87 595 - errors during connection etsbalishment, proxy handshake.
88 596 - errors during TLS negotiation, request sending and header
89 processing.
90 597 - errors during body receiving or processing.
91 598 - user aborted request via "on_header" or "on_body".
92 599 - other, usually nonretryable, errors (garbled URL etc.).
83 93
84 A typical callback might look like this: 94 A typical callback might look like this:
85 95
86 sub { 96 sub {
87 my ($body, $hdr) = @_; 97 my ($body, $hdr) = @_;
101 authentication retries and so on, and how often to do so. 111 authentication retries and so on, and how often to do so.
102 112
103 headers => hashref 113 headers => hashref
104 The request headers to use. Currently, "http_request" may 114 The request headers to use. Currently, "http_request" may
105 provide its own "Host:", "Content-Length:", "Connection:" and 115 provide its own "Host:", "Content-Length:", "Connection:" and
106 "Cookie:" headers and will provide defaults for "User-Agent:" 116 "Cookie:" headers and will provide defaults at least for "TE:",
107 and "Referer:" (this can be suppressed by using "undef" for 117 "Referer:" and "User-Agent:" (this can be suppressed by using
108 these headers in which case they won't be sent at all). 118 "undef" for these headers in which case they won't be sent at
119 all).
120
121 You really should provide your own "User-Agent:" header value
122 that is appropriate for your program - I wouldn't be surprised
123 if the default AnyEvent string gets blocked by webservers sooner
124 or later.
109 125
110 timeout => $seconds 126 timeout => $seconds
111 The time-out to use for various stages - each connect attempt 127 The time-out to use for various stages - each connect attempt
112 will reset the timeout, as will read or write activity, i.e. 128 will reset the timeout, as will read or write activity, i.e.
113 this is not an overall timeout. 129 this is not an overall timeout.
117 proxy => [$host, $port[, $scheme]] or undef 133 proxy => [$host, $port[, $scheme]] or undef
118 Use the given http proxy for all requests. If not specified, 134 Use the given http proxy for all requests. If not specified,
119 then the default proxy (as specified by $ENV{http_proxy}) is 135 then the default proxy (as specified by $ENV{http_proxy}) is
120 used. 136 used.
121 137
122 $scheme must be either missing, "http" for HTTP or "https" for 138 $scheme must be either missing or must be "http" for HTTP.
123 HTTPS.
124 139
125 body => $string 140 body => $string
126 The request body, usually empty. Will be-sent as-is (future 141 The request body, usually empty. Will be sent as-is (future
127 versions of this module might offer more options). 142 versions of this module might offer more options).
128 143
129 cookie_jar => $hash_ref 144 cookie_jar => $hash_ref
130 Passing this parameter enables (simplified) cookie-processing, 145 Passing this parameter enables (simplified) cookie-processing,
131 loosely based on the original netscape specification. 146 loosely based on the original netscape specification.
132 147
133 The $hash_ref must be an (initially empty) hash reference which 148 The $hash_ref must be an (initially empty) hash reference which
134 will get updated automatically. It is possible to save the 149 will get updated automatically. It is possible to save the
135 cookie_jar to persistent storage with something like JSON or 150 cookie jar to persistent storage with something like JSON or
136 Storable, but this is not recommended, as expiry times are 151 Storable - see the "AnyEvent::HTTP::cookie_jar_expire" function
137 currently being ignored. 152 if you wish to remove expired or session-only cookies, and also
153 for documentation on the format of the cookie jar.
138 154
139 Note that this cookie implementation is not of very high 155 Note that this cookie implementation is not meant to be
140 quality, nor meant to be complete. If you want complete cookie 156 complete. If you want complete cookie management you have to do
141 management you have to do that on your own. "cookie_jar" is 157 that on your own. "cookie_jar" is meant as a quick fix to get
142 meant as a quick fix to get some cookie-using sites working. 158 most cookie-using sites working. Cookies are a privacy disaster,
143 Cookies are a privacy disaster, do not use them unless required 159 do not use them unless required to.
144 to. 160
161 When cookie processing is enabled, the "Cookie:" and
162 "Set-Cookie:" headers will be set and handled by this module,
163 otherwise they will be left untouched.
145 164
146 tls_ctx => $scheme | $tls_ctx 165 tls_ctx => $scheme | $tls_ctx
147 Specifies the AnyEvent::TLS context to be used for https 166 Specifies the AnyEvent::TLS context to be used for https
148 connections. This parameter follows the same rules as the 167 connections. This parameter follows the same rules as the
149 "tls_ctx" parameter to AnyEvent::Handle, but additionally, the 168 "tls_ctx" parameter to AnyEvent::Handle, but additionally, the
151 predefined low-security (no verification, highest compatibility) 170 predefined low-security (no verification, highest compatibility)
152 and high-security (CA and common-name verification) TLS context. 171 and high-security (CA and common-name verification) TLS context.
153 172
154 The default for this option is "low", which could be interpreted 173 The default for this option is "low", which could be interpreted
155 as "give me the page, no matter what". 174 as "give me the page, no matter what".
175
176 See also the "sessionid" parameter.
177
178 session => $string
179 The module might reuse connections to the same host internally.
180 Sometimes (e.g. when using TLS), you do not want to reuse
181 connections from other sessions. This can be achieved by setting
182 this parameter to some unique ID (such as the address of an
183 object storing your state data, or the TLS context) - only
184 connections using the same unique ID will be reused.
156 185
157 on_prepare => $callback->($fh) 186 on_prepare => $callback->($fh)
158 In rare cases you need to "tune" the socket before it is used to 187 In rare cases you need to "tune" the socket before it is used to
159 connect (for exmaple, to bind it on a given IP address). This 188 connect (for exmaple, to bind it on a given IP address). This
160 parameter overrides the prepare callback passed to 189 parameter overrides the prepare callback passed to
161 "AnyEvent::Socket::tcp_connect" and behaves exactly the same way 190 "AnyEvent::Socket::tcp_connect" and behaves exactly the same way
162 (e.g. it has to provide a timeout). See the description for the 191 (e.g. it has to provide a timeout). See the description for the
163 $prepare_cb argument of "AnyEvent::Socket::tcp_connect" for 192 $prepare_cb argument of "AnyEvent::Socket::tcp_connect" for
164 details. 193 details.
165 194
195 tcp_connect => $callback->($host, $service, $connect_cb,
196 $prepare_cb)
197 In even rarer cases you want total control over how
198 AnyEvent::HTTP establishes connections. Normally it uses
199 AnyEvent::Socket::tcp_connect to do this, but you can provide
200 your own "tcp_connect" function - obviously, it has to follow
201 the same calling conventions, except that it may always return a
202 connection guard object.
203
204 There are probably lots of weird uses for this function,
205 starting from tracing the hosts "http_request" actually tries to
206 connect, to (inexact but fast) host => IP address caching or
207 even socks protocol support.
208
166 on_header => $callback->($headers) 209 on_header => $callback->($headers)
167 When specified, this callback will be called with the header 210 When specified, this callback will be called with the header
168 hash as soon as headers have been successfully received from the 211 hash as soon as headers have been successfully received from the
169 remote server (not on locally-generated errors). 212 remote server (not on locally-generated errors).
170 213
175 218
176 This callback is useful, among other things, to quickly reject 219 This callback is useful, among other things, to quickly reject
177 unwanted content, which, if it is supposed to be rare, can be 220 unwanted content, which, if it is supposed to be rare, can be
178 faster than first doing a "HEAD" request. 221 faster than first doing a "HEAD" request.
179 222
223 The downside is that cancelling the request makes it impossible
224 to re-use the connection. Also, the "on_header" callback will
225 not receive any trailer (headers sent after the response body).
226
180 Example: cancel the request unless the content-type is 227 Example: cancel the request unless the content-type is
181 "text/html". 228 "text/html".
182 229
183 on_header => sub { 230 on_header => sub {
184 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/ 231 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
191 238
192 It has to return either true (in which case AnyEvent::HTTP will 239 It has to return either true (in which case AnyEvent::HTTP will
193 continue), or false, in which case AnyEvent::HTTP will cancel 240 continue), or false, in which case AnyEvent::HTTP will cancel
194 the download (and call the completion callback with an error 241 the download (and call the completion callback with an error
195 code of 598). 242 code of 598).
243
244 The downside to cancelling the request is that it makes it
245 impossible to re-use the connection.
196 246
197 This callback is useful when the data is too large to be held in 247 This callback is useful when the data is too large to be held in
198 memory (so the callback writes it to a file) or when only some 248 memory (so the callback writes it to a file) or when only some
199 information should be extracted, or when the body should be 249 information should be extracted, or when the body should be
200 processed incrementally. 250 processed incrementally.
214 AnyEvent::Handle object associated with the connection. In error 264 AnyEvent::Handle object associated with the connection. In error
215 cases, "undef" will be passed. When there is no body (e.g. 265 cases, "undef" will be passed. When there is no body (e.g.
216 status 304), the empty string will be passed. 266 status 304), the empty string will be passed.
217 267
218 The handle object might or might not be in TLS mode, might be 268 The handle object might or might not be in TLS mode, might be
219 connected to a proxy, be a persistent connection etc., and 269 connected to a proxy, be a persistent connection, use chunked
220 configured in unspecified ways. The user is responsible for this 270 transfer encoding etc., and configured in unspecified ways. The
221 handle (it will not be used by this module anymore). 271 user is responsible for this handle (it will not be used by this
272 module anymore).
222 273
223 This is useful with some push-type services, where, after the 274 This is useful with some push-type services, where, after the
224 initial headers, an interactive protocol is used (typical 275 initial headers, an interactive protocol is used (typical
225 example would be the push-style twitter API which starts a 276 example would be the push-style twitter API which starts a
226 JSON/XML stream). 277 JSON/XML stream).
227 278
228 If you think you need this, first have a look at "on_body", to 279 If you think you need this, first have a look at "on_body", to
229 see if that doesn't solve your problem in a better way. 280 see if that doesn't solve your problem in a better way.
230 281
282 persistent => $boolean
283 Try to create/reuse a persistent connection. When this flag is
284 set (default: true for idempotent requests, false for all
285 others), then "http_request" tries to re-use an existing
286 (previously-created) persistent connection to the host and,
287 failing that, tries to create a new one.
288
289 Requests failing in certain ways will be automatically retried
290 once, which is dangerous for non-idempotent requests, which is
291 why it defaults to off for them. The reason for this is because
292 the bozos who designed HTTP/1.1 made it impossible to
293 distinguish between a fatal error and a normal connection
294 timeout, so you never know whether there was a problem with your
295 request or not.
296
297 When reusing an existent connection, many parameters (such as
298 TLS context) will be ignored. See the "session" parameter for a
299 workaround.
300
301 keepalive => $boolean
302 Only used when "persistent" is also true. This parameter decides
303 whether "http_request" tries to handshake a HTTP/1.0-style
304 keep-alive connection (as opposed to only a HTTP/1.1 persistent
305 connection).
306
307 The default is true, except when using a proxy, in which case it
308 defaults to false, as HTTP/1.0 proxies cannot support this in a
309 meaningful way.
310
311 handle_params => { key => value ... }
312 The key-value pairs in this hash will be passed to any
313 AnyEvent::Handle constructor that is called - not all requests
314 will create a handle, and sometimes more than one is created, so
315 this parameter is only good for setting hints.
316
317 Example: set the maximum read size to 4096, to potentially
318 conserve memory at the cost of speed.
319
320 handle_params => {
321 max_read_size => 4096,
322 },
323
231 Example: make a simple HTTP GET request for http://www.nethype.de/ 324 Example: do a simple HTTP GET request for http://www.nethype.de/ and
325 print the response body.
232 326
233 http_request GET => "http://www.nethype.de/", sub { 327 http_request GET => "http://www.nethype.de/", sub {
234 my ($body, $hdr) = @_; 328 my ($body, $hdr) = @_;
235 print "$body\n"; 329 print "$body\n";
236 }; 330 };
237 331
238 Example: make a HTTP HEAD request on https://www.google.com/, use a 332 Example: do a HTTP HEAD request on https://www.google.com/, use a
239 timeout of 30 seconds. 333 timeout of 30 seconds.
240 334
241 http_request 335 http_request
242 GET => "https://www.google.com", 336 GET => "https://www.google.com",
337 headers => { "user-agent" => "MySearchClient 1.0" },
243 timeout => 30, 338 timeout => 30,
244 sub { 339 sub {
245 my ($body, $hdr) = @_; 340 my ($body, $hdr) = @_;
246 use Data::Dumper; 341 use Data::Dumper;
247 print Dumper $hdr; 342 print Dumper $hdr;
248 } 343 }
249 ; 344 ;
250 345
251 Example: make another simple HTTP GET request, but immediately try 346 Example: do another simple HTTP GET request, but immediately try to
252 to cancel it. 347 cancel it.
253 348
254 my $request = http_request GET => "http://www.nethype.de/", sub { 349 my $request = http_request GET => "http://www.nethype.de/", sub {
255 my ($body, $hdr) = @_; 350 my ($body, $hdr) = @_;
256 print "$body\n"; 351 print "$body\n";
257 }; 352 };
262 AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for the 357 AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for the
263 actual connection, which in turn uses AnyEvent::DNS to resolve 358 actual connection, which in turn uses AnyEvent::DNS to resolve
264 hostnames. The latter is a simple stub resolver and does no caching on 359 hostnames. The latter is a simple stub resolver and does no caching on
265 its own. If you want DNS caching, you currently have to provide your own 360 its own. If you want DNS caching, you currently have to provide your own
266 default resolver (by storing a suitable resolver object in 361 default resolver (by storing a suitable resolver object in
267 $AnyEvent::DNS::RESOLVER). 362 $AnyEvent::DNS::RESOLVER) or your own "tcp_connect" callback.
268 363
269 GLOBAL FUNCTIONS AND VARIABLES 364 GLOBAL FUNCTIONS AND VARIABLES
270 AnyEvent::HTTP::set_proxy "proxy-url" 365 AnyEvent::HTTP::set_proxy "proxy-url"
271 Sets the default proxy server to use. The proxy-url must begin with 366 Sets the default proxy server to use. The proxy-url must begin with
272 a string of the form "http://host:port" (optionally "https:..."), 367 a string of the form "http://host:port", croaks otherwise.
273 croaks otherwise.
274 368
275 To clear an already-set proxy, use "undef". 369 To clear an already-set proxy, use "undef".
370
371 AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]
372 Remove all cookies from the cookie jar that have been expired. If
373 $session_end is given and true, then additionally remove all session
374 cookies.
375
376 You should call this function (with a true $session_end) before you
377 save cookies to disk, and you should call this function after
378 loading them again. If you have a long-running program you can
379 additonally call this function from time to time.
380
381 A cookie jar is initially an empty hash-reference that is managed by
382 this module. It's format is subject to change, but currently it is
383 like this:
384
385 The key "version" has to contain 1, otherwise the hash gets emptied.
386 All other keys are hostnames or IP addresses pointing to
387 hash-references. The key for these inner hash references is the
388 server path for which this cookie is meant, and the values are again
389 hash-references. The keys of those hash-references is the cookie
390 name, and the value, you guessed it, is another hash-reference, this
391 time with the key-value pairs from the cookie, except for "expires"
392 and "max-age", which have been replaced by a "_expires" key that
393 contains the cookie expiry timestamp.
394
395 Here is an example of a cookie jar with a single cookie, so you have
396 a chance of understanding the above paragraph:
397
398 {
399 version => 1,
400 "10.0.0.1" => {
401 "/" => {
402 "mythweb_id" => {
403 _expires => 1293917923,
404 value => "ooRung9dThee3ooyXooM1Ohm",
405 },
406 },
407 },
408 }
409
410 $date = AnyEvent::HTTP::format_date $timestamp
411 Takes a POSIX timestamp (seconds since the epoch) and formats it as
412 a HTTP Date (RFC 2616).
413
414 $timestamp = AnyEvent::HTTP::parse_date $date
415 Takes a HTTP Date (RFC 2616) or a Cookie date (netscape cookie spec)
416 or a bunch of minor variations of those, and returns the
417 corresponding POSIX timestamp, or "undef" if the date cannot be
418 parsed.
276 419
277 $AnyEvent::HTTP::MAX_RECURSE 420 $AnyEvent::HTTP::MAX_RECURSE
278 The default value for the "recurse" request parameter (default: 10). 421 The default value for the "recurse" request parameter (default: 10).
422
423 $AnyEvent::HTTP::TIMEOUT
424 The default timeout for conenction operations (default: 300).
279 425
280 $AnyEvent::HTTP::USERAGENT 426 $AnyEvent::HTTP::USERAGENT
281 The default value for the "User-Agent" header (the default is 427 The default value for the "User-Agent" header (the default is
282 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; 428 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
283 +http://software.schmorp.de/pkg/AnyEvent)"). 429 +http://software.schmorp.de/pkg/AnyEvent)").
284 430
285 $AnyEvent::HTTP::MAX_PER_HOST 431 $AnyEvent::HTTP::MAX_PER_HOST
286 The maximum number of concurrent connections to the same host 432 The maximum number of concurrent connections to the same host
287 (identified by the hostname). If the limit is exceeded, then the 433 (identified by the hostname). If the limit is exceeded, then the
288 additional requests are queued until previous connections are 434 additional requests are queued until previous connections are
289 closed. 435 closed. Both persistent and non-persistent connections are counted
436 in this limit.
290 437
291 The default value for this is 4, and it is highly advisable to not 438 The default value for this is 4, and it is highly advisable to not
292 increase it. 439 increase it much.
440
441 For comparison: the RFC's recommend 4 non-persistent or 2 persistent
442 connections, older browsers used 2, newers (such as firefox 3)
443 typically use 6, and Opera uses 8 because like, they have the
444 fastest browser and give a shit for everybody else on the planet.
445
446 $AnyEvent::HTTP::PERSISTENT_TIMEOUT
447 The time after which idle persistent conenctions get closed by
448 AnyEvent::HTTP (default: 3).
293 449
294 $AnyEvent::HTTP::ACTIVE 450 $AnyEvent::HTTP::ACTIVE
295 The number of active connections. This is not the number of 451 The number of active connections. This is not the number of
296 currently running requests, but the number of currently open and 452 currently running requests, but the number of currently open and
297 non-idle TCP connections. This number of can be useful for 453 non-idle TCP connections. This number can be useful for
298 load-leveling. 454 load-leveling.
455
456 SOCKS PROXIES
457 Socks proxies are not directly supported by AnyEvent::HTTP. You can
458 compile your perl to support socks, or use an external program such as
459 socksify (dante) or tsocks to make your program use a socks proxy
460 transparently.
461
462 Alternatively, for AnyEvent::HTTP only, you can use your own
463 "tcp_connect" function that does the proxy handshake - here is an
464 example that works with socks4a proxies:
465
466 use Errno;
467 use AnyEvent::Util;
468 use AnyEvent::Socket;
469 use AnyEvent::Handle;
470
471 # host, port and username of/for your socks4a proxy
472 my $socks_host = "10.0.0.23";
473 my $socks_port = 9050;
474 my $socks_user = "";
475
476 sub socks4a_connect {
477 my ($host, $port, $connect_cb, $prepare_cb) = @_;
478
479 my $hdl = new AnyEvent::Handle
480 connect => [$socks_host, $socks_port],
481 on_prepare => sub { $prepare_cb->($_[0]{fh}) },
482 on_error => sub { $connect_cb->() },
483 ;
484
485 $hdl->push_write (pack "CCnNZ*Z*", 4, 1, $port, 1, $socks_user, $host);
486
487 $hdl->push_read (chunk => 8, sub {
488 my ($hdl, $chunk) = @_;
489 my ($status, $port, $ipn) = unpack "xCna4", $chunk;
490
491 if ($status == 0x5a) {
492 $connect_cb->($hdl->{fh}, (format_address $ipn) . ":$port");
493 } else {
494 $! = Errno::ENXIO; $connect_cb->();
495 }
496 });
497
498 $hdl
499 }
500
501 Use "socks4a_connect" instead of "tcp_connect" when doing
502 "http_request"s, possibly after switching off other proxy types:
503
504 AnyEvent::HTTP::set_proxy undef; # usually you do not want other proxies
505
506 http_get 'http://www.google.com', tcp_connect => \&socks4a_connect, sub {
507 my ($data, $headers) = @_;
508 ...
509 };
299 510
300SEE ALSO 511SEE ALSO
301 AnyEvent. 512 AnyEvent.
302 513
303AUTHOR 514AUTHOR

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines