ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/README
(Generate patch)

Comparing AnyEvent-HTTP/README (file contents):
Revision 1.4 by root, Thu Jul 24 06:01:10 2008 UTC vs.
Revision 1.14 by root, Fri Dec 31 03:47:32 2010 UTC

… …
28 limited support. 28 limited support.
29 29
30 METHODS 30 METHODS
31 http_get $url, key => value..., $cb->($data, $headers) 31 http_get $url, key => value..., $cb->($data, $headers)
32 Executes an HTTP-GET request. See the http_request function for 32 Executes an HTTP-GET request. See the http_request function for
33 details on additional parameters. 33 details on additional parameters and the return value.
34 34
35 http_head $url, key => value..., $cb->($data, $headers) 35 http_head $url, key => value..., $cb->($data, $headers)
36 Executes an HTTP-HEAD request. See the http_request function for 36 Executes an HTTP-HEAD request. See the http_request function for
37 details on additional parameters. 37 details on additional parameters and the return value.
38 38
39 http_post $url, $body, key => value..., $cb->($data, $headers) 39 http_post $url, $body, key => value..., $cb->($data, $headers)
40 Executes an HTTP-POST request with a request body of $body. See the 40 Executes an HTTP-POST request with a request body of $body. See the
41 http_request function for details on additional parameters. 41 http_request function for details on additional parameters and the
42 return value.
42 43
43 http_request $method => $url, key => value..., $cb->($data, $headers) 44 http_request $method => $url, key => value..., $cb->($data, $headers)
44 Executes a HTTP request of type $method (e.g. "GET", "POST"). The 45 Executes a HTTP request of type $method (e.g. "GET", "POST"). The
45 URL must be an absolute http or https URL. 46 URL must be an absolute http or https URL.
46 47
48 When called in void context, nothing is returned. In other contexts,
49 "http_request" returns a "cancellation guard" - you have to keep the
50 object at least alive until the callback get called. If the object
51 gets destroyed before the callback is called, the request will be
52 cancelled.
53
47 The callback will be called with the response data as first argument 54 The callback will be called with the response body data as first
48 (or "undef" if it wasn't available due to errors), and a hash-ref 55 argument (or "undef" if an error occured), and a hash-ref with
49 with response headers as second argument. 56 response headers as second argument.
50 57
51 All the headers in that hash are lowercased. In addition to the 58 All the headers in that hash are lowercased. In addition to the
52 response headers, the "pseudo-headers" "HTTPVersion", "Status" and 59 response headers, the "pseudo-headers" (uppercase to avoid clashing
60 with possible response headers) "HTTPVersion", "Status" and "Reason"
53 "Reason" contain the three parts of the HTTP Status-Line of the same 61 contain the three parts of the HTTP Status-Line of the same name. If
62 an error occurs during the body phase of a request, then the
63 original "Status" and "Reason" values from the header are available
64 as "OrigStatus" and "OrigReason".
65
54 name. The pseudo-header "URL" contains the original URL (which can 66 The pseudo-header "URL" contains the actual URL (which can differ
55 differ from the requested URL when following redirects). 67 from the requested URL when following redirects - for example, you
68 might get an error that your URL scheme is not supported even though
69 your URL is a valid http URL because it redirected to an ftp URL, in
70 which case you can look at the URL pseudo header).
56 71
72 The pseudo-header "Redirect" only exists when the request was a
73 result of an internal redirect. In that case it is an array
74 reference with the "($data, $headers)" from the redirect response.
75 Note that this response could in turn be the result of a redirect
76 itself, and "$headers->{Redirect}[1]{Redirect}" will then contain
77 the original response, and so on.
78
57 If the server sends a header multiple lines, then their contents 79 If the server sends a header multiple times, then their contents
58 will be joined together with "\x00". 80 will be joined together with a comma (","), as per the HTTP spec.
59 81
60 If an internal error occurs, such as not being able to resolve a 82 If an internal error occurs, such as not being able to resolve a
61 hostname, then $data will be "undef", "$headers->{Status}" will be 83 hostname, then $data will be "undef", "$headers->{Status}" will be
62 599 and the "Reason" pseudo-header will contain an error message. 84 "59x" (usually 599) and the "Reason" pseudo-header will contain an
85 error message.
63 86
64 A typical callback might look like this: 87 A typical callback might look like this:
65 88
66 sub { 89 sub {
67 my ($body, $hdr) = @_; 90 my ($body, $hdr) = @_;
… …
82 105
83 headers => hashref 106 headers => hashref
84 The request headers to use. Currently, "http_request" may 107 The request headers to use. Currently, "http_request" may
85 provide its own "Host:", "Content-Length:", "Connection:" and 108 provide its own "Host:", "Content-Length:", "Connection:" and
86 "Cookie:" headers and will provide defaults for "User-Agent:" 109 "Cookie:" headers and will provide defaults for "User-Agent:"
87 and "Referer:". 110 and "Referer:" (this can be suppressed by using "undef" for
111 these headers in which case they won't be sent at all).
88 112
89 timeout => $seconds 113 timeout => $seconds
90 The time-out to use for various stages - each connect attempt 114 The time-out to use for various stages - each connect attempt
91 will reset the timeout, as will read or write activity. Default 115 will reset the timeout, as will read or write activity, i.e.
116 this is not an overall timeout.
117
92 timeout is 5 minutes. 118 Default timeout is 5 minutes.
93 119
94 proxy => [$host, $port[, $scheme]] or undef 120 proxy => [$host, $port[, $scheme]] or undef
95 Use the given http proxy for all requests. If not specified, 121 Use the given http proxy for all requests. If not specified,
96 then the default proxy (as specified by $ENV{http_proxy}) is 122 then the default proxy (as specified by $ENV{http_proxy}) is
97 used. 123 used.
98 124
99 $scheme must be either missing or "http" for HTTP, or "https" 125 $scheme must be either missing, "http" for HTTP or "https" for
100 for HTTPS. 126 HTTPS.
101 127
102 body => $string 128 body => $string
103 The request body, usually empty. Will be-sent as-is (future 129 The request body, usually empty. Will be-sent as-is (future
104 versions of this module might offer more options). 130 versions of this module might offer more options).
105 131
… …
108 loosely based on the original netscape specification. 134 loosely based on the original netscape specification.
109 135
110 The $hash_ref must be an (initially empty) hash reference which 136 The $hash_ref must be an (initially empty) hash reference which
111 will get updated automatically. It is possible to save the 137 will get updated automatically. It is possible to save the
112 cookie_jar to persistent storage with something like JSON or 138 cookie_jar to persistent storage with something like JSON or
113 Storable, but this is not recommended, as expire times are 139 Storable, but this is not recommended, as expiry times are
114 currently being ignored. 140 currently being ignored.
115 141
116 Note that this cookie implementation is not of very high 142 Note that this cookie implementation is not of very high
117 quality, nor meant to be complete. If you want complete cookie 143 quality, nor meant to be complete. If you want complete cookie
118 management you have to do that on your own. "cookie_jar" is 144 management you have to do that on your own. "cookie_jar" is
119 meant as a quick fix to get some cookie-using sites working. 145 meant as a quick fix to get some cookie-using sites working.
120 Cookies are a privacy disaster, do not use them unless required 146 Cookies are a privacy disaster, do not use them unless required
121 to. 147 to.
148
149 tls_ctx => $scheme | $tls_ctx
150 Specifies the AnyEvent::TLS context to be used for https
151 connections. This parameter follows the same rules as the
152 "tls_ctx" parameter to AnyEvent::Handle, but additionally, the
153 two strings "low" or "high" can be specified, which give you a
154 predefined low-security (no verification, highest compatibility)
155 and high-security (CA and common-name verification) TLS context.
156
157 The default for this option is "low", which could be interpreted
158 as "give me the page, no matter what".
159
160 on_prepare => $callback->($fh)
161 In rare cases you need to "tune" the socket before it is used to
162 connect (for exmaple, to bind it on a given IP address). This
163 parameter overrides the prepare callback passed to
164 "AnyEvent::Socket::tcp_connect" and behaves exactly the same way
165 (e.g. it has to provide a timeout). See the description for the
166 $prepare_cb argument of "AnyEvent::Socket::tcp_connect" for
167 details.
168
169 tcp_connect => $callback->($host, $service, $connect_cb,
170 $prepare_cb)
171 In even rarer cases you want total control over how
172 AnyEvent::HTTP establishes connections. Normally it uses
173 AnyEvent::Socket::tcp_connect to do this, but you can provide
174 your own "tcp_connect" function - obviously, it has to follow
175 the same calling conventions, except that it may always return a
176 connection guard object.
177
178 There are probably lots of weird uses for this function,
179 starting from tracing the hosts "http_request" actually tries to
180 connect, to (inexact but fast) host => IP address caching or
181 even socks protocol support.
182
183 on_header => $callback->($headers)
184 When specified, this callback will be called with the header
185 hash as soon as headers have been successfully received from the
186 remote server (not on locally-generated errors).
187
188 It has to return either true (in which case AnyEvent::HTTP will
189 continue), or false, in which case AnyEvent::HTTP will cancel
190 the download (and call the finish callback with an error code of
191 598).
192
193 This callback is useful, among other things, to quickly reject
194 unwanted content, which, if it is supposed to be rare, can be
195 faster than first doing a "HEAD" request.
196
197 Example: cancel the request unless the content-type is
198 "text/html".
199
200 on_header => sub {
201 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
202 },
203
204 on_body => $callback->($partial_body, $headers)
205 When specified, all body data will be passed to this callback
206 instead of to the completion callback. The completion callback
207 will get the empty string instead of the body data.
208
209 It has to return either true (in which case AnyEvent::HTTP will
210 continue), or false, in which case AnyEvent::HTTP will cancel
211 the download (and call the completion callback with an error
212 code of 598).
213
214 This callback is useful when the data is too large to be held in
215 memory (so the callback writes it to a file) or when only some
216 information should be extracted, or when the body should be
217 processed incrementally.
218
219 It is usually preferred over doing your own body handling via
220 "want_body_handle", but in case of streaming APIs, where HTTP is
221 only used to create a connection, "want_body_handle" is the
222 better alternative, as it allows you to install your own event
223 handler, reducing resource usage.
224
225 want_body_handle => $enable
226 When enabled (default is disabled), the behaviour of
227 AnyEvent::HTTP changes considerably: after parsing the headers,
228 and instead of downloading the body (if any), the completion
229 callback will be called. Instead of the $body argument
230 containing the body data, the callback will receive the
231 AnyEvent::Handle object associated with the connection. In error
232 cases, "undef" will be passed. When there is no body (e.g.
233 status 304), the empty string will be passed.
234
235 The handle object might or might not be in TLS mode, might be
236 connected to a proxy, be a persistent connection etc., and
237 configured in unspecified ways. The user is responsible for this
238 handle (it will not be used by this module anymore).
239
240 This is useful with some push-type services, where, after the
241 initial headers, an interactive protocol is used (typical
242 example would be the push-style twitter API which starts a
243 JSON/XML stream).
244
245 If you think you need this, first have a look at "on_body", to
246 see if that doesn't solve your problem in a better way.
122 247
123 Example: make a simple HTTP GET request for http://www.nethype.de/ 248 Example: make a simple HTTP GET request for http://www.nethype.de/
124 249
125 http_request GET => "http://www.nethype.de/", sub { 250 http_request GET => "http://www.nethype.de/", sub {
126 my ($body, $hdr) = @_; 251 my ($body, $hdr) = @_;
… …
138 use Data::Dumper; 263 use Data::Dumper;
139 print Dumper $hdr; 264 print Dumper $hdr;
140 } 265 }
141 ; 266 ;
142 267
268 Example: make another simple HTTP GET request, but immediately try
269 to cancel it.
270
271 my $request = http_request GET => "http://www.nethype.de/", sub {
272 my ($body, $hdr) = @_;
273 print "$body\n";
274 };
275
276 undef $request;
277
278 DNS CACHING
279 AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for the
280 actual connection, which in turn uses AnyEvent::DNS to resolve
281 hostnames. The latter is a simple stub resolver and does no caching on
282 its own. If you want DNS caching, you currently have to provide your own
283 default resolver (by storing a suitable resolver object in
284 $AnyEvent::DNS::RESOLVER).
285
143 GLOBAL FUNCTIONS AND VARIABLES 286 GLOBAL FUNCTIONS AND VARIABLES
144 AnyEvent::HTTP::set_proxy "proxy-url" 287 AnyEvent::HTTP::set_proxy "proxy-url"
145 Sets the default proxy server to use. The proxy-url must begin with 288 Sets the default proxy server to use. The proxy-url must begin with
146 a string of the form "http://host:port" (optionally "https:..."). 289 a string of the form "http://host:port" (optionally "https:..."),
290 croaks otherwise.
291
292 To clear an already-set proxy, use "undef".
293
294 $date = AnyEvent::HTTP::format_date $timestamp
295 Takes a POSIX timestamp (seconds since the epoch) and formats it as
296 a HTTP Date (RFC 2616).
297
298 $timestamp = AnyEvent::HTTP::parse_date $date
299 Takes a HTTP Date (RFC 2616) and returns the corresponding POSIX
300 timestamp, or "undef" if the date cannot be parsed.
147 301
148 $AnyEvent::HTTP::MAX_RECURSE 302 $AnyEvent::HTTP::MAX_RECURSE
149 The default value for the "recurse" request parameter (default: 10). 303 The default value for the "recurse" request parameter (default: 10).
150 304
151 $AnyEvent::HTTP::USERAGENT 305 $AnyEvent::HTTP::USERAGENT
152 The default value for the "User-Agent" header (the default is 306 The default value for the "User-Agent" header (the default is
153 "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; 307 "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
154 +http://software.schmorp.de/pkg/AnyEvent)"). 308 +http://software.schmorp.de/pkg/AnyEvent)").
155 309
156 $AnyEvent::HTTP::MAX_PERSISTENT 310 $AnyEvent::HTTP::MAX_PER_HOST
157 The maximum number of persistent connections to keep open (default: 311 The maximum number of concurrent connections to the same host
158 8). 312 (identified by the hostname). If the limit is exceeded, then the
313 additional requests are queued until previous connections are
314 closed.
159 315
160 Not implemented currently. 316 The default value for this is 4, and it is highly advisable to not
161 317 increase it.
162 $AnyEvent::HTTP::PERSISTENT_TIMEOUT
163 The maximum time to cache a persistent connection, in seconds
164 (default: 2).
165
166 Not implemented currently.
167 318
168 $AnyEvent::HTTP::ACTIVE 319 $AnyEvent::HTTP::ACTIVE
169 The number of active connections. This is not the number of 320 The number of active connections. This is not the number of
170 currently running requests, but the number of currently open and 321 currently running requests, but the number of currently open and
171 non-idle TCP connections. This number of can be useful for 322 non-idle TCP connections. This number of can be useful for
172 load-leveling. 323 load-leveling.
173 324
325 SOCKS PROXIES
326 Socks proxies are not directly supported by AnyEvent::HTTP. You can
327 compile your perl to support socks, or use an external program such as
328 socksify (dante) or tsocks to make your program use a socks proxy
329 transparently.
330
331 Alternatively, for AnyEvent::HTTP only, you can use your own
332 "tcp_connect" function that does the proxy handshake - here is an
333 example that works with socks4a proxies:
334
335 use Errno;
336 use AnyEvent::Util;
337 use AnyEvent::Socket;
338 use AnyEvent::Handle;
339
340 # host, port and username of/for your socks4a proxy
341 my $socks_host = "10.0.0.23";
342 my $socks_port = 9050;
343 my $socks_user = "";
344
345 sub socks4a_connect {
346 my ($host, $port, $connect_cb, $prepare_cb) = @_;
347
348 my $hdl = new AnyEvent::Handle
349 connect => [$socks_host, $socks_port],
350 on_prepare => sub { $prepare_cb->($_[0]{fh}) },
351 on_error => sub { $connect_cb->() },
352 ;
353
354 $hdl->push_write (pack "CCnNZ*Z*", 4, 1, $port, 1, $socks_user, $host);
355
356 $hdl->push_read (chunk => 8, sub {
357 my ($hdl, $chunk) = @_;
358 my ($status, $port, $ipn) = unpack "xCna4", $chunk;
359
360 if ($status == 0x5a) {
361 $connect_cb->($hdl->{fh}, (format_address $ipn) . ":$port");
362 } else {
363 $! = Errno::ENXIO; $connect_cb->();
364 }
365 });
366
367 $hdl
368 }
369
370 Use "socks4a_connect" instead of "tcp_connect" when doing
371 "http_request"s, possibly after switching off other proxy types:
372
373 AnyEvent::HTTP::set_proxy undef; # usually you do not want other proxies
374
375 http_get 'http://www.google.com', tcp_connect => \&socks4a_connect, sub {
376 my ($data, $headers) = @_;
377 ...
378 };
379
174SEE ALSO 380SEE ALSO
175 AnyEvent. 381 AnyEvent.
176 382
177AUTHOR 383AUTHOR
178 Marc Lehmann <schmorp@schmorp.de> 384 Marc Lehmann <schmorp@schmorp.de>
179 http://home.schmorp.de/ 385 http://home.schmorp.de/
180 386
387 With many thanks to Дмитрий Шалашов, who provided
388 countless testcases and bugreports.
389

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines