[ViewVC] Annotation of: cvs/AnyEvent-HTTP/README

NAME
    AnyEvent::HTTP - simple but non-blocking HTTP/HTTPS client

SYNOPSIS
       use AnyEvent::HTTP;

       http_get "http://www.nethype.de/", sub { print $_[1] };

       # ... do something else here

DESCRIPTION
    This module is an AnyEvent user, you need to make sure that you use and
    run a supported event loop.

    This module implements a simple, stateless and non-blocking HTTP client.
    It supports GET, POST and other request methods, cookies and more, all
    on a very low level. It can follow redirects supports proxies and
    automatically limits the number of connections to the values specified
    in the RFC.

    It should generally be a "good client" that is enough for most HTTP
    tasks. Simple tasks should be simple, but complex tasks should still be
    possible as the user retains control over request and response headers.

    The caller is responsible for authentication management, cookies (if the
    simplistic implementation in this module doesn't suffice), referer and
    other high-level protocol details for which this module offers only
    limited support.

  METHODS
    http_get $url, key => value..., $cb->($data, $headers)
        Executes an HTTP-GET request. See the http_request function for
        details on additional parameters and the return value.

    http_head $url, key => value..., $cb->($data, $headers)
        Executes an HTTP-HEAD request. See the http_request function for
        details on additional parameters and the return value.

    http_post $url, $body, key => value..., $cb->($data, $headers)
        Executes an HTTP-POST request with a request body of $body. See the
        http_request function for details on additional parameters and the
        return value.

    http_request $method => $url, key => value..., $cb->($data, $headers)
        Executes a HTTP request of type $method (e.g. "GET", "POST"). The
        URL must be an absolute http or https URL.

        When called in void context, nothing is returned. In other contexts,
        "http_request" returns a "cancellation guard" - you have to keep the
        object at least alive until the callback get called. If the object
        gets destroyed before the callbakc is called, the request will be
        cancelled.

        The callback will be called with the response body data as first
        argument (or "undef" if an error occured), and a hash-ref with
        response headers as second argument.

        All the headers in that hash are lowercased. In addition to the
        response headers, the "pseudo-headers" "HTTPVersion", "Status" and
        "Reason" contain the three parts of the HTTP Status-Line of the same
        name. The pseudo-header "URL" contains the original URL (which can
        differ from the requested URL when following redirects).

        If the server sends a header multiple times, then their contents
        will be joined together with a comma (","), as per the HTTP spec.

        If an internal error occurs, such as not being able to resolve a
        hostname, then $data will be "undef", "$headers->{Status}" will be
        "59x" (usually 599) and the "Reason" pseudo-header will contain an
        error message.

        A typical callback might look like this:

           sub {
              my ($body, $hdr) = @_;

              if ($hdr->{Status} =~ /^2/) {
                 ... everything should be ok
              } else {
                 print "error, $hdr->{Status} $hdr->{Reason}\n";
              }
           }

        Additional parameters are key-value pairs, and are fully optional.
        They include:

        recurse => $count (default: $MAX_RECURSE)
            Whether to recurse requests or not, e.g. on redirects,
            authentication retries and so on, and how often to do so.

        headers => hashref
            The request headers to use. Currently, "http_request" may
            provide its own "Host:", "Content-Length:", "Connection:" and
            "Cookie:" headers and will provide defaults for "User-Agent:"
            and "Referer:" (this can be suppressed by using "undef" for
            these headers in which case they won't be sent at all).

        timeout => $seconds
            The time-out to use for various stages - each connect attempt
            will reset the timeout, as will read or write activity, i.e.
            this is not an overall timeout.

            Default timeout is 5 minutes.

        proxy => [$host, $port[, $scheme]] or undef
            Use the given http proxy for all requests. If not specified,
            then the default proxy (as specified by $ENV{http_proxy}) is
            used.

            $scheme must be either missing, "http" for HTTP or "https" for
            HTTPS.

        body => $string
            The request body, usually empty. Will be-sent as-is (future
            versions of this module might offer more options).

        cookie_jar => $hash_ref
            Passing this parameter enables (simplified) cookie-processing,
            loosely based on the original netscape specification.

            The $hash_ref must be an (initially empty) hash reference which
            will get updated automatically. It is possible to save the
            cookie_jar to persistent storage with something like JSON or
            Storable, but this is not recommended, as expiry times are
            currently being ignored.

            Note that this cookie implementation is not of very high
            quality, nor meant to be complete. If you want complete cookie
            management you have to do that on your own. "cookie_jar" is
            meant as a quick fix to get some cookie-using sites working.
            Cookies are a privacy disaster, do not use them unless required
            to.

        tls_ctx => $scheme | $tls_ctx
            Specifies the AnyEvent::TLS context to be used for https
            connections. This parameter follows the same rules as the
            "tls_ctx" parameter to AnyEvent::Handle, but additionally, the
            two strings "low" or "high" can be specified, which give you a
            predefined low-security (no verification, highest compatibility)
            and high-security (CA and common-name verification) TLS context.

            The default for this option is "low", which could be interpreted
            as "give me the page, no matter what".

        on_prepare => $callback->($fh)
            In rare cases you need to "tune" the socket before it is used to
            connect (for exmaple, to bind it on a given IP address). This
            parameter overrides the prepare callback passed to
            "AnyEvent::Socket::tcp_connect" and behaves exactly the same way
            (e.g. it has to provide a timeout). See the description for the
            $prepare_cb argument of "AnyEvent::Socket::tcp_connect" for
            details.

        on_header => $callback->($headers)
            When specified, this callback will be called with the header
            hash as soon as headers have been successfully received from the
            remote server (not on locally-generated errors).

            It has to return either true (in which case AnyEvent::HTTP will
            continue), or false, in which case AnyEvent::HTTP will cancel
            the download (and call the finish callback with an error code of
            598).

            This callback is useful, among other things, to quickly reject
            unwanted content, which, if it is supposed to be rare, can be
            faster than first doing a "HEAD" request.

            Example: cancel the request unless the content-type is
            "text/html".

               on_header => sub {
                  $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
               },

        on_body => $callback->($partial_body, $headers)
            When specified, all body data will be passed to this callback
            instead of to the completion callback. The completion callback
            will get the empty string instead of the body data.

            It has to return either true (in which case AnyEvent::HTTP will
            continue), or false, in which case AnyEvent::HTTP will cancel
            the download (and call the completion callback with an error
            code of 598).

            This callback is useful when the data is too large to be held in
            memory (so the callback writes it to a file) or when only some
            information should be extracted, or when the body should be
            processed incrementally.

            It is usually preferred over doing your own body handling via
            "want_body_handle", but in case of streaming APIs, where HTTP is
            only used to create a connection, "want_body_handle" is the
            better alternative, as it allows you to install your own event
            handler, reducing resource usage.

        want_body_handle => $enable
            When enabled (default is disabled), the behaviour of
            AnyEvent::HTTP changes considerably: after parsing the headers,
            and instead of downloading the body (if any), the completion
            callback will be called. Instead of the $body argument
            containing the body data, the callback will receive the
            AnyEvent::Handle object associated with the connection. In error
            cases, "undef" will be passed. When there is no body (e.g.
            status 304), the empty string will be passed.

            The handle object might or might not be in TLS mode, might be
            connected to a proxy, be a persistent connection etc., and
            configured in unspecified ways. The user is responsible for this
            handle (it will not be used by this module anymore).

            This is useful with some push-type services, where, after the
            initial headers, an interactive protocol is used (typical
            example would be the push-style twitter API which starts a
            JSON/XML stream).

            If you think you need this, first have a look at "on_body", to
            see if that doesn't solve your problem in a better way.

        Example: make a simple HTTP GET request for http://www.nethype.de/

           http_request GET => "http://www.nethype.de/", sub {
              my ($body, $hdr) = @_;
              print "$body\n";
           };

        Example: make a HTTP HEAD request on https://www.google.com/, use a
        timeout of 30 seconds.

           http_request
              GET     => "https://www.google.com",
              timeout => 30,
              sub {
                 my ($body, $hdr) = @_;
                 use Data::Dumper;
                 print Dumper $hdr;
              }
           ;

        Example: make another simple HTTP GET request, but immediately try
        to cancel it.

           my $request = http_request GET => "http://www.nethype.de/", sub {
              my ($body, $hdr) = @_;
              print "$body\n";
           };

           undef $request;

  GLOBAL FUNCTIONS AND VARIABLES
    AnyEvent::HTTP::set_proxy "proxy-url"
        Sets the default proxy server to use. The proxy-url must begin with
        a string of the form "http://host:port" (optionally "https:...").

    $AnyEvent::HTTP::MAX_RECURSE
        The default value for the "recurse" request parameter (default: 10).

    $AnyEvent::HTTP::USERAGENT
        The default value for the "User-Agent" header (the default is
        "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
        +http://software.schmorp.de/pkg/AnyEvent)").

    $AnyEvent::HTTP::MAX_PER_HOST
        The maximum number of concurrent connections to the same host
        (identified by the hostname). If the limit is exceeded, then the
        additional requests are queued until previous connections are
        closed.

        The default value for this is 4, and it is highly advisable to not
        increase it.

    $AnyEvent::HTTP::ACTIVE
        The number of active connections. This is not the number of
        currently running requests, but the number of currently open and
        non-idle TCP connections. This number of can be useful for
        load-leveling.

SEE ALSO
    AnyEvent.

AUTHOR
       Marc Lehmann <schmorp@schmorp.de>
       http://home.schmorp.de/

    With many thanks to Дмитрий Шалашов, who provided
    countless testcases and bugreports.

Revision:	1.11
Committed:	Fri Aug 14 15:21:33 2009 UTC (14 years, 9 months ago) by root
Branch:	MAIN
CVS Tags:	rel-1_43
Changes since 1.10:	+13 -2 lines
Log Message:	1.43
#	User	Rev	Content
1	root	1.1	NAME
2	root	1.2	AnyEvent::HTTP - simple but non-blocking HTTP/HTTPS client
3	root	1.1
4			SYNOPSIS
5	root	1.2	use AnyEvent::HTTP;
6	root	1.1
7	root	1.3	http_get "http://www.nethype.de/", sub { print $_[1] };
8
9			# ... do something else here
10
11	root	1.1	DESCRIPTION
12			This module is an AnyEvent user, you need to make sure that you use and
13			run a supported event loop.
14
15	root	1.2	This module implements a simple, stateless and non-blocking HTTP client.
16			It supports GET, POST and other request methods, cookies and more, all
17			on a very low level. It can follow redirects supports proxies and
18			automatically limits the number of connections to the values specified
19			in the RFC.
20
21			It should generally be a "good client" that is enough for most HTTP
22			tasks. Simple tasks should be simple, but complex tasks should still be
23			possible as the user retains control over request and response headers.
24
25			The caller is responsible for authentication management, cookies (if the
26			simplistic implementation in this module doesn't suffice), referer and
27			other high-level protocol details for which this module offers only
28			limited support.
29
30			METHODS
31			http_get $url, key => value..., $cb->($data, $headers)
32			Executes an HTTP-GET request. See the http_request function for
33	root	1.5	details on additional parameters and the return value.
34	root	1.2
35			http_head $url, key => value..., $cb->($data, $headers)
36			Executes an HTTP-HEAD request. See the http_request function for
37	root	1.5	details on additional parameters and the return value.
38	root	1.2
39			http_post $url, $body, key => value..., $cb->($data, $headers)
40	root	1.4	Executes an HTTP-POST request with a request body of $body. See the
41	root	1.5	http_request function for details on additional parameters and the
42			return value.
43	root	1.2
44			http_request $method => $url, key => value..., $cb->($data, $headers)
45			Executes a HTTP request of type $method (e.g. "GET", "POST"). The
46			URL must be an absolute http or https URL.
47
48	root	1.5	When called in void context, nothing is returned. In other contexts,
49			"http_request" returns a "cancellation guard" - you have to keep the
50			object at least alive until the callback get called. If the object
51			gets destroyed before the callbakc is called, the request will be
52			cancelled.
53
54	root	1.8	The callback will be called with the response body data as first
55			argument (or "undef" if an error occured), and a hash-ref with
56			response headers as second argument.
57	root	1.2
58			All the headers in that hash are lowercased. In addition to the
59	root	1.3	response headers, the "pseudo-headers" "HTTPVersion", "Status" and
60			"Reason" contain the three parts of the HTTP Status-Line of the same
61			name. The pseudo-header "URL" contains the original URL (which can
62			differ from the requested URL when following redirects).
63
64	root	1.6	If the server sends a header multiple times, then their contents
65			will be joined together with a comma (","), as per the HTTP spec.
66	root	1.2
67			If an internal error occurs, such as not being able to resolve a
68			hostname, then $data will be "undef", "$headers->{Status}" will be
69	root	1.8	"59x" (usually 599) and the "Reason" pseudo-header will contain an
70			error message.
71	root	1.2
72			A typical callback might look like this:
73
74			sub {
75			my ($body, $hdr) = @_;
76
77			if ($hdr->{Status} =~ /^2/) {
78			... everything should be ok
79			} else {
80			print "error, $hdr->{Status} $hdr->{Reason}\n";
81			}
82			}
83
84			Additional parameters are key-value pairs, and are fully optional.
85			They include:
86
87			recurse => $count (default: $MAX_RECURSE)
88			Whether to recurse requests or not, e.g. on redirects,
89			authentication retries and so on, and how often to do so.
90
91			headers => hashref
92			The request headers to use. Currently, "http_request" may
93			provide its own "Host:", "Content-Length:", "Connection:" and
94			"Cookie:" headers and will provide defaults for "User-Agent:"
95	root	1.10	and "Referer:" (this can be suppressed by using "undef" for
96			these headers in which case they won't be sent at all).
97	root	1.2
98			timeout => $seconds
99			The time-out to use for various stages - each connect attempt
100	root	1.11	will reset the timeout, as will read or write activity, i.e.
101			this is not an overall timeout.
102
103			Default timeout is 5 minutes.
104	root	1.2
105			proxy => [$host, $port[, $scheme]] or undef
106			Use the given http proxy for all requests. If not specified,
107			then the default proxy (as specified by $ENV{http_proxy}) is
108			used.
109
110	root	1.10	$scheme must be either missing, "http" for HTTP or "https" for
111			HTTPS.
112	root	1.2
113			body => $string
114			The request body, usually empty. Will be-sent as-is (future
115			versions of this module might offer more options).
116
117			cookie_jar => $hash_ref
118			Passing this parameter enables (simplified) cookie-processing,
119			loosely based on the original netscape specification.
120
121			The $hash_ref must be an (initially empty) hash reference which
122			will get updated automatically. It is possible to save the
123			cookie_jar to persistent storage with something like JSON or
124	root	1.8	Storable, but this is not recommended, as expiry times are
125	root	1.2	currently being ignored.
126
127			Note that this cookie implementation is not of very high
128			quality, nor meant to be complete. If you want complete cookie
129			management you have to do that on your own. "cookie_jar" is
130			meant as a quick fix to get some cookie-using sites working.
131			Cookies are a privacy disaster, do not use them unless required
132			to.
133
134	root	1.8	tls_ctx => $scheme \| $tls_ctx
135			Specifies the AnyEvent::TLS context to be used for https
136			connections. This parameter follows the same rules as the
137			"tls_ctx" parameter to AnyEvent::Handle, but additionally, the
138			two strings "low" or "high" can be specified, which give you a
139			predefined low-security (no verification, highest compatibility)
140			and high-security (CA and common-name verification) TLS context.
141
142			The default for this option is "low", which could be interpreted
143			as "give me the page, no matter what".
144
145	root	1.11	on_prepare => $callback->($fh)
146			In rare cases you need to "tune" the socket before it is used to
147			connect (for exmaple, to bind it on a given IP address). This
148			parameter overrides the prepare callback passed to
149			"AnyEvent::Socket::tcp_connect" and behaves exactly the same way
150			(e.g. it has to provide a timeout). See the description for the
151			$prepare_cb argument of "AnyEvent::Socket::tcp_connect" for
152			details.
153
154	root	1.8	on_header => $callback->($headers)
155			When specified, this callback will be called with the header
156			hash as soon as headers have been successfully received from the
157			remote server (not on locally-generated errors).
158
159			It has to return either true (in which case AnyEvent::HTTP will
160			continue), or false, in which case AnyEvent::HTTP will cancel
161			the download (and call the finish callback with an error code of
162			598).
163
164			This callback is useful, among other things, to quickly reject
165			unwanted content, which, if it is supposed to be rare, can be
166			faster than first doing a "HEAD" request.
167
168			Example: cancel the request unless the content-type is
169			"text/html".
170
171			on_header => sub {
172			$_[0]{"content-type"} =~ /^text\/html\s*(?:;\|$)/
173			},
174
175			on_body => $callback->($partial_body, $headers)
176			When specified, all body data will be passed to this callback
177			instead of to the completion callback. The completion callback
178			will get the empty string instead of the body data.
179
180			It has to return either true (in which case AnyEvent::HTTP will
181			continue), or false, in which case AnyEvent::HTTP will cancel
182			the download (and call the completion callback with an error
183			code of 598).
184
185			This callback is useful when the data is too large to be held in
186			memory (so the callback writes it to a file) or when only some
187			information should be extracted, or when the body should be
188			processed incrementally.
189
190			It is usually preferred over doing your own body handling via
191	root	1.9	"want_body_handle", but in case of streaming APIs, where HTTP is
192			only used to create a connection, "want_body_handle" is the
193			better alternative, as it allows you to install your own event
194			handler, reducing resource usage.
195	root	1.8
196			want_body_handle => $enable
197			When enabled (default is disabled), the behaviour of
198			AnyEvent::HTTP changes considerably: after parsing the headers,
199			and instead of downloading the body (if any), the completion
200			callback will be called. Instead of the $body argument
201			containing the body data, the callback will receive the
202			AnyEvent::Handle object associated with the connection. In error
203			cases, "undef" will be passed. When there is no body (e.g.
204			status 304), the empty string will be passed.
205
206			The handle object might or might not be in TLS mode, might be
207			connected to a proxy, be a persistent connection etc., and
208			configured in unspecified ways. The user is responsible for this
209			handle (it will not be used by this module anymore).
210
211			This is useful with some push-type services, where, after the
212			initial headers, an interactive protocol is used (typical
213			example would be the push-style twitter API which starts a
214			JSON/XML stream).
215
216			If you think you need this, first have a look at "on_body", to
217	root	1.9	see if that doesn't solve your problem in a better way.
218	root	1.8
219	root	1.2	Example: make a simple HTTP GET request for http://www.nethype.de/
220
221			http_request GET => "http://www.nethype.de/", sub {
222			my ($body, $hdr) = @_;
223			print "$body\n";
224			};
225
226			Example: make a HTTP HEAD request on https://www.google.com/, use a
227			timeout of 30 seconds.
228
229			http_request
230			GET => "https://www.google.com",
231			timeout => 30,
232			sub {
233			my ($body, $hdr) = @_;
234			use Data::Dumper;
235			print Dumper $hdr;
236			}
237			;
238
239	root	1.5	Example: make another simple HTTP GET request, but immediately try
240			to cancel it.
241
242			my $request = http_request GET => "http://www.nethype.de/", sub {
243			my ($body, $hdr) = @_;
244			print "$body\n";
245			};
246
247			undef $request;
248
249	root	1.2	GLOBAL FUNCTIONS AND VARIABLES
250			AnyEvent::HTTP::set_proxy "proxy-url"
251			Sets the default proxy server to use. The proxy-url must begin with
252			a string of the form "http://host:port" (optionally "https:...").
253
254			$AnyEvent::HTTP::MAX_RECURSE
255			The default value for the "recurse" request parameter (default: 10).
256
257			$AnyEvent::HTTP::USERAGENT
258			The default value for the "User-Agent" header (the default is
259	root	1.8	"Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
260	root	1.2	+http://software.schmorp.de/pkg/AnyEvent)").
261
262	root	1.8	$AnyEvent::HTTP::MAX_PER_HOST
263	root	1.10	The maximum number of concurrent connections to the same host
264	root	1.8	(identified by the hostname). If the limit is exceeded, then the
265			additional requests are queued until previous connections are
266			closed.
267	root	1.2
268	root	1.8	The default value for this is 4, and it is highly advisable to not
269			increase it.
270	root	1.2
271			$AnyEvent::HTTP::ACTIVE
272			The number of active connections. This is not the number of
273			currently running requests, but the number of currently open and
274			non-idle TCP connections. This number of can be useful for
275			load-leveling.
276	root	1.1
277			SEE ALSO
278	root	1.2	AnyEvent.
279	root	1.1
280			AUTHOR
281	root	1.3	Marc Lehmann <schmorp@schmorp.de>
282			http://home.schmorp.de/
283	root	1.1
284	root	1.7	With many thanks to Дмитрий Шалашов, who provided
285			countless testcases and bugreports.
286