[ViewVC] Diff of: cvs/AnyEvent-HTTP/README

Comparing AnyEvent-HTTP/README (file contents):
Revision 1.11 by root, Fri Aug 14 15:21:33 2009 UTC vs.
Revision 1.15 by root, Tue Jan 4 08:29:28 2011 UTC

…		…
46	URL must be an absolute http or https URL.	46	URL must be an absolute http or https URL.
47		47
48	When called in void context, nothing is returned. In other contexts,	48	When called in void context, nothing is returned. In other contexts,
49	"http_request" returns a "cancellation guard" - you have to keep the	49	"http_request" returns a "cancellation guard" - you have to keep the
50	object at least alive until the callback get called. If the object	50	object at least alive until the callback get called. If the object
51	gets destroyed before the callbakc is called, the request will be	51	gets destroyed before the callback is called, the request will be
52	cancelled.	52	cancelled.
53		53
54	The callback will be called with the response body data as first	54	The callback will be called with the response body data as first
55	argument (or "undef" if an error occured), and a hash-ref with	55	argument (or "undef" if an error occured), and a hash-ref with
56	response headers as second argument.	56	response headers (and trailers) as second argument.
57		57
58	All the headers in that hash are lowercased. In addition to the	58	All the headers in that hash are lowercased. In addition to the
59	response headers, the "pseudo-headers" "HTTPVersion", "Status" and	59	response headers, the "pseudo-headers" (uppercase to avoid clashing
		60	with possible response headers) "HTTPVersion", "Status" and "Reason"
60	"Reason" contain the three parts of the HTTP Status-Line of the same	61	contain the three parts of the HTTP Status-Line of the same name. If
		62	an error occurs during the body phase of a request, then the
		63	original "Status" and "Reason" values from the header are available
		64	as "OrigStatus" and "OrigReason".
		65
61	name. The pseudo-header "URL" contains the original URL (which can	66	The pseudo-header "URL" contains the actual URL (which can differ
62	differ from the requested URL when following redirects).	67	from the requested URL when following redirects - for example, you
		68	might get an error that your URL scheme is not supported even though
		69	your URL is a valid http URL because it redirected to an ftp URL, in
		70	which case you can look at the URL pseudo header).
		71
		72	The pseudo-header "Redirect" only exists when the request was a
		73	result of an internal redirect. In that case it is an array
		74	reference with the "($data, $headers)" from the redirect response.
		75	Note that this response could in turn be the result of a redirect
		76	itself, and "$headers->{Redirect}[1]{Redirect}" will then contain
		77	the original response, and so on.
63		78
64	If the server sends a header multiple times, then their contents	79	If the server sends a header multiple times, then their contents
65	will be joined together with a comma (","), as per the HTTP spec.	80	will be joined together with a comma (","), as per the HTTP spec.
66		81
67	If an internal error occurs, such as not being able to resolve a	82	If an internal error occurs, such as not being able to resolve a
68	hostname, then $data will be "undef", "$headers->{Status}" will be	83	hostname, then $data will be "undef", "$headers->{Status}" will be
69	"59x" (usually 599) and the "Reason" pseudo-header will contain an	84	590-599 and the "Reason" pseudo-header will contain an error
70	error message.	85	message. Currently the following status codes are used:
		86
		87	595 - errors during connection etsbalishment, proxy handshake.
		88	596 - errors during TLS negotiation, request sending and header
		89	processing.
		90	597 - errors during body receiving or processing.
		91	598 - user aborted request via "on_header" or "on_body".
		92	599 - other, usually nonretryable, errors (garbled URL etc.).
71		93
72	A typical callback might look like this:	94	A typical callback might look like this:
73		95
74	sub {	96	sub {
75	my ($body, $hdr) = @_;	97	my ($body, $hdr) = @_;
…		…
89	authentication retries and so on, and how often to do so.	111	authentication retries and so on, and how often to do so.
90		112
91	headers => hashref	113	headers => hashref
92	The request headers to use. Currently, "http_request" may	114	The request headers to use. Currently, "http_request" may
93	provide its own "Host:", "Content-Length:", "Connection:" and	115	provide its own "Host:", "Content-Length:", "Connection:" and
94	"Cookie:" headers and will provide defaults for "User-Agent:"	116	"Cookie:" headers and will provide defaults at least for "TE:",
95	and "Referer:" (this can be suppressed by using "undef" for	117	"Referer:" and "User-Agent:" (this can be suppressed by using
96	these headers in which case they won't be sent at all).	118	"undef" for these headers in which case they won't be sent at
		119	all).
		120
		121	You really should provide your own "User-Agent:" header value
		122	that is appropriate for your program - I wouldn't be surprised
		123	if the default AnyEvent string gets blocked by webservers sooner
		124	or later.
97		125
98	timeout => $seconds	126	timeout => $seconds
99	The time-out to use for various stages - each connect attempt	127	The time-out to use for various stages - each connect attempt
100	will reset the timeout, as will read or write activity, i.e.	128	will reset the timeout, as will read or write activity, i.e.
101	this is not an overall timeout.	129	this is not an overall timeout.
…		…
105	proxy => [$host, $port[, $scheme]] or undef	133	proxy => [$host, $port[, $scheme]] or undef
106	Use the given http proxy for all requests. If not specified,	134	Use the given http proxy for all requests. If not specified,
107	then the default proxy (as specified by $ENV{http_proxy}) is	135	then the default proxy (as specified by $ENV{http_proxy}) is
108	used.	136	used.
109		137
110	$scheme must be either missing, "http" for HTTP or "https" for	138	$scheme must be either missing or must be "http" for HTTP.
111	HTTPS.
112		139
113	body => $string	140	body => $string
114	The request body, usually empty. Will be-sent as-is (future	141	The request body, usually empty. Will be sent as-is (future
115	versions of this module might offer more options).	142	versions of this module might offer more options).
116		143
117	cookie_jar => $hash_ref	144	cookie_jar => $hash_ref
118	Passing this parameter enables (simplified) cookie-processing,	145	Passing this parameter enables (simplified) cookie-processing,
119	loosely based on the original netscape specification.	146	loosely based on the original netscape specification.
120		147
121	The $hash_ref must be an (initially empty) hash reference which	148	The $hash_ref must be an (initially empty) hash reference which
122	will get updated automatically. It is possible to save the	149	will get updated automatically. It is possible to save the
123	cookie_jar to persistent storage with something like JSON or	150	cookie jar to persistent storage with something like JSON or
124	Storable, but this is not recommended, as expiry times are	151	Storable - see the "AnyEvent::HTTP::cookie_jar_expire" function
125	currently being ignored.	152	if you wish to remove expired or session-only cookies, and also
		153	for documentation on the format of the cookie jar.
126		154
127	Note that this cookie implementation is not of very high	155	Note that this cookie implementation is not meant to be
128	quality, nor meant to be complete. If you want complete cookie	156	complete. If you want complete cookie management you have to do
129	management you have to do that on your own. "cookie_jar" is	157	that on your own. "cookie_jar" is meant as a quick fix to get
130	meant as a quick fix to get some cookie-using sites working.	158	most cookie-using sites working. Cookies are a privacy disaster,
131	Cookies are a privacy disaster, do not use them unless required	159	do not use them unless required to.
132	to.	160
		161	When cookie processing is enabled, the "Cookie:" and
		162	"Set-Cookie:" headers will be set and handled by this module,
		163	otherwise they will be left untouched.
133		164
134	tls_ctx => $scheme \| $tls_ctx	165	tls_ctx => $scheme \| $tls_ctx
135	Specifies the AnyEvent::TLS context to be used for https	166	Specifies the AnyEvent::TLS context to be used for https
136	connections. This parameter follows the same rules as the	167	connections. This parameter follows the same rules as the
137	"tls_ctx" parameter to AnyEvent::Handle, but additionally, the	168	"tls_ctx" parameter to AnyEvent::Handle, but additionally, the
…		…
139	predefined low-security (no verification, highest compatibility)	170	predefined low-security (no verification, highest compatibility)
140	and high-security (CA and common-name verification) TLS context.	171	and high-security (CA and common-name verification) TLS context.
141		172
142	The default for this option is "low", which could be interpreted	173	The default for this option is "low", which could be interpreted
143	as "give me the page, no matter what".	174	as "give me the page, no matter what".
		175
		176	See also the "sessionid" parameter.
		177
		178	session => $string
		179	The module might reuse connections to the same host internally.
		180	Sometimes (e.g. when using TLS), you do not want to reuse
		181	connections from other sessions. This can be achieved by setting
		182	this parameter to some unique ID (such as the address of an
		183	object storing your state data, or the TLS context) - only
		184	connections using the same unique ID will be reused.
144		185
145	on_prepare => $callback->($fh)	186	on_prepare => $callback->($fh)
146	In rare cases you need to "tune" the socket before it is used to	187	In rare cases you need to "tune" the socket before it is used to
147	connect (for exmaple, to bind it on a given IP address). This	188	connect (for exmaple, to bind it on a given IP address). This
148	parameter overrides the prepare callback passed to	189	parameter overrides the prepare callback passed to
149	"AnyEvent::Socket::tcp_connect" and behaves exactly the same way	190	"AnyEvent::Socket::tcp_connect" and behaves exactly the same way
150	(e.g. it has to provide a timeout). See the description for the	191	(e.g. it has to provide a timeout). See the description for the
151	$prepare_cb argument of "AnyEvent::Socket::tcp_connect" for	192	$prepare_cb argument of "AnyEvent::Socket::tcp_connect" for
152	details.	193	details.
153		194
		195	tcp_connect => $callback->($host, $service, $connect_cb,
		196	$prepare_cb)
		197	In even rarer cases you want total control over how
		198	AnyEvent::HTTP establishes connections. Normally it uses
		199	AnyEvent::Socket::tcp_connect to do this, but you can provide
		200	your own "tcp_connect" function - obviously, it has to follow
		201	the same calling conventions, except that it may always return a
		202	connection guard object.
		203
		204	There are probably lots of weird uses for this function,
		205	starting from tracing the hosts "http_request" actually tries to
		206	connect, to (inexact but fast) host => IP address caching or
		207	even socks protocol support.
		208
154	on_header => $callback->($headers)	209	on_header => $callback->($headers)
155	When specified, this callback will be called with the header	210	When specified, this callback will be called with the header
156	hash as soon as headers have been successfully received from the	211	hash as soon as headers have been successfully received from the
157	remote server (not on locally-generated errors).	212	remote server (not on locally-generated errors).
158		213
…		…
163		218
164	This callback is useful, among other things, to quickly reject	219	This callback is useful, among other things, to quickly reject
165	unwanted content, which, if it is supposed to be rare, can be	220	unwanted content, which, if it is supposed to be rare, can be
166	faster than first doing a "HEAD" request.	221	faster than first doing a "HEAD" request.
167		222
		223	The downside is that cancelling the request makes it impossible
		224	to re-use the connection. Also, the "on_header" callback will
		225	not receive any trailer (headers sent after the response body).
		226
168	Example: cancel the request unless the content-type is	227	Example: cancel the request unless the content-type is
169	"text/html".	228	"text/html".
170		229
171	on_header => sub {	230	on_header => sub {
172	$_[0]{"content-type"} =~ /^text\/html\s*(?:;\|$)/	231	$_[0]{"content-type"} =~ /^text\/html\s*(?:;\|$)/
…		…
179		238
180	It has to return either true (in which case AnyEvent::HTTP will	239	It has to return either true (in which case AnyEvent::HTTP will
181	continue), or false, in which case AnyEvent::HTTP will cancel	240	continue), or false, in which case AnyEvent::HTTP will cancel
182	the download (and call the completion callback with an error	241	the download (and call the completion callback with an error
183	code of 598).	242	code of 598).
		243
		244	The downside to cancelling the request is that it makes it
		245	impossible to re-use the connection.
184		246
185	This callback is useful when the data is too large to be held in	247	This callback is useful when the data is too large to be held in
186	memory (so the callback writes it to a file) or when only some	248	memory (so the callback writes it to a file) or when only some
187	information should be extracted, or when the body should be	249	information should be extracted, or when the body should be
188	processed incrementally.	250	processed incrementally.
…		…
202	AnyEvent::Handle object associated with the connection. In error	264	AnyEvent::Handle object associated with the connection. In error
203	cases, "undef" will be passed. When there is no body (e.g.	265	cases, "undef" will be passed. When there is no body (e.g.
204	status 304), the empty string will be passed.	266	status 304), the empty string will be passed.
205		267
206	The handle object might or might not be in TLS mode, might be	268	The handle object might or might not be in TLS mode, might be
207	connected to a proxy, be a persistent connection etc., and	269	connected to a proxy, be a persistent connection, use chunked
208	configured in unspecified ways. The user is responsible for this	270	transfer encoding etc., and configured in unspecified ways. The
209	handle (it will not be used by this module anymore).	271	user is responsible for this handle (it will not be used by this
		272	module anymore).
210		273
211	This is useful with some push-type services, where, after the	274	This is useful with some push-type services, where, after the
212	initial headers, an interactive protocol is used (typical	275	initial headers, an interactive protocol is used (typical
213	example would be the push-style twitter API which starts a	276	example would be the push-style twitter API which starts a
214	JSON/XML stream).	277	JSON/XML stream).
215		278
216	If you think you need this, first have a look at "on_body", to	279	If you think you need this, first have a look at "on_body", to
217	see if that doesn't solve your problem in a better way.	280	see if that doesn't solve your problem in a better way.
218		281
		282	persistent => $boolean
		283	Try to create/reuse a persistent connection. When this flag is
		284	set (default: true for idempotent requests, false for all
		285	others), then "http_request" tries to re-use an existing
		286	(previously-created) persistent connection to the host and,
		287	failing that, tries to create a new one.
		288
		289	Requests failing in certain ways will be automatically retried
		290	once, which is dangerous for non-idempotent requests, which is
		291	why it defaults to off for them. The reason for this is because
		292	the bozos who designed HTTP/1.1 made it impossible to
		293	distinguish between a fatal error and a normal connection
		294	timeout, so you never know whether there was a problem with your
		295	request or not.
		296
		297	When reusing an existent connection, many parameters (such as
		298	TLS context) will be ignored. See the "session" parameter for a
		299	workaround.
		300
		301	keepalive => $boolean
		302	Only used when "persistent" is also true. This parameter decides
		303	whether "http_request" tries to handshake a HTTP/1.0-style
		304	keep-alive connection (as opposed to only a HTTP/1.1 persistent
		305	connection).
		306
		307	The default is true, except when using a proxy, in which case it
		308	defaults to false, as HTTP/1.0 proxies cannot support this in a
		309	meaningful way.
		310
		311	handle_params => { key => value ... }
		312	The key-value pairs in this hash will be passed to any
		313	AnyEvent::Handle constructor that is called - not all requests
		314	will create a handle, and sometimes more than one is created, so
		315	this parameter is only good for setting hints.
		316
		317	Example: set the maximum read size to 4096, to potentially
		318	conserve memory at the cost of speed.
		319
		320	handle_params => {
		321	max_read_size => 4096,
		322	},
		323
219	Example: make a simple HTTP GET request for http://www.nethype.de/	324	Example: do a simple HTTP GET request for http://www.nethype.de/ and
		325	print the response body.
220		326
221	http_request GET => "http://www.nethype.de/", sub {	327	http_request GET => "http://www.nethype.de/", sub {
222	my ($body, $hdr) = @_;	328	my ($body, $hdr) = @_;
223	print "$body\n";	329	print "$body\n";
224	};	330	};
225		331
226	Example: make a HTTP HEAD request on https://www.google.com/, use a	332	Example: do a HTTP HEAD request on https://www.google.com/, use a
227	timeout of 30 seconds.	333	timeout of 30 seconds.
228		334
229	http_request	335	http_request
230	GET => "https://www.google.com",	336	GET => "https://www.google.com",
		337	headers => { "user-agent" => "MySearchClient 1.0" },
231	timeout => 30,	338	timeout => 30,
232	sub {	339	sub {
233	my ($body, $hdr) = @_;	340	my ($body, $hdr) = @_;
234	use Data::Dumper;	341	use Data::Dumper;
235	print Dumper $hdr;	342	print Dumper $hdr;
236	}	343	}
237	;	344	;
238		345
239	Example: make another simple HTTP GET request, but immediately try	346	Example: do another simple HTTP GET request, but immediately try to
240	to cancel it.	347	cancel it.
241		348
242	my $request = http_request GET => "http://www.nethype.de/", sub {	349	my $request = http_request GET => "http://www.nethype.de/", sub {
243	my ($body, $hdr) = @_;	350	my ($body, $hdr) = @_;
244	print "$body\n";	351	print "$body\n";
245	};	352	};
246		353
247	undef $request;	354	undef $request;
248		355
		356	DNS CACHING
		357	AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for the
		358	actual connection, which in turn uses AnyEvent::DNS to resolve
		359	hostnames. The latter is a simple stub resolver and does no caching on
		360	its own. If you want DNS caching, you currently have to provide your own
		361	default resolver (by storing a suitable resolver object in
		362	$AnyEvent::DNS::RESOLVER) or your own "tcp_connect" callback.
		363
249	GLOBAL FUNCTIONS AND VARIABLES	364	GLOBAL FUNCTIONS AND VARIABLES
250	AnyEvent::HTTP::set_proxy "proxy-url"	365	AnyEvent::HTTP::set_proxy "proxy-url"
251	Sets the default proxy server to use. The proxy-url must begin with	366	Sets the default proxy server to use. The proxy-url must begin with
252	a string of the form "http://host:port" (optionally "https:...").	367	a string of the form "http://host:port", croaks otherwise.
		368
		369	To clear an already-set proxy, use "undef".
		370
		371	AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]
		372	Remove all cookies from the cookie jar that have been expired. If
		373	$session_end is given and true, then additionally remove all session
		374	cookies.
		375
		376	You should call this function (with a true $session_end) before you
		377	save cookies to disk, and you should call this function after
		378	loading them again. If you have a long-running program you can
		379	additonally call this function from time to time.
		380
		381	A cookie jar is initially an empty hash-reference that is managed by
		382	this module. It's format is subject to change, but currently it is
		383	like this:
		384
		385	The key "version" has to contain 1, otherwise the hash gets emptied.
		386	All other keys are hostnames or IP addresses pointing to
		387	hash-references. The key for these inner hash references is the
		388	server path for which this cookie is meant, and the values are again
		389	hash-references. The keys of those hash-references is the cookie
		390	name, and the value, you guessed it, is another hash-reference, this
		391	time with the key-value pairs from the cookie, except for "expires"
		392	and "max-age", which have been replaced by a "_expires" key that
		393	contains the cookie expiry timestamp.
		394
		395	Here is an example of a cookie jar with a single cookie, so you have
		396	a chance of understanding the above paragraph:
		397
		398	{
		399	version => 1,
		400	"10.0.0.1" => {
		401	"/" => {
		402	"mythweb_id" => {
		403	_expires => 1293917923,
		404	value => "ooRung9dThee3ooyXooM1Ohm",
		405	},
		406	},
		407	},
		408	}
		409
		410	$date = AnyEvent::HTTP::format_date $timestamp
		411	Takes a POSIX timestamp (seconds since the epoch) and formats it as
		412	a HTTP Date (RFC 2616).
		413
		414	$timestamp = AnyEvent::HTTP::parse_date $date
		415	Takes a HTTP Date (RFC 2616) or a Cookie date (netscape cookie spec)
		416	or a bunch of minor variations of those, and returns the
		417	corresponding POSIX timestamp, or "undef" if the date cannot be
		418	parsed.
253		419
254	$AnyEvent::HTTP::MAX_RECURSE	420	$AnyEvent::HTTP::MAX_RECURSE
255	The default value for the "recurse" request parameter (default: 10).	421	The default value for the "recurse" request parameter (default: 10).
		422
		423	$AnyEvent::HTTP::TIMEOUT
		424	The default timeout for conenction operations (default: 300).
256		425
257	$AnyEvent::HTTP::USERAGENT	426	$AnyEvent::HTTP::USERAGENT
258	The default value for the "User-Agent" header (the default is	427	The default value for the "User-Agent" header (the default is
259	"Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;	428	"Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION;
260	+http://software.schmorp.de/pkg/AnyEvent)").	429	+http://software.schmorp.de/pkg/AnyEvent)").
261		430
262	$AnyEvent::HTTP::MAX_PER_HOST	431	$AnyEvent::HTTP::MAX_PER_HOST
263	The maximum number of concurrent connections to the same host	432	The maximum number of concurrent connections to the same host
264	(identified by the hostname). If the limit is exceeded, then the	433	(identified by the hostname). If the limit is exceeded, then the
265	additional requests are queued until previous connections are	434	additional requests are queued until previous connections are
266	closed.	435	closed. Both persistent and non-persistent connections are counted
		436	in this limit.
267		437
268	The default value for this is 4, and it is highly advisable to not	438	The default value for this is 4, and it is highly advisable to not
269	increase it.	439	increase it much.
		440
		441	For comparison: the RFC's recommend 4 non-persistent or 2 persistent
		442	connections, older browsers used 2, newers (such as firefox 3)
		443	typically use 6, and Opera uses 8 because like, they have the
		444	fastest browser and give a shit for everybody else on the planet.
		445
		446	$AnyEvent::HTTP::PERSISTENT_TIMEOUT
		447	The time after which idle persistent conenctions get closed by
		448	AnyEvent::HTTP (default: 3).
270		449
271	$AnyEvent::HTTP::ACTIVE	450	$AnyEvent::HTTP::ACTIVE
272	The number of active connections. This is not the number of	451	The number of active connections. This is not the number of
273	currently running requests, but the number of currently open and	452	currently running requests, but the number of currently open and
274	non-idle TCP connections. This number of can be useful for	453	non-idle TCP connections. This number can be useful for
275	load-leveling.	454	load-leveling.
		455
		456	SOCKS PROXIES
		457	Socks proxies are not directly supported by AnyEvent::HTTP. You can
		458	compile your perl to support socks, or use an external program such as
		459	socksify (dante) or tsocks to make your program use a socks proxy
		460	transparently.
		461
		462	Alternatively, for AnyEvent::HTTP only, you can use your own
		463	"tcp_connect" function that does the proxy handshake - here is an
		464	example that works with socks4a proxies:
		465
		466	use Errno;
		467	use AnyEvent::Util;
		468	use AnyEvent::Socket;
		469	use AnyEvent::Handle;
		470
		471	# host, port and username of/for your socks4a proxy
		472	my $socks_host = "10.0.0.23";
		473	my $socks_port = 9050;
		474	my $socks_user = "";
		475
		476	sub socks4a_connect {
		477	my ($host, $port, $connect_cb, $prepare_cb) = @_;
		478
		479	my $hdl = new AnyEvent::Handle
		480	connect => [$socks_host, $socks_port],
		481	on_prepare => sub { $prepare_cb->($_[0]{fh}) },
		482	on_error => sub { $connect_cb->() },
		483	;
		484
		485	$hdl->push_write (pack "CCnNZZ", 4, 1, $port, 1, $socks_user, $host);
		486
		487	$hdl->push_read (chunk => 8, sub {
		488	my ($hdl, $chunk) = @_;
		489	my ($status, $port, $ipn) = unpack "xCna4", $chunk;
		490
		491	if ($status == 0x5a) {
		492	$connect_cb->($hdl->{fh}, (format_address $ipn) . ":$port");
		493	} else {
		494	$! = Errno::ENXIO; $connect_cb->();
		495	}
		496	});
		497
		498	$hdl
		499	}
		500
		501	Use "socks4a_connect" instead of "tcp_connect" when doing
		502	"http_request"s, possibly after switching off other proxy types:
		503
		504	AnyEvent::HTTP::set_proxy undef; # usually you do not want other proxies
		505
		506	http_get 'http://www.google.com', tcp_connect => \&socks4a_connect, sub {
		507	my ($data, $headers) = @_;
		508	...
		509	};
276		510
277	SEE ALSO	511	SEE ALSO
278	AnyEvent.	512	AnyEvent.
279		513
280	AUTHOR	514	AUTHOR

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-HTTP/README (file contents): Revision 1.11 by root, Fri Aug 14 15:21:33 2009 UTC vs. Revision 1.15 by root, Tue Jan 4 08:29:28 2011 UTC

Diff Legend

Comparing AnyEvent-HTTP/README (file contents):
Revision 1.11 by root, Fri Aug 14 15:21:33 2009 UTC vs.
Revision 1.15 by root, Tue Jan 4 08:29:28 2011 UTC