[ViewVC] Diff of: cvs/AnyEvent-HTTP/HTTP.pm

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.90 by root, Mon Jan 3 00:41:25 2011 UTC vs.
Revision 1.111 by root, Sun Apr 22 12:58:38 2012 UTC

…		…
15	This module is an L<AnyEvent> user, you need to make sure that you use and	15	This module is an L<AnyEvent> user, you need to make sure that you use and
16	run a supported event loop.	16	run a supported event loop.
17		17
18	This module implements a simple, stateless and non-blocking HTTP	18	This module implements a simple, stateless and non-blocking HTTP
19	client. It supports GET, POST and other request methods, cookies and more,	19	client. It supports GET, POST and other request methods, cookies and more,
20	all on a very low level. It can follow redirects supports proxies and	20	all on a very low level. It can follow redirects, supports proxies, and
21	automatically limits the number of connections to the values specified in	21	automatically limits the number of connections to the values specified in
22	the RFC.	22	the RFC.
23		23
24	It should generally be a "good client" that is enough for most HTTP	24	It should generally be a "good client" that is enough for most HTTP
25	tasks. Simple tasks should be simple, but complex tasks should still be	25	tasks. Simple tasks should be simple, but complex tasks should still be
…		…
46	use AnyEvent::Util ();	46	use AnyEvent::Util ();
47	use AnyEvent::Handle ();	47	use AnyEvent::Handle ();
48		48
49	use base Exporter::;	49	use base Exporter::;
50		50
51	our $VERSION = '1.5';	51	our $VERSION = '2.14';
52		52
53	our @EXPORT = qw(http_get http_post http_head http_request);	53	our @EXPORT = qw(http_get http_post http_head http_request);
54		54
55	our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";	55	our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
56	our $MAX_RECURSE = 10;	56	our $MAX_RECURSE = 10;
57	our $MAX_PERSISTENT = 8;
58	our $PERSISTENT_TIMEOUT = 2;	57	our $PERSISTENT_TIMEOUT = 3;
59	our $TIMEOUT = 300;	58	our $TIMEOUT = 300;
60		59	our $MAX_PER_HOST = 4; # changing this is evil
61	# changing these is evil
62	our $MAX_PERSISTENT_PER_HOST = 2;
63	our $MAX_PER_HOST = 4;
64		60
65	our $PROXY;	61	our $PROXY;
66	our $ACTIVE = 0;	62	our $ACTIVE = 0;
67		63
68	my %KA_COUNT; # number of open keep-alive connections per host	64	my %KA_CACHE; # indexed by uhost currently, points to [$handle...] array
69	my %CO_SLOT; # number of open connections, and wait queue, per host	65	my %CO_SLOT; # number of open connections, and wait queue, per host
70		66
71	=item http_get $url, key => value..., $cb->($data, $headers)	67	=item http_get $url, key => value..., $cb->($data, $headers)
72		68
73	Executes an HTTP-GET request. See the http_request function for details on	69	Executes an HTTP-GET request. See the http_request function for details on
…		…
173		169
174	You really should provide your own C<User-Agent:> header value that is	170	You really should provide your own C<User-Agent:> header value that is
175	appropriate for your program - I wouldn't be surprised if the default	171	appropriate for your program - I wouldn't be surprised if the default
176	AnyEvent string gets blocked by webservers sooner or later.	172	AnyEvent string gets blocked by webservers sooner or later.
177		173
		174	Also, make sure that your headers names and values do not contain any
		175	embedded newlines.
		176
178	=item timeout => $seconds	177	=item timeout => $seconds
179		178
180	The time-out to use for various stages - each connect attempt will reset	179	The time-out to use for various stages - each connect attempt will reset
181	the timeout, as will read or write activity, i.e. this is not an overall	180	the timeout, as will read or write activity, i.e. this is not an overall
182	timeout.	181	timeout.
183		182
184	Default timeout is 5 minutes.	183	Default timeout is 5 minutes.
185		184
186	=item proxy => [$host, $port[, $scheme]] or undef	185	=item proxy => [$host, $port[, $scheme]] or undef
187		186
188	Use the given http proxy for all requests. If not specified, then the	187	Use the given http proxy for all requests, or no proxy if C<undef> is
189	default proxy (as specified by C<$ENV{http_proxy}>) is used.	188	used.
190		189
191	C<$scheme> must be either missing, C<http> for HTTP or C<https> for	190	C<$scheme> must be either missing or must be C<http> for HTTP.
192	HTTPS.	191
		192	If not specified, then the default proxy is used (see
		193	C<AnyEvent::HTTP::set_proxy>).
193		194
194	=item body => $string	195	=item body => $string
195		196
196	The request body, usually empty. Will be sent as-is (future versions of	197	The request body, usually empty. Will be sent as-is (future versions of
197	this module might offer more options).	198	this module might offer more options).
…		…
227	verification, highest compatibility) and high-security (CA and common-name	228	verification, highest compatibility) and high-security (CA and common-name
228	verification) TLS context.	229	verification) TLS context.
229		230
230	The default for this option is C<low>, which could be interpreted as "give	231	The default for this option is C<low>, which could be interpreted as "give
231	me the page, no matter what".	232	me the page, no matter what".
		233
		234	See also the C<sessionid> parameter.
		235
		236	=item session => $string
		237
		238	The module might reuse connections to the same host internally. Sometimes
		239	(e.g. when using TLS), you do not want to reuse connections from other
		240	sessions. This can be achieved by setting this parameter to some unique
		241	ID (such as the address of an object storing your state data, or the TLS
		242	context) - only connections using the same unique ID will be reused.
232		243
233	=item on_prepare => $callback->($fh)	244	=item on_prepare => $callback->($fh)
234		245
235	In rare cases you need to "tune" the socket before it is used to	246	In rare cases you need to "tune" the socket before it is used to
236	connect (for exmaple, to bind it on a given IP address). This parameter	247	connect (for exmaple, to bind it on a given IP address). This parameter
…		…
306	called. Instead of the C<$body> argument containing the body data, the	317	called. Instead of the C<$body> argument containing the body data, the
307	callback will receive the L<AnyEvent::Handle> object associated with the	318	callback will receive the L<AnyEvent::Handle> object associated with the
308	connection. In error cases, C<undef> will be passed. When there is no body	319	connection. In error cases, C<undef> will be passed. When there is no body
309	(e.g. status C<304>), the empty string will be passed.	320	(e.g. status C<304>), the empty string will be passed.
310		321
311	The handle object might or might not be in TLS mode, might be connected to	322	The handle object might or might not be in TLS mode, might be connected
312	a proxy, be a persistent connection etc., and configured in unspecified	323	to a proxy, be a persistent connection, use chunked transfer encoding
313	ways. The user is responsible for this handle (it will not be used by this	324	etc., and configured in unspecified ways. The user is responsible for this
314	module anymore).	325	handle (it will not be used by this module anymore).
315		326
316	This is useful with some push-type services, where, after the initial	327	This is useful with some push-type services, where, after the initial
317	headers, an interactive protocol is used (typical example would be the	328	headers, an interactive protocol is used (typical example would be the
318	push-style twitter API which starts a JSON/XML stream).	329	push-style twitter API which starts a JSON/XML stream).
319		330
320	If you think you need this, first have a look at C<on_body>, to see if	331	If you think you need this, first have a look at C<on_body>, to see if
321	that doesn't solve your problem in a better way.	332	that doesn't solve your problem in a better way.
		333
		334	=item persistent => $boolean
		335
		336	Try to create/reuse a persistent connection. When this flag is set
		337	(default: true for idempotent requests, false for all others), then
		338	C<http_request> tries to re-use an existing (previously-created)
		339	persistent connection to the host and, failing that, tries to create a new
		340	one.
		341
		342	Requests failing in certain ways will be automatically retried once, which
		343	is dangerous for non-idempotent requests, which is why it defaults to off
		344	for them. The reason for this is because the bozos who designed HTTP/1.1
		345	made it impossible to distinguish between a fatal error and a normal
		346	connection timeout, so you never know whether there was a problem with
		347	your request or not.
		348
		349	When reusing an existent connection, many parameters (such as TLS context)
		350	will be ignored. See the C<session> parameter for a workaround.
		351
		352	=item keepalive => $boolean
		353
		354	Only used when C<persistent> is also true. This parameter decides whether
		355	C<http_request> tries to handshake a HTTP/1.0-style keep-alive connection
		356	(as opposed to only a HTTP/1.1 persistent connection).
		357
		358	The default is true, except when using a proxy, in which case it defaults
		359	to false, as HTTP/1.0 proxies cannot support this in a meaningful way.
		360
		361	=item handle_params => { key => value ... }
		362
		363	The key-value pairs in this hash will be passed to any L<AnyEvent::Handle>
		364	constructor that is called - not all requests will create a handle, and
		365	sometimes more than one is created, so this parameter is only good for
		366	setting hints.
		367
		368	Example: set the maximum read size to 4096, to potentially conserve memory
		369	at the cost of speed.
		370
		371	handle_params => {
		372	max_read_size => 4096,
		373	},
322		374
323	=back	375	=back
324		376
325	Example: do a simple HTTP GET request for http://www.nethype.de/ and print	377	Example: do a simple HTTP GET request for http://www.nethype.de/ and print
326	the response body.	378	the response body.
…		…
332		384
333	Example: do a HTTP HEAD request on https://www.google.com/, use a	385	Example: do a HTTP HEAD request on https://www.google.com/, use a
334	timeout of 30 seconds.	386	timeout of 30 seconds.
335		387
336	http_request	388	http_request
337	GET => "https://www.google.com",	389	HEAD => "https://www.google.com",
338	headers => { "user-agent" => "MySearchClient 1.0" },	390	headers => { "user-agent" => "MySearchClient 1.0" },
339	timeout => 30,	391	timeout => 30,
340	sub {	392	sub {
341	my ($body, $hdr) = @_;	393	my ($body, $hdr) = @_;
342	use Data::Dumper;	394	use Data::Dumper;
…		…
353	};	405	};
354		406
355	undef $request;	407	undef $request;
356		408
357	=cut	409	=cut
		410
		411	#############################################################################
		412	# wait queue/slots
358		413
359	sub _slot_schedule;	414	sub _slot_schedule;
360	sub _slot_schedule($) {	415	sub _slot_schedule($) {
361	my $host = shift;	416	my $host = shift;
362		417
…		…
385		440
386	_slot_schedule $_[0];	441	_slot_schedule $_[0];
387	}	442	}
388		443
389	#############################################################################	444	#############################################################################
		445	# cookie handling
390		446
391	# expire cookies	447	# expire cookies
392	sub cookie_jar_expire($;$) {	448	sub cookie_jar_expire($;$) {
393	my ($jar, $session_end) = @_;	449	my ($jar, $session_end) = @_;
394		450
…		…
418	}	474	}
419	}	475	}
420		476
421	# extract cookies from jar	477	# extract cookies from jar
422	sub cookie_jar_extract($$$$) {	478	sub cookie_jar_extract($$$$) {
423	my ($jar, $uscheme, $uhost, $upath) = @_;	479	my ($jar, $scheme, $host, $path) = @_;
424		480
425	%$jar = () if $jar->{version} != 1;	481	%$jar = () if $jar->{version} != 1;
426		482
427	my @cookies;	483	my @cookies;
428		484
429	while (my ($chost, $paths) = each %$jar) {	485	while (my ($chost, $paths) = each %$jar) {
430	next unless ref $paths;	486	next unless ref $paths;
431		487
432	if ($chost =~ /^\./) {	488	if ($chost =~ /^\./) {
433	next unless $chost eq substr $uhost, -length $chost;	489	next unless $chost eq substr $host, -length $chost;
434	} elsif ($chost =~ /\./) {	490	} elsif ($chost =~ /\./) {
435	next unless $chost eq $uhost;	491	next unless $chost eq $host;
436	} else {	492	} else {
437	next;	493	next;
438	}	494	}
439		495
440	while (my ($cpath, $cookies) = each %$paths) {	496	while (my ($cpath, $cookies) = each %$paths) {
441	next unless $cpath eq substr $upath, 0, length $cpath;	497	next unless $cpath eq substr $path, 0, length $cpath;
442		498
443	while (my ($cookie, $kv) = each %$cookies) {	499	while (my ($cookie, $kv) = each %$cookies) {
444	next if $uscheme ne "https" && exists $kv->{secure};	500	next if $scheme ne "https" && exists $kv->{secure};
445		501
446	if (exists $kv->{_expires} and AE::now > $kv->{_expires}) {	502	if (exists $kv->{_expires} and AE::now > $kv->{_expires}) {
447	delete $cookies->{$cookie};	503	delete $cookies->{$cookie};
448	next;	504	next;
449	}	505	}
…		…
463	\@cookies	519	\@cookies
464	}	520	}
465		521
466	# parse set_cookie header into jar	522	# parse set_cookie header into jar
467	sub cookie_jar_set_cookie($$$$) {	523	sub cookie_jar_set_cookie($$$$) {
468	my ($jar, $set_cookie, $uhost, $date) = @_;	524	my ($jar, $set_cookie, $host, $date) = @_;
469		525
470	my $anow = int AE::now;	526	my $anow = int AE::now;
471	my $snow; # server-now	527	my $snow; # server-now
472		528
473	for ($set_cookie) {	529	for ($set_cookie) {
…		…
479	while (	535	while (
480	m{	536	m{
481	\G\s*	537	\G\s*
482	(?:	538	(?:
483	expires \s=\s ([A-Z][a-z][a-z]+,\ [^,;]+)	539	expires \s=\s ([A-Z][a-z][a-z]+,\ [^,;]+)
484	\| ([^=;,[:space:]]+) (?: \s=\s (?: "((?:[^\\"]+\|\\.))" \| ([^=;,[:space:]]) ) )?	540	\| ([^=;,[:space:]]+) (?: \s=\s (?: "((?:[^\\"]+\|\\.))" \| ([^;,[:space:]]) ) )?
485	)	541	)
486	}gcxsi	542	}gcxsi
487	) {	543	) {
488	my $name = $2;	544	my $name = $2;
489	my $value = $4;	545	my $value = $4;
…		…
496	# quoted	552	# quoted
497	$value = $3;	553	$value = $3;
498	$value =~ s/\\(.)/$1/gs;	554	$value =~ s/\\(.)/$1/gs;
499	}	555	}
500		556
501	push @kv, lc $name, $value;	557	push @kv, @kv ? lc $name : $name, $value;
502		558
503	last unless /\G\s*;/gc;	559	last unless /\G\s*;/gc;
504	}	560	}
505		561
506	last unless @kv;	562	last unless @kv;
…		…
529		585
530	# this is not rfc-like and not netscape-like. go figure.	586	# this is not rfc-like and not netscape-like. go figure.
531	my $ndots = $cdom =~ y/.//;	587	my $ndots = $cdom =~ y/.//;
532	next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);	588	next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
533	} else {	589	} else {
534	$cdom = $uhost;	590	$cdom = $host;
535	}	591	}
536		592
537	# store it	593	# store it
538	$jar->{version} = 1;	594	$jar->{version} = 1;
539	$jar->{lc $cdom}{$cpath}{$name} = \%kv;	595	$jar->{lc $cdom}{$cpath}{$name} = \%kv;
540		596
541	redo if /\G\s*,/gc;	597	redo if /\G\s*,/gc;
542	}	598	}
543	}	599	}
544		600
		601	#############################################################################
		602	# keepalive/persistent connection cache
		603
		604	# fetch a connection from the keepalive cache
		605	sub ka_fetch($) {
		606	my $ka_key = shift;
		607
		608	my $hdl = pop @{ $KA_CACHE{$ka_key} }; # currently we reuse the MOST RECENTLY USED connection
		609	delete $KA_CACHE{$ka_key}
		610	unless @{ $KA_CACHE{$ka_key} };
		611
		612	$hdl
		613	}
		614
		615	sub ka_store($$) {
		616	my ($ka_key, $hdl) = @_;
		617
		618	my $kaa = $KA_CACHE{$ka_key} \|\|= [];
		619
		620	my $destroy = sub {
		621	my @ka = grep $_ != $hdl, @{ $KA_CACHE{$ka_key} };
		622
		623	$hdl->destroy;
		624
		625	@ka
		626	? $KA_CACHE{$ka_key} = \@ka
		627	: delete $KA_CACHE{$ka_key};
		628	};
		629
		630	# on error etc., destroy
		631	$hdl->on_error ($destroy);
		632	$hdl->on_eof ($destroy);
		633	$hdl->on_read ($destroy);
		634	$hdl->timeout ($PERSISTENT_TIMEOUT);
		635
		636	push @$kaa, $hdl;
		637	shift @$kaa while @$kaa > $MAX_PER_HOST;
		638	}
		639
		640	#############################################################################
		641	# utilities
		642
545	# continue to parse $_ for headers and place them into the arg	643	# continue to parse $_ for headers and place them into the arg
546	sub parse_hdr() {	644	sub _parse_hdr() {
547	my %hdr;	645	my %hdr;
548		646
549	# things seen, not parsed:	647	# things seen, not parsed:
550	# p3pP="NON CUR OTPi OUR NOR UNI"	648	# p3pP="NON CUR OTPi OUR NOR UNI"
551		649
…		…
565	for values %hdr;	663	for values %hdr;
566		664
567	\%hdr	665	\%hdr
568	}	666	}
569		667
		668	#############################################################################
		669	# http_get
		670
570	our $qr_nlnl = qr{(?<![^\012])\015?\012};	671	our $qr_nlnl = qr{(?<![^\012])\015?\012};
571		672
572	our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };	673	our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
573	our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };	674	our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
		675
		676	# maybe it should just become a normal object :/
		677
		678	sub _destroy_state(\%) {
		679	my ($state) = @_;
		680
		681	$state->{handle}->destroy if $state->{handle};
		682	%$state = ();
		683	}
		684
		685	sub _error(\%$$) {
		686	my ($state, $cb, $hdr) = @_;
		687
		688	&_destroy_state ($state);
		689
		690	$cb->(undef, $hdr);
		691	()
		692	}
574		693
575	sub http_request($$@) {	694	sub http_request($$@) {
576	my $cb = pop;	695	my $cb = pop;
577	my ($method, $url, %arg) = @_;	696	my ($method, $url, %arg) = @_;
578		697
…		…
596	my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;	715	my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;
597		716
598	return $cb->(undef, { @pseudo, Status => 599, Reason => "Too many redirections" })	717	return $cb->(undef, { @pseudo, Status => 599, Reason => "Too many redirections" })
599	if $recurse < 0;	718	if $recurse < 0;
600		719
601	my $proxy = $arg{proxy} \|\| $PROXY;	720	my $proxy = exists $arg{proxy} ? $arg{proxy} : $PROXY;
602	my $timeout = $arg{timeout} \|\| $TIMEOUT;	721	my $timeout = $arg{timeout} \|\| $TIMEOUT;
603		722
604	my ($uscheme, $uauthority, $upath, $query, $fragment) =	723	my ($uscheme, $uauthority, $upath, $query, undef) = # ignore fragment
605	$url =~ m\|(?:([^:/?#]+):)?(?://([^/?#]))?([^?#])(?:(\?[^#]))?(?:#(.))?\|;	724	$url =~ m\|^([^:]+):(?://([^/?#]))?([^?#])(?:(\?[^#]))?(?:#(.))?$\|;
606		725
607	$uscheme = lc $uscheme;	726	$uscheme = lc $uscheme;
608		727
609	my $uport = $uscheme eq "http" ? 80	728	my $uport = $uscheme eq "http" ? 80
610	: $uscheme eq "https" ? 443	729	: $uscheme eq "https" ? 443
…		…
657	if length $arg{body} \|\| $method ne "GET";	776	if length $arg{body} \|\| $method ne "GET";
658		777
659	my $idempotent = $method =~ /^(?:GET\|HEAD\|PUT\|DELETE\|OPTIONS\|TRACE)$/;	778	my $idempotent = $method =~ /^(?:GET\|HEAD\|PUT\|DELETE\|OPTIONS\|TRACE)$/;
660		779
661	# default value for keepalive is true iff the request is for an idempotent method	780	# default value for keepalive is true iff the request is for an idempotent method
662	my $keepalive = exists $arg{keepalive}	781	my $persistent = exists $arg{persistent} ? !!$arg{persistent} : $idempotent;
663	? $arg{keepalive}*1	782	my $keepalive = exists $arg{keepalive} ? !!$arg{keepalive} : !$proxy;
664	: $idempotent ? $PERSISTENT_TIMEOUT : 0;	783	my $was_persistent; # true if this is actually a recycled connection
665		784
		785	# the key to use in the keepalive cache
		786	my $ka_key = "$uscheme\x00$uhost\x00$uport\x00$arg{sessionid}";
		787
666	$hdr{connection} = ($keepalive ? "" : "close ") . "Te"; #1.1	788	$hdr{connection} = ($persistent ? $keepalive ? "keep-alive " : "" : "close ") . "Te"; #1.1
667	$hdr{te} = "trailers" unless exists $hdr{te}; #1.1	789	$hdr{te} = "trailers" unless exists $hdr{te}; #1.1
668		790
669	my %state = (connect_guard => 1);	791	my %state = (connect_guard => 1);
670		792
671	my $ae_error = 595; # connecting	793	my $ae_error = 595; # connecting
672		794
673	# handle actual, non-tunneled, request	795	# handle actual, non-tunneled, request
674	my $handle_actual_request = sub {	796	my $handle_actual_request = sub {
675	$ae_error = 596; # request phase	797	$ae_error = 596; # request phase
676		798
		799	my $hdl = $state{handle};
		800
677	$state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};	801	$hdl->starttls ("connect") if $uscheme eq "https" && !exists $hdl->{tls};
678		802
679	# send request	803	# send request
680	$state{handle}->push_write (	804	$hdl->push_write (
681	"$method $rpath HTTP/1.1\015\012"	805	"$method $rpath HTTP/1.1\015\012"
682	. (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)	806	. (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
683	. "\015\012"	807	. "\015\012"
684	. (delete $arg{body})	808	. (delete $arg{body})
685	);	809	);
…		…
690	# reduce memory usage, save a kitten, also re-use it for the response headers.	814	# reduce memory usage, save a kitten, also re-use it for the response headers.
691	%hdr = ();	815	%hdr = ();
692		816
693	# status line and headers	817	# status line and headers
694	$state{read_response} = sub {	818	$state{read_response} = sub {
		819	return unless %state;
		820
695	for ("$_[1]") {	821	for ("$_[1]") {
696	y/\015//d; # weed out any \015, as they show up in the weirdest of places.	822	y/\015//d; # weed out any \015, as they show up in the weirdest of places.
697		823
698	/^HTTP\/0([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\012]) )? \012/gxci	824	/^HTTP\/0([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\012]) )? \012/gxci
699	or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid server response" }));	825	or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Invalid server response" };
700		826
701	# 100 Continue handling	827	# 100 Continue handling
702	# should not happen as we don't send expect: 100-continue,	828	# should not happen as we don't send expect: 100-continue,
703	# but we handle it just in case.	829	# but we handle it just in case.
704	# since we send the request body regardless, if we get an error	830	# since we send the request body regardless, if we get an error
…		…
710	HTTPVersion => $1,	836	HTTPVersion => $1,
711	Status => $2,	837	Status => $2,
712	Reason => $3,	838	Reason => $3,
713	;	839	;
714		840
715	my $hdr = parse_hdr	841	my $hdr = _parse_hdr
716	or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Garbled response headers" }));	842	or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Garbled response headers" };
717		843
718	%hdr = (%$hdr, @pseudo);	844	%hdr = (%$hdr, @pseudo);
719	}	845	}
720		846
721	# redirect handling	847	# redirect handling
…		…
751	} elsif ($status == 307) {	877	} elsif ($status == 307) {
752	$redirect = 1;	878	$redirect = 1;
753	}	879	}
754	}	880	}
755		881
756	my $finish = sub { # ($data, $err_status, $err_reason[, $keepalive])	882	my $finish = sub { # ($data, $err_status, $err_reason[, $persistent])
757	my $may_keep_alive = $_[3];	883	if ($state{handle}) {
758		884	# handle keepalive
		885	if (
		886	$persistent
		887	&& $_[3]
		888	&& ($hdr{HTTPVersion} < 1.1
		889	? $hdr{connection} =~ /\bkeep-?alive\b/i
		890	: $hdr{connection} !~ /\bclose\b/i)
		891	) {
		892	ka_store $ka_key, delete $state{handle};
		893	} else {
		894	# no keepalive, destroy the handle
759	$state{handle}->destroy if $state{handle};	895	$state{handle}->destroy;
		896	}
		897	}
		898
760	%state = ();	899	%state = ();
761		900
762	if (defined $_[1]) {	901	if (defined $_[1]) {
763	$hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1];	902	$hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1];
764	$hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2];	903	$hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2];
…		…
771		910
772	if ($redirect && exists $hdr{location}) {	911	if ($redirect && exists $hdr{location}) {
773	# we ignore any errors, as it is very common to receive	912	# we ignore any errors, as it is very common to receive
774	# Content-Length != 0 but no actual body	913	# Content-Length != 0 but no actual body
775	# we also access %hdr, as $_[1] might be an erro	914	# we also access %hdr, as $_[1] might be an erro
		915	$state{recurse} =
776	http_request (	916	http_request (
777	$method => $hdr{location},	917	$method => $hdr{location},
778	%arg,	918	%arg,
779	recurse => $recurse - 1,	919	recurse => $recurse - 1,
780	Redirect => [$_[0], \%hdr],	920	Redirect => [$_[0], \%hdr],
		921	sub {
		922	%state = ();
		923	&$cb
		924	},
781	$cb);	925	);
782	} else {	926	} else {
783	$cb->($_[0], \%hdr);	927	$cb->($_[0], \%hdr);
784	}	928	}
785	};	929	};
786		930
787	$ae_error = 597; # body phase	931	$ae_error = 597; # body phase
788		932
		933	my $chunked = $hdr{"transfer-encoding"} =~ /\bchunked\b/i; # not quite correct...
		934
789	my $len = $hdr{"content-length"};	935	my $len = $chunked ? undef : $hdr{"content-length"};
790		936
791	# body handling, many different code paths	937	# body handling, many different code paths
792	# - no body expected	938	# - no body expected
793	# - want_body_handle	939	# - want_body_handle
794	# - te chunked	940	# - te chunked
…		…
809	$_[0]->on_error (undef);	955	$_[0]->on_error (undef);
810	$_[0]->on_read (undef);	956	$_[0]->on_read (undef);
811		957
812	$finish->(delete $state{handle});	958	$finish->(delete $state{handle});
813		959
814	} elsif ($hdr{"transfer-encoding"} =~ /\bchunked\b/i) {	960	} elsif ($chunked) {
815	my $cl = 0;	961	my $cl = 0;
816	my $body = undef;	962	my $body = "";
817	my $on_body = $arg{on_body} \|\| sub { $body .= shift; 1 };	963	my $on_body = $arg{on_body} \|\| sub { $body .= shift; 1 };
818		964
819	$state{read_chunk} = sub {	965	$state{read_chunk} = sub {
820	$_[1] =~ /^([0-9a-fA-F]+)/	966	$_[1] =~ /^([0-9a-fA-F]+)/
821	or $finish->(undef, $ae_error => "Garbled chunked transfer encoding");	967	or return $finish->(undef, $ae_error => "Garbled chunked transfer encoding");
822		968
823	my $len = hex $1;	969	my $len = hex $1;
824		970
825	if ($len) {	971	if ($len) {
826	$cl += $len;	972	$cl += $len;
…		…
841	$_[0]->push_read (line => $qr_nlnl, sub {	987	$_[0]->push_read (line => $qr_nlnl, sub {
842	if (length $_[1]) {	988	if (length $_[1]) {
843	for ("$_[1]") {	989	for ("$_[1]") {
844	y/\015//d; # weed out any \015, as they show up in the weirdest of places.	990	y/\015//d; # weed out any \015, as they show up in the weirdest of places.
845		991
846	my $hdr = parse_hdr	992	my $hdr = _parse_hdr
847	or return $finish->(undef, $ae_error => "Garbled response trailers");	993	or return $finish->(undef, $ae_error => "Garbled response trailers");
848		994
849	%hdr = (%hdr, %$hdr);	995	%hdr = (%hdr, %$hdr);
850	}	996	}
851	}	997	}
…		…
894	$_[0]->on_read (sub { });	1040	$_[0]->on_read (sub { });
895	}	1041	}
896	}	1042	}
897	};	1043	};
898		1044
		1045	# if keepalive is enabled, then the server closing the connection
		1046	# before a response can happen legally - we retry on idempotent methods.
		1047	if ($was_persistent && $idempotent) {
		1048	my $old_eof = $hdl->{on_eof};
		1049	$hdl->{on_eof} = sub {
		1050	_destroy_state %state;
		1051
		1052	%state = ();
		1053	$state{recurse} =
		1054	http_request (
		1055	$method => $url,
		1056	%arg,
		1057	keepalive => 0,
		1058	sub {
		1059	%state = ();
		1060	&$cb
		1061	}
		1062	);
		1063	};
		1064	$hdl->on_read (sub {
		1065	return unless %state;
		1066
		1067	# as soon as we receive something, a connection close
		1068	# once more becomes a hard error
		1069	$hdl->{on_eof} = $old_eof;
		1070	$hdl->push_read (line => $qr_nlnl, $state{read_response});
		1071	});
		1072	} else {
899	$state{handle}->push_read (line => $qr_nlnl, $state{read_response});	1073	$hdl->push_read (line => $qr_nlnl, $state{read_response});
		1074	}
900	};	1075	};
901		1076
		1077	my $prepare_handle = sub {
		1078	my ($hdl) = $state{handle};
		1079
		1080	$hdl->on_error (sub {
		1081	_error %state, $cb, { @pseudo, Status => $ae_error, Reason => $_[2] };
		1082	});
		1083	$hdl->on_eof (sub {
		1084	_error %state, $cb, { @pseudo, Status => $ae_error, Reason => "Unexpected end-of-file" };
		1085	});
		1086	$hdl->timeout_reset;
		1087	$hdl->timeout ($timeout);
		1088	};
		1089
		1090	# connected to proxy (or origin server)
902	my $connect_cb = sub {	1091	my $connect_cb = sub {
903	$state{fh} = shift	1092	my $fh = shift
904	or do {
905	my $err = "$!";
906	%state = ();
907	return $cb->(undef, { @pseudo, Status => $ae_error, Reason => $err });	1093	or return _error %state, $cb, { @pseudo, Status => $ae_error, Reason => "$!" };
908	};
909		1094
910	return unless delete $state{connect_guard};	1095	return unless delete $state{connect_guard};
911		1096
912	# get handle	1097	# get handle
913	$state{handle} = new AnyEvent::Handle	1098	$state{handle} = new AnyEvent::Handle
		1099	%{ $arg{handle_params} },
914	fh => $state{fh},	1100	fh => $fh,
915	peername => $rhost,	1101	peername => $uhost,
916	tls_ctx => $arg{tls_ctx},	1102	tls_ctx => $arg{tls_ctx},
917	# these need to be reconfigured on keepalive handles
918	timeout => $timeout,
919	on_error => sub {
920	%state = ();
921	$cb->(undef, { @pseudo, Status => $ae_error, Reason => $_[2] });
922	},
923	on_eof => sub {
924	%state = ();
925	$cb->(undef, { @pseudo, Status => $ae_error, Reason => "Unexpected end-of-file" });
926	},
927	;	1103	;
928		1104
929	# limit the number of persistent connections	1105	$prepare_handle->();
930	# keepalive not yet supported
931	# if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
932	# ++$KA_COUNT{$_[1]};
933	# $state{handle}{ka_count_guard} = AnyEvent::Util::guard {
934	# --$KA_COUNT{$_[1]}
935	# };
936	# $hdr{connection} = "keep-alive";
937	# }
938		1106
939	$state{handle}->starttls ("connect") if $rscheme eq "https";	1107	#$state{handle}->starttls ("connect") if $rscheme eq "https";
940		1108
941	# now handle proxy-CONNECT method	1109	# now handle proxy-CONNECT method
942	if ($proxy && $uscheme eq "https") {	1110	if ($proxy && $uscheme eq "https") {
943	# oh dear, we have to wrap it into a connect request	1111	# oh dear, we have to wrap it into a connect request
944		1112
945	# maybe re-use $uauthority with patched port?	1113	# maybe re-use $uauthority with patched port?
946	$state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012\015\012");	1114	$state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012\015\012");
947	$state{handle}->push_read (line => $qr_nlnl, sub {	1115	$state{handle}->push_read (line => $qr_nlnl, sub {
948	$_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix	1116	$_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
949	or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid proxy connect response ($_[1])" }));	1117	or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Invalid proxy connect response ($_[1])" };
950		1118
951	if ($2 == 200) {	1119	if ($2 == 200) {
952	$rpath = $upath;	1120	$rpath = $upath;
953	$handle_actual_request->();	1121	$handle_actual_request->();
954	} else {	1122	} else {
955	%state = ();
956	$cb->(undef, { @pseudo, Status => $2, Reason => $3 });	1123	_error %state, $cb, { @pseudo, Status => $2, Reason => $3 };
957	}	1124	}
958	});	1125	});
959	} else {	1126	} else {
960	$handle_actual_request->();	1127	$handle_actual_request->();
961	}	1128	}
…		…
964	_get_slot $uhost, sub {	1131	_get_slot $uhost, sub {
965	$state{slot_guard} = shift;	1132	$state{slot_guard} = shift;
966		1133
967	return unless $state{connect_guard};	1134	return unless $state{connect_guard};
968		1135
		1136	# try to use an existing keepalive connection, but only if we, ourselves, plan
		1137	# on a keepalive request (in theory, this should be a separate config option).
		1138	if ($persistent && $KA_CACHE{$ka_key}) {
		1139	$was_persistent = 1;
		1140
		1141	$state{handle} = ka_fetch $ka_key;
		1142	$state{handle}->destroyed
		1143	and die "AnyEvent::HTTP: unexpectedly got a destructed handle (1), please report.";#d#
		1144	$prepare_handle->();
		1145	$state{handle}->destroyed
		1146	and die "AnyEvent::HTTP: unexpectedly got a destructed handle (2), please report.";#d#
		1147	$handle_actual_request->();
		1148
		1149	} else {
969	my $tcp_connect = $arg{tcp_connect}	1150	my $tcp_connect = $arg{tcp_connect}
970	\|\| do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };	1151	\|\| do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };
971		1152
972	$state{connect_guard} = $tcp_connect->($rhost, $rport, $connect_cb, $arg{on_prepare} \|\| sub { $timeout });	1153	$state{connect_guard} = $tcp_connect->($rhost, $rport, $connect_cb, $arg{on_prepare} \|\| sub { $timeout });
		1154	}
973	};	1155	};
974		1156
975	defined wantarray && AnyEvent::Util::guard { %state = () }	1157	defined wantarray && AnyEvent::Util::guard { _destroy_state %state }
976	}	1158	}
977		1159
978	sub http_get($@) {	1160	sub http_get($@) {
979	unshift @_, "GET";	1161	unshift @_, "GET";
980	&http_request	1162	&http_request
…		…
998	AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for	1180	AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for
999	the actual connection, which in turn uses AnyEvent::DNS to resolve	1181	the actual connection, which in turn uses AnyEvent::DNS to resolve
1000	hostnames. The latter is a simple stub resolver and does no caching	1182	hostnames. The latter is a simple stub resolver and does no caching
1001	on its own. If you want DNS caching, you currently have to provide	1183	on its own. If you want DNS caching, you currently have to provide
1002	your own default resolver (by storing a suitable resolver object in	1184	your own default resolver (by storing a suitable resolver object in
1003	C<$AnyEvent::DNS::RESOLVER>).	1185	C<$AnyEvent::DNS::RESOLVER>) or your own C<tcp_connect> callback.
1004		1186
1005	=head2 GLOBAL FUNCTIONS AND VARIABLES	1187	=head2 GLOBAL FUNCTIONS AND VARIABLES
1006		1188
1007	=over 4	1189	=over 4
1008		1190
1009	=item AnyEvent::HTTP::set_proxy "proxy-url"	1191	=item AnyEvent::HTTP::set_proxy "proxy-url"
1010		1192
1011	Sets the default proxy server to use. The proxy-url must begin with a	1193	Sets the default proxy server to use. The proxy-url must begin with a
1012	string of the form C<http://host:port> (optionally C<https:...>), croaks	1194	string of the form C<http://host:port>, croaks otherwise.
1013	otherwise.
1014		1195
1015	To clear an already-set proxy, use C<undef>.	1196	To clear an already-set proxy, use C<undef>.
		1197
		1198	When AnyEvent::HTTP is laoded for the first time it will query the
		1199	default proxy from the operating system, currently by looking at
		1200	C<$ENV{http_proxy>}.
1016		1201
1017	=item AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]	1202	=item AnyEvent::HTTP::cookie_jar_expire $jar[, $session_end]
1018		1203
1019	Remove all cookies from the cookie jar that have been expired. If	1204	Remove all cookies from the cookie jar that have been expired. If
1020	C<$session_end> is given and true, then additionally remove all session	1205	C<$session_end> is given and true, then additionally remove all session
…		…
1066		1251
1067	=item $AnyEvent::HTTP::MAX_RECURSE	1252	=item $AnyEvent::HTTP::MAX_RECURSE
1068		1253
1069	The default value for the C<recurse> request parameter (default: C<10>).	1254	The default value for the C<recurse> request parameter (default: C<10>).
1070		1255
		1256	=item $AnyEvent::HTTP::TIMEOUT
		1257
		1258	The default timeout for connection operations (default: C<300>).
		1259
1071	=item $AnyEvent::HTTP::USERAGENT	1260	=item $AnyEvent::HTTP::USERAGENT
1072		1261
1073	The default value for the C<User-Agent> header (the default is	1262	The default value for the C<User-Agent> header (the default is
1074	C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).	1263	C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
1075		1264
1076	=item $AnyEvent::HTTP::MAX_PER_HOST	1265	=item $AnyEvent::HTTP::MAX_PER_HOST
1077		1266
1078	The maximum number of concurrent connections to the same host (identified	1267	The maximum number of concurrent connections to the same host (identified
1079	by the hostname). If the limit is exceeded, then the additional requests	1268	by the hostname). If the limit is exceeded, then the additional requests
1080	are queued until previous connections are closed.	1269	are queued until previous connections are closed. Both persistent and
		1270	non-persistent connections are counted in this limit.
1081		1271
1082	The default value for this is C<4>, and it is highly advisable to not	1272	The default value for this is C<4>, and it is highly advisable to not
1083	increase it.	1273	increase it much.
		1274
		1275	For comparison: the RFC's recommend 4 non-persistent or 2 persistent
		1276	connections, older browsers used 2, newers (such as firefox 3) typically
		1277	use 6, and Opera uses 8 because like, they have the fastest browser and
		1278	give a shit for everybody else on the planet.
		1279
		1280	=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
		1281
		1282	The time after which idle persistent conenctions get closed by
		1283	AnyEvent::HTTP (default: C<3>).
1084		1284
1085	=item $AnyEvent::HTTP::ACTIVE	1285	=item $AnyEvent::HTTP::ACTIVE
1086		1286
1087	The number of active connections. This is not the number of currently	1287	The number of active connections. This is not the number of currently
1088	running requests, but the number of currently open and non-idle TCP	1288	running requests, but the number of currently open and non-idle TCP
1089	connections. This number of can be useful for load-leveling.	1289	connections. This number can be useful for load-leveling.
1090		1290
1091	=back	1291	=back
1092		1292
1093	=cut	1293	=cut
1094		1294
…		…
1128	# other formats fail in the loop below	1328	# other formats fail in the loop below
1129		1329
1130	for (0..11) {	1330	for (0..11) {
1131	if ($m eq $month[$_]) {	1331	if ($m eq $month[$_]) {
1132	require Time::Local;	1332	require Time::Local;
1133	return Time::Local::timegm ($S, $M, $H, $d, $_, $y);	1333	return eval { Time::Local::timegm ($S, $M, $H, $d, $_, $y) };
1134	}	1334	}
1135	}	1335	}
1136		1336
1137	undef	1337	undef
1138	}	1338	}
1139		1339
1140	sub set_proxy($) {	1340	sub set_proxy($) {
1141	if (length $_[0]) {	1341	if (length $_[0]) {
1142	$_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix	1342	$_[0] =~ m%^(http):// ([^:/]+) (?: : (\d*) )?%ix
1143	or Carp::croak "$_[0]: invalid proxy URL";	1343	or Carp::croak "$_[0]: invalid proxy URL";
1144	$PROXY = [$2, $3 \|\| 3128, $1]	1344	$PROXY = [$2, $3 \|\| 3128, $1]
1145	} else {	1345	} else {
1146	undef $PROXY;	1346	undef $PROXY;
1147	}	1347	}
…		…
1150	# initialise proxy from environment	1350	# initialise proxy from environment
1151	eval {	1351	eval {
1152	set_proxy $ENV{http_proxy};	1352	set_proxy $ENV{http_proxy};
1153	};	1353	};
1154		1354
		1355	=head2 SHOWCASE
		1356
		1357	This section contaisn some more elaborate "real-world" examples or code
		1358	snippets.
		1359
		1360	=head2 HTTP/1.1 FILE DOWNLOAD
		1361
		1362	Downloading files with HTTP can be quite tricky, especially when something
		1363	goes wrong and you want to resume.
		1364
		1365	Here is a function that initiates and resumes a download. It uses the
		1366	last modified time to check for file content changes, and works with many
		1367	HTTP/1.0 servers as well, and usually falls back to a complete re-download
		1368	on older servers.
		1369
		1370	It calls the completion callback with either C<undef>, which means a
		1371	nonretryable error occured, C<0> when the download was partial and should
		1372	be retried, and C<1> if it was successful.
		1373
		1374	use AnyEvent::HTTP;
		1375
		1376	sub download($$$) {
		1377	my ($url, $file, $cb) = @_;
		1378
		1379	open my $fh, "+<", $file
		1380	or die "$file: $!";
		1381
		1382	my %hdr;
		1383	my $ofs = 0;
		1384
		1385	warn stat $fh;
		1386	warn -s _;
		1387	if (stat $fh and -s _) {
		1388	$ofs = -s _;
		1389	warn "-s is ", $ofs;
		1390	$hdr{"if-unmodified-since"} = AnyEvent::HTTP::format_date +(stat _)[9];
		1391	$hdr{"range"} = "bytes=$ofs-";
		1392	}
		1393
		1394	http_get $url,
		1395	headers => \%hdr,
		1396	on_header => sub {
		1397	my ($hdr) = @_;
		1398
		1399	if ($hdr->{Status} == 200 && $ofs) {
		1400	# resume failed
		1401	truncate $fh, $ofs = 0;
		1402	}
		1403
		1404	sysseek $fh, $ofs, 0;
		1405
		1406	1
		1407	},
		1408	on_body => sub {
		1409	my ($data, $hdr) = @_;
		1410
		1411	if ($hdr->{Status} =~ /^2/) {
		1412	length $data == syswrite $fh, $data
		1413	or return; # abort on write errors
		1414	}
		1415
		1416	1
		1417	},
		1418	sub {
		1419	my (undef, $hdr) = @_;
		1420
		1421	my $status = $hdr->{Status};
		1422
		1423	if (my $time = AnyEvent::HTTP::parse_date $hdr->{"last-modified"}) {
		1424	utime $fh, $time, $time;
		1425	}
		1426
		1427	if ($status == 200 \|\| $status == 206 \|\| $status == 416) {
		1428	# download ok \|\| resume ok \|\| file already fully downloaded
		1429	$cb->(1, $hdr);
		1430
		1431	} elsif ($status == 412) {
		1432	# file has changed while resuming, delete and retry
		1433	unlink $file;
		1434	$cb->(0, $hdr);
		1435
		1436	} elsif ($status == 500 or $status == 503 or $status =~ /^59/) {
		1437	# retry later
		1438	$cb->(0, $hdr);
		1439
		1440	} else {
		1441	$cb->(undef, $hdr);
		1442	}
		1443	}
		1444	;
		1445	}
		1446
		1447	download "http://server/somelargefile", "/tmp/somelargefile", sub {
		1448	if ($_[0]) {
		1449	print "OK!\n";
		1450	} elsif (defined $_[0]) {
		1451	print "please retry later\n";
		1452	} else {
		1453	print "ERROR\n";
		1454	}
		1455	};
		1456
1155	=head2 SOCKS PROXIES	1457	=head3 SOCKS PROXIES
1156		1458
1157	Socks proxies are not directly supported by AnyEvent::HTTP. You can	1459	Socks proxies are not directly supported by AnyEvent::HTTP. You can
1158	compile your perl to support socks, or use an external program such as	1460	compile your perl to support socks, or use an external program such as
1159	F<socksify> (dante) or F<tsocks> to make your program use a socks proxy	1461	F<socksify> (dante) or F<tsocks> to make your program use a socks proxy
1160	transparently.	1462	transparently.

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-HTTP/HTTP.pm (file contents): Revision 1.90 by root, Mon Jan 3 00:41:25 2011 UTC vs. Revision 1.111 by root, Sun Apr 22 12:58:38 2012 UTC

Diff Legend

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.90 by root, Mon Jan 3 00:41:25 2011 UTC vs.
Revision 1.111 by root, Sun Apr 22 12:58:38 2012 UTC