[ViewVC] Diff of: cvs/JSON-XS/XS.pm

Comparing JSON-XS/XS.pm (file contents):
Revision 1.92 by root, Fri Mar 21 21:47:43 2008 UTC vs.
Revision 1.100 by root, Sun Mar 30 09:27:16 2008 UTC

…		…
103		103
104	package JSON::XS;	104	package JSON::XS;
105		105
106	use strict;	106	use strict;
107		107
108	our $VERSION = '2.1';	108	our $VERSION = '2.2';
109	our @ISA = qw(Exporter);	109	our @ISA = qw(Exporter);
110		110
111	our @EXPORT = qw(encode_json decode_json to_json from_json);	111	our @EXPORT = qw(encode_json decode_json to_json from_json);
112		112
113	sub to_json($) {	113	sub to_json($) {
…		…
462	Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>,	462	Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>,
463	resulting in an invalid JSON text:	463	resulting in an invalid JSON text:
464		464
465	JSON::XS->new->allow_nonref->encode ("Hello, World!")	465	JSON::XS->new->allow_nonref->encode ("Hello, World!")
466	=> "Hello, World!"	466	=> "Hello, World!"
		467
		468	=item $json = $json->allow_unknown ([$enable])
		469
		470	=item $enabled = $json->get_allow_unknown
		471
		472	If C<$enable> is true (or missing), then C<encode> will I<not> throw an
		473	exception when it encounters values it cannot represent in JSON (for
		474	example, filehandles) but instead will encode a JSON C<null> value. Note
		475	that blessed objects are not included here and are handled separately by
		476	c<allow_nonref>.
		477
		478	If C<$enable> is false (the default), then C<encode> will throw an
		479	exception when it encounters anything it cannot encode as JSON.
		480
		481	This option does not affect C<decode> in any way, and it is recommended to
		482	leave it off unless you know your communications partner.
467		483
468	=item $json = $json->allow_blessed ([$enable])	484	=item $json = $json->allow_blessed ([$enable])
469		485
470	=item $enabled = $json->get_allow_blessed	486	=item $enabled = $json->get_allow_blessed
471		487
…		…
681	=> ([], 3)	697	=> ([], 3)
682		698
683	=back	699	=back
684		700
685		701
		702	=head1 INCREMENTAL PARSING
		703
		704	[This section and the API it details is still EXPERIMENTAL]
		705
		706	In some cases, there is the need for incremental parsing of JSON
		707	texts. While this module always has to keep both JSON text and resulting
		708	Perl data structure in memory at one time, it does allow you to parse a
		709	JSON stream incrementally. It does so by accumulating text until it has
		710	a full JSON object, which it then can decode. This process is similar to
		711	using C<decode_prefix> to see if a full JSON object is available, but is
		712	much more efficient (JSON::XS will only attempt to parse the JSON text
		713	once it is sure it has enough text to get a decisive result, using a very
		714	simple but truly incremental parser).
		715
		716	The following two methods deal with this.
		717
		718	=over 4
		719
		720	=item [void, scalar or list context] = $json->incr_parse ([$string])
		721
		722	This is the central parsing function. It can both append new text and
		723	extract objects from the stream accumulated so far (both of these
		724	functions are optional).
		725
		726	If C<$string> is given, then this string is appended to the already
		727	existing JSON fragment stored in the C<$json> object.
		728
		729	After that, if the function is called in void context, it will simply
		730	return without doing anything further. This can be used to add more text
		731	in as many chunks as you want.
		732
		733	If the method is called in scalar context, then it will try to extract
		734	exactly I<one> JSON object. If that is successful, it will return this
		735	object, otherwise it will return C<undef>. If there is a parse error,
		736	this method will croak just as C<decode> would do (one can then use
		737	C<incr_skip> to skip the errornous part). This is the most common way of
		738	using the method.
		739
		740	And finally, in list context, it will try to extract as many objects
		741	from the stream as it can find and return them, or the empty list
		742	otherwise. For this to work, there must be no separators between the JSON
		743	objects or arrays, instead they must be concatenated back-to-back. If
		744	an error occurs, an exception will be raised as in the scalar context
		745	case. Note that in this case, any previously-parsed JSON texts will be
		746	lost.
		747
		748	=item $lvalue_string = $json->incr_text
		749
		750	This method returns the currently stored JSON fragment as an lvalue, that
		751	is, you can manipulate it. This I<only> works when a preceding call to
		752	C<incr_parse> in I<scalar context> successfully returned an object. Under
		753	all other circumstances you must not call this function (I mean it.
		754	although in simple tests it might actually work, it I<will> fail under
		755	real world conditions). As a special exception, you can also call this
		756	method before having parsed anything.
		757
		758	This function is useful in two cases: a) finding the trailing text after a
		759	JSON object or b) parsing multiple JSON objects separated by non-JSON text
		760	(such as commas).
		761
		762	=item $json->incr_skip
		763
		764	This will reset the state of the incremental parser and will remove the
		765	parsed text from the input buffer. This is useful after C<incr_parse>
		766	died, in which case the input buffer and incremental parser state is left
		767	unchanged, to skip the text parsed so far and to reset the parse state.
		768
		769	=back
		770
		771	=head2 LIMITATIONS
		772
		773	All options that affect decoding are supported, except
		774	C<allow_nonref>. The reason for this is that it cannot be made to
		775	work sensibly: JSON objects and arrays are self-delimited, i.e. you can concatenate
		776	them back to back and still decode them perfectly. This does not hold true
		777	for JSON numbers, however.
		778
		779	For example, is the string C<1> a single JSON number, or is it simply the
		780	start of C<12>? Or is C<12> a single JSON number, or the concatenation
		781	of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
		782	takes the conservative route and disallows this case.
		783
		784	=head2 EXAMPLES
		785
		786	Some examples will make all this clearer. First, a simple example that
		787	works similarly to C<decode_prefix>: We want to decode the JSON object at
		788	the start of a string and identify the portion after the JSON object:
		789
		790	my $text = "[1,2,3] hello";
		791
		792	my $json = new JSON::XS;
		793
		794	my $obj = $json->incr_parse ($text)
		795	or die "expected JSON object or array at beginning of string";
		796
		797	my $tail = $json->incr_text;
		798	# $tail now contains " hello"
		799
		800	Easy, isn't it?
		801
		802	Now for a more complicated example: Imagine a hypothetical protocol where
		803	you read some requests from a TCP stream, and each request is a JSON
		804	array, without any separation between them (in fact, it is often useful to
		805	use newlines as "separators", as these get interpreted as whitespace at
		806	the start of the JSON text, which makes it possible to test said protocol
		807	with C<telnet>...).
		808
		809	Here is how you'd do it (it is trivial to write this in an event-based
		810	manner):
		811
		812	my $json = new JSON::XS;
		813
		814	# read some data from the socket
		815	while (sysread $socket, my $buf, 4096) {
		816
		817	# split and decode as many requests as possible
		818	for my $request ($json->incr_parse ($buf)) {
		819	# act on the $request
		820	}
		821	}
		822
		823	Another complicated example: Assume you have a string with JSON objects
		824	or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
		825	[3]>). To parse them, we have to skip the commas between the JSON texts,
		826	and here is where the lvalue-ness of C<incr_text> comes in useful:
		827
		828	my $text = "[1],[2], [3]";
		829	my $json = new JSON::XS;
		830
		831	# void context, so no parsing done
		832	$json->incr_parse ($text);
		833
		834	# now extract as many objects as possible. note the
		835	# use of scalar context so incr_text can be called.
		836	while (my $obj = $json->incr_parse) {
		837	# do something with $obj
		838
		839	# now skip the optional comma
		840	$json->incr_text =~ s/^ \s* , //x;
		841	}
		842
		843	Now lets go for a very complex example: Assume that you have a gigantic
		844	JSON array-of-objects, many gigabytes in size, and you want to parse it,
		845	but you cannot load it into memory fully (this has actually happened in
		846	the real world :).
		847
		848	Well, you lost, you have to implement your own JSON parser. But JSON::XS
		849	can still help you: You implement a (very simple) array parser and let
		850	JSON decode the array elements, which are all full JSON objects on their
		851	own (this wouldn't work if the array elements could be JSON numbers, for
		852	example):
		853
		854	my $json = new JSON::XS;
		855
		856	# open the monster
		857	open my $fh, "<bigfile.json"
		858	or die "bigfile: $!";
		859
		860	# first parse the initial "["
		861	for (;;) {
		862	sysread $fh, my $buf, 65536
		863	or die "read error: $!";
		864	$json->incr_parse ($buf); # void context, so no parsing
		865
		866	# Exit the loop once we found and removed(!) the initial "[".
		867	# In essence, we are (ab-)using the $json object as a simple scalar
		868	# we append data to.
		869	last if $json->incr_text =~ s/^ \s* \[ //x;
		870	}
		871
		872	# now we have the skipped the initial "[", so continue
		873	# parsing all the elements.
		874	for (;;) {
		875	# in this loop we read data until we got a single JSON object
		876	for (;;) {
		877	if (my $obj = $json->incr_parse) {
		878	# do something with $obj
		879	last;
		880	}
		881
		882	# add more data
		883	sysread $fh, my $buf, 65536
		884	or die "read error: $!";
		885	$json->incr_parse ($buf); # void context, so no parsing
		886	}
		887
		888	# in this loop we read data until we either found and parsed the
		889	# separating "," between elements, or the final "]"
		890	for (;;) {
		891	# first skip whitespace
		892	$json->incr_text =~ s/^\s*//;
		893
		894	# if we find "]", we are done
		895	if ($json->incr_text =~ s/^\]//) {
		896	print "finished.\n";
		897	exit;
		898	}
		899
		900	# if we find ",", we can continue with the next element
		901	if ($json->incr_text =~ s/^,//) {
		902	last;
		903	}
		904
		905	# if we find anything else, we have a parse error!
		906	if (length $json->incr_text) {
		907	die "parse error near ", $json->incr_text;
		908	}
		909
		910	# else add more data
		911	sysread $fh, my $buf, 65536
		912	or die "read error: $!";
		913	$json->incr_parse ($buf); # void context, so no parsing
		914	}
		915
		916	This is a complex example, but most of the complexity comes from the fact
		917	that we are trying to be correct (bear with me if I am wrong, I never ran
		918	the above example :).
		919
		920
		921
686	=head1 MAPPING	922	=head1 MAPPING
687		923
688	This section describes how JSON::XS maps Perl values to JSON values and	924	This section describes how JSON::XS maps Perl values to JSON values and
689	vice versa. These mappings are designed to "do the right thing" in most	925	vice versa. These mappings are designed to "do the right thing" in most
690	circumstances automatically, preserving round-tripping characteristics	926	circumstances automatically, preserving round-tripping characteristics
…		…
929	proper subset of most 8-bit and multibyte encodings in use in the world.	1165	proper subset of most 8-bit and multibyte encodings in use in the world.
930		1166
931	=back	1167	=back
932		1168
933		1169
934	=head1 COMPARISON
935
936	As already mentioned, this module was created because none of the existing
937	JSON modules could be made to work correctly. First I will describe the
938	problems (or pleasures) I encountered with various existing JSON modules,
939	followed by some benchmark values. JSON::XS was designed not to suffer
940	from any of these problems or limitations.
941
942	=over 4
943
944	=item JSON 2.xx
945
946	A marvellous piece of engineering, this module either uses JSON::XS
947	directly when available (so will be 100% compatible with it, including
948	speed), or it uses JSON::PP, which is basically JSON::XS translated to
949	Pure Perl, which should be 100% compatible with JSON::XS, just a bit
950	slower.
951
952	You cannot really lose by using this module, especially as it tries very
953	hard to work even with ancient Perl versions, while JSON::XS does not.
954
955	=item JSON 1.07
956
957	Slow (but very portable, as it is written in pure Perl).
958
959	Undocumented/buggy Unicode handling (how JSON handles Unicode values is
960	undocumented. One can get far by feeding it Unicode strings and doing
961	en-/decoding oneself, but Unicode escapes are not working properly).
962
963	No round-tripping (strings get clobbered if they look like numbers, e.g.
964	the string C<2.0> will encode to C<2.0> instead of C<"2.0">, and that will
965	decode into the number 2.
966
967	=item JSON::PC 0.01
968
969	Very fast.
970
971	Undocumented/buggy Unicode handling.
972
973	No round-tripping.
974
975	Has problems handling many Perl values (e.g. regex results and other magic
976	values will make it croak).
977
978	Does not even generate valid JSON (C<{1,2}> gets converted to C<{1:2}>
979	which is not a valid JSON text.
980
981	Unmaintained (maintainer unresponsive for many months, bugs are not
982	getting fixed).
983
984	=item JSON::Syck 0.21
985
986	Very buggy (often crashes).
987
988	Very inflexible (no human-readable format supported, format pretty much
989	undocumented. I need at least a format for easy reading by humans and a
990	single-line compact format for use in a protocol, and preferably a way to
991	generate ASCII-only JSON texts).
992
993	Completely broken (and confusingly documented) Unicode handling (Unicode
994	escapes are not working properly, you need to set ImplicitUnicode to
995	I<different> values on en- and decoding to get symmetric behaviour).
996
997	No round-tripping (simple cases work, but this depends on whether the scalar
998	value was used in a numeric context or not).
999
1000	Dumping hashes may skip hash values depending on iterator state.
1001
1002	Unmaintained (maintainer unresponsive for many months, bugs are not
1003	getting fixed).
1004
1005	Does not check input for validity (i.e. will accept non-JSON input and
1006	return "something" instead of raising an exception. This is a security
1007	issue: imagine two banks transferring money between each other using
1008	JSON. One bank might parse a given non-JSON request and deduct money,
1009	while the other might reject the transaction with a syntax error. While a
1010	good protocol will at least recover, that is extra unnecessary work and
1011	the transaction will still not succeed).
1012
1013	=item JSON::DWIW 0.04
1014
1015	Very fast. Very natural. Very nice.
1016
1017	Undocumented Unicode handling (but the best of the pack. Unicode escapes
1018	still don't get parsed properly).
1019
1020	Very inflexible.
1021
1022	No round-tripping.
1023
1024	Does not generate valid JSON texts (key strings are often unquoted, empty keys
1025	result in nothing being output)
1026
1027	Does not check input for validity.
1028
1029	=back
1030
1031
1032	=head2 JSON and YAML	1170	=head2 JSON and YAML
1033		1171
1034	You often hear that JSON is a subset of YAML. This is, however, a mass	1172	You often hear that JSON is a subset of YAML. This is, however, a mass
1035	hysteria(*) and very far from the truth (as of the time of this writing),	1173	hysteria(*) and very far from the truth (as of the time of this writing),
1036	so let me state it clearly: I<in general, there is no way to configure	1174	so let me state it clearly: I<in general, there is no way to configure
…		…
1092		1230
1093	First comes a comparison between various modules using	1231	First comes a comparison between various modules using
1094	a very short single-line JSON string (also available at	1232	a very short single-line JSON string (also available at
1095	L<http://dist.schmorp.de/misc/json/short.json>).	1233	L<http://dist.schmorp.de/misc/json/short.json>).
1096		1234
1097	{"method": "handleMessage", "params": ["user1", "we were just talking"], \	1235	{"method": "handleMessage", "params": ["user1",
1098	"id": null, "array":[1,11,234,-5,1e5,1e7, true, false]}	1236	"we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
		1237	true, false]}
1099		1238
1100	It shows the number of encodes/decodes per second (JSON::XS uses	1239	It shows the number of encodes/decodes per second (JSON::XS uses
1101	the functional interface, while JSON::XS/2 uses the OO interface	1240	the functional interface, while JSON::XS/2 uses the OO interface
1102	with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables	1241	with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
1103	shrink). Higher is better:	1242	shrink). Higher is better:
…		…
1233	"--" => sub { $_[0] = ${$_[0]} - 1 },	1372	"--" => sub { $_[0] = ${$_[0]} - 1 },
1234	fallback => 1;	1373	fallback => 1;
1235		1374
1236	1;	1375	1;
1237		1376
		1377	=head1 SEE ALSO
		1378
		1379	The F<json_xs> command line utility for quick experiments.
		1380
1238	=head1 AUTHOR	1381	=head1 AUTHOR
1239		1382
1240	Marc Lehmann <schmorp@schmorp.de>	1383	Marc Lehmann <schmorp@schmorp.de>
1241	http://home.schmorp.de/	1384	http://home.schmorp.de/
1242		1385

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing JSON-XS/XS.pm (file contents): Revision 1.92 by root, Fri Mar 21 21:47:43 2008 UTC vs. Revision 1.100 by root, Sun Mar 30 09:27:16 2008 UTC

Diff Legend

Comparing JSON-XS/XS.pm (file contents):
Revision 1.92 by root, Fri Mar 21 21:47:43 2008 UTC vs.
Revision 1.100 by root, Sun Mar 30 09:27:16 2008 UTC