ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.84 by root, Wed Mar 19 02:52:15 2008 UTC vs.
Revision 1.134 by root, Mon Feb 21 15:38:06 2011 UTC

1=head1 NAME
2
3JSON::XS - JSON serialising/deserialising, done correctly and fast
4
1=encoding utf-8 5=encoding utf-8
2
3=head1 NAME
4
5JSON::XS - JSON serialising/deserialising, done correctly and fast
6 6
7JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ 7JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ
8 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html) 8 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html)
9 9
10=head1 SYNOPSIS 10=head1 SYNOPSIS
37primary goal is to be I<correct> and its secondary goal is to be 37primary goal is to be I<correct> and its secondary goal is to be
38I<fast>. To reach the latter goal it was written in C. 38I<fast>. To reach the latter goal it was written in C.
39 39
40Beginning with version 2.0 of the JSON module, when both JSON and 40Beginning with version 2.0 of the JSON module, when both JSON and
41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be 41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42overriden) with no overhead due to emulation (by inheritign constructor 42overridden) with no overhead due to emulation (by inheriting constructor
43and methods). If JSON::XS is not available, it will fall back to the 43and methods). If JSON::XS is not available, it will fall back to the
44compatible JSON::PP module as backend, so using JSON instead of JSON::XS 44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45gives you a portable JSON API that can be fast when you need and doesn't 45gives you a portable JSON API that can be fast when you need and doesn't
46require a C compiler when that is a problem. 46require a C compiler when that is a problem.
47 47
49to write yet another JSON module? While it seems there are many JSON 49to write yet another JSON module? While it seems there are many JSON
50modules, none of them correctly handle all corner cases, and in most cases 50modules, none of them correctly handle all corner cases, and in most cases
51their maintainers are unresponsive, gone missing, or not listening to bug 51their maintainers are unresponsive, gone missing, or not listening to bug
52reports for other reasons. 52reports for other reasons.
53 53
54See COMPARISON, below, for a comparison to some other JSON modules.
55
56See MAPPING, below, on how JSON::XS maps perl values to JSON values and 54See MAPPING, below, on how JSON::XS maps perl values to JSON values and
57vice versa. 55vice versa.
58 56
59=head2 FEATURES 57=head2 FEATURES
60 58
65This module knows how to handle Unicode, documents how and when it does 63This module knows how to handle Unicode, documents how and when it does
66so, and even documents what "correct" means. 64so, and even documents what "correct" means.
67 65
68=item * round-trip integrity 66=item * round-trip integrity
69 67
70When you serialise a perl data structure using only datatypes supported 68When you serialise a perl data structure using only data types supported
71by JSON, the deserialised data structure is identical on the Perl level. 69by JSON and Perl, the deserialised data structure is identical on the Perl
72(e.g. the string "2.0" doesn't suddenly become "2" just because it looks 70level. (e.g. the string "2.0" doesn't suddenly become "2" just because
73like a number). There minor I<are> exceptions to this, read the MAPPING 71it looks like a number). There I<are> minor exceptions to this, read the
74section below to learn about those. 72MAPPING section below to learn about those.
75 73
76=item * strict checking of JSON correctness 74=item * strict checking of JSON correctness
77 75
78There is no guessing, no generating of illegal JSON texts by default, 76There is no guessing, no generating of illegal JSON texts by default,
79and only JSON is accepted as input by default (the latter is a security 77and only JSON is accepted as input by default (the latter is a security
84Compared to other JSON modules and other serialisers such as Storable, 82Compared to other JSON modules and other serialisers such as Storable,
85this module usually compares favourably in terms of speed, too. 83this module usually compares favourably in terms of speed, too.
86 84
87=item * simple to use 85=item * simple to use
88 86
89This module has both a simple functional interface as well as an objetc 87This module has both a simple functional interface as well as an object
90oriented interface interface. 88oriented interface interface.
91 89
92=item * reasonably versatile output formats 90=item * reasonably versatile output formats
93 91
94You can choose between the most compact guaranteed-single-line format 92You can choose between the most compact guaranteed-single-line format
95possible (nice for simple line-based protocols), a pure-ascii format 93possible (nice for simple line-based protocols), a pure-ASCII format
96(for when your transport is not 8-bit clean, still supports the whole 94(for when your transport is not 8-bit clean, still supports the whole
97Unicode range), or a pretty-printed format (for when you want to read that 95Unicode range), or a pretty-printed format (for when you want to read that
98stuff). Or you can combine those features in whatever way you like. 96stuff). Or you can combine those features in whatever way you like.
99 97
100=back 98=back
101 99
102=cut 100=cut
103 101
104package JSON::XS; 102package JSON::XS;
105 103
106use strict; 104use common::sense;
107 105
108our $VERSION = '2.01'; 106our $VERSION = '2.3';
109our @ISA = qw(Exporter); 107our @ISA = qw(Exporter);
110 108
111our @EXPORT = qw(encode_json decode_json to_json from_json); 109our @EXPORT = qw(encode_json decode_json to_json from_json);
112 110
113sub to_json($) { 111sub to_json($) {
137 135
138This function call is functionally identical to: 136This function call is functionally identical to:
139 137
140 $json_text = JSON::XS->new->utf8->encode ($perl_scalar) 138 $json_text = JSON::XS->new->utf8->encode ($perl_scalar)
141 139
142except being faster. 140Except being faster.
143 141
144=item $perl_scalar = decode_json $json_text 142=item $perl_scalar = decode_json $json_text
145 143
146The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries 144The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries
147to parse that as an UTF-8 encoded JSON text, returning the resulting 145to parse that as an UTF-8 encoded JSON text, returning the resulting
149 147
150This function call is functionally identical to: 148This function call is functionally identical to:
151 149
152 $perl_scalar = JSON::XS->new->utf8->decode ($json_text) 150 $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
153 151
154except being faster. 152Except being faster.
155 153
156=item $is_boolean = JSON::XS::is_bool $scalar 154=item $is_boolean = JSON::XS::is_bool $scalar
157 155
158Returns true if the passed scalar represents either JSON::XS::true or 156Returns true if the passed scalar represents either JSON::XS::true or
159JSON::XS::false, two constants that act like C<1> and C<0>, respectively 157JSON::XS::false, two constants that act like C<1> and C<0>, respectively
197 195
198If you didn't know about that flag, just the better, pretend it doesn't 196If you didn't know about that flag, just the better, pretend it doesn't
199exist. 197exist.
200 198
201=item 4. A "Unicode String" is simply a string where each character can be 199=item 4. A "Unicode String" is simply a string where each character can be
202validly interpreted as a Unicode codepoint. 200validly interpreted as a Unicode code point.
203 201
204If you have UTF-8 encoded data, it is no longer a Unicode string, but a 202If you have UTF-8 encoded data, it is no longer a Unicode string, but a
205Unicode string encoded in UTF-8, giving you a binary string. 203Unicode string encoded in UTF-8, giving you a binary string.
206 204
207=item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string. 205=item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string.
245 243
246If C<$enable> is false, then the C<encode> method will not escape Unicode 244If C<$enable> is false, then the C<encode> method will not escape Unicode
247characters unless required by the JSON syntax or other flags. This results 245characters unless required by the JSON syntax or other flags. This results
248in a faster and more compact format. 246in a faster and more compact format.
249 247
248See also the section I<ENCODING/CODESET FLAG NOTES> later in this
249document.
250
250The main use for this flag is to produce JSON texts that can be 251The main use for this flag is to produce JSON texts that can be
251transmitted over a 7-bit channel, as the encoded JSON texts will not 252transmitted over a 7-bit channel, as the encoded JSON texts will not
252contain any 8 bit characters. 253contain any 8 bit characters.
253 254
254 JSON::XS->new->ascii (1)->encode ([chr 0x10401]) 255 JSON::XS->new->ascii (1)->encode ([chr 0x10401])
265will not be affected in any way by this flag, as C<decode> by default 266will not be affected in any way by this flag, as C<decode> by default
266expects Unicode, which is a strict superset of latin1. 267expects Unicode, which is a strict superset of latin1.
267 268
268If C<$enable> is false, then the C<encode> method will not escape Unicode 269If C<$enable> is false, then the C<encode> method will not escape Unicode
269characters unless required by the JSON syntax or other flags. 270characters unless required by the JSON syntax or other flags.
271
272See also the section I<ENCODING/CODESET FLAG NOTES> later in this
273document.
270 274
271The main use for this flag is efficiently encoding binary data as JSON 275The main use for this flag is efficiently encoding binary data as JSON
272text, as most octets will not be escaped, resulting in a smaller encoded 276text, as most octets will not be escaped, resulting in a smaller encoded
273size. The disadvantage is that the resulting JSON text is encoded 277size. The disadvantage is that the resulting JSON text is encoded
274in latin1 (and must correctly be treated as such when storing and 278in latin1 (and must correctly be treated as such when storing and
293 297
294If C<$enable> is false, then the C<encode> method will return the JSON 298If C<$enable> is false, then the C<encode> method will return the JSON
295string as a (non-encoded) Unicode string, while C<decode> expects thus a 299string as a (non-encoded) Unicode string, while C<decode> expects thus a
296Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs 300Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs
297to be done yourself, e.g. using the Encode module. 301to be done yourself, e.g. using the Encode module.
302
303See also the section I<ENCODING/CODESET FLAG NOTES> later in this
304document.
298 305
299Example, output UTF-16BE-encoded JSON: 306Example, output UTF-16BE-encoded JSON:
300 307
301 use Encode; 308 use Encode;
302 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object); 309 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object);
434the same hash might be encoded differently even if contains the same data, 441the same hash might be encoded differently even if contains the same data,
435as key-value pairs have no inherent ordering in Perl. 442as key-value pairs have no inherent ordering in Perl.
436 443
437This setting has no effect when decoding JSON texts. 444This setting has no effect when decoding JSON texts.
438 445
446This setting has currently no effect on tied hashes.
447
439=item $json = $json->allow_nonref ([$enable]) 448=item $json = $json->allow_nonref ([$enable])
440 449
441=item $enabled = $json->get_allow_nonref 450=item $enabled = $json->get_allow_nonref
442 451
443If C<$enable> is true (or missing), then the C<encode> method can convert a 452If C<$enable> is true (or missing), then the C<encode> method can convert a
453Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>, 462Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>,
454resulting in an invalid JSON text: 463resulting in an invalid JSON text:
455 464
456 JSON::XS->new->allow_nonref->encode ("Hello, World!") 465 JSON::XS->new->allow_nonref->encode ("Hello, World!")
457 => "Hello, World!" 466 => "Hello, World!"
467
468=item $json = $json->allow_unknown ([$enable])
469
470=item $enabled = $json->get_allow_unknown
471
472If C<$enable> is true (or missing), then C<encode> will I<not> throw an
473exception when it encounters values it cannot represent in JSON (for
474example, filehandles) but instead will encode a JSON C<null> value. Note
475that blessed objects are not included here and are handled separately by
476c<allow_nonref>.
477
478If C<$enable> is false (the default), then C<encode> will throw an
479exception when it encounters anything it cannot encode as JSON.
480
481This option does not affect C<decode> in any way, and it is recommended to
482leave it off unless you know your communications partner.
458 483
459=item $json = $json->allow_blessed ([$enable]) 484=item $json = $json->allow_blessed ([$enable])
460 485
461=item $enabled = $json->get_allow_blessed 486=item $enabled = $json->get_allow_blessed
462 487
603=item $json = $json->max_depth ([$maximum_nesting_depth]) 628=item $json = $json->max_depth ([$maximum_nesting_depth])
604 629
605=item $max_depth = $json->get_max_depth 630=item $max_depth = $json->get_max_depth
606 631
607Sets the maximum nesting level (default C<512>) accepted while encoding 632Sets the maximum nesting level (default C<512>) accepted while encoding
608or decoding. If the JSON text or Perl data structure has an equal or 633or decoding. If a higher nesting level is detected in JSON text or a Perl
609higher nesting level then this limit, then the encoder and decoder will 634data structure, then the encoder and decoder will stop and croak at that
610stop and croak at that point. 635point.
611 636
612Nesting level is defined by number of hash- or arrayrefs that the encoder 637Nesting level is defined by number of hash- or arrayrefs that the encoder
613needs to traverse to reach a given point or the number of C<{> or C<[> 638needs to traverse to reach a given point or the number of C<{> or C<[>
614characters without their matching closing parenthesis crossed to reach a 639characters without their matching closing parenthesis crossed to reach a
615given character in a string. 640given character in a string.
616 641
617Setting the maximum depth to one disallows any nesting, so that ensures 642Setting the maximum depth to one disallows any nesting, so that ensures
618that the object is only a single hash/object or array. 643that the object is only a single hash/object or array.
619 644
620The argument to C<max_depth> will be rounded up to the next highest power
621of two. If no argument is given, the highest possible setting will be 645If no argument is given, the highest possible setting will be used, which
622used, which is rarely useful. 646is rarely useful.
647
648Note that nesting is implemented by recursion in C. The default value has
649been chosen to be as large as typical operating systems allow without
650crashing.
623 651
624See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 652See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
625 653
626=item $json = $json->max_size ([$maximum_string_size]) 654=item $json = $json->max_size ([$maximum_string_size])
627 655
628=item $max_size = $json->get_max_size 656=item $max_size = $json->get_max_size
629 657
630Set the maximum length a JSON text may have (in bytes) where decoding is 658Set the maximum length a JSON text may have (in bytes) where decoding is
631being attempted. The default is C<0>, meaning no limit. When C<decode> 659being attempted. The default is C<0>, meaning no limit. When C<decode>
632is called on a string longer then this number of characters it will not 660is called on a string that is longer then this many bytes, it will not
633attempt to decode the string but throw an exception. This setting has no 661attempt to decode the string but throw an exception. This setting has no
634effect on C<encode> (yet). 662effect on C<encode> (yet).
635 663
636The argument to C<max_size> will be rounded up to the next B<highest> 664If no argument is given, the limit check will be deactivated (same as when
637power of two (so may be more than requested). If no argument is given, the 665C<0> is specified).
638limit check will be deactivated (same as when C<0> is specified).
639 666
640See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 667See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
641 668
642=item $json_text = $json->encode ($perl_scalar) 669=item $json_text = $json->encode ($perl_scalar)
643 670
672 => ([], 3) 699 => ([], 3)
673 700
674=back 701=back
675 702
676 703
704=head1 INCREMENTAL PARSING
705
706In some cases, there is the need for incremental parsing of JSON
707texts. While this module always has to keep both JSON text and resulting
708Perl data structure in memory at one time, it does allow you to parse a
709JSON stream incrementally. It does so by accumulating text until it has
710a full JSON object, which it then can decode. This process is similar to
711using C<decode_prefix> to see if a full JSON object is available, but
712is much more efficient (and can be implemented with a minimum of method
713calls).
714
715JSON::XS will only attempt to parse the JSON text once it is sure it
716has enough text to get a decisive result, using a very simple but
717truly incremental parser. This means that it sometimes won't stop as
718early as the full parser, for example, it doesn't detect mismatched
719parentheses. The only thing it guarantees is that it starts decoding as
720soon as a syntactically valid JSON text has been seen. This means you need
721to set resource limits (e.g. C<max_size>) to ensure the parser will stop
722parsing in the presence if syntax errors.
723
724The following methods implement this incremental parser.
725
726=over 4
727
728=item [void, scalar or list context] = $json->incr_parse ([$string])
729
730This is the central parsing function. It can both append new text and
731extract objects from the stream accumulated so far (both of these
732functions are optional).
733
734If C<$string> is given, then this string is appended to the already
735existing JSON fragment stored in the C<$json> object.
736
737After that, if the function is called in void context, it will simply
738return without doing anything further. This can be used to add more text
739in as many chunks as you want.
740
741If the method is called in scalar context, then it will try to extract
742exactly I<one> JSON object. If that is successful, it will return this
743object, otherwise it will return C<undef>. If there is a parse error,
744this method will croak just as C<decode> would do (one can then use
745C<incr_skip> to skip the errornous part). This is the most common way of
746using the method.
747
748And finally, in list context, it will try to extract as many objects
749from the stream as it can find and return them, or the empty list
750otherwise. For this to work, there must be no separators between the JSON
751objects or arrays, instead they must be concatenated back-to-back. If
752an error occurs, an exception will be raised as in the scalar context
753case. Note that in this case, any previously-parsed JSON texts will be
754lost.
755
756Example: Parse some JSON arrays/objects in a given string and return
757them.
758
759 my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]");
760
761=item $lvalue_string = $json->incr_text
762
763This method returns the currently stored JSON fragment as an lvalue, that
764is, you can manipulate it. This I<only> works when a preceding call to
765C<incr_parse> in I<scalar context> successfully returned an object. Under
766all other circumstances you must not call this function (I mean it.
767although in simple tests it might actually work, it I<will> fail under
768real world conditions). As a special exception, you can also call this
769method before having parsed anything.
770
771This function is useful in two cases: a) finding the trailing text after a
772JSON object or b) parsing multiple JSON objects separated by non-JSON text
773(such as commas).
774
775=item $json->incr_skip
776
777This will reset the state of the incremental parser and will remove
778the parsed text from the input buffer so far. This is useful after
779C<incr_parse> died, in which case the input buffer and incremental parser
780state is left unchanged, to skip the text parsed so far and to reset the
781parse state.
782
783The difference to C<incr_reset> is that only text until the parse error
784occured is removed.
785
786=item $json->incr_reset
787
788This completely resets the incremental parser, that is, after this call,
789it will be as if the parser had never parsed anything.
790
791This is useful if you want to repeatedly parse JSON objects and want to
792ignore any trailing data, which means you have to reset the parser after
793each successful decode.
794
795=back
796
797=head2 LIMITATIONS
798
799All options that affect decoding are supported, except
800C<allow_nonref>. The reason for this is that it cannot be made to
801work sensibly: JSON objects and arrays are self-delimited, i.e. you can concatenate
802them back to back and still decode them perfectly. This does not hold true
803for JSON numbers, however.
804
805For example, is the string C<1> a single JSON number, or is it simply the
806start of C<12>? Or is C<12> a single JSON number, or the concatenation
807of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
808takes the conservative route and disallows this case.
809
810=head2 EXAMPLES
811
812Some examples will make all this clearer. First, a simple example that
813works similarly to C<decode_prefix>: We want to decode the JSON object at
814the start of a string and identify the portion after the JSON object:
815
816 my $text = "[1,2,3] hello";
817
818 my $json = new JSON::XS;
819
820 my $obj = $json->incr_parse ($text)
821 or die "expected JSON object or array at beginning of string";
822
823 my $tail = $json->incr_text;
824 # $tail now contains " hello"
825
826Easy, isn't it?
827
828Now for a more complicated example: Imagine a hypothetical protocol where
829you read some requests from a TCP stream, and each request is a JSON
830array, without any separation between them (in fact, it is often useful to
831use newlines as "separators", as these get interpreted as whitespace at
832the start of the JSON text, which makes it possible to test said protocol
833with C<telnet>...).
834
835Here is how you'd do it (it is trivial to write this in an event-based
836manner):
837
838 my $json = new JSON::XS;
839
840 # read some data from the socket
841 while (sysread $socket, my $buf, 4096) {
842
843 # split and decode as many requests as possible
844 for my $request ($json->incr_parse ($buf)) {
845 # act on the $request
846 }
847 }
848
849Another complicated example: Assume you have a string with JSON objects
850or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
851[3]>). To parse them, we have to skip the commas between the JSON texts,
852and here is where the lvalue-ness of C<incr_text> comes in useful:
853
854 my $text = "[1],[2], [3]";
855 my $json = new JSON::XS;
856
857 # void context, so no parsing done
858 $json->incr_parse ($text);
859
860 # now extract as many objects as possible. note the
861 # use of scalar context so incr_text can be called.
862 while (my $obj = $json->incr_parse) {
863 # do something with $obj
864
865 # now skip the optional comma
866 $json->incr_text =~ s/^ \s* , //x;
867 }
868
869Now lets go for a very complex example: Assume that you have a gigantic
870JSON array-of-objects, many gigabytes in size, and you want to parse it,
871but you cannot load it into memory fully (this has actually happened in
872the real world :).
873
874Well, you lost, you have to implement your own JSON parser. But JSON::XS
875can still help you: You implement a (very simple) array parser and let
876JSON decode the array elements, which are all full JSON objects on their
877own (this wouldn't work if the array elements could be JSON numbers, for
878example):
879
880 my $json = new JSON::XS;
881
882 # open the monster
883 open my $fh, "<bigfile.json"
884 or die "bigfile: $!";
885
886 # first parse the initial "["
887 for (;;) {
888 sysread $fh, my $buf, 65536
889 or die "read error: $!";
890 $json->incr_parse ($buf); # void context, so no parsing
891
892 # Exit the loop once we found and removed(!) the initial "[".
893 # In essence, we are (ab-)using the $json object as a simple scalar
894 # we append data to.
895 last if $json->incr_text =~ s/^ \s* \[ //x;
896 }
897
898 # now we have the skipped the initial "[", so continue
899 # parsing all the elements.
900 for (;;) {
901 # in this loop we read data until we got a single JSON object
902 for (;;) {
903 if (my $obj = $json->incr_parse) {
904 # do something with $obj
905 last;
906 }
907
908 # add more data
909 sysread $fh, my $buf, 65536
910 or die "read error: $!";
911 $json->incr_parse ($buf); # void context, so no parsing
912 }
913
914 # in this loop we read data until we either found and parsed the
915 # separating "," between elements, or the final "]"
916 for (;;) {
917 # first skip whitespace
918 $json->incr_text =~ s/^\s*//;
919
920 # if we find "]", we are done
921 if ($json->incr_text =~ s/^\]//) {
922 print "finished.\n";
923 exit;
924 }
925
926 # if we find ",", we can continue with the next element
927 if ($json->incr_text =~ s/^,//) {
928 last;
929 }
930
931 # if we find anything else, we have a parse error!
932 if (length $json->incr_text) {
933 die "parse error near ", $json->incr_text;
934 }
935
936 # else add more data
937 sysread $fh, my $buf, 65536
938 or die "read error: $!";
939 $json->incr_parse ($buf); # void context, so no parsing
940 }
941
942This is a complex example, but most of the complexity comes from the fact
943that we are trying to be correct (bear with me if I am wrong, I never ran
944the above example :).
945
946
947
677=head1 MAPPING 948=head1 MAPPING
678 949
679This section describes how JSON::XS maps Perl values to JSON values and 950This section describes how JSON::XS maps Perl values to JSON values and
680vice versa. These mappings are designed to "do the right thing" in most 951vice versa. These mappings are designed to "do the right thing" in most
681circumstances automatically, preserving round-tripping characteristics 952circumstances automatically, preserving round-tripping characteristics
722 993
723Numbers containing a fractional or exponential part will always be 994Numbers containing a fractional or exponential part will always be
724represented as numeric (floating point) values, possibly at a loss of 995represented as numeric (floating point) values, possibly at a loss of
725precision (in which case you might lose perfect roundtripping ability, but 996precision (in which case you might lose perfect roundtripping ability, but
726the JSON number will still be re-encoded as a JSON number). 997the JSON number will still be re-encoded as a JSON number).
998
999Note that precision is not accuracy - binary floating point values cannot
1000represent most decimal fractions exactly, and when converting from and to
1001floating point, JSON::XS only guarantees precision up to but not including
1002the leats significant bit.
727 1003
728=item true, false 1004=item true, false
729 1005
730These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>, 1006These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>,
731respectively. They are overloaded to act almost exactly like the numbers 1007respectively. They are overloaded to act almost exactly like the numbers
768Other unblessed references are generally not allowed and will cause an 1044Other unblessed references are generally not allowed and will cause an
769exception to be thrown, except for references to the integers C<0> and 1045exception to be thrown, except for references to the integers C<0> and
770C<1>, which get turned into C<false> and C<true> atoms in JSON. You can 1046C<1>, which get turned into C<false> and C<true> atoms in JSON. You can
771also use C<JSON::XS::false> and C<JSON::XS::true> to improve readability. 1047also use C<JSON::XS::false> and C<JSON::XS::true> to improve readability.
772 1048
773 encode_json [\0,JSON::XS::true] # yields [false,true] 1049 encode_json [\0, JSON::XS::true] # yields [false,true]
774 1050
775=item JSON::XS::true, JSON::XS::false 1051=item JSON::XS::true, JSON::XS::false
776 1052
777These special values become JSON true and JSON false values, 1053These special values become JSON true and JSON false values,
778respectively. You can also use C<\1> and C<\0> directly if you want. 1054respectively. You can also use C<\1> and C<\0> directly if you want.
816 my $x = "3"; # some variable containing a string 1092 my $x = "3"; # some variable containing a string
817 $x += 0; # numify it, ensuring it will be dumped as a number 1093 $x += 0; # numify it, ensuring it will be dumped as a number
818 $x *= 1; # same thing, the choice is yours. 1094 $x *= 1; # same thing, the choice is yours.
819 1095
820You can not currently force the type in other, less obscure, ways. Tell me 1096You can not currently force the type in other, less obscure, ways. Tell me
821if you need this capability (but don't forget to explain why its needed 1097if you need this capability (but don't forget to explain why it's needed
822:). 1098:).
1099
1100Note that numerical precision has the same meaning as under Perl (so
1101binary to decimal conversion follows the same rules as in Perl, which
1102can differ to other languages). Also, your perl interpreter might expose
1103extensions to the floating point numbers of your platform, such as
1104infinities or NaN's - these cannot be represented in JSON, and it is an
1105error to pass those in.
823 1106
824=back 1107=back
825 1108
826 1109
827=head1 ENCODING/CODESET FLAG NOTES 1110=head1 ENCODING/CODESET FLAG NOTES
828 1111
829The interested reader might have seen a number of flags that signify 1112The interested reader might have seen a number of flags that signify
830encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be 1113encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be
831some confusion on what these do, so here is a short comparison: 1114some confusion on what these do, so here is a short comparison:
832 1115
833C<utf8> controls wether the JSON text created by C<encode> (and expected 1116C<utf8> controls whether the JSON text created by C<encode> (and expected
834by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only 1117by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only
835control wether C<encode> escapes character values outside their respective 1118control whether C<encode> escapes character values outside their respective
836codeset range. Neither of these flags conflict with each other, although 1119codeset range. Neither of these flags conflict with each other, although
837some combinations make less sense than others. 1120some combinations make less sense than others.
838 1121
839Care has been taken to make all flags symmetrical with respect to 1122Care has been taken to make all flags symmetrical with respect to
840C<encode> and C<decode>, that is, texts encoded with any combination of 1123C<encode> and C<decode>, that is, texts encoded with any combination of
920proper subset of most 8-bit and multibyte encodings in use in the world. 1203proper subset of most 8-bit and multibyte encodings in use in the world.
921 1204
922=back 1205=back
923 1206
924 1207
925=head1 COMPARISON 1208=head2 JSON and ECMAscript
926 1209
927As already mentioned, this module was created because none of the existing 1210JSON syntax is based on how literals are represented in javascript (the
928JSON modules could be made to work correctly. First I will describe the 1211not-standardised predecessor of ECMAscript) which is presumably why it is
929problems (or pleasures) I encountered with various existing JSON modules, 1212called "JavaScript Object Notation".
930followed by some benchmark values. JSON::XS was designed not to suffer
931from any of these problems or limitations.
932 1213
933=over 4 1214However, JSON is not a subset (and also not a superset of course) of
1215ECMAscript (the standard) or javascript (whatever browsers actually
1216implement).
934 1217
935=item JSON 2.xx 1218If you want to use javascript's C<eval> function to "parse" JSON, you
1219might run into parse errors for valid JSON texts, or the resulting data
1220structure might not be queryable:
936 1221
937A marvellous piece of engineering, this module either uses JSON::XS 1222One of the problems is that U+2028 and U+2029 are valid characters inside
938directly when available (so will be 100% compatible with it, including 1223JSON strings, but are not allowed in ECMAscript string literals, so the
939speed), or it uses JSON::PP, which is basically JSON::XS translated to 1224following Perl fragment will not output something that can be guaranteed
940Pure Perl, which should be 100% compatible with JSON::XS, just a bit 1225to be parsable by javascript's C<eval>:
941slower.
942 1226
943You cannot really lose by using this module. 1227 use JSON::XS;
944 1228
945=item JSON 1.07 1229 print encode_json [chr 0x2028];
946 1230
947Slow (but very portable, as it is written in pure Perl). 1231The right fix for this is to use a proper JSON parser in your javascript
1232programs, and not rely on C<eval> (see for example Douglas Crockford's
1233F<json2.js> parser).
948 1234
949Undocumented/buggy Unicode handling (how JSON handles Unicode values is 1235If this is not an option, you can, as a stop-gap measure, simply encode to
950undocumented. One can get far by feeding it Unicode strings and doing 1236ASCII-only JSON:
951en-/decoding oneself, but Unicode escapes are not working properly).
952 1237
953No round-tripping (strings get clobbered if they look like numbers, e.g. 1238 use JSON::XS;
954the string C<2.0> will encode to C<2.0> instead of C<"2.0">, and that will
955decode into the number 2.
956 1239
957=item JSON::PC 0.01 1240 print JSON::XS->new->ascii->encode ([chr 0x2028]);
958 1241
959Very fast. 1242Note that this will enlarge the resulting JSON text quite a bit if you
1243have many non-ASCII characters. You might be tempted to run some regexes
1244to only escape U+2028 and U+2029, e.g.:
960 1245
961Undocumented/buggy Unicode handling. 1246 # DO NOT USE THIS!
1247 my $json = JSON::XS->new->utf8->encode ([chr 0x2028]);
1248 $json =~ s/\xe2\x80\xa8/\\u2028/g; # escape U+2028
1249 $json =~ s/\xe2\x80\xa9/\\u2029/g; # escape U+2029
1250 print $json;
962 1251
963No round-tripping. 1252Note that I<this is a bad idea>: the above only works for U+2028 and
1253U+2029 and thus only for fully ECMAscript-compliant parsers. Many existing
1254javascript implementations, however, have issues with other characters as
1255well - using C<eval> naively simply I<will> cause problems.
964 1256
965Has problems handling many Perl values (e.g. regex results and other magic 1257Another problem is that some javascript implementations reserve
966values will make it croak). 1258some property names for their own purposes (which probably makes
1259them non-ECMAscript-compliant). For example, Iceweasel reserves the
1260C<__proto__> property name for its own purposes.
967 1261
968Does not even generate valid JSON (C<{1,2}> gets converted to C<{1:2}> 1262If that is a problem, you could parse try to filter the resulting JSON
969which is not a valid JSON text. 1263output for these property strings, e.g.:
970 1264
971Unmaintained (maintainer unresponsive for many months, bugs are not 1265 $json =~ s/"__proto__"\s*:/"__proto__renamed":/g;
972getting fixed).
973 1266
974=item JSON::Syck 0.21 1267This works because C<__proto__> is not valid outside of strings, so every
1268occurence of C<"__proto__"\s*:> must be a string used as property name.
975 1269
976Very buggy (often crashes). 1270If you know of other incompatibilities, please let me know.
977
978Very inflexible (no human-readable format supported, format pretty much
979undocumented. I need at least a format for easy reading by humans and a
980single-line compact format for use in a protocol, and preferably a way to
981generate ASCII-only JSON texts).
982
983Completely broken (and confusingly documented) Unicode handling (Unicode
984escapes are not working properly, you need to set ImplicitUnicode to
985I<different> values on en- and decoding to get symmetric behaviour).
986
987No round-tripping (simple cases work, but this depends on whether the scalar
988value was used in a numeric context or not).
989
990Dumping hashes may skip hash values depending on iterator state.
991
992Unmaintained (maintainer unresponsive for many months, bugs are not
993getting fixed).
994
995Does not check input for validity (i.e. will accept non-JSON input and
996return "something" instead of raising an exception. This is a security
997issue: imagine two banks transferring money between each other using
998JSON. One bank might parse a given non-JSON request and deduct money,
999while the other might reject the transaction with a syntax error. While a
1000good protocol will at least recover, that is extra unnecessary work and
1001the transaction will still not succeed).
1002
1003=item JSON::DWIW 0.04
1004
1005Very fast. Very natural. Very nice.
1006
1007Undocumented Unicode handling (but the best of the pack. Unicode escapes
1008still don't get parsed properly).
1009
1010Very inflexible.
1011
1012No round-tripping.
1013
1014Does not generate valid JSON texts (key strings are often unquoted, empty keys
1015result in nothing being output)
1016
1017Does not check input for validity.
1018
1019=back
1020 1271
1021 1272
1022=head2 JSON and YAML 1273=head2 JSON and YAML
1023 1274
1024You often hear that JSON is a subset of YAML. This is, however, a mass 1275You often hear that JSON is a subset of YAML. This is, however, a mass
1025hysteria(*) and very far from the truth. In general, there is no way to 1276hysteria(*) and very far from the truth (as of the time of this writing),
1277so let me state it clearly: I<in general, there is no way to configure
1026configure JSON::XS to output a data structure as valid YAML that works for 1278JSON::XS to output a data structure as valid YAML> that works in all
1027all cases. 1279cases.
1028 1280
1029If you really must use JSON::XS to generate YAML, you should use this 1281If you really must use JSON::XS to generate YAML, you should use this
1030algorithm (subject to change in future versions): 1282algorithm (subject to change in future versions):
1031 1283
1032 my $to_yaml = JSON::XS->new->utf8->space_after (1); 1284 my $to_yaml = JSON::XS->new->utf8->space_after (1);
1033 my $yaml = $to_yaml->encode ($ref) . "\n"; 1285 my $yaml = $to_yaml->encode ($ref) . "\n";
1034 1286
1035This will I<usually> generate JSON texts that also parse as valid 1287This will I<usually> generate JSON texts that also parse as valid
1036YAML. Please note that YAML has hardcoded limits on (simple) object key 1288YAML. Please note that YAML has hardcoded limits on (simple) object key
1037lengths that JSON doesn't have and also has different and incompatible 1289lengths that JSON doesn't have and also has different and incompatible
1038unicode handling, so you should make sure that your hash keys are 1290unicode character escape syntax, so you should make sure that your hash
1039noticeably shorter than the 1024 "stream characters" YAML allows and that 1291keys are noticeably shorter than the 1024 "stream characters" YAML allows
1040you do not have codepoints with values outside the Unicode BMP (basic 1292and that you do not have characters with codepoint values outside the
1041multilingual page). YAML also does not allow C<\/> sequences in strings 1293Unicode BMP (basic multilingual page). YAML also does not allow C<\/>
1042(which JSON::XS does not I<currently> generate). 1294sequences in strings (which JSON::XS does not I<currently> generate, but
1295other JSON generators might).
1043 1296
1044There might be other incompatibilities that I am not aware of (or the YAML 1297There might be other incompatibilities that I am not aware of (or the YAML
1045specification has been changed yet again - it does so quite often). In 1298specification has been changed yet again - it does so quite often). In
1046general you should not try to generate YAML with a JSON generator or vice 1299general you should not try to generate YAML with a JSON generator or vice
1047versa, or try to parse JSON with a YAML parser or vice versa: chances are 1300versa, or try to parse JSON with a YAML parser or vice versa: chances are
1050 1303
1051=over 4 1304=over 4
1052 1305
1053=item (*) 1306=item (*)
1054 1307
1055This is spread actively by the YAML team, however. For many years now they 1308I have been pressured multiple times by Brian Ingerson (one of the
1056claim YAML were a superset of JSON, even when proven otherwise. 1309authors of the YAML specification) to remove this paragraph, despite him
1310acknowledging that the actual incompatibilities exist. As I was personally
1311bitten by this "JSON is YAML" lie, I refused and said I will continue to
1312educate people about these issues, so others do not run into the same
1313problem again and again. After this, Brian called me a (quote)I<complete
1314and worthless idiot>(unquote).
1057 1315
1058Even the author of this manpage was at some point accused of providing 1316In my opinion, instead of pressuring and insulting people who actually
1059"incorrect" information, despite the evidence presented (claims ranged 1317clarify issues with YAML and the wrong statements of some of its
1060from "your documentation contains inaccurate and negative statements about 1318proponents, I would kindly suggest reading the JSON spec (which is not
1061YAML" (the only negative comment is this footnote, and it didn't exist 1319that difficult or long) and finally make YAML compatible to it, and
1062back then; the question on which claims were inaccurate was never answered 1320educating users about the changes, instead of spreading lies about the
1063etc.) to "the YAML spec is not up-to-date" (the *real* and supposedly 1321real compatibility for many I<years> and trying to silence people who
1064JSON-compatible spec is apparently not currently publicly available) 1322point out that it isn't true.
1065to actual requests to replace this section by *incorrect* information,
1066suppressing information about the real problem).
1067 1323
1068So whenever you are told that YAML was a superset of JSON, first check 1324Addendum/2009: the YAML 1.2 spec is still incomaptible with JSON, even
1069wether it is really true (it might be when you check it, but it certainly 1325though the incompatibilities have been documented (and are known to
1070was not true when this was written). I would much prefer if the YAML team 1326Brian) for many years and the spec makes explicit claims that YAML is a
1071would spent their time on actually making JSON compatibility a truth 1327superset of JSON. It would be so easy to fix, but apparently, bullying and
1072(JSON, after all, has a very small and simple specification) instead of 1328corrupting userdata is so much easier.
1073trying to lobby/force people into reporting untruths.
1074 1329
1075=back 1330=back
1076 1331
1077 1332
1078=head2 SPEED 1333=head2 SPEED
1080It seems that JSON::XS is surprisingly fast, as shown in the following 1335It seems that JSON::XS is surprisingly fast, as shown in the following
1081tables. They have been generated with the help of the C<eg/bench> program 1336tables. They have been generated with the help of the C<eg/bench> program
1082in the JSON::XS distribution, to make it easy to compare on your own 1337in the JSON::XS distribution, to make it easy to compare on your own
1083system. 1338system.
1084 1339
1085First comes a comparison between various modules using a very short 1340First comes a comparison between various modules using
1086single-line JSON string: 1341a very short single-line JSON string (also available at
1342L<http://dist.schmorp.de/misc/json/short.json>).
1087 1343
1088 {"method": "handleMessage", "params": ["user1", "we were just talking"], \ 1344 {"method": "handleMessage", "params": ["user1",
1089 "id": null, "array":[1,11,234,-5,1e5,1e7, true, false]} 1345 "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
1346 1, 0]}
1090 1347
1091It shows the number of encodes/decodes per second (JSON::XS uses 1348It shows the number of encodes/decodes per second (JSON::XS uses
1092the functional interface, while JSON::XS/2 uses the OO interface 1349the functional interface, while JSON::XS/2 uses the OO interface
1093with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables 1350with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
1094shrink). Higher is better: 1351shrink. JSON::DWIW/DS uses the deserialise function, while JSON::DWIW::FJ
1352uses the from_json method). Higher is better:
1095 1353
1096 module | encode | decode | 1354 module | encode | decode |
1097 -----------|------------|------------| 1355 --------------|------------|------------|
1098 JSON 1.x | 4990.842 | 4088.813 | 1356 JSON::DWIW/DS | 86302.551 | 102300.098 |
1099 JSON::DWIW | 51653.990 | 71575.154 | 1357 JSON::DWIW/FJ | 86302.551 | 75983.768 |
1100 JSON::PC | 65948.176 | 74631.744 | 1358 JSON::PP | 15827.562 | 6638.658 |
1101 JSON::PP | 8931.652 | 3817.168 | 1359 JSON::Syck | 63358.066 | 47662.545 |
1102 JSON::Syck | 24877.248 | 27776.848 | 1360 JSON::XS | 511500.488 | 511500.488 |
1103 JSON::XS | 388361.481 | 227951.304 | 1361 JSON::XS/2 | 291271.111 | 388361.481 |
1104 JSON::XS/2 | 227951.304 | 218453.333 | 1362 JSON::XS/3 | 361577.931 | 361577.931 |
1105 JSON::XS/3 | 338250.323 | 218453.333 | 1363 Storable | 66788.280 | 265462.278 |
1106 Storable | 16500.016 | 135300.129 |
1107 -----------+------------+------------+ 1364 --------------+------------+------------+
1108 1365
1109That is, JSON::XS is about five times faster than JSON::DWIW on encoding, 1366That is, JSON::XS is almost six times faster than JSON::DWIW on encoding,
1110about three times faster on decoding, and over forty times faster 1367about five times faster on decoding, and over thirty to seventy times
1111than JSON, even with pretty-printing and key sorting. It also compares 1368faster than JSON's pure perl implementation. It also compares favourably
1112favourably to Storable for small amounts of data. 1369to Storable for small amounts of data.
1113 1370
1114Using a longer test string (roughly 18KB, generated from Yahoo! Locals 1371Using a longer test string (roughly 18KB, generated from Yahoo! Locals
1115search API (http://nanoref.com/yahooapis/mgPdGg): 1372search API (L<http://dist.schmorp.de/misc/json/long.json>).
1116 1373
1117 module | encode | decode | 1374 module | encode | decode |
1118 -----------|------------|------------| 1375 --------------|------------|------------|
1119 JSON 1.x | 55.260 | 34.971 | 1376 JSON::DWIW/DS | 1647.927 | 2673.916 |
1120 JSON::DWIW | 825.228 | 1082.513 | 1377 JSON::DWIW/FJ | 1630.249 | 2596.128 |
1121 JSON::PC | 3571.444 | 2394.829 |
1122 JSON::PP | 210.987 | 32.574 | 1378 JSON::PP | 400.640 | 62.311 |
1123 JSON::Syck | 552.551 | 787.544 | 1379 JSON::Syck | 1481.040 | 1524.869 |
1124 JSON::XS | 5780.463 | 4854.519 | 1380 JSON::XS | 20661.596 | 9541.183 |
1125 JSON::XS/2 | 3869.998 | 4798.975 | 1381 JSON::XS/2 | 10683.403 | 9416.938 |
1126 JSON::XS/3 | 5862.880 | 4798.975 | 1382 JSON::XS/3 | 20661.596 | 9400.054 |
1127 Storable | 4445.002 | 5235.027 | 1383 Storable | 19765.806 | 10000.725 |
1128 -----------+------------+------------+ 1384 --------------+------------+------------+
1129 1385
1130Again, JSON::XS leads by far (except for Storable which non-surprisingly 1386Again, JSON::XS leads by far (except for Storable which non-surprisingly
1131decodes faster). 1387decodes a bit faster).
1132 1388
1133On large strings containing lots of high Unicode characters, some modules 1389On large strings containing lots of high Unicode characters, some modules
1134(such as JSON::PC) seem to decode faster than JSON::XS, but the result 1390(such as JSON::PC) seem to decode faster than JSON::XS, but the result
1135will be broken due to missing (or wrong) Unicode handling. Others refuse 1391will be broken due to missing (or wrong) Unicode handling. Others refuse
1136to decode or encode properly, so it was impossible to prepare a fair 1392to decode or encode properly, so it was impossible to prepare a fair
1162to free the temporary). If that is exceeded, the program crashes. To be 1418to free the temporary). If that is exceeded, the program crashes. To be
1163conservative, the default nesting limit is set to 512. If your process 1419conservative, the default nesting limit is set to 512. If your process
1164has a smaller stack, you should adjust this setting accordingly with the 1420has a smaller stack, you should adjust this setting accordingly with the
1165C<max_depth> method. 1421C<max_depth> method.
1166 1422
1167And last but least, something else could bomb you that I forgot to think 1423Something else could bomb you, too, that I forgot to think of. In that
1168of. In that case, you get to keep the pieces. I am always open for hints, 1424case, you get to keep the pieces. I am always open for hints, though...
1169though... 1425
1426Also keep in mind that JSON::XS might leak contents of your Perl data
1427structures in its error messages, so when you serialise sensitive
1428information you might want to make sure that exceptions thrown by JSON::XS
1429will not end up in front of untrusted eyes.
1170 1430
1171If you are using JSON::XS to return packets to consumption 1431If you are using JSON::XS to return packets to consumption
1172by JavaScript scripts in a browser you should have a look at 1432by JavaScript scripts in a browser you should have a look at
1173L<http://jpsykes.com/47/practical-csrf-and-json-security> to see whether 1433L<http://blog.archive.jpsykes.com/47/practical-csrf-and-json-security/> to
1174you are vulnerable to some common attack vectors (which really are browser 1434see whether you are vulnerable to some common attack vectors (which really
1175design bugs, but it is still you who will have to deal with it, as major 1435are browser design bugs, but it is still you who will have to deal with
1176browser developers care only for features, not about getting security 1436it, as major browser developers care only for features, not about getting
1177right). 1437security right).
1178 1438
1179 1439
1180=head1 THREADS 1440=head1 THREADS
1181 1441
1182This module is I<not> guaranteed to be thread safe and there are no 1442This module is I<not> guaranteed to be thread safe and there are no
1183plans to change this until Perl gets thread support (as opposed to the 1443plans to change this until Perl gets thread support (as opposed to the
1184horribly slow so-called "threads" which are simply slow and bloated 1444horribly slow so-called "threads" which are simply slow and bloated
1185process simulations - use fork, its I<much> faster, cheaper, better). 1445process simulations - use fork, it's I<much> faster, cheaper, better).
1186 1446
1187(It might actually work, but you have been warned). 1447(It might actually work, but you have been warned).
1188 1448
1189 1449
1190=head1 BUGS 1450=head1 BUGS
1191 1451
1192While the goal of this module is to be correct, that unfortunately does 1452While the goal of this module is to be correct, that unfortunately does
1193not mean its bug-free, only that I think its design is bug-free. It is 1453not mean it's bug-free, only that I think its design is bug-free. If you
1194still relatively early in its development. If you keep reporting bugs they 1454keep reporting bugs they will be fixed swiftly, though.
1195will be fixed swiftly, though.
1196 1455
1197Please refrain from using rt.cpan.org or any other bug reporting 1456Please refrain from using rt.cpan.org or any other bug reporting
1198service. I put the contact address into my modules for a reason. 1457service. I put the contact address into my modules for a reason.
1199 1458
1200=cut 1459=cut
1220 "--" => sub { $_[0] = ${$_[0]} - 1 }, 1479 "--" => sub { $_[0] = ${$_[0]} - 1 },
1221 fallback => 1; 1480 fallback => 1;
1222 1481
12231; 14821;
1224 1483
1484=head1 SEE ALSO
1485
1486The F<json_xs> command line utility for quick experiments.
1487
1225=head1 AUTHOR 1488=head1 AUTHOR
1226 1489
1227 Marc Lehmann <schmorp@schmorp.de> 1490 Marc Lehmann <schmorp@schmorp.de>
1228 http://home.schmorp.de/ 1491 http://home.schmorp.de/
1229 1492

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines