ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.89 by root, Wed Mar 19 13:54:53 2008 UTC vs.
Revision 1.121 by root, Mon Jul 13 22:13:17 2009 UTC

1=head1 NAME
2
3JSON::XS - JSON serialising/deserialising, done correctly and fast
4
1=encoding utf-8 5=encoding utf-8
2
3=head1 NAME
4
5JSON::XS - JSON serialising/deserialising, done correctly and fast
6 6
7JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ 7JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ
8 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html) 8 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html)
9 9
10=head1 SYNOPSIS 10=head1 SYNOPSIS
37primary goal is to be I<correct> and its secondary goal is to be 37primary goal is to be I<correct> and its secondary goal is to be
38I<fast>. To reach the latter goal it was written in C. 38I<fast>. To reach the latter goal it was written in C.
39 39
40Beginning with version 2.0 of the JSON module, when both JSON and 40Beginning with version 2.0 of the JSON module, when both JSON and
41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be 41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42overriden) with no overhead due to emulation (by inheritign constructor 42overridden) with no overhead due to emulation (by inheriting constructor
43and methods). If JSON::XS is not available, it will fall back to the 43and methods). If JSON::XS is not available, it will fall back to the
44compatible JSON::PP module as backend, so using JSON instead of JSON::XS 44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45gives you a portable JSON API that can be fast when you need and doesn't 45gives you a portable JSON API that can be fast when you need and doesn't
46require a C compiler when that is a problem. 46require a C compiler when that is a problem.
47 47
49to write yet another JSON module? While it seems there are many JSON 49to write yet another JSON module? While it seems there are many JSON
50modules, none of them correctly handle all corner cases, and in most cases 50modules, none of them correctly handle all corner cases, and in most cases
51their maintainers are unresponsive, gone missing, or not listening to bug 51their maintainers are unresponsive, gone missing, or not listening to bug
52reports for other reasons. 52reports for other reasons.
53 53
54See COMPARISON, below, for a comparison to some other JSON modules.
55
56See MAPPING, below, on how JSON::XS maps perl values to JSON values and 54See MAPPING, below, on how JSON::XS maps perl values to JSON values and
57vice versa. 55vice versa.
58 56
59=head2 FEATURES 57=head2 FEATURES
60 58
65This module knows how to handle Unicode, documents how and when it does 63This module knows how to handle Unicode, documents how and when it does
66so, and even documents what "correct" means. 64so, and even documents what "correct" means.
67 65
68=item * round-trip integrity 66=item * round-trip integrity
69 67
70When you serialise a perl data structure using only datatypes supported 68When you serialise a perl data structure using only data types supported
71by JSON, the deserialised data structure is identical on the Perl level. 69by JSON, the deserialised data structure is identical on the Perl level.
72(e.g. the string "2.0" doesn't suddenly become "2" just because it looks 70(e.g. the string "2.0" doesn't suddenly become "2" just because it looks
73like a number). There minor I<are> exceptions to this, read the MAPPING 71like a number). There minor I<are> exceptions to this, read the MAPPING
74section below to learn about those. 72section below to learn about those.
75 73
84Compared to other JSON modules and other serialisers such as Storable, 82Compared to other JSON modules and other serialisers such as Storable,
85this module usually compares favourably in terms of speed, too. 83this module usually compares favourably in terms of speed, too.
86 84
87=item * simple to use 85=item * simple to use
88 86
89This module has both a simple functional interface as well as an objetc 87This module has both a simple functional interface as well as an object
90oriented interface interface. 88oriented interface interface.
91 89
92=item * reasonably versatile output formats 90=item * reasonably versatile output formats
93 91
94You can choose between the most compact guaranteed-single-line format 92You can choose between the most compact guaranteed-single-line format
95possible (nice for simple line-based protocols), a pure-ascii format 93possible (nice for simple line-based protocols), a pure-ASCII format
96(for when your transport is not 8-bit clean, still supports the whole 94(for when your transport is not 8-bit clean, still supports the whole
97Unicode range), or a pretty-printed format (for when you want to read that 95Unicode range), or a pretty-printed format (for when you want to read that
98stuff). Or you can combine those features in whatever way you like. 96stuff). Or you can combine those features in whatever way you like.
99 97
100=back 98=back
101 99
102=cut 100=cut
103 101
104package JSON::XS; 102package JSON::XS;
105 103
106use strict; 104use common::sense;
107 105
108our $VERSION = '2.01'; 106our $VERSION = '2.24';
109our @ISA = qw(Exporter); 107our @ISA = qw(Exporter);
110 108
111our @EXPORT = qw(encode_json decode_json to_json from_json); 109our @EXPORT = qw(encode_json decode_json to_json from_json);
112 110
113sub to_json($) { 111sub to_json($) {
137 135
138This function call is functionally identical to: 136This function call is functionally identical to:
139 137
140 $json_text = JSON::XS->new->utf8->encode ($perl_scalar) 138 $json_text = JSON::XS->new->utf8->encode ($perl_scalar)
141 139
142except being faster. 140Except being faster.
143 141
144=item $perl_scalar = decode_json $json_text 142=item $perl_scalar = decode_json $json_text
145 143
146The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries 144The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries
147to parse that as an UTF-8 encoded JSON text, returning the resulting 145to parse that as an UTF-8 encoded JSON text, returning the resulting
149 147
150This function call is functionally identical to: 148This function call is functionally identical to:
151 149
152 $perl_scalar = JSON::XS->new->utf8->decode ($json_text) 150 $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
153 151
154except being faster. 152Except being faster.
155 153
156=item $is_boolean = JSON::XS::is_bool $scalar 154=item $is_boolean = JSON::XS::is_bool $scalar
157 155
158Returns true if the passed scalar represents either JSON::XS::true or 156Returns true if the passed scalar represents either JSON::XS::true or
159JSON::XS::false, two constants that act like C<1> and C<0>, respectively 157JSON::XS::false, two constants that act like C<1> and C<0>, respectively
197 195
198If you didn't know about that flag, just the better, pretend it doesn't 196If you didn't know about that flag, just the better, pretend it doesn't
199exist. 197exist.
200 198
201=item 4. A "Unicode String" is simply a string where each character can be 199=item 4. A "Unicode String" is simply a string where each character can be
202validly interpreted as a Unicode codepoint. 200validly interpreted as a Unicode code point.
203 201
204If you have UTF-8 encoded data, it is no longer a Unicode string, but a 202If you have UTF-8 encoded data, it is no longer a Unicode string, but a
205Unicode string encoded in UTF-8, giving you a binary string. 203Unicode string encoded in UTF-8, giving you a binary string.
206 204
207=item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string. 205=item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string.
245 243
246If C<$enable> is false, then the C<encode> method will not escape Unicode 244If C<$enable> is false, then the C<encode> method will not escape Unicode
247characters unless required by the JSON syntax or other flags. This results 245characters unless required by the JSON syntax or other flags. This results
248in a faster and more compact format. 246in a faster and more compact format.
249 247
248See also the section I<ENCODING/CODESET FLAG NOTES> later in this
249document.
250
250The main use for this flag is to produce JSON texts that can be 251The main use for this flag is to produce JSON texts that can be
251transmitted over a 7-bit channel, as the encoded JSON texts will not 252transmitted over a 7-bit channel, as the encoded JSON texts will not
252contain any 8 bit characters. 253contain any 8 bit characters.
253 254
254 JSON::XS->new->ascii (1)->encode ([chr 0x10401]) 255 JSON::XS->new->ascii (1)->encode ([chr 0x10401])
265will not be affected in any way by this flag, as C<decode> by default 266will not be affected in any way by this flag, as C<decode> by default
266expects Unicode, which is a strict superset of latin1. 267expects Unicode, which is a strict superset of latin1.
267 268
268If C<$enable> is false, then the C<encode> method will not escape Unicode 269If C<$enable> is false, then the C<encode> method will not escape Unicode
269characters unless required by the JSON syntax or other flags. 270characters unless required by the JSON syntax or other flags.
271
272See also the section I<ENCODING/CODESET FLAG NOTES> later in this
273document.
270 274
271The main use for this flag is efficiently encoding binary data as JSON 275The main use for this flag is efficiently encoding binary data as JSON
272text, as most octets will not be escaped, resulting in a smaller encoded 276text, as most octets will not be escaped, resulting in a smaller encoded
273size. The disadvantage is that the resulting JSON text is encoded 277size. The disadvantage is that the resulting JSON text is encoded
274in latin1 (and must correctly be treated as such when storing and 278in latin1 (and must correctly be treated as such when storing and
293 297
294If C<$enable> is false, then the C<encode> method will return the JSON 298If C<$enable> is false, then the C<encode> method will return the JSON
295string as a (non-encoded) Unicode string, while C<decode> expects thus a 299string as a (non-encoded) Unicode string, while C<decode> expects thus a
296Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs 300Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs
297to be done yourself, e.g. using the Encode module. 301to be done yourself, e.g. using the Encode module.
302
303See also the section I<ENCODING/CODESET FLAG NOTES> later in this
304document.
298 305
299Example, output UTF-16BE-encoded JSON: 306Example, output UTF-16BE-encoded JSON:
300 307
301 use Encode; 308 use Encode;
302 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object); 309 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object);
453Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>, 460Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>,
454resulting in an invalid JSON text: 461resulting in an invalid JSON text:
455 462
456 JSON::XS->new->allow_nonref->encode ("Hello, World!") 463 JSON::XS->new->allow_nonref->encode ("Hello, World!")
457 => "Hello, World!" 464 => "Hello, World!"
465
466=item $json = $json->allow_unknown ([$enable])
467
468=item $enabled = $json->get_allow_unknown
469
470If C<$enable> is true (or missing), then C<encode> will I<not> throw an
471exception when it encounters values it cannot represent in JSON (for
472example, filehandles) but instead will encode a JSON C<null> value. Note
473that blessed objects are not included here and are handled separately by
474c<allow_nonref>.
475
476If C<$enable> is false (the default), then C<encode> will throw an
477exception when it encounters anything it cannot encode as JSON.
478
479This option does not affect C<decode> in any way, and it is recommended to
480leave it off unless you know your communications partner.
458 481
459=item $json = $json->allow_blessed ([$enable]) 482=item $json = $json->allow_blessed ([$enable])
460 483
461=item $enabled = $json->get_allow_blessed 484=item $enabled = $json->get_allow_blessed
462 485
603=item $json = $json->max_depth ([$maximum_nesting_depth]) 626=item $json = $json->max_depth ([$maximum_nesting_depth])
604 627
605=item $max_depth = $json->get_max_depth 628=item $max_depth = $json->get_max_depth
606 629
607Sets the maximum nesting level (default C<512>) accepted while encoding 630Sets the maximum nesting level (default C<512>) accepted while encoding
608or decoding. If the JSON text or Perl data structure has an equal or 631or decoding. If a higher nesting level is detected in JSON text or a Perl
609higher nesting level then this limit, then the encoder and decoder will 632data structure, then the encoder and decoder will stop and croak at that
610stop and croak at that point. 633point.
611 634
612Nesting level is defined by number of hash- or arrayrefs that the encoder 635Nesting level is defined by number of hash- or arrayrefs that the encoder
613needs to traverse to reach a given point or the number of C<{> or C<[> 636needs to traverse to reach a given point or the number of C<{> or C<[>
614characters without their matching closing parenthesis crossed to reach a 637characters without their matching closing parenthesis crossed to reach a
615given character in a string. 638given character in a string.
616 639
617Setting the maximum depth to one disallows any nesting, so that ensures 640Setting the maximum depth to one disallows any nesting, so that ensures
618that the object is only a single hash/object or array. 641that the object is only a single hash/object or array.
619 642
620The argument to C<max_depth> will be rounded up to the next highest power
621of two. If no argument is given, the highest possible setting will be 643If no argument is given, the highest possible setting will be used, which
622used, which is rarely useful. 644is rarely useful.
645
646Note that nesting is implemented by recursion in C. The default value has
647been chosen to be as large as typical operating systems allow without
648crashing.
623 649
624See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 650See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
625 651
626=item $json = $json->max_size ([$maximum_string_size]) 652=item $json = $json->max_size ([$maximum_string_size])
627 653
628=item $max_size = $json->get_max_size 654=item $max_size = $json->get_max_size
629 655
630Set the maximum length a JSON text may have (in bytes) where decoding is 656Set the maximum length a JSON text may have (in bytes) where decoding is
631being attempted. The default is C<0>, meaning no limit. When C<decode> 657being attempted. The default is C<0>, meaning no limit. When C<decode>
632is called on a string longer then this number of characters it will not 658is called on a string that is longer then this many bytes, it will not
633attempt to decode the string but throw an exception. This setting has no 659attempt to decode the string but throw an exception. This setting has no
634effect on C<encode> (yet). 660effect on C<encode> (yet).
635 661
636The argument to C<max_size> will be rounded up to the next B<highest> 662If no argument is given, the limit check will be deactivated (same as when
637power of two (so may be more than requested). If no argument is given, the 663C<0> is specified).
638limit check will be deactivated (same as when C<0> is specified).
639 664
640See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 665See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
641 666
642=item $json_text = $json->encode ($perl_scalar) 667=item $json_text = $json->encode ($perl_scalar)
643 668
670 695
671 JSON::XS->new->decode_prefix ("[1] the tail") 696 JSON::XS->new->decode_prefix ("[1] the tail")
672 => ([], 3) 697 => ([], 3)
673 698
674=back 699=back
700
701
702=head1 INCREMENTAL PARSING
703
704In some cases, there is the need for incremental parsing of JSON
705texts. While this module always has to keep both JSON text and resulting
706Perl data structure in memory at one time, it does allow you to parse a
707JSON stream incrementally. It does so by accumulating text until it has
708a full JSON object, which it then can decode. This process is similar to
709using C<decode_prefix> to see if a full JSON object is available, but
710is much more efficient (and can be implemented with a minimum of method
711calls).
712
713JSON::XS will only attempt to parse the JSON text once it is sure it
714has enough text to get a decisive result, using a very simple but
715truly incremental parser. This means that it sometimes won't stop as
716early as the full parser, for example, it doesn't detect parenthese
717mismatches. The only thing it guarantees is that it starts decoding as
718soon as a syntactically valid JSON text has been seen. This means you need
719to set resource limits (e.g. C<max_size>) to ensure the parser will stop
720parsing in the presence if syntax errors.
721
722The following methods implement this incremental parser.
723
724=over 4
725
726=item [void, scalar or list context] = $json->incr_parse ([$string])
727
728This is the central parsing function. It can both append new text and
729extract objects from the stream accumulated so far (both of these
730functions are optional).
731
732If C<$string> is given, then this string is appended to the already
733existing JSON fragment stored in the C<$json> object.
734
735After that, if the function is called in void context, it will simply
736return without doing anything further. This can be used to add more text
737in as many chunks as you want.
738
739If the method is called in scalar context, then it will try to extract
740exactly I<one> JSON object. If that is successful, it will return this
741object, otherwise it will return C<undef>. If there is a parse error,
742this method will croak just as C<decode> would do (one can then use
743C<incr_skip> to skip the errornous part). This is the most common way of
744using the method.
745
746And finally, in list context, it will try to extract as many objects
747from the stream as it can find and return them, or the empty list
748otherwise. For this to work, there must be no separators between the JSON
749objects or arrays, instead they must be concatenated back-to-back. If
750an error occurs, an exception will be raised as in the scalar context
751case. Note that in this case, any previously-parsed JSON texts will be
752lost.
753
754=item $lvalue_string = $json->incr_text
755
756This method returns the currently stored JSON fragment as an lvalue, that
757is, you can manipulate it. This I<only> works when a preceding call to
758C<incr_parse> in I<scalar context> successfully returned an object. Under
759all other circumstances you must not call this function (I mean it.
760although in simple tests it might actually work, it I<will> fail under
761real world conditions). As a special exception, you can also call this
762method before having parsed anything.
763
764This function is useful in two cases: a) finding the trailing text after a
765JSON object or b) parsing multiple JSON objects separated by non-JSON text
766(such as commas).
767
768=item $json->incr_skip
769
770This will reset the state of the incremental parser and will remove
771the parsed text from the input buffer so far. This is useful after
772C<incr_parse> died, in which case the input buffer and incremental parser
773state is left unchanged, to skip the text parsed so far and to reset the
774parse state.
775
776The difference to C<incr_reset> is that only text until the parse error
777occured is removed.
778
779=item $json->incr_reset
780
781This completely resets the incremental parser, that is, after this call,
782it will be as if the parser had never parsed anything.
783
784This is useful if you want to repeatedly parse JSON objects and want to
785ignore any trailing data, which means you have to reset the parser after
786each successful decode.
787
788=back
789
790=head2 LIMITATIONS
791
792All options that affect decoding are supported, except
793C<allow_nonref>. The reason for this is that it cannot be made to
794work sensibly: JSON objects and arrays are self-delimited, i.e. you can concatenate
795them back to back and still decode them perfectly. This does not hold true
796for JSON numbers, however.
797
798For example, is the string C<1> a single JSON number, or is it simply the
799start of C<12>? Or is C<12> a single JSON number, or the concatenation
800of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
801takes the conservative route and disallows this case.
802
803=head2 EXAMPLES
804
805Some examples will make all this clearer. First, a simple example that
806works similarly to C<decode_prefix>: We want to decode the JSON object at
807the start of a string and identify the portion after the JSON object:
808
809 my $text = "[1,2,3] hello";
810
811 my $json = new JSON::XS;
812
813 my $obj = $json->incr_parse ($text)
814 or die "expected JSON object or array at beginning of string";
815
816 my $tail = $json->incr_text;
817 # $tail now contains " hello"
818
819Easy, isn't it?
820
821Now for a more complicated example: Imagine a hypothetical protocol where
822you read some requests from a TCP stream, and each request is a JSON
823array, without any separation between them (in fact, it is often useful to
824use newlines as "separators", as these get interpreted as whitespace at
825the start of the JSON text, which makes it possible to test said protocol
826with C<telnet>...).
827
828Here is how you'd do it (it is trivial to write this in an event-based
829manner):
830
831 my $json = new JSON::XS;
832
833 # read some data from the socket
834 while (sysread $socket, my $buf, 4096) {
835
836 # split and decode as many requests as possible
837 for my $request ($json->incr_parse ($buf)) {
838 # act on the $request
839 }
840 }
841
842Another complicated example: Assume you have a string with JSON objects
843or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
844[3]>). To parse them, we have to skip the commas between the JSON texts,
845and here is where the lvalue-ness of C<incr_text> comes in useful:
846
847 my $text = "[1],[2], [3]";
848 my $json = new JSON::XS;
849
850 # void context, so no parsing done
851 $json->incr_parse ($text);
852
853 # now extract as many objects as possible. note the
854 # use of scalar context so incr_text can be called.
855 while (my $obj = $json->incr_parse) {
856 # do something with $obj
857
858 # now skip the optional comma
859 $json->incr_text =~ s/^ \s* , //x;
860 }
861
862Now lets go for a very complex example: Assume that you have a gigantic
863JSON array-of-objects, many gigabytes in size, and you want to parse it,
864but you cannot load it into memory fully (this has actually happened in
865the real world :).
866
867Well, you lost, you have to implement your own JSON parser. But JSON::XS
868can still help you: You implement a (very simple) array parser and let
869JSON decode the array elements, which are all full JSON objects on their
870own (this wouldn't work if the array elements could be JSON numbers, for
871example):
872
873 my $json = new JSON::XS;
874
875 # open the monster
876 open my $fh, "<bigfile.json"
877 or die "bigfile: $!";
878
879 # first parse the initial "["
880 for (;;) {
881 sysread $fh, my $buf, 65536
882 or die "read error: $!";
883 $json->incr_parse ($buf); # void context, so no parsing
884
885 # Exit the loop once we found and removed(!) the initial "[".
886 # In essence, we are (ab-)using the $json object as a simple scalar
887 # we append data to.
888 last if $json->incr_text =~ s/^ \s* \[ //x;
889 }
890
891 # now we have the skipped the initial "[", so continue
892 # parsing all the elements.
893 for (;;) {
894 # in this loop we read data until we got a single JSON object
895 for (;;) {
896 if (my $obj = $json->incr_parse) {
897 # do something with $obj
898 last;
899 }
900
901 # add more data
902 sysread $fh, my $buf, 65536
903 or die "read error: $!";
904 $json->incr_parse ($buf); # void context, so no parsing
905 }
906
907 # in this loop we read data until we either found and parsed the
908 # separating "," between elements, or the final "]"
909 for (;;) {
910 # first skip whitespace
911 $json->incr_text =~ s/^\s*//;
912
913 # if we find "]", we are done
914 if ($json->incr_text =~ s/^\]//) {
915 print "finished.\n";
916 exit;
917 }
918
919 # if we find ",", we can continue with the next element
920 if ($json->incr_text =~ s/^,//) {
921 last;
922 }
923
924 # if we find anything else, we have a parse error!
925 if (length $json->incr_text) {
926 die "parse error near ", $json->incr_text;
927 }
928
929 # else add more data
930 sysread $fh, my $buf, 65536
931 or die "read error: $!";
932 $json->incr_parse ($buf); # void context, so no parsing
933 }
934
935This is a complex example, but most of the complexity comes from the fact
936that we are trying to be correct (bear with me if I am wrong, I never ran
937the above example :).
938
675 939
676 940
677=head1 MAPPING 941=head1 MAPPING
678 942
679This section describes how JSON::XS maps Perl values to JSON values and 943This section describes how JSON::XS maps Perl values to JSON values and
768Other unblessed references are generally not allowed and will cause an 1032Other unblessed references are generally not allowed and will cause an
769exception to be thrown, except for references to the integers C<0> and 1033exception to be thrown, except for references to the integers C<0> and
770C<1>, which get turned into C<false> and C<true> atoms in JSON. You can 1034C<1>, which get turned into C<false> and C<true> atoms in JSON. You can
771also use C<JSON::XS::false> and C<JSON::XS::true> to improve readability. 1035also use C<JSON::XS::false> and C<JSON::XS::true> to improve readability.
772 1036
773 encode_json [\0,JSON::XS::true] # yields [false,true] 1037 encode_json [\0, JSON::XS::true] # yields [false,true]
774 1038
775=item JSON::XS::true, JSON::XS::false 1039=item JSON::XS::true, JSON::XS::false
776 1040
777These special values become JSON true and JSON false values, 1041These special values become JSON true and JSON false values,
778respectively. You can also use C<\1> and C<\0> directly if you want. 1042respectively. You can also use C<\1> and C<\0> directly if you want.
816 my $x = "3"; # some variable containing a string 1080 my $x = "3"; # some variable containing a string
817 $x += 0; # numify it, ensuring it will be dumped as a number 1081 $x += 0; # numify it, ensuring it will be dumped as a number
818 $x *= 1; # same thing, the choice is yours. 1082 $x *= 1; # same thing, the choice is yours.
819 1083
820You can not currently force the type in other, less obscure, ways. Tell me 1084You can not currently force the type in other, less obscure, ways. Tell me
821if you need this capability (but don't forget to explain why its needed 1085if you need this capability (but don't forget to explain why it's needed
822:). 1086:).
823 1087
824=back 1088=back
825 1089
826 1090
828 1092
829The interested reader might have seen a number of flags that signify 1093The interested reader might have seen a number of flags that signify
830encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be 1094encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be
831some confusion on what these do, so here is a short comparison: 1095some confusion on what these do, so here is a short comparison:
832 1096
833C<utf8> controls wether the JSON text created by C<encode> (and expected 1097C<utf8> controls whether the JSON text created by C<encode> (and expected
834by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only 1098by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only
835control wether C<encode> escapes character values outside their respective 1099control whether C<encode> escapes character values outside their respective
836codeset range. Neither of these flags conflict with each other, although 1100codeset range. Neither of these flags conflict with each other, although
837some combinations make less sense than others. 1101some combinations make less sense than others.
838 1102
839Care has been taken to make all flags symmetrical with respect to 1103Care has been taken to make all flags symmetrical with respect to
840C<encode> and C<decode>, that is, texts encoded with any combination of 1104C<encode> and C<decode>, that is, texts encoded with any combination of
920proper subset of most 8-bit and multibyte encodings in use in the world. 1184proper subset of most 8-bit and multibyte encodings in use in the world.
921 1185
922=back 1186=back
923 1187
924 1188
925=head1 COMPARISON 1189=head2 JSON and ECMAscript
926 1190
927As already mentioned, this module was created because none of the existing 1191JSON syntax is based on how literals are represented in javascript (the
928JSON modules could be made to work correctly. First I will describe the 1192not-standardised predecessor of ECMAscript) which is presumably why it is
929problems (or pleasures) I encountered with various existing JSON modules, 1193called "JavaScript Object Notation".
930followed by some benchmark values. JSON::XS was designed not to suffer
931from any of these problems or limitations.
932 1194
933=over 4 1195However, JSON is not a subset (and also not a superset of course) of
1196ECMAscript (the standard) or javascript (whatever browsers actually
1197implement).
934 1198
935=item JSON 2.xx 1199If you want to use javascript's C<eval> function to "parse" JSON, you
1200might run into parse errors for valid JSON texts, or the resulting data
1201structure might not be queryable:
936 1202
937A marvellous piece of engineering, this module either uses JSON::XS 1203One of the problems is that U+2028 and U+2029 are valid characters inside
938directly when available (so will be 100% compatible with it, including 1204JSON strings, but are not allowed in ECMAscript string literals, so the
939speed), or it uses JSON::PP, which is basically JSON::XS translated to 1205following Perl fragment will not output something that can be guaranteed
940Pure Perl, which should be 100% compatible with JSON::XS, just a bit 1206to be parsable by javascript's C<eval>:
941slower.
942 1207
943You cannot really lose by using this module, especially as it tries very 1208 use JSON::XS;
944hard to work even with ancient Perl versions, while JSON::XS does not.
945 1209
946=item JSON 1.07 1210 print encode_json [chr 0x2028];
947 1211
948Slow (but very portable, as it is written in pure Perl). 1212The right fix for this is to use a proper JSON parser in your javascript
1213programs, and not rely on C<eval> (see for example Douglas Crockford's
1214F<json2.js> parser).
949 1215
950Undocumented/buggy Unicode handling (how JSON handles Unicode values is 1216If this is not an option, you can, as a stop-gap measure, simply encode to
951undocumented. One can get far by feeding it Unicode strings and doing 1217ASCII-only JSON:
952en-/decoding oneself, but Unicode escapes are not working properly).
953 1218
954No round-tripping (strings get clobbered if they look like numbers, e.g. 1219 use JSON::XS;
955the string C<2.0> will encode to C<2.0> instead of C<"2.0">, and that will
956decode into the number 2.
957 1220
958=item JSON::PC 0.01 1221 print JSON::XS->new->ascii->encode ([chr 0x2028]);
959 1222
960Very fast. 1223Note that this will enlarge the resulting JSON text quite a bit if you
1224have many non-ASCII characters. You might be tempted to run some regexes
1225to only escape U+2028 and U+2029, e.g.:
961 1226
962Undocumented/buggy Unicode handling. 1227 # DO NOT USE THIS!
1228 my $json = JSON::XS->new->utf8->encode ([chr 0x2028]);
1229 $json =~ s/\xe2\x80\xa8/\\u2028/g; # escape U+2028
1230 $json =~ s/\xe2\x80\xa9/\\u2029/g; # escape U+2029
1231 print $json;
963 1232
964No round-tripping. 1233Note that I<this is a bad idea>: the above only works for U+2028 and
1234U+2029 and thus only for fully ECMAscript-compliant parsers. Many existing
1235javascript implementations, however, have issues with other characters as
1236well - using C<eval> naively simply I<will> cause problems.
965 1237
966Has problems handling many Perl values (e.g. regex results and other magic 1238Another problem is that some javascript implementations reserve
967values will make it croak). 1239some property names for their own purposes (which probably makes
1240them non-ECMAscript-compliant). For example, Iceweasel reserves the
1241C<__proto__> property name for it's own purposes.
968 1242
969Does not even generate valid JSON (C<{1,2}> gets converted to C<{1:2}> 1243If that is a problem, you could parse try to filter the resulting JSON
970which is not a valid JSON text. 1244output for these property strings, e.g.:
971 1245
972Unmaintained (maintainer unresponsive for many months, bugs are not 1246 $json =~ s/"__proto__"\s*:/"__proto__renamed":/g;
973getting fixed).
974 1247
975=item JSON::Syck 0.21 1248This works because C<__proto__> is not valid outside of strings, so every
1249occurence of C<"__proto__"\s*:> must be a string used as property name.
976 1250
977Very buggy (often crashes). 1251If you know of other incompatibilities, please let me know.
978
979Very inflexible (no human-readable format supported, format pretty much
980undocumented. I need at least a format for easy reading by humans and a
981single-line compact format for use in a protocol, and preferably a way to
982generate ASCII-only JSON texts).
983
984Completely broken (and confusingly documented) Unicode handling (Unicode
985escapes are not working properly, you need to set ImplicitUnicode to
986I<different> values on en- and decoding to get symmetric behaviour).
987
988No round-tripping (simple cases work, but this depends on whether the scalar
989value was used in a numeric context or not).
990
991Dumping hashes may skip hash values depending on iterator state.
992
993Unmaintained (maintainer unresponsive for many months, bugs are not
994getting fixed).
995
996Does not check input for validity (i.e. will accept non-JSON input and
997return "something" instead of raising an exception. This is a security
998issue: imagine two banks transferring money between each other using
999JSON. One bank might parse a given non-JSON request and deduct money,
1000while the other might reject the transaction with a syntax error. While a
1001good protocol will at least recover, that is extra unnecessary work and
1002the transaction will still not succeed).
1003
1004=item JSON::DWIW 0.04
1005
1006Very fast. Very natural. Very nice.
1007
1008Undocumented Unicode handling (but the best of the pack. Unicode escapes
1009still don't get parsed properly).
1010
1011Very inflexible.
1012
1013No round-tripping.
1014
1015Does not generate valid JSON texts (key strings are often unquoted, empty keys
1016result in nothing being output)
1017
1018Does not check input for validity.
1019
1020=back
1021 1252
1022 1253
1023=head2 JSON and YAML 1254=head2 JSON and YAML
1024 1255
1025You often hear that JSON is a subset of YAML. This is, however, a mass 1256You often hear that JSON is a subset of YAML. This is, however, a mass
1026hysteria(*) and very far from the truth. In general, there is no way to 1257hysteria(*) and very far from the truth (as of the time of this writing),
1258so let me state it clearly: I<in general, there is no way to configure
1027configure JSON::XS to output a data structure as valid YAML that works for 1259JSON::XS to output a data structure as valid YAML> that works in all
1028all cases. 1260cases.
1029 1261
1030If you really must use JSON::XS to generate YAML, you should use this 1262If you really must use JSON::XS to generate YAML, you should use this
1031algorithm (subject to change in future versions): 1263algorithm (subject to change in future versions):
1032 1264
1033 my $to_yaml = JSON::XS->new->utf8->space_after (1); 1265 my $to_yaml = JSON::XS->new->utf8->space_after (1);
1036This will I<usually> generate JSON texts that also parse as valid 1268This will I<usually> generate JSON texts that also parse as valid
1037YAML. Please note that YAML has hardcoded limits on (simple) object key 1269YAML. Please note that YAML has hardcoded limits on (simple) object key
1038lengths that JSON doesn't have and also has different and incompatible 1270lengths that JSON doesn't have and also has different and incompatible
1039unicode handling, so you should make sure that your hash keys are 1271unicode handling, so you should make sure that your hash keys are
1040noticeably shorter than the 1024 "stream characters" YAML allows and that 1272noticeably shorter than the 1024 "stream characters" YAML allows and that
1041you do not have codepoints with values outside the Unicode BMP (basic 1273you do not have characters with codepoint values outside the Unicode BMP
1042multilingual page). YAML also does not allow C<\/> sequences in strings 1274(basic multilingual page). YAML also does not allow C<\/> sequences in
1043(which JSON::XS does not I<currently> generate). 1275strings (which JSON::XS does not I<currently> generate, but other JSON
1276generators might).
1044 1277
1045There might be other incompatibilities that I am not aware of (or the YAML 1278There might be other incompatibilities that I am not aware of (or the YAML
1046specification has been changed yet again - it does so quite often). In 1279specification has been changed yet again - it does so quite often). In
1047general you should not try to generate YAML with a JSON generator or vice 1280general you should not try to generate YAML with a JSON generator or vice
1048versa, or try to parse JSON with a YAML parser or vice versa: chances are 1281versa, or try to parse JSON with a YAML parser or vice versa: chances are
1051 1284
1052=over 4 1285=over 4
1053 1286
1054=item (*) 1287=item (*)
1055 1288
1056This is spread actively by the YAML team, however. For many years now they 1289I have been pressured multiple times by Brian Ingerson (one of the
1057claim YAML were a superset of JSON, even when proven otherwise. 1290authors of the YAML specification) to remove this paragraph, despite him
1291acknowledging that the actual incompatibilities exist. As I was personally
1292bitten by this "JSON is YAML" lie, I refused and said I will continue to
1293educate people about these issues, so others do not run into the same
1294problem again and again. After this, Brian called me a (quote)I<complete
1295and worthless idiot>(unquote).
1058 1296
1059Even the author of this manpage was at some point accused of providing 1297In my opinion, instead of pressuring and insulting people who actually
1060"incorrect" information, despite the evidence presented (claims ranged 1298clarify issues with YAML and the wrong statements of some of its
1061from "your documentation contains inaccurate and negative statements about 1299proponents, I would kindly suggest reading the JSON spec (which is not
1062YAML" (the only negative comment is this footnote, and it didn't exist 1300that difficult or long) and finally make YAML compatible to it, and
1063back then; the question on which claims were inaccurate was never answered 1301educating users about the changes, instead of spreading lies about the
1064etc.) to "the YAML spec is not up-to-date" (the *real* and supposedly 1302real compatibility for many I<years> and trying to silence people who
1065JSON-compatible spec is apparently not currently publicly available) 1303point out that it isn't true.
1066to actual requests to replace this section by *incorrect* information,
1067suppressing information about the real problem).
1068
1069So whenever you are told that YAML was a superset of JSON, first check
1070wether it is really true (it might be when you check it, but it certainly
1071was not true when this was written). I would much prefer if the YAML team
1072would spent their time on actually making JSON compatibility a truth
1073(JSON, after all, has a very small and simple specification) instead of
1074trying to lobby/force people into reporting untruths.
1075 1304
1076=back 1305=back
1077 1306
1078 1307
1079=head2 SPEED 1308=head2 SPEED
1085 1314
1086First comes a comparison between various modules using 1315First comes a comparison between various modules using
1087a very short single-line JSON string (also available at 1316a very short single-line JSON string (also available at
1088L<http://dist.schmorp.de/misc/json/short.json>). 1317L<http://dist.schmorp.de/misc/json/short.json>).
1089 1318
1090 {"method": "handleMessage", "params": ["user1", "we were just talking"], \ 1319 {"method": "handleMessage", "params": ["user1",
1091 "id": null, "array":[1,11,234,-5,1e5,1e7, true, false]} 1320 "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
1321 true, false]}
1092 1322
1093It shows the number of encodes/decodes per second (JSON::XS uses 1323It shows the number of encodes/decodes per second (JSON::XS uses
1094the functional interface, while JSON::XS/2 uses the OO interface 1324the functional interface, while JSON::XS/2 uses the OO interface
1095with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables 1325with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
1096shrink). Higher is better: 1326shrink). Higher is better:
1186=head1 THREADS 1416=head1 THREADS
1187 1417
1188This module is I<not> guaranteed to be thread safe and there are no 1418This module is I<not> guaranteed to be thread safe and there are no
1189plans to change this until Perl gets thread support (as opposed to the 1419plans to change this until Perl gets thread support (as opposed to the
1190horribly slow so-called "threads" which are simply slow and bloated 1420horribly slow so-called "threads" which are simply slow and bloated
1191process simulations - use fork, its I<much> faster, cheaper, better). 1421process simulations - use fork, it's I<much> faster, cheaper, better).
1192 1422
1193(It might actually work, but you have been warned). 1423(It might actually work, but you have been warned).
1194 1424
1195 1425
1196=head1 BUGS 1426=head1 BUGS
1197 1427
1198While the goal of this module is to be correct, that unfortunately does 1428While the goal of this module is to be correct, that unfortunately does
1199not mean its bug-free, only that I think its design is bug-free. It is 1429not mean it's bug-free, only that I think its design is bug-free. If you
1200still relatively early in its development. If you keep reporting bugs they 1430keep reporting bugs they will be fixed swiftly, though.
1201will be fixed swiftly, though.
1202 1431
1203Please refrain from using rt.cpan.org or any other bug reporting 1432Please refrain from using rt.cpan.org or any other bug reporting
1204service. I put the contact address into my modules for a reason. 1433service. I put the contact address into my modules for a reason.
1205 1434
1206=cut 1435=cut
1226 "--" => sub { $_[0] = ${$_[0]} - 1 }, 1455 "--" => sub { $_[0] = ${$_[0]} - 1 },
1227 fallback => 1; 1456 fallback => 1;
1228 1457
12291; 14581;
1230 1459
1460=head1 SEE ALSO
1461
1462The F<json_xs> command line utility for quick experiments.
1463
1231=head1 AUTHOR 1464=head1 AUTHOR
1232 1465
1233 Marc Lehmann <schmorp@schmorp.de> 1466 Marc Lehmann <schmorp@schmorp.de>
1234 http://home.schmorp.de/ 1467 http://home.schmorp.de/
1235 1468

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines