ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.81 by root, Sat Dec 29 17:33:38 2007 UTC vs.
Revision 1.149 by root, Tue Oct 29 00:21:10 2013 UTC

1=head1 NAME 1=head1 NAME
2 2
3JSON::XS - JSON serialising/deserialising, done correctly and fast 3JSON::XS - JSON serialising/deserialising, done correctly and fast
4
5=encoding utf-8
4 6
5JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ 7JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ
6 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html) 8 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html)
7 9
8=head1 SYNOPSIS 10=head1 SYNOPSIS
35primary goal is to be I<correct> and its secondary goal is to be 37primary goal is to be I<correct> and its secondary goal is to be
36I<fast>. To reach the latter goal it was written in C. 38I<fast>. To reach the latter goal it was written in C.
37 39
38Beginning with version 2.0 of the JSON module, when both JSON and 40Beginning with version 2.0 of the JSON module, when both JSON and
39JSON::XS are installed, then JSON will fall back on JSON::XS (this can be 41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
40overriden) with no overhead due to emulation (by inheritign constructor 42overridden) with no overhead due to emulation (by inheriting constructor
41and methods). If JSON::XS is not available, it will fall back to the 43and methods). If JSON::XS is not available, it will fall back to the
42compatible JSON::PP module as backend, so using JSON instead of JSON::XS 44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
43gives you a portable JSON API that can be fast when you need and doesn't 45gives you a portable JSON API that can be fast when you need and doesn't
44require a C compiler when that is a problem. 46require a C compiler when that is a problem.
45 47
47to write yet another JSON module? While it seems there are many JSON 49to write yet another JSON module? While it seems there are many JSON
48modules, none of them correctly handle all corner cases, and in most cases 50modules, none of them correctly handle all corner cases, and in most cases
49their maintainers are unresponsive, gone missing, or not listening to bug 51their maintainers are unresponsive, gone missing, or not listening to bug
50reports for other reasons. 52reports for other reasons.
51 53
52See COMPARISON, below, for a comparison to some other JSON modules.
53
54See MAPPING, below, on how JSON::XS maps perl values to JSON values and 54See MAPPING, below, on how JSON::XS maps perl values to JSON values and
55vice versa. 55vice versa.
56 56
57=head2 FEATURES 57=head2 FEATURES
58 58
59=over 4 59=over 4
60 60
61=item * correct Unicode handling 61=item * correct Unicode handling
62 62
63This module knows how to handle Unicode, and even documents how and when 63This module knows how to handle Unicode, documents how and when it does
64it does so. 64so, and even documents what "correct" means.
65 65
66=item * round-trip integrity 66=item * round-trip integrity
67 67
68When you serialise a perl data structure using only datatypes supported 68When you serialise a perl data structure using only data types supported
69by JSON, the deserialised data structure is identical on the Perl level. 69by JSON and Perl, the deserialised data structure is identical on the Perl
70(e.g. the string "2.0" doesn't suddenly become "2" just because it looks 70level. (e.g. the string "2.0" doesn't suddenly become "2" just because
71like a number). 71it looks like a number). There I<are> minor exceptions to this, read the
72MAPPING section below to learn about those.
72 73
73=item * strict checking of JSON correctness 74=item * strict checking of JSON correctness
74 75
75There is no guessing, no generating of illegal JSON texts by default, 76There is no guessing, no generating of illegal JSON texts by default,
76and only JSON is accepted as input by default (the latter is a security 77and only JSON is accepted as input by default (the latter is a security
77feature). 78feature).
78 79
79=item * fast 80=item * fast
80 81
81Compared to other JSON modules, this module compares favourably in terms 82Compared to other JSON modules and other serialisers such as Storable,
82of speed, too. 83this module usually compares favourably in terms of speed, too.
83 84
84=item * simple to use 85=item * simple to use
85 86
86This module has both a simple functional interface as well as an OO 87This module has both a simple functional interface as well as an object
87interface. 88oriented interface.
88 89
89=item * reasonably versatile output formats 90=item * reasonably versatile output formats
90 91
91You can choose between the most compact guaranteed single-line format 92You can choose between the most compact guaranteed-single-line format
92possible (nice for simple line-based protocols), a pure-ascii format 93possible (nice for simple line-based protocols), a pure-ASCII format
93(for when your transport is not 8-bit clean, still supports the whole 94(for when your transport is not 8-bit clean, still supports the whole
94Unicode range), or a pretty-printed format (for when you want to read that 95Unicode range), or a pretty-printed format (for when you want to read that
95stuff). Or you can combine those features in whatever way you like. 96stuff). Or you can combine those features in whatever way you like.
96 97
97=back 98=back
98 99
99=cut 100=cut
100 101
101package JSON::XS; 102package JSON::XS;
102 103
103use strict; 104use common::sense;
104 105
105our $VERSION = '2.01'; 106our $VERSION = '3.0';
106our @ISA = qw(Exporter); 107our @ISA = qw(Exporter);
107 108
108our @EXPORT = qw(encode_json decode_json to_json from_json); 109our @EXPORT = qw(encode_json decode_json);
109
110sub to_json($) {
111 require Carp;
112 Carp::croak ("JSON::XS::to_json has been renamed to encode_json, either downgrade to pre-2.0 versions of JSON::XS or rename the call");
113}
114
115sub from_json($) {
116 require Carp;
117 Carp::croak ("JSON::XS::from_json has been renamed to decode_json, either downgrade to pre-2.0 versions of JSON::XS or rename the call");
118}
119 110
120use Exporter; 111use Exporter;
121use XSLoader; 112use XSLoader;
122 113
114use Types::Serialiser ();
115
123=head1 FUNCTIONAL INTERFACE 116=head1 FUNCTIONAL INTERFACE
124 117
125The following convenience methods are provided by this module. They are 118The following convenience methods are provided by this module. They are
126exported by default: 119exported by default:
127 120
134 127
135This function call is functionally identical to: 128This function call is functionally identical to:
136 129
137 $json_text = JSON::XS->new->utf8->encode ($perl_scalar) 130 $json_text = JSON::XS->new->utf8->encode ($perl_scalar)
138 131
139except being faster. 132Except being faster.
140 133
141=item $perl_scalar = decode_json $json_text 134=item $perl_scalar = decode_json $json_text
142 135
143The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries 136The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries
144to parse that as an UTF-8 encoded JSON text, returning the resulting 137to parse that as an UTF-8 encoded JSON text, returning the resulting
146 139
147This function call is functionally identical to: 140This function call is functionally identical to:
148 141
149 $perl_scalar = JSON::XS->new->utf8->decode ($json_text) 142 $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
150 143
151except being faster. 144Except being faster.
152
153=item $is_boolean = JSON::XS::is_bool $scalar
154
155Returns true if the passed scalar represents either JSON::XS::true or
156JSON::XS::false, two constants that act like C<1> and C<0>, respectively
157and are used to represent JSON C<true> and C<false> values in Perl.
158
159See MAPPING, below, for more information on how JSON values are mapped to
160Perl.
161 145
162=back 146=back
163 147
164 148
165=head1 A FEW NOTES ON UNICODE AND PERL 149=head1 A FEW NOTES ON UNICODE AND PERL
174This enables you to store Unicode characters as single characters in a 158This enables you to store Unicode characters as single characters in a
175Perl string - very natural. 159Perl string - very natural.
176 160
177=item 2. Perl does I<not> associate an encoding with your strings. 161=item 2. Perl does I<not> associate an encoding with your strings.
178 162
179Unless you force it to, e.g. when matching it against a regex, or printing 163... until you force it to, e.g. when matching it against a regex, or
180the scalar to a file, in which case Perl either interprets your string as 164printing the scalar to a file, in which case Perl either interprets your
181locale-encoded text, octets/binary, or as Unicode, depending on various 165string as locale-encoded text, octets/binary, or as Unicode, depending
182settings. In no case is an encoding stored together with your data, it is 166on various settings. In no case is an encoding stored together with your
183I<use> that decides encoding, not any magical metadata. 167data, it is I<use> that decides encoding, not any magical meta data.
184 168
185=item 3. The internal utf-8 flag has no meaning with regards to the 169=item 3. The internal utf-8 flag has no meaning with regards to the
186encoding of your string. 170encoding of your string.
187 171
188Just ignore that flag unless you debug a Perl bug, a module written in 172Just ignore that flag unless you debug a Perl bug, a module written in
194 178
195If you didn't know about that flag, just the better, pretend it doesn't 179If you didn't know about that flag, just the better, pretend it doesn't
196exist. 180exist.
197 181
198=item 4. A "Unicode String" is simply a string where each character can be 182=item 4. A "Unicode String" is simply a string where each character can be
199validly interpreted as a Unicode codepoint. 183validly interpreted as a Unicode code point.
200 184
201If you have UTF-8 encoded data, it is no longer a Unicode string, but a 185If you have UTF-8 encoded data, it is no longer a Unicode string, but a
202Unicode string encoded in UTF-8, giving you a binary string. 186Unicode string encoded in UTF-8, giving you a binary string.
203 187
204=item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string. 188=item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string.
242 226
243If C<$enable> is false, then the C<encode> method will not escape Unicode 227If C<$enable> is false, then the C<encode> method will not escape Unicode
244characters unless required by the JSON syntax or other flags. This results 228characters unless required by the JSON syntax or other flags. This results
245in a faster and more compact format. 229in a faster and more compact format.
246 230
231See also the section I<ENCODING/CODESET FLAG NOTES> later in this
232document.
233
247The main use for this flag is to produce JSON texts that can be 234The main use for this flag is to produce JSON texts that can be
248transmitted over a 7-bit channel, as the encoded JSON texts will not 235transmitted over a 7-bit channel, as the encoded JSON texts will not
249contain any 8 bit characters. 236contain any 8 bit characters.
250 237
251 JSON::XS->new->ascii (1)->encode ([chr 0x10401]) 238 JSON::XS->new->ascii (1)->encode ([chr 0x10401])
262will not be affected in any way by this flag, as C<decode> by default 249will not be affected in any way by this flag, as C<decode> by default
263expects Unicode, which is a strict superset of latin1. 250expects Unicode, which is a strict superset of latin1.
264 251
265If C<$enable> is false, then the C<encode> method will not escape Unicode 252If C<$enable> is false, then the C<encode> method will not escape Unicode
266characters unless required by the JSON syntax or other flags. 253characters unless required by the JSON syntax or other flags.
254
255See also the section I<ENCODING/CODESET FLAG NOTES> later in this
256document.
267 257
268The main use for this flag is efficiently encoding binary data as JSON 258The main use for this flag is efficiently encoding binary data as JSON
269text, as most octets will not be escaped, resulting in a smaller encoded 259text, as most octets will not be escaped, resulting in a smaller encoded
270size. The disadvantage is that the resulting JSON text is encoded 260size. The disadvantage is that the resulting JSON text is encoded
271in latin1 (and must correctly be treated as such when storing and 261in latin1 (and must correctly be treated as such when storing and
290 280
291If C<$enable> is false, then the C<encode> method will return the JSON 281If C<$enable> is false, then the C<encode> method will return the JSON
292string as a (non-encoded) Unicode string, while C<decode> expects thus a 282string as a (non-encoded) Unicode string, while C<decode> expects thus a
293Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs 283Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs
294to be done yourself, e.g. using the Encode module. 284to be done yourself, e.g. using the Encode module.
285
286See also the section I<ENCODING/CODESET FLAG NOTES> later in this
287document.
295 288
296Example, output UTF-16BE-encoded JSON: 289Example, output UTF-16BE-encoded JSON:
297 290
298 use Encode; 291 use Encode;
299 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object); 292 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object);
422If C<$enable> is true (or missing), then the C<encode> method will output JSON objects 415If C<$enable> is true (or missing), then the C<encode> method will output JSON objects
423by sorting their keys. This is adding a comparatively high overhead. 416by sorting their keys. This is adding a comparatively high overhead.
424 417
425If C<$enable> is false, then the C<encode> method will output key-value 418If C<$enable> is false, then the C<encode> method will output key-value
426pairs in the order Perl stores them (which will likely change between runs 419pairs in the order Perl stores them (which will likely change between runs
427of the same script). 420of the same script, and can change even within the same run from 5.18
421onwards).
428 422
429This option is useful if you want the same data structure to be encoded as 423This option is useful if you want the same data structure to be encoded as
430the same JSON text (given the same overall settings). If it is disabled, 424the same JSON text (given the same overall settings). If it is disabled,
431the same hash might be encoded differently even if contains the same data, 425the same hash might be encoded differently even if contains the same data,
432as key-value pairs have no inherent ordering in Perl. 426as key-value pairs have no inherent ordering in Perl.
433 427
434This setting has no effect when decoding JSON texts. 428This setting has no effect when decoding JSON texts.
435 429
430This setting has currently no effect on tied hashes.
431
436=item $json = $json->allow_nonref ([$enable]) 432=item $json = $json->allow_nonref ([$enable])
437 433
438=item $enabled = $json->get_allow_nonref 434=item $enabled = $json->get_allow_nonref
439 435
440If C<$enable> is true (or missing), then the C<encode> method can convert a 436If C<$enable> is true (or missing), then the C<encode> method can convert a
451resulting in an invalid JSON text: 447resulting in an invalid JSON text:
452 448
453 JSON::XS->new->allow_nonref->encode ("Hello, World!") 449 JSON::XS->new->allow_nonref->encode ("Hello, World!")
454 => "Hello, World!" 450 => "Hello, World!"
455 451
452=item $json = $json->allow_unknown ([$enable])
453
454=item $enabled = $json->get_allow_unknown
455
456If C<$enable> is true (or missing), then C<encode> will I<not> throw an
457exception when it encounters values it cannot represent in JSON (for
458example, filehandles) but instead will encode a JSON C<null> value. Note
459that blessed objects are not included here and are handled separately by
460c<allow_nonref>.
461
462If C<$enable> is false (the default), then C<encode> will throw an
463exception when it encounters anything it cannot encode as JSON.
464
465This option does not affect C<decode> in any way, and it is recommended to
466leave it off unless you know your communications partner.
467
456=item $json = $json->allow_blessed ([$enable]) 468=item $json = $json->allow_blessed ([$enable])
457 469
458=item $enabled = $json->get_allow_blessed 470=item $enabled = $json->get_allow_blessed
459 471
472See L<OBJECT SERIALISATION> for details.
473
460If C<$enable> is true (or missing), then the C<encode> method will not 474If C<$enable> is true (or missing), then the C<encode> method will not
461barf when it encounters a blessed reference. Instead, the value of the 475barf when it encounters a blessed reference that it cannot convert
462B<convert_blessed> option will decide whether C<null> (C<convert_blessed> 476otherwise. Instead, a JSON C<null> value is encoded instead of the object.
463disabled or no C<TO_JSON> method found) or a representation of the
464object (C<convert_blessed> enabled and C<TO_JSON> method found) is being
465encoded. Has no effect on C<decode>.
466 477
467If C<$enable> is false (the default), then C<encode> will throw an 478If C<$enable> is false (the default), then C<encode> will throw an
468exception when it encounters a blessed object. 479exception when it encounters a blessed object that it cannot convert
480otherwise.
481
482This setting has no effect on C<decode>.
469 483
470=item $json = $json->convert_blessed ([$enable]) 484=item $json = $json->convert_blessed ([$enable])
471 485
472=item $enabled = $json->get_convert_blessed 486=item $enabled = $json->get_convert_blessed
487
488See L<OBJECT SERIALISATION> for details.
473 489
474If C<$enable> is true (or missing), then C<encode>, upon encountering a 490If C<$enable> is true (or missing), then C<encode>, upon encountering a
475blessed object, will check for the availability of the C<TO_JSON> method 491blessed object, will check for the availability of the C<TO_JSON> method
476on the object's class. If found, it will be called in scalar context 492on the object's class. If found, it will be called in scalar context and
477and the resulting scalar will be encoded instead of the object. If no 493the resulting scalar will be encoded instead of the object.
478C<TO_JSON> method is found, the value of C<allow_blessed> will decide what
479to do.
480 494
481The C<TO_JSON> method may safely call die if it wants. If C<TO_JSON> 495The C<TO_JSON> method may safely call die if it wants. If C<TO_JSON>
482returns other blessed objects, those will be handled in the same 496returns other blessed objects, those will be handled in the same
483way. C<TO_JSON> must take care of not causing an endless recursion cycle 497way. C<TO_JSON> must take care of not causing an endless recursion cycle
484(== crash) in this case. The name of C<TO_JSON> was chosen because other 498(== crash) in this case. The name of C<TO_JSON> was chosen because other
485methods called by the Perl core (== not by the user of the object) are 499methods called by the Perl core (== not by the user of the object) are
486usually in upper case letters and to avoid collisions with any C<to_json> 500usually in upper case letters and to avoid collisions with any C<to_json>
487function or method. 501function or method.
488 502
489This setting does not yet influence C<decode> in any way, but in the 503If C<$enable> is false (the default), then C<encode> will not consider
490future, global hooks might get installed that influence C<decode> and are 504this type of conversion.
491enabled by this setting.
492 505
493If C<$enable> is false, then the C<allow_blessed> setting will decide what 506This setting has no effect on C<decode>.
494to do when a blessed object is found. 507
508=item $json = $json->allow_tags ([$enable])
509
510=item $enabled = $json->allow_tags
511
512See L<OBJECT SERIALISATION> for details.
513
514If C<$enable> is true (or missing), then C<encode>, upon encountering a
515blessed object, will check for the availability of the C<FREEZE> method on
516the object's class. If found, it will be used to serialise the object into
517a nonstandard tagged JSON value (that JSON decoders cannot decode).
518
519It also causes C<decode> to parse such tagged JSON values and deserialise
520them via a call to the C<THAW> method.
521
522If C<$enable> is false (the default), then C<encode> will not consider
523this type of conversion, and tagged JSON values will cause a parse error
524in C<decode>, as if tags were not part of the grammar.
495 525
496=item $json = $json->filter_json_object ([$coderef->($hashref)]) 526=item $json = $json->filter_json_object ([$coderef->($hashref)])
497 527
498When C<$coderef> is specified, it will be called from C<decode> each 528When C<$coderef> is specified, it will be called from C<decode> each
499time it decodes a JSON object. The only argument is a reference to the 529time it decodes a JSON object. The only argument is a reference to the
600=item $json = $json->max_depth ([$maximum_nesting_depth]) 630=item $json = $json->max_depth ([$maximum_nesting_depth])
601 631
602=item $max_depth = $json->get_max_depth 632=item $max_depth = $json->get_max_depth
603 633
604Sets the maximum nesting level (default C<512>) accepted while encoding 634Sets the maximum nesting level (default C<512>) accepted while encoding
605or decoding. If the JSON text or Perl data structure has an equal or 635or decoding. If a higher nesting level is detected in JSON text or a Perl
606higher nesting level then this limit, then the encoder and decoder will 636data structure, then the encoder and decoder will stop and croak at that
607stop and croak at that point. 637point.
608 638
609Nesting level is defined by number of hash- or arrayrefs that the encoder 639Nesting level is defined by number of hash- or arrayrefs that the encoder
610needs to traverse to reach a given point or the number of C<{> or C<[> 640needs to traverse to reach a given point or the number of C<{> or C<[>
611characters without their matching closing parenthesis crossed to reach a 641characters without their matching closing parenthesis crossed to reach a
612given character in a string. 642given character in a string.
613 643
614Setting the maximum depth to one disallows any nesting, so that ensures 644Setting the maximum depth to one disallows any nesting, so that ensures
615that the object is only a single hash/object or array. 645that the object is only a single hash/object or array.
616 646
617The argument to C<max_depth> will be rounded up to the next highest power
618of two. If no argument is given, the highest possible setting will be 647If no argument is given, the highest possible setting will be used, which
619used, which is rarely useful. 648is rarely useful.
649
650Note that nesting is implemented by recursion in C. The default value has
651been chosen to be as large as typical operating systems allow without
652crashing.
620 653
621See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 654See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
622 655
623=item $json = $json->max_size ([$maximum_string_size]) 656=item $json = $json->max_size ([$maximum_string_size])
624 657
625=item $max_size = $json->get_max_size 658=item $max_size = $json->get_max_size
626 659
627Set the maximum length a JSON text may have (in bytes) where decoding is 660Set the maximum length a JSON text may have (in bytes) where decoding is
628being attempted. The default is C<0>, meaning no limit. When C<decode> 661being attempted. The default is C<0>, meaning no limit. When C<decode>
629is called on a string longer then this number of characters it will not 662is called on a string that is longer then this many bytes, it will not
630attempt to decode the string but throw an exception. This setting has no 663attempt to decode the string but throw an exception. This setting has no
631effect on C<encode> (yet). 664effect on C<encode> (yet).
632 665
633The argument to C<max_size> will be rounded up to the next B<highest> 666If no argument is given, the limit check will be deactivated (same as when
634power of two (so may be more than requested). If no argument is given, the 667C<0> is specified).
635limit check will be deactivated (same as when C<0> is specified).
636 668
637See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 669See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
638 670
639=item $json_text = $json->encode ($perl_scalar) 671=item $json_text = $json->encode ($perl_scalar)
640 672
641Converts the given Perl data structure (a simple scalar or a reference 673Converts the given Perl value or data structure to its JSON
642to a hash or array) to its JSON representation. Simple scalars will be 674representation. Croaks on error.
643converted into JSON string or number sequences, while references to arrays
644become JSON arrays and references to hashes become JSON objects. Undefined
645Perl values (e.g. C<undef>) become JSON C<null> values. Neither C<true>
646nor C<false> values will be generated.
647 675
648=item $perl_scalar = $json->decode ($json_text) 676=item $perl_scalar = $json->decode ($json_text)
649 677
650The opposite of C<encode>: expects a JSON text and tries to parse it, 678The opposite of C<encode>: expects a JSON text and tries to parse it,
651returning the resulting simple scalar or reference. Croaks on error. 679returning the resulting simple scalar or reference. Croaks on error.
652
653JSON numbers and strings become simple Perl scalars. JSON arrays become
654Perl arrayrefs and JSON objects become Perl hashrefs. C<true> becomes
655C<1>, C<false> becomes C<0> and C<null> becomes C<undef>.
656 680
657=item ($perl_scalar, $characters) = $json->decode_prefix ($json_text) 681=item ($perl_scalar, $characters) = $json->decode_prefix ($json_text)
658 682
659This works like the C<decode> method, but instead of raising an exception 683This works like the C<decode> method, but instead of raising an exception
660when there is trailing garbage after the first JSON object, it will 684when there is trailing garbage after the first JSON object, it will
661silently stop parsing there and return the number of characters consumed 685silently stop parsing there and return the number of characters consumed
662so far. 686so far.
663 687
664This is useful if your JSON texts are not delimited by an outer protocol 688This is useful if your JSON texts are not delimited by an outer protocol
665(which is not the brightest thing to do in the first place) and you need
666to know where the JSON text ends. 689and you need to know where the JSON text ends.
667 690
668 JSON::XS->new->decode_prefix ("[1] the tail") 691 JSON::XS->new->decode_prefix ("[1] the tail")
669 => ([], 3) 692 => ([], 3)
670 693
671=back 694=back
695
696
697=head1 INCREMENTAL PARSING
698
699In some cases, there is the need for incremental parsing of JSON
700texts. While this module always has to keep both JSON text and resulting
701Perl data structure in memory at one time, it does allow you to parse a
702JSON stream incrementally. It does so by accumulating text until it has
703a full JSON object, which it then can decode. This process is similar to
704using C<decode_prefix> to see if a full JSON object is available, but
705is much more efficient (and can be implemented with a minimum of method
706calls).
707
708JSON::XS will only attempt to parse the JSON text once it is sure it
709has enough text to get a decisive result, using a very simple but
710truly incremental parser. This means that it sometimes won't stop as
711early as the full parser, for example, it doesn't detect mismatched
712parentheses. The only thing it guarantees is that it starts decoding as
713soon as a syntactically valid JSON text has been seen. This means you need
714to set resource limits (e.g. C<max_size>) to ensure the parser will stop
715parsing in the presence if syntax errors.
716
717The following methods implement this incremental parser.
718
719=over 4
720
721=item [void, scalar or list context] = $json->incr_parse ([$string])
722
723This is the central parsing function. It can both append new text and
724extract objects from the stream accumulated so far (both of these
725functions are optional).
726
727If C<$string> is given, then this string is appended to the already
728existing JSON fragment stored in the C<$json> object.
729
730After that, if the function is called in void context, it will simply
731return without doing anything further. This can be used to add more text
732in as many chunks as you want.
733
734If the method is called in scalar context, then it will try to extract
735exactly I<one> JSON object. If that is successful, it will return this
736object, otherwise it will return C<undef>. If there is a parse error,
737this method will croak just as C<decode> would do (one can then use
738C<incr_skip> to skip the erroneous part). This is the most common way of
739using the method.
740
741And finally, in list context, it will try to extract as many objects
742from the stream as it can find and return them, or the empty list
743otherwise. For this to work, there must be no separators between the JSON
744objects or arrays, instead they must be concatenated back-to-back. If
745an error occurs, an exception will be raised as in the scalar context
746case. Note that in this case, any previously-parsed JSON texts will be
747lost.
748
749Example: Parse some JSON arrays/objects in a given string and return
750them.
751
752 my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]");
753
754=item $lvalue_string = $json->incr_text
755
756This method returns the currently stored JSON fragment as an lvalue, that
757is, you can manipulate it. This I<only> works when a preceding call to
758C<incr_parse> in I<scalar context> successfully returned an object. Under
759all other circumstances you must not call this function (I mean it.
760although in simple tests it might actually work, it I<will> fail under
761real world conditions). As a special exception, you can also call this
762method before having parsed anything.
763
764This function is useful in two cases: a) finding the trailing text after a
765JSON object or b) parsing multiple JSON objects separated by non-JSON text
766(such as commas).
767
768=item $json->incr_skip
769
770This will reset the state of the incremental parser and will remove
771the parsed text from the input buffer so far. This is useful after
772C<incr_parse> died, in which case the input buffer and incremental parser
773state is left unchanged, to skip the text parsed so far and to reset the
774parse state.
775
776The difference to C<incr_reset> is that only text until the parse error
777occurred is removed.
778
779=item $json->incr_reset
780
781This completely resets the incremental parser, that is, after this call,
782it will be as if the parser had never parsed anything.
783
784This is useful if you want to repeatedly parse JSON objects and want to
785ignore any trailing data, which means you have to reset the parser after
786each successful decode.
787
788=back
789
790=head2 LIMITATIONS
791
792All options that affect decoding are supported, except
793C<allow_nonref>. The reason for this is that it cannot be made to work
794sensibly: JSON objects and arrays are self-delimited, i.e. you can
795concatenate them back to back and still decode them perfectly. This does
796not hold true for JSON numbers, however.
797
798For example, is the string C<1> a single JSON number, or is it simply the
799start of C<12>? Or is C<12> a single JSON number, or the concatenation
800of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
801takes the conservative route and disallows this case.
802
803=head2 EXAMPLES
804
805Some examples will make all this clearer. First, a simple example that
806works similarly to C<decode_prefix>: We want to decode the JSON object at
807the start of a string and identify the portion after the JSON object:
808
809 my $text = "[1,2,3] hello";
810
811 my $json = new JSON::XS;
812
813 my $obj = $json->incr_parse ($text)
814 or die "expected JSON object or array at beginning of string";
815
816 my $tail = $json->incr_text;
817 # $tail now contains " hello"
818
819Easy, isn't it?
820
821Now for a more complicated example: Imagine a hypothetical protocol where
822you read some requests from a TCP stream, and each request is a JSON
823array, without any separation between them (in fact, it is often useful to
824use newlines as "separators", as these get interpreted as whitespace at
825the start of the JSON text, which makes it possible to test said protocol
826with C<telnet>...).
827
828Here is how you'd do it (it is trivial to write this in an event-based
829manner):
830
831 my $json = new JSON::XS;
832
833 # read some data from the socket
834 while (sysread $socket, my $buf, 4096) {
835
836 # split and decode as many requests as possible
837 for my $request ($json->incr_parse ($buf)) {
838 # act on the $request
839 }
840 }
841
842Another complicated example: Assume you have a string with JSON objects
843or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
844[3]>). To parse them, we have to skip the commas between the JSON texts,
845and here is where the lvalue-ness of C<incr_text> comes in useful:
846
847 my $text = "[1],[2], [3]";
848 my $json = new JSON::XS;
849
850 # void context, so no parsing done
851 $json->incr_parse ($text);
852
853 # now extract as many objects as possible. note the
854 # use of scalar context so incr_text can be called.
855 while (my $obj = $json->incr_parse) {
856 # do something with $obj
857
858 # now skip the optional comma
859 $json->incr_text =~ s/^ \s* , //x;
860 }
861
862Now lets go for a very complex example: Assume that you have a gigantic
863JSON array-of-objects, many gigabytes in size, and you want to parse it,
864but you cannot load it into memory fully (this has actually happened in
865the real world :).
866
867Well, you lost, you have to implement your own JSON parser. But JSON::XS
868can still help you: You implement a (very simple) array parser and let
869JSON decode the array elements, which are all full JSON objects on their
870own (this wouldn't work if the array elements could be JSON numbers, for
871example):
872
873 my $json = new JSON::XS;
874
875 # open the monster
876 open my $fh, "<bigfile.json"
877 or die "bigfile: $!";
878
879 # first parse the initial "["
880 for (;;) {
881 sysread $fh, my $buf, 65536
882 or die "read error: $!";
883 $json->incr_parse ($buf); # void context, so no parsing
884
885 # Exit the loop once we found and removed(!) the initial "[".
886 # In essence, we are (ab-)using the $json object as a simple scalar
887 # we append data to.
888 last if $json->incr_text =~ s/^ \s* \[ //x;
889 }
890
891 # now we have the skipped the initial "[", so continue
892 # parsing all the elements.
893 for (;;) {
894 # in this loop we read data until we got a single JSON object
895 for (;;) {
896 if (my $obj = $json->incr_parse) {
897 # do something with $obj
898 last;
899 }
900
901 # add more data
902 sysread $fh, my $buf, 65536
903 or die "read error: $!";
904 $json->incr_parse ($buf); # void context, so no parsing
905 }
906
907 # in this loop we read data until we either found and parsed the
908 # separating "," between elements, or the final "]"
909 for (;;) {
910 # first skip whitespace
911 $json->incr_text =~ s/^\s*//;
912
913 # if we find "]", we are done
914 if ($json->incr_text =~ s/^\]//) {
915 print "finished.\n";
916 exit;
917 }
918
919 # if we find ",", we can continue with the next element
920 if ($json->incr_text =~ s/^,//) {
921 last;
922 }
923
924 # if we find anything else, we have a parse error!
925 if (length $json->incr_text) {
926 die "parse error near ", $json->incr_text;
927 }
928
929 # else add more data
930 sysread $fh, my $buf, 65536
931 or die "read error: $!";
932 $json->incr_parse ($buf); # void context, so no parsing
933 }
934
935This is a complex example, but most of the complexity comes from the fact
936that we are trying to be correct (bear with me if I am wrong, I never ran
937the above example :).
938
672 939
673 940
674=head1 MAPPING 941=head1 MAPPING
675 942
676This section describes how JSON::XS maps Perl values to JSON values and 943This section describes how JSON::XS maps Perl values to JSON values and
706 973
707A JSON number becomes either an integer, numeric (floating point) or 974A JSON number becomes either an integer, numeric (floating point) or
708string scalar in perl, depending on its range and any fractional parts. On 975string scalar in perl, depending on its range and any fractional parts. On
709the Perl level, there is no difference between those as Perl handles all 976the Perl level, there is no difference between those as Perl handles all
710the conversion details, but an integer may take slightly less memory and 977the conversion details, but an integer may take slightly less memory and
711might represent more values exactly than (floating point) numbers. 978might represent more values exactly than floating point numbers.
712 979
713If the number consists of digits only, JSON::XS will try to represent 980If the number consists of digits only, JSON::XS will try to represent
714it as an integer value. If that fails, it will try to represent it as 981it as an integer value. If that fails, it will try to represent it as
715a numeric (floating point) value if that is possible without loss of 982a numeric (floating point) value if that is possible without loss of
716precision. Otherwise it will preserve the number as a string value. 983precision. Otherwise it will preserve the number as a string value (in
984which case you lose roundtripping ability, as the JSON number will be
985re-encoded to a JSON string).
717 986
718Numbers containing a fractional or exponential part will always be 987Numbers containing a fractional or exponential part will always be
719represented as numeric (floating point) values, possibly at a loss of 988represented as numeric (floating point) values, possibly at a loss of
720precision. 989precision (in which case you might lose perfect roundtripping ability, but
990the JSON number will still be re-encoded as a JSON number).
721 991
722This might create round-tripping problems as numbers might become strings, 992Note that precision is not accuracy - binary floating point values cannot
723but as Perl is typeless there is no other way to do it. 993represent most decimal fractions exactly, and when converting from and to
994floating point, JSON::XS only guarantees precision up to but not including
995the least significant bit.
724 996
725=item true, false 997=item true, false
726 998
727These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>, 999These JSON atoms become C<Types::Serialiser::true> and
728respectively. They are overloaded to act almost exactly like the numbers 1000C<Types::Serialiser::false>, respectively. They are overloaded to act
729C<1> and C<0>. You can check whether a scalar is a JSON boolean by using 1001almost exactly like the numbers C<1> and C<0>. You can check whether
730the C<JSON::XS::is_bool> function. 1002a scalar is a JSON boolean by using the C<Types::Serialiser::is_bool>
1003function (after C<use Types::Serialier>, of course).
731 1004
732=item null 1005=item null
733 1006
734A JSON null atom becomes C<undef> in Perl. 1007A JSON null atom becomes C<undef> in Perl.
1008
1009=item shell-style comments (C<< # I<text> >>)
1010
1011As a nonstandard extension to the JSON syntax that is enabled by the
1012C<relaxed> setting, shell-style comments are allowed. They can start
1013anywhere outside strings and go till the end of the line.
1014
1015=item tagged values (C<< (I<tag>)I<value> >>).
1016
1017Another nonstandard extension to the JSON syntax, enabled with the
1018C<allow_tags> setting, are tagged values. In this implementation, the
1019I<tag> must be a perl package/class name encoded as a JSON string, and the
1020I<value> must be a JSON array encoding optional constructor arguments.
1021
1022See L<OBJECT SERIALISATION>, below, for details.
735 1023
736=back 1024=back
737 1025
738 1026
739=head2 PERL -> JSON 1027=head2 PERL -> JSON
744 1032
745=over 4 1033=over 4
746 1034
747=item hash references 1035=item hash references
748 1036
749Perl hash references become JSON objects. As there is no inherent ordering 1037Perl hash references become JSON objects. As there is no inherent
750in hash keys (or JSON objects), they will usually be encoded in a 1038ordering in hash keys (or JSON objects), they will usually be encoded
751pseudo-random order that can change between runs of the same program but 1039in a pseudo-random order. JSON::XS can optionally sort the hash keys
752stays generally the same within a single run of a program. JSON::XS can 1040(determined by the I<canonical> flag), so the same datastructure will
753optionally sort the hash keys (determined by the I<canonical> flag), so 1041serialise to the same JSON text (given same settings and version of
754the same datastructure will serialise to the same JSON text (given same 1042JSON::XS), but this incurs a runtime overhead and is only rarely useful,
755settings and version of JSON::XS), but this incurs a runtime overhead 1043e.g. when you want to compare some JSON text against another for equality.
756and is only rarely useful, e.g. when you want to compare some JSON text
757against another for equality.
758 1044
759=item array references 1045=item array references
760 1046
761Perl array references become JSON arrays. 1047Perl array references become JSON arrays.
762 1048
763=item other references 1049=item other references
764 1050
765Other unblessed references are generally not allowed and will cause an 1051Other unblessed references are generally not allowed and will cause an
766exception to be thrown, except for references to the integers C<0> and 1052exception to be thrown, except for references to the integers C<0> and
767C<1>, which get turned into C<false> and C<true> atoms in JSON. You can 1053C<1>, which get turned into C<false> and C<true> atoms in JSON.
768also use C<JSON::XS::false> and C<JSON::XS::true> to improve readability.
769 1054
1055Since C<JSON::XS> uses the boolean model from L<Types::Serialiser>, you
1056can also C<use Types::Serialiser> and then use C<Types::Serialiser::false>
1057and C<Types::Serialiser::true> to improve readability.
1058
1059 use Types::Serialiser;
770 encode_json [\0,JSON::XS::true] # yields [false,true] 1060 encode_json [\0, Types::Serialiser::true] # yields [false,true]
771 1061
772=item JSON::XS::true, JSON::XS::false 1062=item Types::Serialiser::true, Types::Serialiser::false
773 1063
774These special values become JSON true and JSON false values, 1064These special values from the L<Types::Serialiser> module become JSON true
775respectively. You can also use C<\1> and C<\0> directly if you want. 1065and JSON false values, respectively. You can also use C<\1> and C<\0>
1066directly if you want.
776 1067
777=item blessed objects 1068=item blessed objects
778 1069
779Blessed objects are not allowed. JSON::XS currently tries to encode their 1070Blessed objects are not directly representable in JSON, but C<JSON::XS>
780underlying representation (hash- or arrayref), but this behaviour might 1071allows various ways of handling objects. See L<OBJECT SERIALISATION>,
781change in future versions. 1072below, for details.
782 1073
783=item simple scalars 1074=item simple scalars
784 1075
785Simple Perl scalars (any scalar that is not a reference) are the most 1076Simple Perl scalars (any scalar that is not a reference) are the most
786difficult objects to encode: JSON::XS will encode undefined scalars as 1077difficult objects to encode: JSON::XS will encode undefined scalars as
787JSON null value, scalars that have last been used in a string context 1078JSON C<null> values, scalars that have last been used in a string context
788before encoding as JSON strings and anything else as number value: 1079before encoding as JSON strings, and anything else as number value:
789 1080
790 # dump as number 1081 # dump as number
791 encode_json [2] # yields [2] 1082 encode_json [2] # yields [2]
792 encode_json [-3.0e17] # yields [-3e+17] 1083 encode_json [-3.0e17] # yields [-3e+17]
793 my $value = 5; encode_json [$value] # yields [5] 1084 my $value = 5; encode_json [$value] # yields [5]
811 my $x = "3"; # some variable containing a string 1102 my $x = "3"; # some variable containing a string
812 $x += 0; # numify it, ensuring it will be dumped as a number 1103 $x += 0; # numify it, ensuring it will be dumped as a number
813 $x *= 1; # same thing, the choice is yours. 1104 $x *= 1; # same thing, the choice is yours.
814 1105
815You can not currently force the type in other, less obscure, ways. Tell me 1106You can not currently force the type in other, less obscure, ways. Tell me
816if you need this capability. 1107if you need this capability (but don't forget to explain why it's needed
1108:).
1109
1110Note that numerical precision has the same meaning as under Perl (so
1111binary to decimal conversion follows the same rules as in Perl, which
1112can differ to other languages). Also, your perl interpreter might expose
1113extensions to the floating point numbers of your platform, such as
1114infinities or NaN's - these cannot be represented in JSON, and it is an
1115error to pass those in.
817 1116
818=back 1117=back
819 1118
1119=head2 OBJECT SERIALISATION
820 1120
821=head1 COMPARISON 1121As JSON cannot directly represent Perl objects, you have to choose between
1122a pure JSON representation (without the ability to deserialise the object
1123automatically again), and a nonstandard extension to the JSON syntax,
1124tagged values.
822 1125
823As already mentioned, this module was created because none of the existing 1126=head3 SERIALISATION
824JSON modules could be made to work correctly. First I will describe the 1127
825problems (or pleasures) I encountered with various existing JSON modules, 1128What happens when C<JSON::XS> encounters a Perl object depends on the
826followed by some benchmark values. JSON::XS was designed not to suffer 1129C<allow_blessed>, C<convert_blessed> and C<allow_tags> settings, which are
827from any of these problems or limitations. 1130used in this order:
828 1131
829=over 4 1132=over 4
830 1133
831=item JSON 1.07 1134=item 1. C<allow_tags> is enabled and the object has a C<FREEZE> method.
832 1135
833Slow (but very portable, as it is written in pure Perl). 1136In this case, C<JSON::XS> uses the L<Types::Serialiser> object
1137serialisation protocol to create a tagged JSON value, using a nonstandard
1138extension to the JSON syntax.
834 1139
835Undocumented/buggy Unicode handling (how JSON handles Unicode values is 1140This works by invoking the C<FREEZE> method on the object, with the first
836undocumented. One can get far by feeding it Unicode strings and doing 1141argument being the object to serialise, and the second argument being the
837en-/decoding oneself, but Unicode escapes are not working properly). 1142constant string C<JSON> to distinguish it from other serialisers.
838 1143
839No round-tripping (strings get clobbered if they look like numbers, e.g. 1144The C<FREEZE> method can return any number of values (i.e. zero or
840the string C<2.0> will encode to C<2.0> instead of C<"2.0">, and that will 1145more). These values and the paclkage/classname of the object will then be
841decode into the number 2. 1146encoded as a tagged JSON value in the following format:
842 1147
843=item JSON::PC 0.01 1148 ("classname")[FREEZE return values...]
844 1149
845Very fast. 1150For example, the hypothetical C<My::Object> C<FREEZE> method might use the
1151objects C<type> and C<id> members to encode the object:
846 1152
847Undocumented/buggy Unicode handling. 1153 sub My::Object::FREEZE {
1154 my ($self, $serialiser) = @_;
848 1155
849No round-tripping. 1156 ($self->{type}, $self->{id})
1157 }
850 1158
851Has problems handling many Perl values (e.g. regex results and other magic 1159=item 2. C<convert_blessed> is enabled and the object has a C<TO_JSON> method.
852values will make it croak).
853 1160
854Does not even generate valid JSON (C<{1,2}> gets converted to C<{1:2}> 1161In this case, the C<TO_JSON> method of the object is invoked in scalar
855which is not a valid JSON text. 1162context. It must return a single scalar that can be directly encoded into
1163JSON. This scalar replaces the object in the JSON text.
856 1164
857Unmaintained (maintainer unresponsive for many months, bugs are not 1165For example, the following C<TO_JSON> method will convert all L<URI>
858getting fixed). 1166objects to JSON strings when serialised. The fatc that these values
1167originally were L<URI> objects is lost.
859 1168
860=item JSON::Syck 0.21 1169 sub URI::TO_JSON {
1170 my ($uri) = @_;
1171 $uri->as_string
1172 }
861 1173
862Very buggy (often crashes). 1174=item 3. C<allow_blessed> is enabled.
863 1175
864Very inflexible (no human-readable format supported, format pretty much 1176The object will be serialised as a JSON null value.
865undocumented. I need at least a format for easy reading by humans and a
866single-line compact format for use in a protocol, and preferably a way to
867generate ASCII-only JSON texts).
868 1177
869Completely broken (and confusingly documented) Unicode handling (Unicode 1178=item 4. none of the above
870escapes are not working properly, you need to set ImplicitUnicode to
871I<different> values on en- and decoding to get symmetric behaviour).
872 1179
873No round-tripping (simple cases work, but this depends on whether the scalar 1180If none of the settings are enabled or the respective methods are missing,
874value was used in a numeric context or not). 1181C<JSON::XS> throws an exception.
875
876Dumping hashes may skip hash values depending on iterator state.
877
878Unmaintained (maintainer unresponsive for many months, bugs are not
879getting fixed).
880
881Does not check input for validity (i.e. will accept non-JSON input and
882return "something" instead of raising an exception. This is a security
883issue: imagine two banks transferring money between each other using
884JSON. One bank might parse a given non-JSON request and deduct money,
885while the other might reject the transaction with a syntax error. While a
886good protocol will at least recover, that is extra unnecessary work and
887the transaction will still not succeed).
888
889=item JSON::DWIW 0.04
890
891Very fast. Very natural. Very nice.
892
893Undocumented Unicode handling (but the best of the pack. Unicode escapes
894still don't get parsed properly).
895
896Very inflexible.
897
898No round-tripping.
899
900Does not generate valid JSON texts (key strings are often unquoted, empty keys
901result in nothing being output)
902
903Does not check input for validity.
904 1182
905=back 1183=back
906 1184
1185=head3 DESERIALISATION
1186
1187For deserialisation there are only two cases to consider: either
1188nonstandard tagging was used, in which case C<allow_tags> decides,
1189or objects cannot be automatically be deserialised, in which
1190case you can use postprocessing or the C<filter_json_object> or
1191C<filter_json_single_key_object> callbacks to get some real objects our of
1192your JSON.
1193
1194This section only considers the tagged value case: I a tagged JSON object
1195is encountered during decoding and C<allow_tags> is disabled, a parse
1196error will result (as if tagged values were not part of the grammar).
1197
1198If C<allow_tags> is enabled, C<JSON::XS> will look up the C<THAW> method
1199of the package/classname used during serialisation (it will not attempt
1200to load the package as a Perl module). If there is no such method, the
1201decoding will fail with an error.
1202
1203Otherwise, the C<THAW> method is invoked with the classname as first
1204argument, the constant string C<JSON> as second argument, and all the
1205values from the JSON array (the values originally returned by the
1206C<FREEZE> method) as remaining arguments.
1207
1208The method must then return the object. While technically you can return
1209any Perl scalar, you might have to enable the C<enable_nonref> setting to
1210make that work in all cases, so better return an actual blessed reference.
1211
1212As an example, let's implement a C<THAW> function that regenerates the
1213C<My::Object> from the C<FREEZE> example earlier:
1214
1215 sub My::Object::THAW {
1216 my ($class, $serialiser, $type, $id) = @_;
1217
1218 $class->new (type => $type, id => $id)
1219 }
1220
1221
1222=head1 ENCODING/CODESET FLAG NOTES
1223
1224The interested reader might have seen a number of flags that signify
1225encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be
1226some confusion on what these do, so here is a short comparison:
1227
1228C<utf8> controls whether the JSON text created by C<encode> (and expected
1229by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only
1230control whether C<encode> escapes character values outside their respective
1231codeset range. Neither of these flags conflict with each other, although
1232some combinations make less sense than others.
1233
1234Care has been taken to make all flags symmetrical with respect to
1235C<encode> and C<decode>, that is, texts encoded with any combination of
1236these flag values will be correctly decoded when the same flags are used
1237- in general, if you use different flag settings while encoding vs. when
1238decoding you likely have a bug somewhere.
1239
1240Below comes a verbose discussion of these flags. Note that a "codeset" is
1241simply an abstract set of character-codepoint pairs, while an encoding
1242takes those codepoint numbers and I<encodes> them, in our case into
1243octets. Unicode is (among other things) a codeset, UTF-8 is an encoding,
1244and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at
1245the same time, which can be confusing.
1246
1247=over 4
1248
1249=item C<utf8> flag disabled
1250
1251When C<utf8> is disabled (the default), then C<encode>/C<decode> generate
1252and expect Unicode strings, that is, characters with high ordinal Unicode
1253values (> 255) will be encoded as such characters, and likewise such
1254characters are decoded as-is, no changes to them will be done, except
1255"(re-)interpreting" them as Unicode codepoints or Unicode characters,
1256respectively (to Perl, these are the same thing in strings unless you do
1257funny/weird/dumb stuff).
1258
1259This is useful when you want to do the encoding yourself (e.g. when you
1260want to have UTF-16 encoded JSON texts) or when some other layer does
1261the encoding for you (for example, when printing to a terminal using a
1262filehandle that transparently encodes to UTF-8 you certainly do NOT want
1263to UTF-8 encode your data first and have Perl encode it another time).
1264
1265=item C<utf8> flag enabled
1266
1267If the C<utf8>-flag is enabled, C<encode>/C<decode> will encode all
1268characters using the corresponding UTF-8 multi-byte sequence, and will
1269expect your input strings to be encoded as UTF-8, that is, no "character"
1270of the input string must have any value > 255, as UTF-8 does not allow
1271that.
1272
1273The C<utf8> flag therefore switches between two modes: disabled means you
1274will get a Unicode string in Perl, enabled means you get an UTF-8 encoded
1275octet/binary string in Perl.
1276
1277=item C<latin1> or C<ascii> flags enabled
1278
1279With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters
1280with ordinal values > 255 (> 127 with C<ascii>) and encode the remaining
1281characters as specified by the C<utf8> flag.
1282
1283If C<utf8> is disabled, then the result is also correctly encoded in those
1284character sets (as both are proper subsets of Unicode, meaning that a
1285Unicode string with all character values < 256 is the same thing as a
1286ISO-8859-1 string, and a Unicode string with all character values < 128 is
1287the same thing as an ASCII string in Perl).
1288
1289If C<utf8> is enabled, you still get a correct UTF-8-encoded string,
1290regardless of these flags, just some more characters will be escaped using
1291C<\uXXXX> then before.
1292
1293Note that ISO-8859-1-I<encoded> strings are not compatible with UTF-8
1294encoding, while ASCII-encoded strings are. That is because the ISO-8859-1
1295encoding is NOT a subset of UTF-8 (despite the ISO-8859-1 I<codeset> being
1296a subset of Unicode), while ASCII is.
1297
1298Surprisingly, C<decode> will ignore these flags and so treat all input
1299values as governed by the C<utf8> flag. If it is disabled, this allows you
1300to decode ISO-8859-1- and ASCII-encoded strings, as both strict subsets of
1301Unicode. If it is enabled, you can correctly decode UTF-8 encoded strings.
1302
1303So neither C<latin1> nor C<ascii> are incompatible with the C<utf8> flag -
1304they only govern when the JSON output engine escapes a character or not.
1305
1306The main use for C<latin1> is to relatively efficiently store binary data
1307as JSON, at the expense of breaking compatibility with most JSON decoders.
1308
1309The main use for C<ascii> is to force the output to not contain characters
1310with values > 127, which means you can interpret the resulting string
1311as UTF-8, ISO-8859-1, ASCII, KOI8-R or most about any character set and
13128-bit-encoding, and still get the same data structure back. This is useful
1313when your channel for JSON transfer is not 8-bit clean or the encoding
1314might be mangled in between (e.g. in mail), and works because ASCII is a
1315proper subset of most 8-bit and multibyte encodings in use in the world.
1316
1317=back
1318
1319
1320=head2 JSON and ECMAscript
1321
1322JSON syntax is based on how literals are represented in javascript (the
1323not-standardised predecessor of ECMAscript) which is presumably why it is
1324called "JavaScript Object Notation".
1325
1326However, JSON is not a subset (and also not a superset of course) of
1327ECMAscript (the standard) or javascript (whatever browsers actually
1328implement).
1329
1330If you want to use javascript's C<eval> function to "parse" JSON, you
1331might run into parse errors for valid JSON texts, or the resulting data
1332structure might not be queryable:
1333
1334One of the problems is that U+2028 and U+2029 are valid characters inside
1335JSON strings, but are not allowed in ECMAscript string literals, so the
1336following Perl fragment will not output something that can be guaranteed
1337to be parsable by javascript's C<eval>:
1338
1339 use JSON::XS;
1340
1341 print encode_json [chr 0x2028];
1342
1343The right fix for this is to use a proper JSON parser in your javascript
1344programs, and not rely on C<eval> (see for example Douglas Crockford's
1345F<json2.js> parser).
1346
1347If this is not an option, you can, as a stop-gap measure, simply encode to
1348ASCII-only JSON:
1349
1350 use JSON::XS;
1351
1352 print JSON::XS->new->ascii->encode ([chr 0x2028]);
1353
1354Note that this will enlarge the resulting JSON text quite a bit if you
1355have many non-ASCII characters. You might be tempted to run some regexes
1356to only escape U+2028 and U+2029, e.g.:
1357
1358 # DO NOT USE THIS!
1359 my $json = JSON::XS->new->utf8->encode ([chr 0x2028]);
1360 $json =~ s/\xe2\x80\xa8/\\u2028/g; # escape U+2028
1361 $json =~ s/\xe2\x80\xa9/\\u2029/g; # escape U+2029
1362 print $json;
1363
1364Note that I<this is a bad idea>: the above only works for U+2028 and
1365U+2029 and thus only for fully ECMAscript-compliant parsers. Many existing
1366javascript implementations, however, have issues with other characters as
1367well - using C<eval> naively simply I<will> cause problems.
1368
1369Another problem is that some javascript implementations reserve
1370some property names for their own purposes (which probably makes
1371them non-ECMAscript-compliant). For example, Iceweasel reserves the
1372C<__proto__> property name for its own purposes.
1373
1374If that is a problem, you could parse try to filter the resulting JSON
1375output for these property strings, e.g.:
1376
1377 $json =~ s/"__proto__"\s*:/"__proto__renamed":/g;
1378
1379This works because C<__proto__> is not valid outside of strings, so every
1380occurrence of C<"__proto__"\s*:> must be a string used as property name.
1381
1382If you know of other incompatibilities, please let me know.
1383
907 1384
908=head2 JSON and YAML 1385=head2 JSON and YAML
909 1386
910You often hear that JSON is a subset of YAML. This is, however, a mass 1387You often hear that JSON is a subset of YAML. This is, however, a mass
911hysteria and very far from the truth. In general, there is no way to 1388hysteria(*) and very far from the truth (as of the time of this writing),
1389so let me state it clearly: I<in general, there is no way to configure
912configure JSON::XS to output a data structure as valid YAML that works for 1390JSON::XS to output a data structure as valid YAML> that works in all
913all cases. 1391cases.
914 1392
915If you really must use JSON::XS to generate YAML, you should use this 1393If you really must use JSON::XS to generate YAML, you should use this
916algorithm (subject to change in future versions): 1394algorithm (subject to change in future versions):
917 1395
918 my $to_yaml = JSON::XS->new->utf8->space_after (1); 1396 my $to_yaml = JSON::XS->new->utf8->space_after (1);
919 my $yaml = $to_yaml->encode ($ref) . "\n"; 1397 my $yaml = $to_yaml->encode ($ref) . "\n";
920 1398
921This will usually generate JSON texts that also parse as valid 1399This will I<usually> generate JSON texts that also parse as valid
922YAML. Please note that YAML has hardcoded limits on (simple) object key 1400YAML. Please note that YAML has hardcoded limits on (simple) object key
923lengths that JSON doesn't have and also has different and incompatible 1401lengths that JSON doesn't have and also has different and incompatible
924unicode handling, so you should make sure that your hash keys are 1402unicode character escape syntax, so you should make sure that your hash
925noticeably shorter than the 1024 "stream characters" YAML allows and that 1403keys are noticeably shorter than the 1024 "stream characters" YAML allows
926you do not have codepoints with values outside the Unicode BMP (basic 1404and that you do not have characters with codepoint values outside the
927multilingual page). YAML also does not allow C<\/> sequences in strings 1405Unicode BMP (basic multilingual page). YAML also does not allow C<\/>
928(which JSON::XS does not I<currently> generate). 1406sequences in strings (which JSON::XS does not I<currently> generate, but
1407other JSON generators might).
929 1408
930There might be other incompatibilities that I am not aware of. In general 1409There might be other incompatibilities that I am not aware of (or the YAML
1410specification has been changed yet again - it does so quite often). In
931you should not try to generate YAML with a JSON generator or vice versa, 1411general you should not try to generate YAML with a JSON generator or vice
932or try to parse JSON with a YAML parser or vice versa: chances are high 1412versa, or try to parse JSON with a YAML parser or vice versa: chances are
933that you will run into severe interoperability problems when you least 1413high that you will run into severe interoperability problems when you
934expect it. 1414least expect it.
1415
1416=over 4
1417
1418=item (*)
1419
1420I have been pressured multiple times by Brian Ingerson (one of the
1421authors of the YAML specification) to remove this paragraph, despite him
1422acknowledging that the actual incompatibilities exist. As I was personally
1423bitten by this "JSON is YAML" lie, I refused and said I will continue to
1424educate people about these issues, so others do not run into the same
1425problem again and again. After this, Brian called me a (quote)I<complete
1426and worthless idiot>(unquote).
1427
1428In my opinion, instead of pressuring and insulting people who actually
1429clarify issues with YAML and the wrong statements of some of its
1430proponents, I would kindly suggest reading the JSON spec (which is not
1431that difficult or long) and finally make YAML compatible to it, and
1432educating users about the changes, instead of spreading lies about the
1433real compatibility for many I<years> and trying to silence people who
1434point out that it isn't true.
1435
1436Addendum/2009: the YAML 1.2 spec is still incompatible with JSON, even
1437though the incompatibilities have been documented (and are known to Brian)
1438for many years and the spec makes explicit claims that YAML is a superset
1439of JSON. It would be so easy to fix, but apparently, bullying people and
1440corrupting userdata is so much easier.
1441
1442=back
935 1443
936 1444
937=head2 SPEED 1445=head2 SPEED
938 1446
939It seems that JSON::XS is surprisingly fast, as shown in the following 1447It seems that JSON::XS is surprisingly fast, as shown in the following
940tables. They have been generated with the help of the C<eg/bench> program 1448tables. They have been generated with the help of the C<eg/bench> program
941in the JSON::XS distribution, to make it easy to compare on your own 1449in the JSON::XS distribution, to make it easy to compare on your own
942system. 1450system.
943 1451
944First comes a comparison between various modules using a very short 1452First comes a comparison between various modules using
945single-line JSON string: 1453a very short single-line JSON string (also available at
1454L<http://dist.schmorp.de/misc/json/short.json>).
946 1455
947 {"method": "handleMessage", "params": ["user1", "we were just talking"], \ 1456 {"method": "handleMessage", "params": ["user1",
948 "id": null, "array":[1,11,234,-5,1e5,1e7, true, false]} 1457 "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
1458 1, 0]}
949 1459
950It shows the number of encodes/decodes per second (JSON::XS uses 1460It shows the number of encodes/decodes per second (JSON::XS uses
951the functional interface, while JSON::XS/2 uses the OO interface 1461the functional interface, while JSON::XS/2 uses the OO interface
952with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables 1462with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
953shrink). Higher is better: 1463shrink. JSON::DWIW/DS uses the deserialise function, while JSON::DWIW::FJ
1464uses the from_json method). Higher is better:
954 1465
955 module | encode | decode | 1466 module | encode | decode |
956 -----------|------------|------------| 1467 --------------|------------|------------|
957 JSON 1.x | 4990.842 | 4088.813 | 1468 JSON::DWIW/DS | 86302.551 | 102300.098 |
958 JSON::DWIW | 51653.990 | 71575.154 | 1469 JSON::DWIW/FJ | 86302.551 | 75983.768 |
959 JSON::PC | 65948.176 | 74631.744 | 1470 JSON::PP | 15827.562 | 6638.658 |
960 JSON::PP | 8931.652 | 3817.168 | 1471 JSON::Syck | 63358.066 | 47662.545 |
961 JSON::Syck | 24877.248 | 27776.848 | 1472 JSON::XS | 511500.488 | 511500.488 |
962 JSON::XS | 388361.481 | 227951.304 | 1473 JSON::XS/2 | 291271.111 | 388361.481 |
963 JSON::XS/2 | 227951.304 | 218453.333 | 1474 JSON::XS/3 | 361577.931 | 361577.931 |
964 JSON::XS/3 | 338250.323 | 218453.333 | 1475 Storable | 66788.280 | 265462.278 |
965 Storable | 16500.016 | 135300.129 |
966 -----------+------------+------------+ 1476 --------------+------------+------------+
967 1477
968That is, JSON::XS is about five times faster than JSON::DWIW on encoding, 1478That is, JSON::XS is almost six times faster than JSON::DWIW on encoding,
969about three times faster on decoding, and over forty times faster 1479about five times faster on decoding, and over thirty to seventy times
970than JSON, even with pretty-printing and key sorting. It also compares 1480faster than JSON's pure perl implementation. It also compares favourably
971favourably to Storable for small amounts of data. 1481to Storable for small amounts of data.
972 1482
973Using a longer test string (roughly 18KB, generated from Yahoo! Locals 1483Using a longer test string (roughly 18KB, generated from Yahoo! Locals
974search API (http://nanoref.com/yahooapis/mgPdGg): 1484search API (L<http://dist.schmorp.de/misc/json/long.json>).
975 1485
976 module | encode | decode | 1486 module | encode | decode |
977 -----------|------------|------------| 1487 --------------|------------|------------|
978 JSON 1.x | 55.260 | 34.971 | 1488 JSON::DWIW/DS | 1647.927 | 2673.916 |
979 JSON::DWIW | 825.228 | 1082.513 | 1489 JSON::DWIW/FJ | 1630.249 | 2596.128 |
980 JSON::PC | 3571.444 | 2394.829 |
981 JSON::PP | 210.987 | 32.574 | 1490 JSON::PP | 400.640 | 62.311 |
982 JSON::Syck | 552.551 | 787.544 | 1491 JSON::Syck | 1481.040 | 1524.869 |
983 JSON::XS | 5780.463 | 4854.519 | 1492 JSON::XS | 20661.596 | 9541.183 |
984 JSON::XS/2 | 3869.998 | 4798.975 | 1493 JSON::XS/2 | 10683.403 | 9416.938 |
985 JSON::XS/3 | 5862.880 | 4798.975 | 1494 JSON::XS/3 | 20661.596 | 9400.054 |
986 Storable | 4445.002 | 5235.027 | 1495 Storable | 19765.806 | 10000.725 |
987 -----------+------------+------------+ 1496 --------------+------------+------------+
988 1497
989Again, JSON::XS leads by far (except for Storable which non-surprisingly 1498Again, JSON::XS leads by far (except for Storable which non-surprisingly
990decodes faster). 1499decodes a bit faster).
991 1500
992On large strings containing lots of high Unicode characters, some modules 1501On large strings containing lots of high Unicode characters, some modules
993(such as JSON::PC) seem to decode faster than JSON::XS, but the result 1502(such as JSON::PC) seem to decode faster than JSON::XS, but the result
994will be broken due to missing (or wrong) Unicode handling. Others refuse 1503will be broken due to missing (or wrong) Unicode handling. Others refuse
995to decode or encode properly, so it was impossible to prepare a fair 1504to decode or encode properly, so it was impossible to prepare a fair
1021to free the temporary). If that is exceeded, the program crashes. To be 1530to free the temporary). If that is exceeded, the program crashes. To be
1022conservative, the default nesting limit is set to 512. If your process 1531conservative, the default nesting limit is set to 512. If your process
1023has a smaller stack, you should adjust this setting accordingly with the 1532has a smaller stack, you should adjust this setting accordingly with the
1024C<max_depth> method. 1533C<max_depth> method.
1025 1534
1026And last but least, something else could bomb you that I forgot to think 1535Something else could bomb you, too, that I forgot to think of. In that
1027of. In that case, you get to keep the pieces. I am always open for hints, 1536case, you get to keep the pieces. I am always open for hints, though...
1028though... 1537
1538Also keep in mind that JSON::XS might leak contents of your Perl data
1539structures in its error messages, so when you serialise sensitive
1540information you might want to make sure that exceptions thrown by JSON::XS
1541will not end up in front of untrusted eyes.
1029 1542
1030If you are using JSON::XS to return packets to consumption 1543If you are using JSON::XS to return packets to consumption
1031by JavaScript scripts in a browser you should have a look at 1544by JavaScript scripts in a browser you should have a look at
1032L<http://jpsykes.com/47/practical-csrf-and-json-security> to see whether 1545L<http://blog.archive.jpsykes.com/47/practical-csrf-and-json-security/> to
1033you are vulnerable to some common attack vectors (which really are browser 1546see whether you are vulnerable to some common attack vectors (which really
1034design bugs, but it is still you who will have to deal with it, as major 1547are browser design bugs, but it is still you who will have to deal with
1035browser developers care only for features, not about getting security 1548it, as major browser developers care only for features, not about getting
1036right). 1549security right).
1550
1551
1552=head1 INTEROPERABILITY WITH OTHER MODULES
1553
1554C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
1555constants. That means that the JSON true and false values will be
1556comaptible to true and false values of iother modules that do the same,
1557such as L<JSON::PP> and L<CBOR::XS>.
1037 1558
1038 1559
1039=head1 THREADS 1560=head1 THREADS
1040 1561
1041This module is I<not> guaranteed to be thread safe and there are no 1562This module is I<not> guaranteed to be thread safe and there are no
1042plans to change this until Perl gets thread support (as opposed to the 1563plans to change this until Perl gets thread support (as opposed to the
1043horribly slow so-called "threads" which are simply slow and bloated 1564horribly slow so-called "threads" which are simply slow and bloated
1044process simulations - use fork, its I<much> faster, cheaper, better). 1565process simulations - use fork, it's I<much> faster, cheaper, better).
1045 1566
1046(It might actually work, but you have been warned). 1567(It might actually work, but you have been warned).
1047 1568
1048 1569
1570=head1 THE PERILS OF SETLOCALE
1571
1572Sometimes people avoid the Perl locale support and directly call the
1573system's setlocale function with C<LC_ALL>.
1574
1575This breaks both perl and modules such as JSON::XS, as stringification of
1576numbers no longer works correctly (e.g. C<$x = 0.1; print "$x"+1> might
1577print C<1>, and JSON::XS might output illegal JSON as JSON::XS relies on
1578perl to stringify numbers).
1579
1580The solution is simple: don't call C<setlocale>, or use it for only those
1581categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>.
1582
1583If you need C<LC_NUMERIC>, you should enable it only around the code that
1584actually needs it (avoiding stringification of numbers), and restore it
1585afterwards.
1586
1587
1049=head1 BUGS 1588=head1 BUGS
1050 1589
1051While the goal of this module is to be correct, that unfortunately does 1590While the goal of this module is to be correct, that unfortunately does
1052not mean its bug-free, only that I think its design is bug-free. It is 1591not mean it's bug-free, only that I think its design is bug-free. If you
1053still relatively early in its development. If you keep reporting bugs they 1592keep reporting bugs they will be fixed swiftly, though.
1054will be fixed swiftly, though.
1055 1593
1056Please refrain from using rt.cpan.org or any other bug reporting 1594Please refrain from using rt.cpan.org or any other bug reporting
1057service. I put the contact address into my modules for a reason. 1595service. I put the contact address into my modules for a reason.
1058 1596
1059=cut 1597=cut
1060 1598
1061our $true = do { bless \(my $dummy = 1), "JSON::XS::Boolean" }; 1599BEGIN {
1062our $false = do { bless \(my $dummy = 0), "JSON::XS::Boolean" }; 1600 *true = \$Types::Serialiser::true;
1601 *true = \&Types::Serialiser::true;
1602 *false = \$Types::Serialiser::false;
1603 *false = \&Types::Serialiser::false;
1604 *is_bool = \&Types::Serialiser::is_bool;
1063 1605
1064sub true() { $true } 1606 *JSON::XS::Boolean:: = *Types::Serialiser::Boolean::;
1065sub false() { $false }
1066
1067sub is_bool($) {
1068 UNIVERSAL::isa $_[0], "JSON::XS::Boolean"
1069# or UNIVERSAL::isa $_[0], "JSON::Literal"
1070} 1607}
1071 1608
1072XSLoader::load "JSON::XS", $VERSION; 1609XSLoader::load "JSON::XS", $VERSION;
1073 1610
1074package JSON::XS::Boolean; 1611=head1 SEE ALSO
1075 1612
1076use overload 1613The F<json_xs> command line utility for quick experiments.
1077 "0+" => sub { ${$_[0]} },
1078 "++" => sub { $_[0] = ${$_[0]} + 1 },
1079 "--" => sub { $_[0] = ${$_[0]} - 1 },
1080 fallback => 1;
1081
10821;
1083 1614
1084=head1 AUTHOR 1615=head1 AUTHOR
1085 1616
1086 Marc Lehmann <schmorp@schmorp.de> 1617 Marc Lehmann <schmorp@schmorp.de>
1087 http://home.schmorp.de/ 1618 http://home.schmorp.de/
1088 1619
1089=cut 1620=cut
1090 1621
16221
1623

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines