ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
Revision: 1.144
Committed: Mon Oct 28 23:19:54 2013 UTC (10 years, 6 months ago) by root
Branch: MAIN
Changes since 1.143: +35 -37 lines
Log Message:
*** empty log message ***

File Contents

# Content
1 =head1 NAME
2
3 JSON::XS - JSON serialising/deserialising, done correctly and fast
4
5 =encoding utf-8
6
7 JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ
8 (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html)
9
10 =head1 SYNOPSIS
11
12 use JSON::XS;
13
14 # exported functions, they croak on error
15 # and expect/generate UTF-8
16
17 $utf8_encoded_json_text = encode_json $perl_hash_or_arrayref;
18 $perl_hash_or_arrayref = decode_json $utf8_encoded_json_text;
19
20 # OO-interface
21
22 $coder = JSON::XS->new->ascii->pretty->allow_nonref;
23 $pretty_printed_unencoded = $coder->encode ($perl_scalar);
24 $perl_scalar = $coder->decode ($unicode_json_text);
25
26 # Note that JSON version 2.0 and above will automatically use JSON::XS
27 # if available, at virtually no speed overhead either, so you should
28 # be able to just:
29
30 use JSON;
31
32 # and do the same things, except that you have a pure-perl fallback now.
33
34 =head1 DESCRIPTION
35
36 This module converts Perl data structures to JSON and vice versa. Its
37 primary goal is to be I<correct> and its secondary goal is to be
38 I<fast>. To reach the latter goal it was written in C.
39
40 Beginning with version 2.0 of the JSON module, when both JSON and
41 JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42 overridden) with no overhead due to emulation (by inheriting constructor
43 and methods). If JSON::XS is not available, it will fall back to the
44 compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45 gives you a portable JSON API that can be fast when you need and doesn't
46 require a C compiler when that is a problem.
47
48 As this is the n-th-something JSON module on CPAN, what was the reason
49 to write yet another JSON module? While it seems there are many JSON
50 modules, none of them correctly handle all corner cases, and in most cases
51 their maintainers are unresponsive, gone missing, or not listening to bug
52 reports for other reasons.
53
54 See MAPPING, below, on how JSON::XS maps perl values to JSON values and
55 vice versa.
56
57 =head2 FEATURES
58
59 =over 4
60
61 =item * correct Unicode handling
62
63 This module knows how to handle Unicode, documents how and when it does
64 so, and even documents what "correct" means.
65
66 =item * round-trip integrity
67
68 When you serialise a perl data structure using only data types supported
69 by JSON and Perl, the deserialised data structure is identical on the Perl
70 level. (e.g. the string "2.0" doesn't suddenly become "2" just because
71 it looks like a number). There I<are> minor exceptions to this, read the
72 MAPPING section below to learn about those.
73
74 =item * strict checking of JSON correctness
75
76 There is no guessing, no generating of illegal JSON texts by default,
77 and only JSON is accepted as input by default (the latter is a security
78 feature).
79
80 =item * fast
81
82 Compared to other JSON modules and other serialisers such as Storable,
83 this module usually compares favourably in terms of speed, too.
84
85 =item * simple to use
86
87 This module has both a simple functional interface as well as an object
88 oriented interface.
89
90 =item * reasonably versatile output formats
91
92 You can choose between the most compact guaranteed-single-line format
93 possible (nice for simple line-based protocols), a pure-ASCII format
94 (for when your transport is not 8-bit clean, still supports the whole
95 Unicode range), or a pretty-printed format (for when you want to read that
96 stuff). Or you can combine those features in whatever way you like.
97
98 =back
99
100 =cut
101
102 package JSON::XS;
103
104 use common::sense;
105
106 our $VERSION = 2.34;
107 our @ISA = qw(Exporter);
108
109 our @EXPORT = qw(encode_json decode_json);
110
111 use Exporter;
112 use XSLoader;
113
114 use Types::Serialiser ();
115
116 =head1 FUNCTIONAL INTERFACE
117
118 The following convenience methods are provided by this module. They are
119 exported by default:
120
121 =over 4
122
123 =item $json_text = encode_json $perl_scalar
124
125 Converts the given Perl data structure to a UTF-8 encoded, binary string
126 (that is, the string contains octets only). Croaks on error.
127
128 This function call is functionally identical to:
129
130 $json_text = JSON::XS->new->utf8->encode ($perl_scalar)
131
132 Except being faster.
133
134 =item $perl_scalar = decode_json $json_text
135
136 The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries
137 to parse that as an UTF-8 encoded JSON text, returning the resulting
138 reference. Croaks on error.
139
140 This function call is functionally identical to:
141
142 $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
143
144 Except being faster.
145
146 =back
147
148
149 =head1 A FEW NOTES ON UNICODE AND PERL
150
151 Since this often leads to confusion, here are a few very clear words on
152 how Unicode works in Perl, modulo bugs.
153
154 =over 4
155
156 =item 1. Perl strings can store characters with ordinal values > 255.
157
158 This enables you to store Unicode characters as single characters in a
159 Perl string - very natural.
160
161 =item 2. Perl does I<not> associate an encoding with your strings.
162
163 ... until you force it to, e.g. when matching it against a regex, or
164 printing the scalar to a file, in which case Perl either interprets your
165 string as locale-encoded text, octets/binary, or as Unicode, depending
166 on various settings. In no case is an encoding stored together with your
167 data, it is I<use> that decides encoding, not any magical meta data.
168
169 =item 3. The internal utf-8 flag has no meaning with regards to the
170 encoding of your string.
171
172 Just ignore that flag unless you debug a Perl bug, a module written in
173 XS or want to dive into the internals of perl. Otherwise it will only
174 confuse you, as, despite the name, it says nothing about how your string
175 is encoded. You can have Unicode strings with that flag set, with that
176 flag clear, and you can have binary data with that flag set and that flag
177 clear. Other possibilities exist, too.
178
179 If you didn't know about that flag, just the better, pretend it doesn't
180 exist.
181
182 =item 4. A "Unicode String" is simply a string where each character can be
183 validly interpreted as a Unicode code point.
184
185 If you have UTF-8 encoded data, it is no longer a Unicode string, but a
186 Unicode string encoded in UTF-8, giving you a binary string.
187
188 =item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string.
189
190 It's a fact. Learn to live with it.
191
192 =back
193
194 I hope this helps :)
195
196
197 =head1 OBJECT-ORIENTED INTERFACE
198
199 The object oriented interface lets you configure your own encoding or
200 decoding style, within the limits of supported formats.
201
202 =over 4
203
204 =item $json = new JSON::XS
205
206 Creates a new JSON::XS object that can be used to de/encode JSON
207 strings. All boolean flags described below are by default I<disabled>.
208
209 The mutators for flags all return the JSON object again and thus calls can
210 be chained:
211
212 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]})
213 => {"a": [1, 2]}
214
215 =item $json = $json->ascii ([$enable])
216
217 =item $enabled = $json->get_ascii
218
219 If C<$enable> is true (or missing), then the C<encode> method will not
220 generate characters outside the code range C<0..127> (which is ASCII). Any
221 Unicode characters outside that range will be escaped using either a
222 single \uXXXX (BMP characters) or a double \uHHHH\uLLLLL escape sequence,
223 as per RFC4627. The resulting encoded JSON text can be treated as a native
224 Unicode string, an ascii-encoded, latin1-encoded or UTF-8 encoded string,
225 or any other superset of ASCII.
226
227 If C<$enable> is false, then the C<encode> method will not escape Unicode
228 characters unless required by the JSON syntax or other flags. This results
229 in a faster and more compact format.
230
231 See also the section I<ENCODING/CODESET FLAG NOTES> later in this
232 document.
233
234 The main use for this flag is to produce JSON texts that can be
235 transmitted over a 7-bit channel, as the encoded JSON texts will not
236 contain any 8 bit characters.
237
238 JSON::XS->new->ascii (1)->encode ([chr 0x10401])
239 => ["\ud801\udc01"]
240
241 =item $json = $json->latin1 ([$enable])
242
243 =item $enabled = $json->get_latin1
244
245 If C<$enable> is true (or missing), then the C<encode> method will encode
246 the resulting JSON text as latin1 (or iso-8859-1), escaping any characters
247 outside the code range C<0..255>. The resulting string can be treated as a
248 latin1-encoded JSON text or a native Unicode string. The C<decode> method
249 will not be affected in any way by this flag, as C<decode> by default
250 expects Unicode, which is a strict superset of latin1.
251
252 If C<$enable> is false, then the C<encode> method will not escape Unicode
253 characters unless required by the JSON syntax or other flags.
254
255 See also the section I<ENCODING/CODESET FLAG NOTES> later in this
256 document.
257
258 The main use for this flag is efficiently encoding binary data as JSON
259 text, as most octets will not be escaped, resulting in a smaller encoded
260 size. The disadvantage is that the resulting JSON text is encoded
261 in latin1 (and must correctly be treated as such when storing and
262 transferring), a rare encoding for JSON. It is therefore most useful when
263 you want to store data structures known to contain binary data efficiently
264 in files or databases, not when talking to other JSON encoders/decoders.
265
266 JSON::XS->new->latin1->encode (["\x{89}\x{abc}"]
267 => ["\x{89}\\u0abc"] # (perl syntax, U+abc escaped, U+89 not)
268
269 =item $json = $json->utf8 ([$enable])
270
271 =item $enabled = $json->get_utf8
272
273 If C<$enable> is true (or missing), then the C<encode> method will encode
274 the JSON result into UTF-8, as required by many protocols, while the
275 C<decode> method expects to be handled an UTF-8-encoded string. Please
276 note that UTF-8-encoded strings do not contain any characters outside the
277 range C<0..255>, they are thus useful for bytewise/binary I/O. In future
278 versions, enabling this option might enable autodetection of the UTF-16
279 and UTF-32 encoding families, as described in RFC4627.
280
281 If C<$enable> is false, then the C<encode> method will return the JSON
282 string as a (non-encoded) Unicode string, while C<decode> expects thus a
283 Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs
284 to be done yourself, e.g. using the Encode module.
285
286 See also the section I<ENCODING/CODESET FLAG NOTES> later in this
287 document.
288
289 Example, output UTF-16BE-encoded JSON:
290
291 use Encode;
292 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object);
293
294 Example, decode UTF-32LE-encoded JSON:
295
296 use Encode;
297 $object = JSON::XS->new->decode (decode "UTF-32LE", $jsontext);
298
299 =item $json = $json->pretty ([$enable])
300
301 This enables (or disables) all of the C<indent>, C<space_before> and
302 C<space_after> (and in the future possibly more) flags in one call to
303 generate the most readable (or most compact) form possible.
304
305 Example, pretty-print some simple structure:
306
307 my $json = JSON::XS->new->pretty(1)->encode ({a => [1,2]})
308 =>
309 {
310 "a" : [
311 1,
312 2
313 ]
314 }
315
316 =item $json = $json->indent ([$enable])
317
318 =item $enabled = $json->get_indent
319
320 If C<$enable> is true (or missing), then the C<encode> method will use a multiline
321 format as output, putting every array member or object/hash key-value pair
322 into its own line, indenting them properly.
323
324 If C<$enable> is false, no newlines or indenting will be produced, and the
325 resulting JSON text is guaranteed not to contain any C<newlines>.
326
327 This setting has no effect when decoding JSON texts.
328
329 =item $json = $json->space_before ([$enable])
330
331 =item $enabled = $json->get_space_before
332
333 If C<$enable> is true (or missing), then the C<encode> method will add an extra
334 optional space before the C<:> separating keys from values in JSON objects.
335
336 If C<$enable> is false, then the C<encode> method will not add any extra
337 space at those places.
338
339 This setting has no effect when decoding JSON texts. You will also
340 most likely combine this setting with C<space_after>.
341
342 Example, space_before enabled, space_after and indent disabled:
343
344 {"key" :"value"}
345
346 =item $json = $json->space_after ([$enable])
347
348 =item $enabled = $json->get_space_after
349
350 If C<$enable> is true (or missing), then the C<encode> method will add an extra
351 optional space after the C<:> separating keys from values in JSON objects
352 and extra whitespace after the C<,> separating key-value pairs and array
353 members.
354
355 If C<$enable> is false, then the C<encode> method will not add any extra
356 space at those places.
357
358 This setting has no effect when decoding JSON texts.
359
360 Example, space_before and indent disabled, space_after enabled:
361
362 {"key": "value"}
363
364 =item $json = $json->relaxed ([$enable])
365
366 =item $enabled = $json->get_relaxed
367
368 If C<$enable> is true (or missing), then C<decode> will accept some
369 extensions to normal JSON syntax (see below). C<encode> will not be
370 affected in anyway. I<Be aware that this option makes you accept invalid
371 JSON texts as if they were valid!>. I suggest only to use this option to
372 parse application-specific files written by humans (configuration files,
373 resource files etc.)
374
375 If C<$enable> is false (the default), then C<decode> will only accept
376 valid JSON texts.
377
378 Currently accepted extensions are:
379
380 =over 4
381
382 =item * list items can have an end-comma
383
384 JSON I<separates> array elements and key-value pairs with commas. This
385 can be annoying if you write JSON texts manually and want to be able to
386 quickly append elements, so this extension accepts comma at the end of
387 such items not just between them:
388
389 [
390 1,
391 2, <- this comma not normally allowed
392 ]
393 {
394 "k1": "v1",
395 "k2": "v2", <- this comma not normally allowed
396 }
397
398 =item * shell-style '#'-comments
399
400 Whenever JSON allows whitespace, shell-style comments are additionally
401 allowed. They are terminated by the first carriage-return or line-feed
402 character, after which more white-space and comments are allowed.
403
404 [
405 1, # this comment not allowed in JSON
406 # neither this one...
407 ]
408
409 =back
410
411 =item $json = $json->canonical ([$enable])
412
413 =item $enabled = $json->get_canonical
414
415 If C<$enable> is true (or missing), then the C<encode> method will output JSON objects
416 by sorting their keys. This is adding a comparatively high overhead.
417
418 If C<$enable> is false, then the C<encode> method will output key-value
419 pairs in the order Perl stores them (which will likely change between runs
420 of the same script, and can change even within the same run from 5.18
421 onwards).
422
423 This option is useful if you want the same data structure to be encoded as
424 the same JSON text (given the same overall settings). If it is disabled,
425 the same hash might be encoded differently even if contains the same data,
426 as key-value pairs have no inherent ordering in Perl.
427
428 This setting has no effect when decoding JSON texts.
429
430 This setting has currently no effect on tied hashes.
431
432 =item $json = $json->allow_nonref ([$enable])
433
434 =item $enabled = $json->get_allow_nonref
435
436 If C<$enable> is true (or missing), then the C<encode> method can convert a
437 non-reference into its corresponding string, number or null JSON value,
438 which is an extension to RFC4627. Likewise, C<decode> will accept those JSON
439 values instead of croaking.
440
441 If C<$enable> is false, then the C<encode> method will croak if it isn't
442 passed an arrayref or hashref, as JSON texts must either be an object
443 or array. Likewise, C<decode> will croak if given something that is not a
444 JSON object or array.
445
446 Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>,
447 resulting in an invalid JSON text:
448
449 JSON::XS->new->allow_nonref->encode ("Hello, World!")
450 => "Hello, World!"
451
452 =item $json = $json->allow_unknown ([$enable])
453
454 =item $enabled = $json->get_allow_unknown
455
456 If C<$enable> is true (or missing), then C<encode> will I<not> throw an
457 exception when it encounters values it cannot represent in JSON (for
458 example, filehandles) but instead will encode a JSON C<null> value. Note
459 that blessed objects are not included here and are handled separately by
460 c<allow_nonref>.
461
462 If C<$enable> is false (the default), then C<encode> will throw an
463 exception when it encounters anything it cannot encode as JSON.
464
465 This option does not affect C<decode> in any way, and it is recommended to
466 leave it off unless you know your communications partner.
467
468 =item $json = $json->allow_blessed ([$enable])
469
470 =item $enabled = $json->get_allow_blessed
471
472 If C<$enable> is true (or missing), then the C<encode> method will not
473 barf when it encounters a blessed reference. Instead, the value of the
474 B<convert_blessed> option will decide whether C<null> (C<convert_blessed>
475 disabled or no C<TO_JSON> method found) or a representation of the
476 object (C<convert_blessed> enabled and C<TO_JSON> method found) is being
477 encoded. Has no effect on C<decode>.
478
479 If C<$enable> is false (the default), then C<encode> will throw an
480 exception when it encounters a blessed object.
481
482 =item $json = $json->convert_blessed ([$enable])
483
484 =item $enabled = $json->get_convert_blessed
485
486 If C<$enable> is true (or missing), then C<encode>, upon encountering a
487 blessed object, will check for the availability of the C<TO_JSON> method
488 on the object's class. If found, it will be called in scalar context
489 and the resulting scalar will be encoded instead of the object. If no
490 C<TO_JSON> method is found, the value of C<allow_blessed> will decide what
491 to do.
492
493 The C<TO_JSON> method may safely call die if it wants. If C<TO_JSON>
494 returns other blessed objects, those will be handled in the same
495 way. C<TO_JSON> must take care of not causing an endless recursion cycle
496 (== crash) in this case. The name of C<TO_JSON> was chosen because other
497 methods called by the Perl core (== not by the user of the object) are
498 usually in upper case letters and to avoid collisions with any C<to_json>
499 function or method.
500
501 This setting does not yet influence C<decode> in any way, but in the
502 future, global hooks might get installed that influence C<decode> and are
503 enabled by this setting.
504
505 If C<$enable> is false, then the C<allow_blessed> setting will decide what
506 to do when a blessed object is found.
507
508 =item $json = $json->filter_json_object ([$coderef->($hashref)])
509
510 When C<$coderef> is specified, it will be called from C<decode> each
511 time it decodes a JSON object. The only argument is a reference to the
512 newly-created hash. If the code references returns a single scalar (which
513 need not be a reference), this value (i.e. a copy of that scalar to avoid
514 aliasing) is inserted into the deserialised data structure. If it returns
515 an empty list (NOTE: I<not> C<undef>, which is a valid scalar), the
516 original deserialised hash will be inserted. This setting can slow down
517 decoding considerably.
518
519 When C<$coderef> is omitted or undefined, any existing callback will
520 be removed and C<decode> will not change the deserialised hash in any
521 way.
522
523 Example, convert all JSON objects into the integer 5:
524
525 my $js = JSON::XS->new->filter_json_object (sub { 5 });
526 # returns [5]
527 $js->decode ('[{}]')
528 # throw an exception because allow_nonref is not enabled
529 # so a lone 5 is not allowed.
530 $js->decode ('{"a":1, "b":2}');
531
532 =item $json = $json->filter_json_single_key_object ($key [=> $coderef->($value)])
533
534 Works remotely similar to C<filter_json_object>, but is only called for
535 JSON objects having a single key named C<$key>.
536
537 This C<$coderef> is called before the one specified via
538 C<filter_json_object>, if any. It gets passed the single value in the JSON
539 object. If it returns a single value, it will be inserted into the data
540 structure. If it returns nothing (not even C<undef> but the empty list),
541 the callback from C<filter_json_object> will be called next, as if no
542 single-key callback were specified.
543
544 If C<$coderef> is omitted or undefined, the corresponding callback will be
545 disabled. There can only ever be one callback for a given key.
546
547 As this callback gets called less often then the C<filter_json_object>
548 one, decoding speed will not usually suffer as much. Therefore, single-key
549 objects make excellent targets to serialise Perl objects into, especially
550 as single-key JSON objects are as close to the type-tagged value concept
551 as JSON gets (it's basically an ID/VALUE tuple). Of course, JSON does not
552 support this in any way, so you need to make sure your data never looks
553 like a serialised Perl hash.
554
555 Typical names for the single object key are C<__class_whatever__>, or
556 C<$__dollars_are_rarely_used__$> or C<}ugly_brace_placement>, or even
557 things like C<__class_md5sum(classname)__>, to reduce the risk of clashing
558 with real hashes.
559
560 Example, decode JSON objects of the form C<< { "__widget__" => <id> } >>
561 into the corresponding C<< $WIDGET{<id>} >> object:
562
563 # return whatever is in $WIDGET{5}:
564 JSON::XS
565 ->new
566 ->filter_json_single_key_object (__widget__ => sub {
567 $WIDGET{ $_[0] }
568 })
569 ->decode ('{"__widget__": 5')
570
571 # this can be used with a TO_JSON method in some "widget" class
572 # for serialisation to json:
573 sub WidgetBase::TO_JSON {
574 my ($self) = @_;
575
576 unless ($self->{id}) {
577 $self->{id} = ..get..some..id..;
578 $WIDGET{$self->{id}} = $self;
579 }
580
581 { __widget__ => $self->{id} }
582 }
583
584 =item $json = $json->shrink ([$enable])
585
586 =item $enabled = $json->get_shrink
587
588 Perl usually over-allocates memory a bit when allocating space for
589 strings. This flag optionally resizes strings generated by either
590 C<encode> or C<decode> to their minimum size possible. This can save
591 memory when your JSON texts are either very very long or you have many
592 short strings. It will also try to downgrade any strings to octet-form
593 if possible: perl stores strings internally either in an encoding called
594 UTF-X or in octet-form. The latter cannot store everything but uses less
595 space in general (and some buggy Perl or C code might even rely on that
596 internal representation being used).
597
598 The actual definition of what shrink does might change in future versions,
599 but it will always try to save space at the expense of time.
600
601 If C<$enable> is true (or missing), the string returned by C<encode> will
602 be shrunk-to-fit, while all strings generated by C<decode> will also be
603 shrunk-to-fit.
604
605 If C<$enable> is false, then the normal perl allocation algorithms are used.
606 If you work with your data, then this is likely to be faster.
607
608 In the future, this setting might control other things, such as converting
609 strings that look like integers or floats into integers or floats
610 internally (there is no difference on the Perl level), saving space.
611
612 =item $json = $json->max_depth ([$maximum_nesting_depth])
613
614 =item $max_depth = $json->get_max_depth
615
616 Sets the maximum nesting level (default C<512>) accepted while encoding
617 or decoding. If a higher nesting level is detected in JSON text or a Perl
618 data structure, then the encoder and decoder will stop and croak at that
619 point.
620
621 Nesting level is defined by number of hash- or arrayrefs that the encoder
622 needs to traverse to reach a given point or the number of C<{> or C<[>
623 characters without their matching closing parenthesis crossed to reach a
624 given character in a string.
625
626 Setting the maximum depth to one disallows any nesting, so that ensures
627 that the object is only a single hash/object or array.
628
629 If no argument is given, the highest possible setting will be used, which
630 is rarely useful.
631
632 Note that nesting is implemented by recursion in C. The default value has
633 been chosen to be as large as typical operating systems allow without
634 crashing.
635
636 See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
637
638 =item $json = $json->max_size ([$maximum_string_size])
639
640 =item $max_size = $json->get_max_size
641
642 Set the maximum length a JSON text may have (in bytes) where decoding is
643 being attempted. The default is C<0>, meaning no limit. When C<decode>
644 is called on a string that is longer then this many bytes, it will not
645 attempt to decode the string but throw an exception. This setting has no
646 effect on C<encode> (yet).
647
648 If no argument is given, the limit check will be deactivated (same as when
649 C<0> is specified).
650
651 See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
652
653 =item $json_text = $json->encode ($perl_scalar)
654
655 Converts the given Perl value or data structure to its JSON
656 representation. Croaks on error.
657
658 =item $perl_scalar = $json->decode ($json_text)
659
660 The opposite of C<encode>: expects a JSON text and tries to parse it,
661 returning the resulting simple scalar or reference. Croaks on error.
662
663 =item ($perl_scalar, $characters) = $json->decode_prefix ($json_text)
664
665 This works like the C<decode> method, but instead of raising an exception
666 when there is trailing garbage after the first JSON object, it will
667 silently stop parsing there and return the number of characters consumed
668 so far.
669
670 This is useful if your JSON texts are not delimited by an outer protocol
671 and you need to know where the JSON text ends.
672
673 JSON::XS->new->decode_prefix ("[1] the tail")
674 => ([], 3)
675
676 =back
677
678
679 =head1 INCREMENTAL PARSING
680
681 In some cases, there is the need for incremental parsing of JSON
682 texts. While this module always has to keep both JSON text and resulting
683 Perl data structure in memory at one time, it does allow you to parse a
684 JSON stream incrementally. It does so by accumulating text until it has
685 a full JSON object, which it then can decode. This process is similar to
686 using C<decode_prefix> to see if a full JSON object is available, but
687 is much more efficient (and can be implemented with a minimum of method
688 calls).
689
690 JSON::XS will only attempt to parse the JSON text once it is sure it
691 has enough text to get a decisive result, using a very simple but
692 truly incremental parser. This means that it sometimes won't stop as
693 early as the full parser, for example, it doesn't detect mismatched
694 parentheses. The only thing it guarantees is that it starts decoding as
695 soon as a syntactically valid JSON text has been seen. This means you need
696 to set resource limits (e.g. C<max_size>) to ensure the parser will stop
697 parsing in the presence if syntax errors.
698
699 The following methods implement this incremental parser.
700
701 =over 4
702
703 =item [void, scalar or list context] = $json->incr_parse ([$string])
704
705 This is the central parsing function. It can both append new text and
706 extract objects from the stream accumulated so far (both of these
707 functions are optional).
708
709 If C<$string> is given, then this string is appended to the already
710 existing JSON fragment stored in the C<$json> object.
711
712 After that, if the function is called in void context, it will simply
713 return without doing anything further. This can be used to add more text
714 in as many chunks as you want.
715
716 If the method is called in scalar context, then it will try to extract
717 exactly I<one> JSON object. If that is successful, it will return this
718 object, otherwise it will return C<undef>. If there is a parse error,
719 this method will croak just as C<decode> would do (one can then use
720 C<incr_skip> to skip the erroneous part). This is the most common way of
721 using the method.
722
723 And finally, in list context, it will try to extract as many objects
724 from the stream as it can find and return them, or the empty list
725 otherwise. For this to work, there must be no separators between the JSON
726 objects or arrays, instead they must be concatenated back-to-back. If
727 an error occurs, an exception will be raised as in the scalar context
728 case. Note that in this case, any previously-parsed JSON texts will be
729 lost.
730
731 Example: Parse some JSON arrays/objects in a given string and return
732 them.
733
734 my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]");
735
736 =item $lvalue_string = $json->incr_text
737
738 This method returns the currently stored JSON fragment as an lvalue, that
739 is, you can manipulate it. This I<only> works when a preceding call to
740 C<incr_parse> in I<scalar context> successfully returned an object. Under
741 all other circumstances you must not call this function (I mean it.
742 although in simple tests it might actually work, it I<will> fail under
743 real world conditions). As a special exception, you can also call this
744 method before having parsed anything.
745
746 This function is useful in two cases: a) finding the trailing text after a
747 JSON object or b) parsing multiple JSON objects separated by non-JSON text
748 (such as commas).
749
750 =item $json->incr_skip
751
752 This will reset the state of the incremental parser and will remove
753 the parsed text from the input buffer so far. This is useful after
754 C<incr_parse> died, in which case the input buffer and incremental parser
755 state is left unchanged, to skip the text parsed so far and to reset the
756 parse state.
757
758 The difference to C<incr_reset> is that only text until the parse error
759 occurred is removed.
760
761 =item $json->incr_reset
762
763 This completely resets the incremental parser, that is, after this call,
764 it will be as if the parser had never parsed anything.
765
766 This is useful if you want to repeatedly parse JSON objects and want to
767 ignore any trailing data, which means you have to reset the parser after
768 each successful decode.
769
770 =back
771
772 =head2 LIMITATIONS
773
774 All options that affect decoding are supported, except
775 C<allow_nonref>. The reason for this is that it cannot be made to work
776 sensibly: JSON objects and arrays are self-delimited, i.e. you can
777 concatenate them back to back and still decode them perfectly. This does
778 not hold true for JSON numbers, however.
779
780 For example, is the string C<1> a single JSON number, or is it simply the
781 start of C<12>? Or is C<12> a single JSON number, or the concatenation
782 of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
783 takes the conservative route and disallows this case.
784
785 =head2 EXAMPLES
786
787 Some examples will make all this clearer. First, a simple example that
788 works similarly to C<decode_prefix>: We want to decode the JSON object at
789 the start of a string and identify the portion after the JSON object:
790
791 my $text = "[1,2,3] hello";
792
793 my $json = new JSON::XS;
794
795 my $obj = $json->incr_parse ($text)
796 or die "expected JSON object or array at beginning of string";
797
798 my $tail = $json->incr_text;
799 # $tail now contains " hello"
800
801 Easy, isn't it?
802
803 Now for a more complicated example: Imagine a hypothetical protocol where
804 you read some requests from a TCP stream, and each request is a JSON
805 array, without any separation between them (in fact, it is often useful to
806 use newlines as "separators", as these get interpreted as whitespace at
807 the start of the JSON text, which makes it possible to test said protocol
808 with C<telnet>...).
809
810 Here is how you'd do it (it is trivial to write this in an event-based
811 manner):
812
813 my $json = new JSON::XS;
814
815 # read some data from the socket
816 while (sysread $socket, my $buf, 4096) {
817
818 # split and decode as many requests as possible
819 for my $request ($json->incr_parse ($buf)) {
820 # act on the $request
821 }
822 }
823
824 Another complicated example: Assume you have a string with JSON objects
825 or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
826 [3]>). To parse them, we have to skip the commas between the JSON texts,
827 and here is where the lvalue-ness of C<incr_text> comes in useful:
828
829 my $text = "[1],[2], [3]";
830 my $json = new JSON::XS;
831
832 # void context, so no parsing done
833 $json->incr_parse ($text);
834
835 # now extract as many objects as possible. note the
836 # use of scalar context so incr_text can be called.
837 while (my $obj = $json->incr_parse) {
838 # do something with $obj
839
840 # now skip the optional comma
841 $json->incr_text =~ s/^ \s* , //x;
842 }
843
844 Now lets go for a very complex example: Assume that you have a gigantic
845 JSON array-of-objects, many gigabytes in size, and you want to parse it,
846 but you cannot load it into memory fully (this has actually happened in
847 the real world :).
848
849 Well, you lost, you have to implement your own JSON parser. But JSON::XS
850 can still help you: You implement a (very simple) array parser and let
851 JSON decode the array elements, which are all full JSON objects on their
852 own (this wouldn't work if the array elements could be JSON numbers, for
853 example):
854
855 my $json = new JSON::XS;
856
857 # open the monster
858 open my $fh, "<bigfile.json"
859 or die "bigfile: $!";
860
861 # first parse the initial "["
862 for (;;) {
863 sysread $fh, my $buf, 65536
864 or die "read error: $!";
865 $json->incr_parse ($buf); # void context, so no parsing
866
867 # Exit the loop once we found and removed(!) the initial "[".
868 # In essence, we are (ab-)using the $json object as a simple scalar
869 # we append data to.
870 last if $json->incr_text =~ s/^ \s* \[ //x;
871 }
872
873 # now we have the skipped the initial "[", so continue
874 # parsing all the elements.
875 for (;;) {
876 # in this loop we read data until we got a single JSON object
877 for (;;) {
878 if (my $obj = $json->incr_parse) {
879 # do something with $obj
880 last;
881 }
882
883 # add more data
884 sysread $fh, my $buf, 65536
885 or die "read error: $!";
886 $json->incr_parse ($buf); # void context, so no parsing
887 }
888
889 # in this loop we read data until we either found and parsed the
890 # separating "," between elements, or the final "]"
891 for (;;) {
892 # first skip whitespace
893 $json->incr_text =~ s/^\s*//;
894
895 # if we find "]", we are done
896 if ($json->incr_text =~ s/^\]//) {
897 print "finished.\n";
898 exit;
899 }
900
901 # if we find ",", we can continue with the next element
902 if ($json->incr_text =~ s/^,//) {
903 last;
904 }
905
906 # if we find anything else, we have a parse error!
907 if (length $json->incr_text) {
908 die "parse error near ", $json->incr_text;
909 }
910
911 # else add more data
912 sysread $fh, my $buf, 65536
913 or die "read error: $!";
914 $json->incr_parse ($buf); # void context, so no parsing
915 }
916
917 This is a complex example, but most of the complexity comes from the fact
918 that we are trying to be correct (bear with me if I am wrong, I never ran
919 the above example :).
920
921
922
923 =head1 MAPPING
924
925 This section describes how JSON::XS maps Perl values to JSON values and
926 vice versa. These mappings are designed to "do the right thing" in most
927 circumstances automatically, preserving round-tripping characteristics
928 (what you put in comes out as something equivalent).
929
930 For the more enlightened: note that in the following descriptions,
931 lowercase I<perl> refers to the Perl interpreter, while uppercase I<Perl>
932 refers to the abstract Perl language itself.
933
934
935 =head2 JSON -> PERL
936
937 =over 4
938
939 =item object
940
941 A JSON object becomes a reference to a hash in Perl. No ordering of object
942 keys is preserved (JSON does not preserve object key ordering itself).
943
944 =item array
945
946 A JSON array becomes a reference to an array in Perl.
947
948 =item string
949
950 A JSON string becomes a string scalar in Perl - Unicode codepoints in JSON
951 are represented by the same codepoints in the Perl string, so no manual
952 decoding is necessary.
953
954 =item number
955
956 A JSON number becomes either an integer, numeric (floating point) or
957 string scalar in perl, depending on its range and any fractional parts. On
958 the Perl level, there is no difference between those as Perl handles all
959 the conversion details, but an integer may take slightly less memory and
960 might represent more values exactly than floating point numbers.
961
962 If the number consists of digits only, JSON::XS will try to represent
963 it as an integer value. If that fails, it will try to represent it as
964 a numeric (floating point) value if that is possible without loss of
965 precision. Otherwise it will preserve the number as a string value (in
966 which case you lose roundtripping ability, as the JSON number will be
967 re-encoded to a JSON string).
968
969 Numbers containing a fractional or exponential part will always be
970 represented as numeric (floating point) values, possibly at a loss of
971 precision (in which case you might lose perfect roundtripping ability, but
972 the JSON number will still be re-encoded as a JSON number).
973
974 Note that precision is not accuracy - binary floating point values cannot
975 represent most decimal fractions exactly, and when converting from and to
976 floating point, JSON::XS only guarantees precision up to but not including
977 the least significant bit.
978
979 =item true, false
980
981 These JSON atoms become C<Types::Serialiser::true> and
982 C<Types::Serialiser::false>, respectively. They are overloaded to act
983 almost exactly like the numbers C<1> and C<0>. You can check whether
984 a scalar is a JSON boolean by using the C<Types::Serialiser::is_bool>
985 function (after C<use Types::Serialier>, of course).
986
987 =item null
988
989 A JSON null atom becomes C<undef> in Perl.
990
991 =back
992
993
994 =head2 PERL -> JSON
995
996 The mapping from Perl to JSON is slightly more difficult, as Perl is a
997 truly typeless language, so we can only guess which JSON type is meant by
998 a Perl value.
999
1000 =over 4
1001
1002 =item hash references
1003
1004 Perl hash references become JSON objects. As there is no inherent
1005 ordering in hash keys (or JSON objects), they will usually be encoded
1006 in a pseudo-random order. JSON::XS can optionally sort the hash keys
1007 (determined by the I<canonical> flag), so the same datastructure will
1008 serialise to the same JSON text (given same settings and version of
1009 JSON::XS), but this incurs a runtime overhead and is only rarely useful,
1010 e.g. when you want to compare some JSON text against another for equality.
1011
1012 =item array references
1013
1014 Perl array references become JSON arrays.
1015
1016 =item other references
1017
1018 Other unblessed references are generally not allowed and will cause an
1019 exception to be thrown, except for references to the integers C<0> and
1020 C<1>, which get turned into C<false> and C<true> atoms in JSON.
1021
1022 Since C<JSON::XS> uses the boolean model from L<Types::Serialiser>, you
1023 can also C<use Types::Serialiser> and then use C<Types::Serialiser::false>
1024 and C<Types::Serialiser::true> to improve readability.
1025
1026 use Types::Serialiser;
1027 encode_json [\0, Types::Serialiser::true] # yields [false,true]
1028
1029 =item Types::Serialiser::true, Types::Serialiser::false
1030
1031 These special values from the L<Types::Serialiser> module become JSON true
1032 and JSON false values, respectively. You can also use C<\1> and C<\0>
1033 directly if you want.
1034
1035 =item blessed objects
1036
1037 Blessed objects are not directly representable in JSON. See the
1038 C<allow_blessed> and C<convert_blessed> methods on various options on
1039 how to deal with this: basically, you can choose between throwing an
1040 exception, encoding the reference as if it weren't blessed, or provide
1041 your own serialiser method.
1042
1043 =item simple scalars
1044
1045 Simple Perl scalars (any scalar that is not a reference) are the most
1046 difficult objects to encode: JSON::XS will encode undefined scalars as
1047 JSON C<null> values, scalars that have last been used in a string context
1048 before encoding as JSON strings, and anything else as number value:
1049
1050 # dump as number
1051 encode_json [2] # yields [2]
1052 encode_json [-3.0e17] # yields [-3e+17]
1053 my $value = 5; encode_json [$value] # yields [5]
1054
1055 # used as string, so dump as string
1056 print $value;
1057 encode_json [$value] # yields ["5"]
1058
1059 # undef becomes null
1060 encode_json [undef] # yields [null]
1061
1062 You can force the type to be a JSON string by stringifying it:
1063
1064 my $x = 3.1; # some variable containing a number
1065 "$x"; # stringified
1066 $x .= ""; # another, more awkward way to stringify
1067 print $x; # perl does it for you, too, quite often
1068
1069 You can force the type to be a JSON number by numifying it:
1070
1071 my $x = "3"; # some variable containing a string
1072 $x += 0; # numify it, ensuring it will be dumped as a number
1073 $x *= 1; # same thing, the choice is yours.
1074
1075 You can not currently force the type in other, less obscure, ways. Tell me
1076 if you need this capability (but don't forget to explain why it's needed
1077 :).
1078
1079 Note that numerical precision has the same meaning as under Perl (so
1080 binary to decimal conversion follows the same rules as in Perl, which
1081 can differ to other languages). Also, your perl interpreter might expose
1082 extensions to the floating point numbers of your platform, such as
1083 infinities or NaN's - these cannot be represented in JSON, and it is an
1084 error to pass those in.
1085
1086 =back
1087
1088
1089 =head1 ENCODING/CODESET FLAG NOTES
1090
1091 The interested reader might have seen a number of flags that signify
1092 encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be
1093 some confusion on what these do, so here is a short comparison:
1094
1095 C<utf8> controls whether the JSON text created by C<encode> (and expected
1096 by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only
1097 control whether C<encode> escapes character values outside their respective
1098 codeset range. Neither of these flags conflict with each other, although
1099 some combinations make less sense than others.
1100
1101 Care has been taken to make all flags symmetrical with respect to
1102 C<encode> and C<decode>, that is, texts encoded with any combination of
1103 these flag values will be correctly decoded when the same flags are used
1104 - in general, if you use different flag settings while encoding vs. when
1105 decoding you likely have a bug somewhere.
1106
1107 Below comes a verbose discussion of these flags. Note that a "codeset" is
1108 simply an abstract set of character-codepoint pairs, while an encoding
1109 takes those codepoint numbers and I<encodes> them, in our case into
1110 octets. Unicode is (among other things) a codeset, UTF-8 is an encoding,
1111 and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at
1112 the same time, which can be confusing.
1113
1114 =over 4
1115
1116 =item C<utf8> flag disabled
1117
1118 When C<utf8> is disabled (the default), then C<encode>/C<decode> generate
1119 and expect Unicode strings, that is, characters with high ordinal Unicode
1120 values (> 255) will be encoded as such characters, and likewise such
1121 characters are decoded as-is, no changes to them will be done, except
1122 "(re-)interpreting" them as Unicode codepoints or Unicode characters,
1123 respectively (to Perl, these are the same thing in strings unless you do
1124 funny/weird/dumb stuff).
1125
1126 This is useful when you want to do the encoding yourself (e.g. when you
1127 want to have UTF-16 encoded JSON texts) or when some other layer does
1128 the encoding for you (for example, when printing to a terminal using a
1129 filehandle that transparently encodes to UTF-8 you certainly do NOT want
1130 to UTF-8 encode your data first and have Perl encode it another time).
1131
1132 =item C<utf8> flag enabled
1133
1134 If the C<utf8>-flag is enabled, C<encode>/C<decode> will encode all
1135 characters using the corresponding UTF-8 multi-byte sequence, and will
1136 expect your input strings to be encoded as UTF-8, that is, no "character"
1137 of the input string must have any value > 255, as UTF-8 does not allow
1138 that.
1139
1140 The C<utf8> flag therefore switches between two modes: disabled means you
1141 will get a Unicode string in Perl, enabled means you get an UTF-8 encoded
1142 octet/binary string in Perl.
1143
1144 =item C<latin1> or C<ascii> flags enabled
1145
1146 With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters
1147 with ordinal values > 255 (> 127 with C<ascii>) and encode the remaining
1148 characters as specified by the C<utf8> flag.
1149
1150 If C<utf8> is disabled, then the result is also correctly encoded in those
1151 character sets (as both are proper subsets of Unicode, meaning that a
1152 Unicode string with all character values < 256 is the same thing as a
1153 ISO-8859-1 string, and a Unicode string with all character values < 128 is
1154 the same thing as an ASCII string in Perl).
1155
1156 If C<utf8> is enabled, you still get a correct UTF-8-encoded string,
1157 regardless of these flags, just some more characters will be escaped using
1158 C<\uXXXX> then before.
1159
1160 Note that ISO-8859-1-I<encoded> strings are not compatible with UTF-8
1161 encoding, while ASCII-encoded strings are. That is because the ISO-8859-1
1162 encoding is NOT a subset of UTF-8 (despite the ISO-8859-1 I<codeset> being
1163 a subset of Unicode), while ASCII is.
1164
1165 Surprisingly, C<decode> will ignore these flags and so treat all input
1166 values as governed by the C<utf8> flag. If it is disabled, this allows you
1167 to decode ISO-8859-1- and ASCII-encoded strings, as both strict subsets of
1168 Unicode. If it is enabled, you can correctly decode UTF-8 encoded strings.
1169
1170 So neither C<latin1> nor C<ascii> are incompatible with the C<utf8> flag -
1171 they only govern when the JSON output engine escapes a character or not.
1172
1173 The main use for C<latin1> is to relatively efficiently store binary data
1174 as JSON, at the expense of breaking compatibility with most JSON decoders.
1175
1176 The main use for C<ascii> is to force the output to not contain characters
1177 with values > 127, which means you can interpret the resulting string
1178 as UTF-8, ISO-8859-1, ASCII, KOI8-R or most about any character set and
1179 8-bit-encoding, and still get the same data structure back. This is useful
1180 when your channel for JSON transfer is not 8-bit clean or the encoding
1181 might be mangled in between (e.g. in mail), and works because ASCII is a
1182 proper subset of most 8-bit and multibyte encodings in use in the world.
1183
1184 =back
1185
1186
1187 =head2 JSON and ECMAscript
1188
1189 JSON syntax is based on how literals are represented in javascript (the
1190 not-standardised predecessor of ECMAscript) which is presumably why it is
1191 called "JavaScript Object Notation".
1192
1193 However, JSON is not a subset (and also not a superset of course) of
1194 ECMAscript (the standard) or javascript (whatever browsers actually
1195 implement).
1196
1197 If you want to use javascript's C<eval> function to "parse" JSON, you
1198 might run into parse errors for valid JSON texts, or the resulting data
1199 structure might not be queryable:
1200
1201 One of the problems is that U+2028 and U+2029 are valid characters inside
1202 JSON strings, but are not allowed in ECMAscript string literals, so the
1203 following Perl fragment will not output something that can be guaranteed
1204 to be parsable by javascript's C<eval>:
1205
1206 use JSON::XS;
1207
1208 print encode_json [chr 0x2028];
1209
1210 The right fix for this is to use a proper JSON parser in your javascript
1211 programs, and not rely on C<eval> (see for example Douglas Crockford's
1212 F<json2.js> parser).
1213
1214 If this is not an option, you can, as a stop-gap measure, simply encode to
1215 ASCII-only JSON:
1216
1217 use JSON::XS;
1218
1219 print JSON::XS->new->ascii->encode ([chr 0x2028]);
1220
1221 Note that this will enlarge the resulting JSON text quite a bit if you
1222 have many non-ASCII characters. You might be tempted to run some regexes
1223 to only escape U+2028 and U+2029, e.g.:
1224
1225 # DO NOT USE THIS!
1226 my $json = JSON::XS->new->utf8->encode ([chr 0x2028]);
1227 $json =~ s/\xe2\x80\xa8/\\u2028/g; # escape U+2028
1228 $json =~ s/\xe2\x80\xa9/\\u2029/g; # escape U+2029
1229 print $json;
1230
1231 Note that I<this is a bad idea>: the above only works for U+2028 and
1232 U+2029 and thus only for fully ECMAscript-compliant parsers. Many existing
1233 javascript implementations, however, have issues with other characters as
1234 well - using C<eval> naively simply I<will> cause problems.
1235
1236 Another problem is that some javascript implementations reserve
1237 some property names for their own purposes (which probably makes
1238 them non-ECMAscript-compliant). For example, Iceweasel reserves the
1239 C<__proto__> property name for its own purposes.
1240
1241 If that is a problem, you could parse try to filter the resulting JSON
1242 output for these property strings, e.g.:
1243
1244 $json =~ s/"__proto__"\s*:/"__proto__renamed":/g;
1245
1246 This works because C<__proto__> is not valid outside of strings, so every
1247 occurrence of C<"__proto__"\s*:> must be a string used as property name.
1248
1249 If you know of other incompatibilities, please let me know.
1250
1251
1252 =head2 JSON and YAML
1253
1254 You often hear that JSON is a subset of YAML. This is, however, a mass
1255 hysteria(*) and very far from the truth (as of the time of this writing),
1256 so let me state it clearly: I<in general, there is no way to configure
1257 JSON::XS to output a data structure as valid YAML> that works in all
1258 cases.
1259
1260 If you really must use JSON::XS to generate YAML, you should use this
1261 algorithm (subject to change in future versions):
1262
1263 my $to_yaml = JSON::XS->new->utf8->space_after (1);
1264 my $yaml = $to_yaml->encode ($ref) . "\n";
1265
1266 This will I<usually> generate JSON texts that also parse as valid
1267 YAML. Please note that YAML has hardcoded limits on (simple) object key
1268 lengths that JSON doesn't have and also has different and incompatible
1269 unicode character escape syntax, so you should make sure that your hash
1270 keys are noticeably shorter than the 1024 "stream characters" YAML allows
1271 and that you do not have characters with codepoint values outside the
1272 Unicode BMP (basic multilingual page). YAML also does not allow C<\/>
1273 sequences in strings (which JSON::XS does not I<currently> generate, but
1274 other JSON generators might).
1275
1276 There might be other incompatibilities that I am not aware of (or the YAML
1277 specification has been changed yet again - it does so quite often). In
1278 general you should not try to generate YAML with a JSON generator or vice
1279 versa, or try to parse JSON with a YAML parser or vice versa: chances are
1280 high that you will run into severe interoperability problems when you
1281 least expect it.
1282
1283 =over 4
1284
1285 =item (*)
1286
1287 I have been pressured multiple times by Brian Ingerson (one of the
1288 authors of the YAML specification) to remove this paragraph, despite him
1289 acknowledging that the actual incompatibilities exist. As I was personally
1290 bitten by this "JSON is YAML" lie, I refused and said I will continue to
1291 educate people about these issues, so others do not run into the same
1292 problem again and again. After this, Brian called me a (quote)I<complete
1293 and worthless idiot>(unquote).
1294
1295 In my opinion, instead of pressuring and insulting people who actually
1296 clarify issues with YAML and the wrong statements of some of its
1297 proponents, I would kindly suggest reading the JSON spec (which is not
1298 that difficult or long) and finally make YAML compatible to it, and
1299 educating users about the changes, instead of spreading lies about the
1300 real compatibility for many I<years> and trying to silence people who
1301 point out that it isn't true.
1302
1303 Addendum/2009: the YAML 1.2 spec is still incompatible with JSON, even
1304 though the incompatibilities have been documented (and are known to Brian)
1305 for many years and the spec makes explicit claims that YAML is a superset
1306 of JSON. It would be so easy to fix, but apparently, bullying people and
1307 corrupting userdata is so much easier.
1308
1309 =back
1310
1311
1312 =head2 SPEED
1313
1314 It seems that JSON::XS is surprisingly fast, as shown in the following
1315 tables. They have been generated with the help of the C<eg/bench> program
1316 in the JSON::XS distribution, to make it easy to compare on your own
1317 system.
1318
1319 First comes a comparison between various modules using
1320 a very short single-line JSON string (also available at
1321 L<http://dist.schmorp.de/misc/json/short.json>).
1322
1323 {"method": "handleMessage", "params": ["user1",
1324 "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
1325 1, 0]}
1326
1327 It shows the number of encodes/decodes per second (JSON::XS uses
1328 the functional interface, while JSON::XS/2 uses the OO interface
1329 with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
1330 shrink. JSON::DWIW/DS uses the deserialise function, while JSON::DWIW::FJ
1331 uses the from_json method). Higher is better:
1332
1333 module | encode | decode |
1334 --------------|------------|------------|
1335 JSON::DWIW/DS | 86302.551 | 102300.098 |
1336 JSON::DWIW/FJ | 86302.551 | 75983.768 |
1337 JSON::PP | 15827.562 | 6638.658 |
1338 JSON::Syck | 63358.066 | 47662.545 |
1339 JSON::XS | 511500.488 | 511500.488 |
1340 JSON::XS/2 | 291271.111 | 388361.481 |
1341 JSON::XS/3 | 361577.931 | 361577.931 |
1342 Storable | 66788.280 | 265462.278 |
1343 --------------+------------+------------+
1344
1345 That is, JSON::XS is almost six times faster than JSON::DWIW on encoding,
1346 about five times faster on decoding, and over thirty to seventy times
1347 faster than JSON's pure perl implementation. It also compares favourably
1348 to Storable for small amounts of data.
1349
1350 Using a longer test string (roughly 18KB, generated from Yahoo! Locals
1351 search API (L<http://dist.schmorp.de/misc/json/long.json>).
1352
1353 module | encode | decode |
1354 --------------|------------|------------|
1355 JSON::DWIW/DS | 1647.927 | 2673.916 |
1356 JSON::DWIW/FJ | 1630.249 | 2596.128 |
1357 JSON::PP | 400.640 | 62.311 |
1358 JSON::Syck | 1481.040 | 1524.869 |
1359 JSON::XS | 20661.596 | 9541.183 |
1360 JSON::XS/2 | 10683.403 | 9416.938 |
1361 JSON::XS/3 | 20661.596 | 9400.054 |
1362 Storable | 19765.806 | 10000.725 |
1363 --------------+------------+------------+
1364
1365 Again, JSON::XS leads by far (except for Storable which non-surprisingly
1366 decodes a bit faster).
1367
1368 On large strings containing lots of high Unicode characters, some modules
1369 (such as JSON::PC) seem to decode faster than JSON::XS, but the result
1370 will be broken due to missing (or wrong) Unicode handling. Others refuse
1371 to decode or encode properly, so it was impossible to prepare a fair
1372 comparison table for that case.
1373
1374
1375 =head1 SECURITY CONSIDERATIONS
1376
1377 When you are using JSON in a protocol, talking to untrusted potentially
1378 hostile creatures requires relatively few measures.
1379
1380 First of all, your JSON decoder should be secure, that is, should not have
1381 any buffer overflows. Obviously, this module should ensure that and I am
1382 trying hard on making that true, but you never know.
1383
1384 Second, you need to avoid resource-starving attacks. That means you should
1385 limit the size of JSON texts you accept, or make sure then when your
1386 resources run out, that's just fine (e.g. by using a separate process that
1387 can crash safely). The size of a JSON text in octets or characters is
1388 usually a good indication of the size of the resources required to decode
1389 it into a Perl structure. While JSON::XS can check the size of the JSON
1390 text, it might be too late when you already have it in memory, so you
1391 might want to check the size before you accept the string.
1392
1393 Third, JSON::XS recurses using the C stack when decoding objects and
1394 arrays. The C stack is a limited resource: for instance, on my amd64
1395 machine with 8MB of stack size I can decode around 180k nested arrays but
1396 only 14k nested JSON objects (due to perl itself recursing deeply on croak
1397 to free the temporary). If that is exceeded, the program crashes. To be
1398 conservative, the default nesting limit is set to 512. If your process
1399 has a smaller stack, you should adjust this setting accordingly with the
1400 C<max_depth> method.
1401
1402 Something else could bomb you, too, that I forgot to think of. In that
1403 case, you get to keep the pieces. I am always open for hints, though...
1404
1405 Also keep in mind that JSON::XS might leak contents of your Perl data
1406 structures in its error messages, so when you serialise sensitive
1407 information you might want to make sure that exceptions thrown by JSON::XS
1408 will not end up in front of untrusted eyes.
1409
1410 If you are using JSON::XS to return packets to consumption
1411 by JavaScript scripts in a browser you should have a look at
1412 L<http://blog.archive.jpsykes.com/47/practical-csrf-and-json-security/> to
1413 see whether you are vulnerable to some common attack vectors (which really
1414 are browser design bugs, but it is still you who will have to deal with
1415 it, as major browser developers care only for features, not about getting
1416 security right).
1417
1418
1419 =head1 INTEROPERABILITY WITH OTHER MODULES
1420
1421 C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
1422 constants. That means that the JSON true and false values will be
1423 comaptible to true and false values of iother modules that do the same,
1424 such as L<JSON::PP> and L<CBOR::XS>.
1425
1426
1427 =head1 THREADS
1428
1429 This module is I<not> guaranteed to be thread safe and there are no
1430 plans to change this until Perl gets thread support (as opposed to the
1431 horribly slow so-called "threads" which are simply slow and bloated
1432 process simulations - use fork, it's I<much> faster, cheaper, better).
1433
1434 (It might actually work, but you have been warned).
1435
1436
1437 =head1 THE PERILS OF SETLOCALE
1438
1439 Sometimes people avoid the Perl locale support and directly call the
1440 system's setlocale function with C<LC_ALL>.
1441
1442 This breaks both perl and modules such as JSON::XS, as stringification of
1443 numbers no longer works correctly (e.g. C<$x = 0.1; print "$x"+1> might
1444 print C<1>, and JSON::XS might output illegal JSON as JSON::XS relies on
1445 perl to stringify numbers).
1446
1447 The solution is simple: don't call C<setlocale>, or use it for only those
1448 categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>.
1449
1450 If you need C<LC_NUMERIC>, you should enable it only around the code that
1451 actually needs it (avoiding stringification of numbers), and restore it
1452 afterwards.
1453
1454
1455 =head1 BUGS
1456
1457 While the goal of this module is to be correct, that unfortunately does
1458 not mean it's bug-free, only that I think its design is bug-free. If you
1459 keep reporting bugs they will be fixed swiftly, though.
1460
1461 Please refrain from using rt.cpan.org or any other bug reporting
1462 service. I put the contact address into my modules for a reason.
1463
1464 =cut
1465
1466 BEGIN {
1467 *true = \$Types::Serialiser::true;
1468 *true = \&Types::Serialiser::true;
1469 *false = \$Types::Serialiser::false;
1470 *false = \&Types::Serialiser::false;
1471 *is_bool = \&Types::Serialiser::is_bool;
1472
1473 *JSON::XS::Boolean:: = *Types::Serialiser::Boolean::;
1474 }
1475
1476 XSLoader::load "JSON::XS", $VERSION;
1477
1478 =head1 SEE ALSO
1479
1480 The F<json_xs> command line utility for quick experiments.
1481
1482 =head1 AUTHOR
1483
1484 Marc Lehmann <schmorp@schmorp.de>
1485 http://home.schmorp.de/
1486
1487 =cut
1488
1489 1
1490