ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
Revision: 1.144
Committed: Mon Oct 28 23:19:54 2013 UTC (10 years, 6 months ago) by root
Branch: MAIN
Changes since 1.143: +35 -37 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.92 =head1 NAME
2    
3 root 1.102 JSON::XS - JSON serialising/deserialising, done correctly and fast
4    
5 root 1.84 =encoding utf-8
6    
7 root 1.62 JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ
8     (http://fleur.hio.jp/perldoc/mix/lib/JSON/XS.html)
9    
10 root 1.1 =head1 SYNOPSIS
11    
12     use JSON::XS;
13    
14 root 1.22 # exported functions, they croak on error
15     # and expect/generate UTF-8
16 root 1.12
17 root 1.78 $utf8_encoded_json_text = encode_json $perl_hash_or_arrayref;
18     $perl_hash_or_arrayref = decode_json $utf8_encoded_json_text;
19 root 1.12
20 root 1.22 # OO-interface
21 root 1.12
22     $coder = JSON::XS->new->ascii->pretty->allow_nonref;
23     $pretty_printed_unencoded = $coder->encode ($perl_scalar);
24     $perl_scalar = $coder->decode ($unicode_json_text);
25    
26 root 1.77 # Note that JSON version 2.0 and above will automatically use JSON::XS
27     # if available, at virtually no speed overhead either, so you should
28     # be able to just:
29    
30     use JSON;
31    
32     # and do the same things, except that you have a pure-perl fallback now.
33    
34 root 1.1 =head1 DESCRIPTION
35    
36 root 1.2 This module converts Perl data structures to JSON and vice versa. Its
37     primary goal is to be I<correct> and its secondary goal is to be
38     I<fast>. To reach the latter goal it was written in C.
39    
40 root 1.77 Beginning with version 2.0 of the JSON module, when both JSON and
41     JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42 root 1.105 overridden) with no overhead due to emulation (by inheriting constructor
43 root 1.77 and methods). If JSON::XS is not available, it will fall back to the
44     compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45     gives you a portable JSON API that can be fast when you need and doesn't
46     require a C compiler when that is a problem.
47    
48 root 1.2 As this is the n-th-something JSON module on CPAN, what was the reason
49     to write yet another JSON module? While it seems there are many JSON
50     modules, none of them correctly handle all corner cases, and in most cases
51     their maintainers are unresponsive, gone missing, or not listening to bug
52     reports for other reasons.
53    
54 root 1.10 See MAPPING, below, on how JSON::XS maps perl values to JSON values and
55     vice versa.
56    
57 root 1.2 =head2 FEATURES
58    
59 root 1.1 =over 4
60    
61 root 1.68 =item * correct Unicode handling
62 root 1.2
63 root 1.84 This module knows how to handle Unicode, documents how and when it does
64     so, and even documents what "correct" means.
65 root 1.2
66     =item * round-trip integrity
67    
68 root 1.105 When you serialise a perl data structure using only data types supported
69 root 1.131 by JSON and Perl, the deserialised data structure is identical on the Perl
70     level. (e.g. the string "2.0" doesn't suddenly become "2" just because
71     it looks like a number). There I<are> minor exceptions to this, read the
72     MAPPING section below to learn about those.
73 root 1.2
74     =item * strict checking of JSON correctness
75    
76 root 1.16 There is no guessing, no generating of illegal JSON texts by default,
77 root 1.10 and only JSON is accepted as input by default (the latter is a security
78     feature).
79 root 1.2
80     =item * fast
81    
82 root 1.84 Compared to other JSON modules and other serialisers such as Storable,
83     this module usually compares favourably in terms of speed, too.
84 root 1.2
85     =item * simple to use
86    
87 root 1.105 This module has both a simple functional interface as well as an object
88 root 1.140 oriented interface.
89 root 1.2
90     =item * reasonably versatile output formats
91    
92 root 1.84 You can choose between the most compact guaranteed-single-line format
93 root 1.105 possible (nice for simple line-based protocols), a pure-ASCII format
94 root 1.21 (for when your transport is not 8-bit clean, still supports the whole
95 root 1.68 Unicode range), or a pretty-printed format (for when you want to read that
96 root 1.21 stuff). Or you can combine those features in whatever way you like.
97 root 1.2
98     =back
99    
100 root 1.1 =cut
101    
102     package JSON::XS;
103    
104 root 1.121 use common::sense;
105 root 1.20
106 root 1.139 our $VERSION = 2.34;
107 root 1.43 our @ISA = qw(Exporter);
108 root 1.1
109 root 1.141 our @EXPORT = qw(encode_json decode_json);
110 root 1.1
111 root 1.43 use Exporter;
112     use XSLoader;
113 root 1.1
114 root 1.144 use Types::Serialiser ();
115    
116 root 1.2 =head1 FUNCTIONAL INTERFACE
117    
118 root 1.68 The following convenience methods are provided by this module. They are
119 root 1.2 exported by default:
120    
121     =over 4
122    
123 root 1.78 =item $json_text = encode_json $perl_scalar
124 root 1.2
125 root 1.63 Converts the given Perl data structure to a UTF-8 encoded, binary string
126     (that is, the string contains octets only). Croaks on error.
127 root 1.2
128 root 1.16 This function call is functionally identical to:
129 root 1.2
130 root 1.16 $json_text = JSON::XS->new->utf8->encode ($perl_scalar)
131    
132 root 1.105 Except being faster.
133 root 1.16
134 root 1.78 =item $perl_scalar = decode_json $json_text
135 root 1.2
136 root 1.78 The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries
137 root 1.63 to parse that as an UTF-8 encoded JSON text, returning the resulting
138     reference. Croaks on error.
139 root 1.2
140 root 1.16 This function call is functionally identical to:
141    
142     $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
143    
144 root 1.105 Except being faster.
145 root 1.2
146     =back
147    
148 root 1.23
149 root 1.63 =head1 A FEW NOTES ON UNICODE AND PERL
150    
151     Since this often leads to confusion, here are a few very clear words on
152     how Unicode works in Perl, modulo bugs.
153    
154     =over 4
155    
156     =item 1. Perl strings can store characters with ordinal values > 255.
157    
158 root 1.68 This enables you to store Unicode characters as single characters in a
159 root 1.63 Perl string - very natural.
160    
161     =item 2. Perl does I<not> associate an encoding with your strings.
162    
163 root 1.84 ... until you force it to, e.g. when matching it against a regex, or
164     printing the scalar to a file, in which case Perl either interprets your
165     string as locale-encoded text, octets/binary, or as Unicode, depending
166     on various settings. In no case is an encoding stored together with your
167     data, it is I<use> that decides encoding, not any magical meta data.
168 root 1.63
169     =item 3. The internal utf-8 flag has no meaning with regards to the
170     encoding of your string.
171    
172     Just ignore that flag unless you debug a Perl bug, a module written in
173     XS or want to dive into the internals of perl. Otherwise it will only
174     confuse you, as, despite the name, it says nothing about how your string
175 root 1.68 is encoded. You can have Unicode strings with that flag set, with that
176 root 1.63 flag clear, and you can have binary data with that flag set and that flag
177     clear. Other possibilities exist, too.
178    
179     If you didn't know about that flag, just the better, pretend it doesn't
180     exist.
181    
182     =item 4. A "Unicode String" is simply a string where each character can be
183 root 1.105 validly interpreted as a Unicode code point.
184 root 1.63
185     If you have UTF-8 encoded data, it is no longer a Unicode string, but a
186     Unicode string encoded in UTF-8, giving you a binary string.
187    
188     =item 5. A string containing "high" (> 255) character values is I<not> a UTF-8 string.
189    
190 root 1.68 It's a fact. Learn to live with it.
191 root 1.63
192     =back
193    
194     I hope this helps :)
195    
196    
197 root 1.2 =head1 OBJECT-ORIENTED INTERFACE
198    
199     The object oriented interface lets you configure your own encoding or
200     decoding style, within the limits of supported formats.
201    
202     =over 4
203    
204     =item $json = new JSON::XS
205    
206     Creates a new JSON::XS object that can be used to de/encode JSON
207     strings. All boolean flags described below are by default I<disabled>.
208 root 1.1
209 root 1.2 The mutators for flags all return the JSON object again and thus calls can
210     be chained:
211    
212 root 1.16 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]})
213 root 1.3 => {"a": [1, 2]}
214 root 1.2
215 root 1.7 =item $json = $json->ascii ([$enable])
216 root 1.2
217 root 1.72 =item $enabled = $json->get_ascii
218    
219 root 1.16 If C<$enable> is true (or missing), then the C<encode> method will not
220     generate characters outside the code range C<0..127> (which is ASCII). Any
221 root 1.68 Unicode characters outside that range will be escaped using either a
222 root 1.16 single \uXXXX (BMP characters) or a double \uHHHH\uLLLLL escape sequence,
223 root 1.32 as per RFC4627. The resulting encoded JSON text can be treated as a native
224 root 1.68 Unicode string, an ascii-encoded, latin1-encoded or UTF-8 encoded string,
225 root 1.32 or any other superset of ASCII.
226 root 1.2
227     If C<$enable> is false, then the C<encode> method will not escape Unicode
228 root 1.33 characters unless required by the JSON syntax or other flags. This results
229     in a faster and more compact format.
230    
231 root 1.90 See also the section I<ENCODING/CODESET FLAG NOTES> later in this
232     document.
233    
234 root 1.33 The main use for this flag is to produce JSON texts that can be
235     transmitted over a 7-bit channel, as the encoded JSON texts will not
236     contain any 8 bit characters.
237 root 1.2
238 root 1.16 JSON::XS->new->ascii (1)->encode ([chr 0x10401])
239     => ["\ud801\udc01"]
240 root 1.3
241 root 1.33 =item $json = $json->latin1 ([$enable])
242    
243 root 1.72 =item $enabled = $json->get_latin1
244    
245 root 1.33 If C<$enable> is true (or missing), then the C<encode> method will encode
246     the resulting JSON text as latin1 (or iso-8859-1), escaping any characters
247     outside the code range C<0..255>. The resulting string can be treated as a
248 root 1.68 latin1-encoded JSON text or a native Unicode string. The C<decode> method
249 root 1.33 will not be affected in any way by this flag, as C<decode> by default
250 root 1.68 expects Unicode, which is a strict superset of latin1.
251 root 1.33
252     If C<$enable> is false, then the C<encode> method will not escape Unicode
253     characters unless required by the JSON syntax or other flags.
254    
255 root 1.90 See also the section I<ENCODING/CODESET FLAG NOTES> later in this
256     document.
257    
258 root 1.33 The main use for this flag is efficiently encoding binary data as JSON
259     text, as most octets will not be escaped, resulting in a smaller encoded
260     size. The disadvantage is that the resulting JSON text is encoded
261     in latin1 (and must correctly be treated as such when storing and
262 root 1.68 transferring), a rare encoding for JSON. It is therefore most useful when
263 root 1.33 you want to store data structures known to contain binary data efficiently
264     in files or databases, not when talking to other JSON encoders/decoders.
265    
266     JSON::XS->new->latin1->encode (["\x{89}\x{abc}"]
267     => ["\x{89}\\u0abc"] # (perl syntax, U+abc escaped, U+89 not)
268    
269 root 1.7 =item $json = $json->utf8 ([$enable])
270 root 1.2
271 root 1.72 =item $enabled = $json->get_utf8
272    
273 root 1.7 If C<$enable> is true (or missing), then the C<encode> method will encode
274 root 1.16 the JSON result into UTF-8, as required by many protocols, while the
275 root 1.7 C<decode> method expects to be handled an UTF-8-encoded string. Please
276     note that UTF-8-encoded strings do not contain any characters outside the
277 root 1.16 range C<0..255>, they are thus useful for bytewise/binary I/O. In future
278     versions, enabling this option might enable autodetection of the UTF-16
279     and UTF-32 encoding families, as described in RFC4627.
280 root 1.2
281     If C<$enable> is false, then the C<encode> method will return the JSON
282 root 1.68 string as a (non-encoded) Unicode string, while C<decode> expects thus a
283     Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs
284 root 1.2 to be done yourself, e.g. using the Encode module.
285    
286 root 1.90 See also the section I<ENCODING/CODESET FLAG NOTES> later in this
287     document.
288    
289 root 1.16 Example, output UTF-16BE-encoded JSON:
290    
291     use Encode;
292     $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object);
293    
294     Example, decode UTF-32LE-encoded JSON:
295    
296     use Encode;
297     $object = JSON::XS->new->decode (decode "UTF-32LE", $jsontext);
298 root 1.12
299 root 1.7 =item $json = $json->pretty ([$enable])
300 root 1.2
301     This enables (or disables) all of the C<indent>, C<space_before> and
302 root 1.3 C<space_after> (and in the future possibly more) flags in one call to
303 root 1.2 generate the most readable (or most compact) form possible.
304    
305 root 1.12 Example, pretty-print some simple structure:
306    
307 root 1.3 my $json = JSON::XS->new->pretty(1)->encode ({a => [1,2]})
308     =>
309     {
310     "a" : [
311     1,
312     2
313     ]
314     }
315    
316 root 1.7 =item $json = $json->indent ([$enable])
317 root 1.2
318 root 1.72 =item $enabled = $json->get_indent
319    
320 root 1.7 If C<$enable> is true (or missing), then the C<encode> method will use a multiline
321 root 1.2 format as output, putting every array member or object/hash key-value pair
322 root 1.68 into its own line, indenting them properly.
323 root 1.2
324     If C<$enable> is false, no newlines or indenting will be produced, and the
325 root 1.68 resulting JSON text is guaranteed not to contain any C<newlines>.
326 root 1.2
327 root 1.16 This setting has no effect when decoding JSON texts.
328 root 1.2
329 root 1.7 =item $json = $json->space_before ([$enable])
330 root 1.2
331 root 1.72 =item $enabled = $json->get_space_before
332    
333 root 1.7 If C<$enable> is true (or missing), then the C<encode> method will add an extra
334 root 1.2 optional space before the C<:> separating keys from values in JSON objects.
335    
336     If C<$enable> is false, then the C<encode> method will not add any extra
337     space at those places.
338    
339 root 1.16 This setting has no effect when decoding JSON texts. You will also
340     most likely combine this setting with C<space_after>.
341 root 1.2
342 root 1.12 Example, space_before enabled, space_after and indent disabled:
343    
344     {"key" :"value"}
345    
346 root 1.7 =item $json = $json->space_after ([$enable])
347 root 1.2
348 root 1.72 =item $enabled = $json->get_space_after
349    
350 root 1.7 If C<$enable> is true (or missing), then the C<encode> method will add an extra
351 root 1.2 optional space after the C<:> separating keys from values in JSON objects
352     and extra whitespace after the C<,> separating key-value pairs and array
353     members.
354    
355     If C<$enable> is false, then the C<encode> method will not add any extra
356     space at those places.
357    
358 root 1.16 This setting has no effect when decoding JSON texts.
359 root 1.2
360 root 1.12 Example, space_before and indent disabled, space_after enabled:
361    
362     {"key": "value"}
363    
364 root 1.59 =item $json = $json->relaxed ([$enable])
365    
366 root 1.72 =item $enabled = $json->get_relaxed
367    
368 root 1.59 If C<$enable> is true (or missing), then C<decode> will accept some
369     extensions to normal JSON syntax (see below). C<encode> will not be
370     affected in anyway. I<Be aware that this option makes you accept invalid
371     JSON texts as if they were valid!>. I suggest only to use this option to
372     parse application-specific files written by humans (configuration files,
373     resource files etc.)
374    
375     If C<$enable> is false (the default), then C<decode> will only accept
376     valid JSON texts.
377    
378     Currently accepted extensions are:
379    
380     =over 4
381    
382     =item * list items can have an end-comma
383    
384     JSON I<separates> array elements and key-value pairs with commas. This
385     can be annoying if you write JSON texts manually and want to be able to
386     quickly append elements, so this extension accepts comma at the end of
387     such items not just between them:
388    
389     [
390     1,
391     2, <- this comma not normally allowed
392     ]
393     {
394     "k1": "v1",
395     "k2": "v2", <- this comma not normally allowed
396     }
397    
398 root 1.60 =item * shell-style '#'-comments
399    
400     Whenever JSON allows whitespace, shell-style comments are additionally
401     allowed. They are terminated by the first carriage-return or line-feed
402     character, after which more white-space and comments are allowed.
403    
404     [
405     1, # this comment not allowed in JSON
406     # neither this one...
407     ]
408    
409 root 1.59 =back
410    
411 root 1.7 =item $json = $json->canonical ([$enable])
412 root 1.2
413 root 1.72 =item $enabled = $json->get_canonical
414    
415 root 1.7 If C<$enable> is true (or missing), then the C<encode> method will output JSON objects
416 root 1.2 by sorting their keys. This is adding a comparatively high overhead.
417    
418     If C<$enable> is false, then the C<encode> method will output key-value
419     pairs in the order Perl stores them (which will likely change between runs
420 root 1.139 of the same script, and can change even within the same run from 5.18
421     onwards).
422 root 1.2
423     This option is useful if you want the same data structure to be encoded as
424 root 1.16 the same JSON text (given the same overall settings). If it is disabled,
425 root 1.68 the same hash might be encoded differently even if contains the same data,
426 root 1.2 as key-value pairs have no inherent ordering in Perl.
427    
428 root 1.16 This setting has no effect when decoding JSON texts.
429 root 1.2
430 root 1.122 This setting has currently no effect on tied hashes.
431    
432 root 1.7 =item $json = $json->allow_nonref ([$enable])
433 root 1.3
434 root 1.72 =item $enabled = $json->get_allow_nonref
435    
436 root 1.7 If C<$enable> is true (or missing), then the C<encode> method can convert a
437 root 1.3 non-reference into its corresponding string, number or null JSON value,
438     which is an extension to RFC4627. Likewise, C<decode> will accept those JSON
439     values instead of croaking.
440    
441     If C<$enable> is false, then the C<encode> method will croak if it isn't
442 root 1.16 passed an arrayref or hashref, as JSON texts must either be an object
443 root 1.3 or array. Likewise, C<decode> will croak if given something that is not a
444     JSON object or array.
445    
446 root 1.12 Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>,
447     resulting in an invalid JSON text:
448    
449     JSON::XS->new->allow_nonref->encode ("Hello, World!")
450     => "Hello, World!"
451    
452 root 1.99 =item $json = $json->allow_unknown ([$enable])
453    
454     =item $enabled = $json->get_allow_unknown
455    
456     If C<$enable> is true (or missing), then C<encode> will I<not> throw an
457     exception when it encounters values it cannot represent in JSON (for
458     example, filehandles) but instead will encode a JSON C<null> value. Note
459     that blessed objects are not included here and are handled separately by
460     c<allow_nonref>.
461    
462     If C<$enable> is false (the default), then C<encode> will throw an
463     exception when it encounters anything it cannot encode as JSON.
464    
465     This option does not affect C<decode> in any way, and it is recommended to
466     leave it off unless you know your communications partner.
467    
468 root 1.44 =item $json = $json->allow_blessed ([$enable])
469    
470 root 1.75 =item $enabled = $json->get_allow_blessed
471 root 1.72
472 root 1.44 If C<$enable> is true (or missing), then the C<encode> method will not
473     barf when it encounters a blessed reference. Instead, the value of the
474 root 1.68 B<convert_blessed> option will decide whether C<null> (C<convert_blessed>
475 root 1.76 disabled or no C<TO_JSON> method found) or a representation of the
476     object (C<convert_blessed> enabled and C<TO_JSON> method found) is being
477 root 1.44 encoded. Has no effect on C<decode>.
478    
479     If C<$enable> is false (the default), then C<encode> will throw an
480     exception when it encounters a blessed object.
481    
482     =item $json = $json->convert_blessed ([$enable])
483    
484 root 1.72 =item $enabled = $json->get_convert_blessed
485    
486 root 1.44 If C<$enable> is true (or missing), then C<encode>, upon encountering a
487     blessed object, will check for the availability of the C<TO_JSON> method
488     on the object's class. If found, it will be called in scalar context
489     and the resulting scalar will be encoded instead of the object. If no
490     C<TO_JSON> method is found, the value of C<allow_blessed> will decide what
491     to do.
492    
493     The C<TO_JSON> method may safely call die if it wants. If C<TO_JSON>
494     returns other blessed objects, those will be handled in the same
495     way. C<TO_JSON> must take care of not causing an endless recursion cycle
496     (== crash) in this case. The name of C<TO_JSON> was chosen because other
497 root 1.46 methods called by the Perl core (== not by the user of the object) are
498 root 1.78 usually in upper case letters and to avoid collisions with any C<to_json>
499     function or method.
500 root 1.44
501 root 1.45 This setting does not yet influence C<decode> in any way, but in the
502     future, global hooks might get installed that influence C<decode> and are
503     enabled by this setting.
504    
505 root 1.44 If C<$enable> is false, then the C<allow_blessed> setting will decide what
506     to do when a blessed object is found.
507    
508 root 1.52 =item $json = $json->filter_json_object ([$coderef->($hashref)])
509 root 1.51
510     When C<$coderef> is specified, it will be called from C<decode> each
511     time it decodes a JSON object. The only argument is a reference to the
512     newly-created hash. If the code references returns a single scalar (which
513     need not be a reference), this value (i.e. a copy of that scalar to avoid
514     aliasing) is inserted into the deserialised data structure. If it returns
515     an empty list (NOTE: I<not> C<undef>, which is a valid scalar), the
516     original deserialised hash will be inserted. This setting can slow down
517     decoding considerably.
518    
519 root 1.52 When C<$coderef> is omitted or undefined, any existing callback will
520     be removed and C<decode> will not change the deserialised hash in any
521     way.
522 root 1.51
523     Example, convert all JSON objects into the integer 5:
524    
525     my $js = JSON::XS->new->filter_json_object (sub { 5 });
526     # returns [5]
527     $js->decode ('[{}]')
528 root 1.52 # throw an exception because allow_nonref is not enabled
529     # so a lone 5 is not allowed.
530 root 1.51 $js->decode ('{"a":1, "b":2}');
531    
532 root 1.52 =item $json = $json->filter_json_single_key_object ($key [=> $coderef->($value)])
533 root 1.51
534 root 1.52 Works remotely similar to C<filter_json_object>, but is only called for
535     JSON objects having a single key named C<$key>.
536 root 1.51
537     This C<$coderef> is called before the one specified via
538 root 1.52 C<filter_json_object>, if any. It gets passed the single value in the JSON
539     object. If it returns a single value, it will be inserted into the data
540     structure. If it returns nothing (not even C<undef> but the empty list),
541     the callback from C<filter_json_object> will be called next, as if no
542     single-key callback were specified.
543    
544     If C<$coderef> is omitted or undefined, the corresponding callback will be
545     disabled. There can only ever be one callback for a given key.
546 root 1.51
547     As this callback gets called less often then the C<filter_json_object>
548     one, decoding speed will not usually suffer as much. Therefore, single-key
549     objects make excellent targets to serialise Perl objects into, especially
550     as single-key JSON objects are as close to the type-tagged value concept
551 root 1.68 as JSON gets (it's basically an ID/VALUE tuple). Of course, JSON does not
552 root 1.51 support this in any way, so you need to make sure your data never looks
553     like a serialised Perl hash.
554    
555     Typical names for the single object key are C<__class_whatever__>, or
556     C<$__dollars_are_rarely_used__$> or C<}ugly_brace_placement>, or even
557     things like C<__class_md5sum(classname)__>, to reduce the risk of clashing
558     with real hashes.
559    
560     Example, decode JSON objects of the form C<< { "__widget__" => <id> } >>
561     into the corresponding C<< $WIDGET{<id>} >> object:
562    
563     # return whatever is in $WIDGET{5}:
564     JSON::XS
565     ->new
566 root 1.52 ->filter_json_single_key_object (__widget__ => sub {
567     $WIDGET{ $_[0] }
568 root 1.51 })
569     ->decode ('{"__widget__": 5')
570    
571     # this can be used with a TO_JSON method in some "widget" class
572     # for serialisation to json:
573     sub WidgetBase::TO_JSON {
574     my ($self) = @_;
575    
576     unless ($self->{id}) {
577     $self->{id} = ..get..some..id..;
578     $WIDGET{$self->{id}} = $self;
579     }
580    
581     { __widget__ => $self->{id} }
582     }
583    
584 root 1.7 =item $json = $json->shrink ([$enable])
585    
586 root 1.72 =item $enabled = $json->get_shrink
587    
588 root 1.7 Perl usually over-allocates memory a bit when allocating space for
589 root 1.24 strings. This flag optionally resizes strings generated by either
590 root 1.7 C<encode> or C<decode> to their minimum size possible. This can save
591 root 1.16 memory when your JSON texts are either very very long or you have many
592 root 1.8 short strings. It will also try to downgrade any strings to octet-form
593     if possible: perl stores strings internally either in an encoding called
594     UTF-X or in octet-form. The latter cannot store everything but uses less
595 root 1.24 space in general (and some buggy Perl or C code might even rely on that
596     internal representation being used).
597 root 1.7
598 root 1.24 The actual definition of what shrink does might change in future versions,
599     but it will always try to save space at the expense of time.
600    
601     If C<$enable> is true (or missing), the string returned by C<encode> will
602     be shrunk-to-fit, while all strings generated by C<decode> will also be
603     shrunk-to-fit.
604 root 1.7
605     If C<$enable> is false, then the normal perl allocation algorithms are used.
606     If you work with your data, then this is likely to be faster.
607    
608     In the future, this setting might control other things, such as converting
609     strings that look like integers or floats into integers or floats
610     internally (there is no difference on the Perl level), saving space.
611    
612 root 1.23 =item $json = $json->max_depth ([$maximum_nesting_depth])
613    
614 root 1.72 =item $max_depth = $json->get_max_depth
615    
616 root 1.28 Sets the maximum nesting level (default C<512>) accepted while encoding
617 root 1.101 or decoding. If a higher nesting level is detected in JSON text or a Perl
618     data structure, then the encoder and decoder will stop and croak at that
619     point.
620 root 1.23
621     Nesting level is defined by number of hash- or arrayrefs that the encoder
622     needs to traverse to reach a given point or the number of C<{> or C<[>
623     characters without their matching closing parenthesis crossed to reach a
624     given character in a string.
625    
626     Setting the maximum depth to one disallows any nesting, so that ensures
627     that the object is only a single hash/object or array.
628    
629 root 1.101 If no argument is given, the highest possible setting will be used, which
630     is rarely useful.
631    
632     Note that nesting is implemented by recursion in C. The default value has
633     been chosen to be as large as typical operating systems allow without
634     crashing.
635 root 1.47
636     See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
637    
638     =item $json = $json->max_size ([$maximum_string_size])
639    
640 root 1.72 =item $max_size = $json->get_max_size
641    
642 root 1.47 Set the maximum length a JSON text may have (in bytes) where decoding is
643     being attempted. The default is C<0>, meaning no limit. When C<decode>
644 root 1.101 is called on a string that is longer then this many bytes, it will not
645 root 1.47 attempt to decode the string but throw an exception. This setting has no
646     effect on C<encode> (yet).
647    
648 root 1.101 If no argument is given, the limit check will be deactivated (same as when
649     C<0> is specified).
650 root 1.23
651     See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
652    
653 root 1.16 =item $json_text = $json->encode ($perl_scalar)
654 root 1.2
655 root 1.141 Converts the given Perl value or data structure to its JSON
656     representation. Croaks on error.
657 root 1.1
658 root 1.16 =item $perl_scalar = $json->decode ($json_text)
659 root 1.1
660 root 1.16 The opposite of C<encode>: expects a JSON text and tries to parse it,
661 root 1.2 returning the resulting simple scalar or reference. Croaks on error.
662 root 1.1
663 root 1.34 =item ($perl_scalar, $characters) = $json->decode_prefix ($json_text)
664    
665     This works like the C<decode> method, but instead of raising an exception
666     when there is trailing garbage after the first JSON object, it will
667     silently stop parsing there and return the number of characters consumed
668     so far.
669    
670     This is useful if your JSON texts are not delimited by an outer protocol
671 root 1.143 and you need to know where the JSON text ends.
672 root 1.34
673     JSON::XS->new->decode_prefix ("[1] the tail")
674     => ([], 3)
675    
676 root 1.1 =back
677    
678 root 1.23
679 root 1.94 =head1 INCREMENTAL PARSING
680    
681     In some cases, there is the need for incremental parsing of JSON
682     texts. While this module always has to keep both JSON text and resulting
683     Perl data structure in memory at one time, it does allow you to parse a
684     JSON stream incrementally. It does so by accumulating text until it has
685     a full JSON object, which it then can decode. This process is similar to
686 root 1.108 using C<decode_prefix> to see if a full JSON object is available, but
687     is much more efficient (and can be implemented with a minimum of method
688     calls).
689    
690     JSON::XS will only attempt to parse the JSON text once it is sure it
691     has enough text to get a decisive result, using a very simple but
692     truly incremental parser. This means that it sometimes won't stop as
693 root 1.134 early as the full parser, for example, it doesn't detect mismatched
694     parentheses. The only thing it guarantees is that it starts decoding as
695 root 1.108 soon as a syntactically valid JSON text has been seen. This means you need
696     to set resource limits (e.g. C<max_size>) to ensure the parser will stop
697     parsing in the presence if syntax errors.
698 root 1.94
699 root 1.108 The following methods implement this incremental parser.
700 root 1.94
701     =over 4
702    
703     =item [void, scalar or list context] = $json->incr_parse ([$string])
704    
705     This is the central parsing function. It can both append new text and
706     extract objects from the stream accumulated so far (both of these
707     functions are optional).
708    
709     If C<$string> is given, then this string is appended to the already
710     existing JSON fragment stored in the C<$json> object.
711    
712     After that, if the function is called in void context, it will simply
713     return without doing anything further. This can be used to add more text
714     in as many chunks as you want.
715    
716     If the method is called in scalar context, then it will try to extract
717     exactly I<one> JSON object. If that is successful, it will return this
718 root 1.96 object, otherwise it will return C<undef>. If there is a parse error,
719     this method will croak just as C<decode> would do (one can then use
720 root 1.140 C<incr_skip> to skip the erroneous part). This is the most common way of
721 root 1.94 using the method.
722    
723     And finally, in list context, it will try to extract as many objects
724     from the stream as it can find and return them, or the empty list
725     otherwise. For this to work, there must be no separators between the JSON
726 root 1.96 objects or arrays, instead they must be concatenated back-to-back. If
727     an error occurs, an exception will be raised as in the scalar context
728     case. Note that in this case, any previously-parsed JSON texts will be
729     lost.
730    
731 root 1.130 Example: Parse some JSON arrays/objects in a given string and return
732     them.
733    
734     my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]");
735    
736 root 1.94 =item $lvalue_string = $json->incr_text
737    
738     This method returns the currently stored JSON fragment as an lvalue, that
739     is, you can manipulate it. This I<only> works when a preceding call to
740     C<incr_parse> in I<scalar context> successfully returned an object. Under
741     all other circumstances you must not call this function (I mean it.
742     although in simple tests it might actually work, it I<will> fail under
743     real world conditions). As a special exception, you can also call this
744     method before having parsed anything.
745    
746     This function is useful in two cases: a) finding the trailing text after a
747     JSON object or b) parsing multiple JSON objects separated by non-JSON text
748     (such as commas).
749    
750 root 1.97 =item $json->incr_skip
751    
752 root 1.114 This will reset the state of the incremental parser and will remove
753     the parsed text from the input buffer so far. This is useful after
754     C<incr_parse> died, in which case the input buffer and incremental parser
755     state is left unchanged, to skip the text parsed so far and to reset the
756     parse state.
757    
758     The difference to C<incr_reset> is that only text until the parse error
759 root 1.140 occurred is removed.
760 root 1.97
761 root 1.106 =item $json->incr_reset
762    
763     This completely resets the incremental parser, that is, after this call,
764     it will be as if the parser had never parsed anything.
765    
766 root 1.114 This is useful if you want to repeatedly parse JSON objects and want to
767 root 1.106 ignore any trailing data, which means you have to reset the parser after
768     each successful decode.
769    
770 root 1.94 =back
771    
772     =head2 LIMITATIONS
773    
774     All options that affect decoding are supported, except
775 root 1.143 C<allow_nonref>. The reason for this is that it cannot be made to work
776     sensibly: JSON objects and arrays are self-delimited, i.e. you can
777     concatenate them back to back and still decode them perfectly. This does
778     not hold true for JSON numbers, however.
779 root 1.94
780     For example, is the string C<1> a single JSON number, or is it simply the
781     start of C<12>? Or is C<12> a single JSON number, or the concatenation
782     of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
783     takes the conservative route and disallows this case.
784    
785     =head2 EXAMPLES
786    
787     Some examples will make all this clearer. First, a simple example that
788     works similarly to C<decode_prefix>: We want to decode the JSON object at
789     the start of a string and identify the portion after the JSON object:
790    
791     my $text = "[1,2,3] hello";
792    
793     my $json = new JSON::XS;
794    
795     my $obj = $json->incr_parse ($text)
796     or die "expected JSON object or array at beginning of string";
797    
798     my $tail = $json->incr_text;
799     # $tail now contains " hello"
800    
801     Easy, isn't it?
802    
803     Now for a more complicated example: Imagine a hypothetical protocol where
804     you read some requests from a TCP stream, and each request is a JSON
805     array, without any separation between them (in fact, it is often useful to
806     use newlines as "separators", as these get interpreted as whitespace at
807     the start of the JSON text, which makes it possible to test said protocol
808     with C<telnet>...).
809    
810     Here is how you'd do it (it is trivial to write this in an event-based
811     manner):
812    
813     my $json = new JSON::XS;
814    
815     # read some data from the socket
816     while (sysread $socket, my $buf, 4096) {
817    
818     # split and decode as many requests as possible
819     for my $request ($json->incr_parse ($buf)) {
820     # act on the $request
821     }
822     }
823    
824     Another complicated example: Assume you have a string with JSON objects
825     or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
826     [3]>). To parse them, we have to skip the commas between the JSON texts,
827     and here is where the lvalue-ness of C<incr_text> comes in useful:
828    
829     my $text = "[1],[2], [3]";
830     my $json = new JSON::XS;
831    
832     # void context, so no parsing done
833     $json->incr_parse ($text);
834    
835     # now extract as many objects as possible. note the
836     # use of scalar context so incr_text can be called.
837     while (my $obj = $json->incr_parse) {
838     # do something with $obj
839    
840     # now skip the optional comma
841     $json->incr_text =~ s/^ \s* , //x;
842     }
843    
844     Now lets go for a very complex example: Assume that you have a gigantic
845     JSON array-of-objects, many gigabytes in size, and you want to parse it,
846     but you cannot load it into memory fully (this has actually happened in
847     the real world :).
848    
849     Well, you lost, you have to implement your own JSON parser. But JSON::XS
850     can still help you: You implement a (very simple) array parser and let
851     JSON decode the array elements, which are all full JSON objects on their
852     own (this wouldn't work if the array elements could be JSON numbers, for
853     example):
854    
855     my $json = new JSON::XS;
856    
857     # open the monster
858     open my $fh, "<bigfile.json"
859     or die "bigfile: $!";
860    
861     # first parse the initial "["
862     for (;;) {
863     sysread $fh, my $buf, 65536
864     or die "read error: $!";
865     $json->incr_parse ($buf); # void context, so no parsing
866    
867     # Exit the loop once we found and removed(!) the initial "[".
868     # In essence, we are (ab-)using the $json object as a simple scalar
869     # we append data to.
870     last if $json->incr_text =~ s/^ \s* \[ //x;
871     }
872    
873     # now we have the skipped the initial "[", so continue
874     # parsing all the elements.
875     for (;;) {
876     # in this loop we read data until we got a single JSON object
877     for (;;) {
878     if (my $obj = $json->incr_parse) {
879     # do something with $obj
880     last;
881     }
882    
883     # add more data
884     sysread $fh, my $buf, 65536
885     or die "read error: $!";
886     $json->incr_parse ($buf); # void context, so no parsing
887     }
888    
889     # in this loop we read data until we either found and parsed the
890     # separating "," between elements, or the final "]"
891     for (;;) {
892     # first skip whitespace
893     $json->incr_text =~ s/^\s*//;
894    
895     # if we find "]", we are done
896     if ($json->incr_text =~ s/^\]//) {
897     print "finished.\n";
898     exit;
899     }
900    
901     # if we find ",", we can continue with the next element
902     if ($json->incr_text =~ s/^,//) {
903     last;
904     }
905    
906     # if we find anything else, we have a parse error!
907     if (length $json->incr_text) {
908     die "parse error near ", $json->incr_text;
909     }
910    
911     # else add more data
912     sysread $fh, my $buf, 65536
913     or die "read error: $!";
914     $json->incr_parse ($buf); # void context, so no parsing
915     }
916    
917     This is a complex example, but most of the complexity comes from the fact
918     that we are trying to be correct (bear with me if I am wrong, I never ran
919     the above example :).
920    
921    
922    
923 root 1.10 =head1 MAPPING
924    
925     This section describes how JSON::XS maps Perl values to JSON values and
926     vice versa. These mappings are designed to "do the right thing" in most
927     circumstances automatically, preserving round-tripping characteristics
928     (what you put in comes out as something equivalent).
929    
930     For the more enlightened: note that in the following descriptions,
931 root 1.68 lowercase I<perl> refers to the Perl interpreter, while uppercase I<Perl>
932 root 1.10 refers to the abstract Perl language itself.
933    
934 root 1.39
935 root 1.10 =head2 JSON -> PERL
936    
937     =over 4
938    
939     =item object
940    
941     A JSON object becomes a reference to a hash in Perl. No ordering of object
942 root 1.68 keys is preserved (JSON does not preserve object key ordering itself).
943 root 1.10
944     =item array
945    
946     A JSON array becomes a reference to an array in Perl.
947    
948     =item string
949    
950     A JSON string becomes a string scalar in Perl - Unicode codepoints in JSON
951     are represented by the same codepoints in the Perl string, so no manual
952     decoding is necessary.
953    
954     =item number
955    
956 root 1.56 A JSON number becomes either an integer, numeric (floating point) or
957     string scalar in perl, depending on its range and any fractional parts. On
958     the Perl level, there is no difference between those as Perl handles all
959     the conversion details, but an integer may take slightly less memory and
960 root 1.84 might represent more values exactly than floating point numbers.
961 root 1.56
962     If the number consists of digits only, JSON::XS will try to represent
963     it as an integer value. If that fails, it will try to represent it as
964     a numeric (floating point) value if that is possible without loss of
965 root 1.84 precision. Otherwise it will preserve the number as a string value (in
966     which case you lose roundtripping ability, as the JSON number will be
967 root 1.140 re-encoded to a JSON string).
968 root 1.56
969     Numbers containing a fractional or exponential part will always be
970     represented as numeric (floating point) values, possibly at a loss of
971 root 1.84 precision (in which case you might lose perfect roundtripping ability, but
972     the JSON number will still be re-encoded as a JSON number).
973 root 1.10
974 root 1.131 Note that precision is not accuracy - binary floating point values cannot
975     represent most decimal fractions exactly, and when converting from and to
976     floating point, JSON::XS only guarantees precision up to but not including
977 root 1.140 the least significant bit.
978 root 1.131
979 root 1.10 =item true, false
980    
981 root 1.144 These JSON atoms become C<Types::Serialiser::true> and
982     C<Types::Serialiser::false>, respectively. They are overloaded to act
983     almost exactly like the numbers C<1> and C<0>. You can check whether
984     a scalar is a JSON boolean by using the C<Types::Serialiser::is_bool>
985     function (after C<use Types::Serialier>, of course).
986 root 1.10
987     =item null
988    
989     A JSON null atom becomes C<undef> in Perl.
990    
991     =back
992    
993 root 1.39
994 root 1.10 =head2 PERL -> JSON
995    
996     The mapping from Perl to JSON is slightly more difficult, as Perl is a
997     truly typeless language, so we can only guess which JSON type is meant by
998     a Perl value.
999    
1000     =over 4
1001    
1002     =item hash references
1003    
1004 root 1.142 Perl hash references become JSON objects. As there is no inherent
1005     ordering in hash keys (or JSON objects), they will usually be encoded
1006     in a pseudo-random order. JSON::XS can optionally sort the hash keys
1007     (determined by the I<canonical> flag), so the same datastructure will
1008     serialise to the same JSON text (given same settings and version of
1009     JSON::XS), but this incurs a runtime overhead and is only rarely useful,
1010     e.g. when you want to compare some JSON text against another for equality.
1011 root 1.10
1012     =item array references
1013    
1014     Perl array references become JSON arrays.
1015    
1016 root 1.25 =item other references
1017    
1018     Other unblessed references are generally not allowed and will cause an
1019     exception to be thrown, except for references to the integers C<0> and
1020 root 1.144 C<1>, which get turned into C<false> and C<true> atoms in JSON.
1021 root 1.25
1022 root 1.144 Since C<JSON::XS> uses the boolean model from L<Types::Serialiser>, you
1023     can also C<use Types::Serialiser> and then use C<Types::Serialiser::false>
1024     and C<Types::Serialiser::true> to improve readability.
1025 root 1.25
1026 root 1.144 use Types::Serialiser;
1027     encode_json [\0, Types::Serialiser::true] # yields [false,true]
1028 root 1.43
1029 root 1.144 =item Types::Serialiser::true, Types::Serialiser::false
1030    
1031     These special values from the L<Types::Serialiser> module become JSON true
1032     and JSON false values, respectively. You can also use C<\1> and C<\0>
1033     directly if you want.
1034 root 1.43
1035 root 1.10 =item blessed objects
1036    
1037 root 1.83 Blessed objects are not directly representable in JSON. See the
1038     C<allow_blessed> and C<convert_blessed> methods on various options on
1039     how to deal with this: basically, you can choose between throwing an
1040     exception, encoding the reference as if it weren't blessed, or provide
1041     your own serialiser method.
1042 root 1.10
1043     =item simple scalars
1044    
1045     Simple Perl scalars (any scalar that is not a reference) are the most
1046     difficult objects to encode: JSON::XS will encode undefined scalars as
1047 root 1.83 JSON C<null> values, scalars that have last been used in a string context
1048     before encoding as JSON strings, and anything else as number value:
1049 root 1.10
1050     # dump as number
1051 root 1.78 encode_json [2] # yields [2]
1052     encode_json [-3.0e17] # yields [-3e+17]
1053     my $value = 5; encode_json [$value] # yields [5]
1054 root 1.10
1055     # used as string, so dump as string
1056     print $value;
1057 root 1.78 encode_json [$value] # yields ["5"]
1058 root 1.10
1059     # undef becomes null
1060 root 1.78 encode_json [undef] # yields [null]
1061 root 1.10
1062 root 1.68 You can force the type to be a JSON string by stringifying it:
1063 root 1.10
1064     my $x = 3.1; # some variable containing a number
1065     "$x"; # stringified
1066     $x .= ""; # another, more awkward way to stringify
1067     print $x; # perl does it for you, too, quite often
1068    
1069 root 1.68 You can force the type to be a JSON number by numifying it:
1070 root 1.10
1071     my $x = "3"; # some variable containing a string
1072     $x += 0; # numify it, ensuring it will be dumped as a number
1073 root 1.68 $x *= 1; # same thing, the choice is yours.
1074 root 1.10
1075 root 1.68 You can not currently force the type in other, less obscure, ways. Tell me
1076 root 1.91 if you need this capability (but don't forget to explain why it's needed
1077 root 1.83 :).
1078 root 1.10
1079 root 1.131 Note that numerical precision has the same meaning as under Perl (so
1080     binary to decimal conversion follows the same rules as in Perl, which
1081     can differ to other languages). Also, your perl interpreter might expose
1082     extensions to the floating point numbers of your platform, such as
1083     infinities or NaN's - these cannot be represented in JSON, and it is an
1084     error to pass those in.
1085    
1086 root 1.10 =back
1087    
1088 root 1.23
1089 root 1.84 =head1 ENCODING/CODESET FLAG NOTES
1090    
1091     The interested reader might have seen a number of flags that signify
1092     encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be
1093     some confusion on what these do, so here is a short comparison:
1094    
1095 root 1.91 C<utf8> controls whether the JSON text created by C<encode> (and expected
1096 root 1.84 by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only
1097 root 1.91 control whether C<encode> escapes character values outside their respective
1098 root 1.84 codeset range. Neither of these flags conflict with each other, although
1099     some combinations make less sense than others.
1100    
1101     Care has been taken to make all flags symmetrical with respect to
1102     C<encode> and C<decode>, that is, texts encoded with any combination of
1103     these flag values will be correctly decoded when the same flags are used
1104     - in general, if you use different flag settings while encoding vs. when
1105     decoding you likely have a bug somewhere.
1106    
1107     Below comes a verbose discussion of these flags. Note that a "codeset" is
1108     simply an abstract set of character-codepoint pairs, while an encoding
1109     takes those codepoint numbers and I<encodes> them, in our case into
1110     octets. Unicode is (among other things) a codeset, UTF-8 is an encoding,
1111     and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at
1112     the same time, which can be confusing.
1113    
1114     =over 4
1115    
1116     =item C<utf8> flag disabled
1117    
1118     When C<utf8> is disabled (the default), then C<encode>/C<decode> generate
1119     and expect Unicode strings, that is, characters with high ordinal Unicode
1120     values (> 255) will be encoded as such characters, and likewise such
1121 root 1.140 characters are decoded as-is, no changes to them will be done, except
1122 root 1.84 "(re-)interpreting" them as Unicode codepoints or Unicode characters,
1123     respectively (to Perl, these are the same thing in strings unless you do
1124     funny/weird/dumb stuff).
1125    
1126     This is useful when you want to do the encoding yourself (e.g. when you
1127     want to have UTF-16 encoded JSON texts) or when some other layer does
1128     the encoding for you (for example, when printing to a terminal using a
1129     filehandle that transparently encodes to UTF-8 you certainly do NOT want
1130     to UTF-8 encode your data first and have Perl encode it another time).
1131    
1132     =item C<utf8> flag enabled
1133    
1134     If the C<utf8>-flag is enabled, C<encode>/C<decode> will encode all
1135     characters using the corresponding UTF-8 multi-byte sequence, and will
1136     expect your input strings to be encoded as UTF-8, that is, no "character"
1137     of the input string must have any value > 255, as UTF-8 does not allow
1138     that.
1139    
1140     The C<utf8> flag therefore switches between two modes: disabled means you
1141     will get a Unicode string in Perl, enabled means you get an UTF-8 encoded
1142     octet/binary string in Perl.
1143    
1144     =item C<latin1> or C<ascii> flags enabled
1145    
1146     With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters
1147     with ordinal values > 255 (> 127 with C<ascii>) and encode the remaining
1148     characters as specified by the C<utf8> flag.
1149    
1150     If C<utf8> is disabled, then the result is also correctly encoded in those
1151     character sets (as both are proper subsets of Unicode, meaning that a
1152     Unicode string with all character values < 256 is the same thing as a
1153     ISO-8859-1 string, and a Unicode string with all character values < 128 is
1154     the same thing as an ASCII string in Perl).
1155    
1156     If C<utf8> is enabled, you still get a correct UTF-8-encoded string,
1157     regardless of these flags, just some more characters will be escaped using
1158     C<\uXXXX> then before.
1159    
1160     Note that ISO-8859-1-I<encoded> strings are not compatible with UTF-8
1161     encoding, while ASCII-encoded strings are. That is because the ISO-8859-1
1162     encoding is NOT a subset of UTF-8 (despite the ISO-8859-1 I<codeset> being
1163     a subset of Unicode), while ASCII is.
1164    
1165     Surprisingly, C<decode> will ignore these flags and so treat all input
1166     values as governed by the C<utf8> flag. If it is disabled, this allows you
1167     to decode ISO-8859-1- and ASCII-encoded strings, as both strict subsets of
1168     Unicode. If it is enabled, you can correctly decode UTF-8 encoded strings.
1169    
1170     So neither C<latin1> nor C<ascii> are incompatible with the C<utf8> flag -
1171     they only govern when the JSON output engine escapes a character or not.
1172    
1173     The main use for C<latin1> is to relatively efficiently store binary data
1174     as JSON, at the expense of breaking compatibility with most JSON decoders.
1175    
1176     The main use for C<ascii> is to force the output to not contain characters
1177     with values > 127, which means you can interpret the resulting string
1178     as UTF-8, ISO-8859-1, ASCII, KOI8-R or most about any character set and
1179     8-bit-encoding, and still get the same data structure back. This is useful
1180     when your channel for JSON transfer is not 8-bit clean or the encoding
1181     might be mangled in between (e.g. in mail), and works because ASCII is a
1182     proper subset of most 8-bit and multibyte encodings in use in the world.
1183    
1184     =back
1185    
1186    
1187 root 1.115 =head2 JSON and ECMAscript
1188    
1189     JSON syntax is based on how literals are represented in javascript (the
1190     not-standardised predecessor of ECMAscript) which is presumably why it is
1191     called "JavaScript Object Notation".
1192    
1193     However, JSON is not a subset (and also not a superset of course) of
1194     ECMAscript (the standard) or javascript (whatever browsers actually
1195     implement).
1196    
1197     If you want to use javascript's C<eval> function to "parse" JSON, you
1198     might run into parse errors for valid JSON texts, or the resulting data
1199     structure might not be queryable:
1200    
1201     One of the problems is that U+2028 and U+2029 are valid characters inside
1202     JSON strings, but are not allowed in ECMAscript string literals, so the
1203     following Perl fragment will not output something that can be guaranteed
1204     to be parsable by javascript's C<eval>:
1205    
1206     use JSON::XS;
1207    
1208     print encode_json [chr 0x2028];
1209    
1210     The right fix for this is to use a proper JSON parser in your javascript
1211 root 1.117 programs, and not rely on C<eval> (see for example Douglas Crockford's
1212     F<json2.js> parser).
1213 root 1.115
1214     If this is not an option, you can, as a stop-gap measure, simply encode to
1215     ASCII-only JSON:
1216    
1217     use JSON::XS;
1218    
1219     print JSON::XS->new->ascii->encode ([chr 0x2028]);
1220    
1221 root 1.117 Note that this will enlarge the resulting JSON text quite a bit if you
1222     have many non-ASCII characters. You might be tempted to run some regexes
1223     to only escape U+2028 and U+2029, e.g.:
1224 root 1.115
1225 root 1.117 # DO NOT USE THIS!
1226 root 1.115 my $json = JSON::XS->new->utf8->encode ([chr 0x2028]);
1227     $json =~ s/\xe2\x80\xa8/\\u2028/g; # escape U+2028
1228     $json =~ s/\xe2\x80\xa9/\\u2029/g; # escape U+2029
1229     print $json;
1230    
1231 root 1.117 Note that I<this is a bad idea>: the above only works for U+2028 and
1232     U+2029 and thus only for fully ECMAscript-compliant parsers. Many existing
1233     javascript implementations, however, have issues with other characters as
1234     well - using C<eval> naively simply I<will> cause problems.
1235 root 1.116
1236 root 1.115 Another problem is that some javascript implementations reserve
1237     some property names for their own purposes (which probably makes
1238     them non-ECMAscript-compliant). For example, Iceweasel reserves the
1239 root 1.134 C<__proto__> property name for its own purposes.
1240 root 1.115
1241     If that is a problem, you could parse try to filter the resulting JSON
1242     output for these property strings, e.g.:
1243    
1244     $json =~ s/"__proto__"\s*:/"__proto__renamed":/g;
1245    
1246     This works because C<__proto__> is not valid outside of strings, so every
1247 root 1.140 occurrence of C<"__proto__"\s*:> must be a string used as property name.
1248 root 1.115
1249     If you know of other incompatibilities, please let me know.
1250    
1251    
1252 root 1.39 =head2 JSON and YAML
1253    
1254 root 1.80 You often hear that JSON is a subset of YAML. This is, however, a mass
1255 root 1.90 hysteria(*) and very far from the truth (as of the time of this writing),
1256     so let me state it clearly: I<in general, there is no way to configure
1257     JSON::XS to output a data structure as valid YAML> that works in all
1258     cases.
1259 root 1.39
1260 root 1.41 If you really must use JSON::XS to generate YAML, you should use this
1261 root 1.39 algorithm (subject to change in future versions):
1262    
1263     my $to_yaml = JSON::XS->new->utf8->space_after (1);
1264     my $yaml = $to_yaml->encode ($ref) . "\n";
1265    
1266 root 1.83 This will I<usually> generate JSON texts that also parse as valid
1267 root 1.41 YAML. Please note that YAML has hardcoded limits on (simple) object key
1268 root 1.80 lengths that JSON doesn't have and also has different and incompatible
1269 root 1.124 unicode character escape syntax, so you should make sure that your hash
1270     keys are noticeably shorter than the 1024 "stream characters" YAML allows
1271     and that you do not have characters with codepoint values outside the
1272     Unicode BMP (basic multilingual page). YAML also does not allow C<\/>
1273     sequences in strings (which JSON::XS does not I<currently> generate, but
1274     other JSON generators might).
1275 root 1.39
1276 root 1.83 There might be other incompatibilities that I am not aware of (or the YAML
1277     specification has been changed yet again - it does so quite often). In
1278     general you should not try to generate YAML with a JSON generator or vice
1279     versa, or try to parse JSON with a YAML parser or vice versa: chances are
1280     high that you will run into severe interoperability problems when you
1281     least expect it.
1282 root 1.39
1283 root 1.82 =over 4
1284    
1285     =item (*)
1286    
1287 root 1.90 I have been pressured multiple times by Brian Ingerson (one of the
1288     authors of the YAML specification) to remove this paragraph, despite him
1289     acknowledging that the actual incompatibilities exist. As I was personally
1290     bitten by this "JSON is YAML" lie, I refused and said I will continue to
1291     educate people about these issues, so others do not run into the same
1292     problem again and again. After this, Brian called me a (quote)I<complete
1293     and worthless idiot>(unquote).
1294    
1295     In my opinion, instead of pressuring and insulting people who actually
1296     clarify issues with YAML and the wrong statements of some of its
1297     proponents, I would kindly suggest reading the JSON spec (which is not
1298     that difficult or long) and finally make YAML compatible to it, and
1299     educating users about the changes, instead of spreading lies about the
1300     real compatibility for many I<years> and trying to silence people who
1301     point out that it isn't true.
1302 root 1.82
1303 root 1.135 Addendum/2009: the YAML 1.2 spec is still incompatible with JSON, even
1304     though the incompatibilities have been documented (and are known to Brian)
1305     for many years and the spec makes explicit claims that YAML is a superset
1306     of JSON. It would be so easy to fix, but apparently, bullying people and
1307 root 1.124 corrupting userdata is so much easier.
1308    
1309 root 1.82 =back
1310    
1311 root 1.39
1312 root 1.3 =head2 SPEED
1313    
1314 root 1.4 It seems that JSON::XS is surprisingly fast, as shown in the following
1315     tables. They have been generated with the help of the C<eg/bench> program
1316     in the JSON::XS distribution, to make it easy to compare on your own
1317     system.
1318    
1319 root 1.88 First comes a comparison between various modules using
1320     a very short single-line JSON string (also available at
1321 root 1.89 L<http://dist.schmorp.de/misc/json/short.json>).
1322 root 1.18
1323 root 1.100 {"method": "handleMessage", "params": ["user1",
1324     "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
1325 root 1.129 1, 0]}
1326 root 1.18
1327 root 1.39 It shows the number of encodes/decodes per second (JSON::XS uses
1328     the functional interface, while JSON::XS/2 uses the OO interface
1329     with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
1330 root 1.129 shrink. JSON::DWIW/DS uses the deserialise function, while JSON::DWIW::FJ
1331     uses the from_json method). Higher is better:
1332 root 1.4
1333 root 1.129 module | encode | decode |
1334     --------------|------------|------------|
1335     JSON::DWIW/DS | 86302.551 | 102300.098 |
1336     JSON::DWIW/FJ | 86302.551 | 75983.768 |
1337     JSON::PP | 15827.562 | 6638.658 |
1338     JSON::Syck | 63358.066 | 47662.545 |
1339     JSON::XS | 511500.488 | 511500.488 |
1340     JSON::XS/2 | 291271.111 | 388361.481 |
1341     JSON::XS/3 | 361577.931 | 361577.931 |
1342     Storable | 66788.280 | 265462.278 |
1343     --------------+------------+------------+
1344    
1345     That is, JSON::XS is almost six times faster than JSON::DWIW on encoding,
1346     about five times faster on decoding, and over thirty to seventy times
1347     faster than JSON's pure perl implementation. It also compares favourably
1348     to Storable for small amounts of data.
1349 root 1.4
1350 root 1.13 Using a longer test string (roughly 18KB, generated from Yahoo! Locals
1351 root 1.89 search API (L<http://dist.schmorp.de/misc/json/long.json>).
1352 root 1.4
1353 root 1.129 module | encode | decode |
1354     --------------|------------|------------|
1355     JSON::DWIW/DS | 1647.927 | 2673.916 |
1356     JSON::DWIW/FJ | 1630.249 | 2596.128 |
1357     JSON::PP | 400.640 | 62.311 |
1358     JSON::Syck | 1481.040 | 1524.869 |
1359     JSON::XS | 20661.596 | 9541.183 |
1360     JSON::XS/2 | 10683.403 | 9416.938 |
1361     JSON::XS/3 | 20661.596 | 9400.054 |
1362     Storable | 19765.806 | 10000.725 |
1363     --------------+------------+------------+
1364 root 1.4
1365 root 1.40 Again, JSON::XS leads by far (except for Storable which non-surprisingly
1366 root 1.129 decodes a bit faster).
1367 root 1.4
1368 root 1.68 On large strings containing lots of high Unicode characters, some modules
1369 root 1.18 (such as JSON::PC) seem to decode faster than JSON::XS, but the result
1370 root 1.68 will be broken due to missing (or wrong) Unicode handling. Others refuse
1371 root 1.18 to decode or encode properly, so it was impossible to prepare a fair
1372     comparison table for that case.
1373 root 1.13
1374 root 1.11
1375 root 1.23 =head1 SECURITY CONSIDERATIONS
1376    
1377     When you are using JSON in a protocol, talking to untrusted potentially
1378     hostile creatures requires relatively few measures.
1379    
1380     First of all, your JSON decoder should be secure, that is, should not have
1381     any buffer overflows. Obviously, this module should ensure that and I am
1382     trying hard on making that true, but you never know.
1383    
1384     Second, you need to avoid resource-starving attacks. That means you should
1385     limit the size of JSON texts you accept, or make sure then when your
1386 root 1.68 resources run out, that's just fine (e.g. by using a separate process that
1387 root 1.23 can crash safely). The size of a JSON text in octets or characters is
1388     usually a good indication of the size of the resources required to decode
1389 root 1.47 it into a Perl structure. While JSON::XS can check the size of the JSON
1390     text, it might be too late when you already have it in memory, so you
1391     might want to check the size before you accept the string.
1392 root 1.23
1393     Third, JSON::XS recurses using the C stack when decoding objects and
1394     arrays. The C stack is a limited resource: for instance, on my amd64
1395 root 1.28 machine with 8MB of stack size I can decode around 180k nested arrays but
1396     only 14k nested JSON objects (due to perl itself recursing deeply on croak
1397 root 1.79 to free the temporary). If that is exceeded, the program crashes. To be
1398 root 1.28 conservative, the default nesting limit is set to 512. If your process
1399     has a smaller stack, you should adjust this setting accordingly with the
1400     C<max_depth> method.
1401 root 1.23
1402 root 1.86 Something else could bomb you, too, that I forgot to think of. In that
1403     case, you get to keep the pieces. I am always open for hints, though...
1404    
1405     Also keep in mind that JSON::XS might leak contents of your Perl data
1406     structures in its error messages, so when you serialise sensitive
1407     information you might want to make sure that exceptions thrown by JSON::XS
1408     will not end up in front of untrusted eyes.
1409 root 1.23
1410 root 1.42 If you are using JSON::XS to return packets to consumption
1411 root 1.68 by JavaScript scripts in a browser you should have a look at
1412 root 1.127 L<http://blog.archive.jpsykes.com/47/practical-csrf-and-json-security/> to
1413     see whether you are vulnerable to some common attack vectors (which really
1414     are browser design bugs, but it is still you who will have to deal with
1415     it, as major browser developers care only for features, not about getting
1416     security right).
1417 root 1.42
1418 root 1.11
1419 root 1.144 =head1 INTEROPERABILITY WITH OTHER MODULES
1420    
1421     C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
1422     constants. That means that the JSON true and false values will be
1423     comaptible to true and false values of iother modules that do the same,
1424     such as L<JSON::PP> and L<CBOR::XS>.
1425    
1426    
1427 root 1.64 =head1 THREADS
1428    
1429 root 1.68 This module is I<not> guaranteed to be thread safe and there are no
1430 root 1.64 plans to change this until Perl gets thread support (as opposed to the
1431     horribly slow so-called "threads" which are simply slow and bloated
1432 root 1.91 process simulations - use fork, it's I<much> faster, cheaper, better).
1433 root 1.64
1434 root 1.68 (It might actually work, but you have been warned).
1435 root 1.64
1436    
1437 root 1.139 =head1 THE PERILS OF SETLOCALE
1438    
1439     Sometimes people avoid the Perl locale support and directly call the
1440     system's setlocale function with C<LC_ALL>.
1441    
1442     This breaks both perl and modules such as JSON::XS, as stringification of
1443 root 1.140 numbers no longer works correctly (e.g. C<$x = 0.1; print "$x"+1> might
1444 root 1.139 print C<1>, and JSON::XS might output illegal JSON as JSON::XS relies on
1445     perl to stringify numbers).
1446    
1447     The solution is simple: don't call C<setlocale>, or use it for only those
1448     categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>.
1449    
1450     If you need C<LC_NUMERIC>, you should enable it only around the code that
1451     actually needs it (avoiding stringification of numbers), and restore it
1452     afterwards.
1453    
1454    
1455 root 1.4 =head1 BUGS
1456    
1457     While the goal of this module is to be correct, that unfortunately does
1458 root 1.103 not mean it's bug-free, only that I think its design is bug-free. If you
1459     keep reporting bugs they will be fixed swiftly, though.
1460 root 1.4
1461 root 1.64 Please refrain from using rt.cpan.org or any other bug reporting
1462     service. I put the contact address into my modules for a reason.
1463    
1464 root 1.2 =cut
1465    
1466 root 1.144 BEGIN {
1467     *true = \$Types::Serialiser::true;
1468     *true = \&Types::Serialiser::true;
1469     *false = \$Types::Serialiser::false;
1470     *false = \&Types::Serialiser::false;
1471     *is_bool = \&Types::Serialiser::is_bool;
1472 root 1.43
1473 root 1.144 *JSON::XS::Boolean:: = *Types::Serialiser::Boolean::;
1474 root 1.43 }
1475    
1476     XSLoader::load "JSON::XS", $VERSION;
1477    
1478 root 1.93 =head1 SEE ALSO
1479    
1480     The F<json_xs> command line utility for quick experiments.
1481    
1482 root 1.1 =head1 AUTHOR
1483    
1484     Marc Lehmann <schmorp@schmorp.de>
1485     http://home.schmorp.de/
1486    
1487     =cut
1488    
1489 root 1.144 1
1490