ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.150 by root, Tue Oct 29 00:48:07 2013 UTC vs.
Revision 1.169 by root, Thu Nov 15 20:49:12 2018 UTC

… …
40Beginning with version 2.0 of the JSON module, when both JSON and 40Beginning with version 2.0 of the JSON module, when both JSON and
41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be 41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42overridden) with no overhead due to emulation (by inheriting constructor 42overridden) with no overhead due to emulation (by inheriting constructor
43and methods). If JSON::XS is not available, it will fall back to the 43and methods). If JSON::XS is not available, it will fall back to the
44compatible JSON::PP module as backend, so using JSON instead of JSON::XS 44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45gives you a portable JSON API that can be fast when you need and doesn't 45gives you a portable JSON API that can be fast when you need it and
46require a C compiler when that is a problem. 46doesn't require a C compiler when that is a problem.
47 47
48As this is the n-th-something JSON module on CPAN, what was the reason 48As this is the n-th-something JSON module on CPAN, what was the reason
49to write yet another JSON module? While it seems there are many JSON 49to write yet another JSON module? While it seems there are many JSON
50modules, none of them correctly handle all corner cases, and in most cases 50modules, none of them correctly handle all corner cases, and in most cases
51their maintainers are unresponsive, gone missing, or not listening to bug 51their maintainers are unresponsive, gone missing, or not listening to bug
… …
101 101
102package JSON::XS; 102package JSON::XS;
103 103
104use common::sense; 104use common::sense;
105 105
106our $VERSION = '3.0'; 106our $VERSION = 3.04;
107our @ISA = qw(Exporter); 107our @ISA = qw(Exporter);
108 108
109our @EXPORT = qw(encode_json decode_json); 109our @EXPORT = qw(encode_json decode_json);
110 110
111use Exporter; 111use Exporter;
… …
131 131
132Except being faster. 132Except being faster.
133 133
134=item $perl_scalar = decode_json $json_text 134=item $perl_scalar = decode_json $json_text
135 135
136The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries 136The opposite of C<encode_json>: expects a UTF-8 (binary) string and tries
137to parse that as an UTF-8 encoded JSON text, returning the resulting 137to parse that as a UTF-8 encoded JSON text, returning the resulting
138reference. Croaks on error. 138reference. Croaks on error.
139 139
140This function call is functionally identical to: 140This function call is functionally identical to:
141 141
142 $perl_scalar = JSON::XS->new->utf8->decode ($json_text) 142 $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
… …
202=over 4 202=over 4
203 203
204=item $json = new JSON::XS 204=item $json = new JSON::XS
205 205
206Creates a new JSON::XS object that can be used to de/encode JSON 206Creates a new JSON::XS object that can be used to de/encode JSON
207strings. All boolean flags described below are by default I<disabled>. 207strings. All boolean flags described below are by default I<disabled>
208(with the exception of C<allow_nonref>, which defaults to I<enabled> since
209version C<4.0>).
208 210
209The mutators for flags all return the JSON object again and thus calls can 211The mutators for flags all return the JSON object again and thus calls can
210be chained: 212be chained:
211 213
212 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]}) 214 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]})
… …
270 272
271=item $enabled = $json->get_utf8 273=item $enabled = $json->get_utf8
272 274
273If C<$enable> is true (or missing), then the C<encode> method will encode 275If C<$enable> is true (or missing), then the C<encode> method will encode
274the JSON result into UTF-8, as required by many protocols, while the 276the JSON result into UTF-8, as required by many protocols, while the
275C<decode> method expects to be handled an UTF-8-encoded string. Please 277C<decode> method expects to be handed a UTF-8-encoded string. Please
276note that UTF-8-encoded strings do not contain any characters outside the 278note that UTF-8-encoded strings do not contain any characters outside the
277range C<0..255>, they are thus useful for bytewise/binary I/O. In future 279range C<0..255>, they are thus useful for bytewise/binary I/O. In future
278versions, enabling this option might enable autodetection of the UTF-16 280versions, enabling this option might enable autodetection of the UTF-16
279and UTF-32 encoding families, as described in RFC4627. 281and UTF-32 encoding families, as described in RFC4627.
280 282
… …
365 367
366=item $enabled = $json->get_relaxed 368=item $enabled = $json->get_relaxed
367 369
368If C<$enable> is true (or missing), then C<decode> will accept some 370If C<$enable> is true (or missing), then C<decode> will accept some
369extensions to normal JSON syntax (see below). C<encode> will not be 371extensions to normal JSON syntax (see below). C<encode> will not be
370affected in anyway. I<Be aware that this option makes you accept invalid 372affected in any way. I<Be aware that this option makes you accept invalid
371JSON texts as if they were valid!>. I suggest only to use this option to 373JSON texts as if they were valid!>. I suggest only to use this option to
372parse application-specific files written by humans (configuration files, 374parse application-specific files written by humans (configuration files,
373resource files etc.) 375resource files etc.)
374 376
375If C<$enable> is false (the default), then C<decode> will only accept 377If C<$enable> is false (the default), then C<decode> will only accept
… …
404 [ 406 [
405 1, # this comment not allowed in JSON 407 1, # this comment not allowed in JSON
406 # neither this one... 408 # neither this one...
407 ] 409 ]
408 410
411=item * literal ASCII TAB characters in strings
412
413Literal ASCII TAB characters are now allowed in strings (and treated as
414C<\t>).
415
416 [
417 "Hello\tWorld",
418 "Hello<TAB>World", # literal <TAB> would not normally be allowed
419 ]
420
409=back 421=back
410 422
411=item $json = $json->canonical ([$enable]) 423=item $json = $json->canonical ([$enable])
412 424
413=item $enabled = $json->get_canonical 425=item $enabled = $json->get_canonical
… …
431 443
432=item $json = $json->allow_nonref ([$enable]) 444=item $json = $json->allow_nonref ([$enable])
433 445
434=item $enabled = $json->get_allow_nonref 446=item $enabled = $json->get_allow_nonref
435 447
448Unlike other boolean options, this opotion is enabled by default beginning
449with version C<4.0>. See L<SECURITY CONSIDERATIONS> for the gory details.
450
436If C<$enable> is true (or missing), then the C<encode> method can convert a 451If C<$enable> is true (or missing), then the C<encode> method can convert a
437non-reference into its corresponding string, number or null JSON value, 452non-reference into its corresponding string, number or null JSON value,
438which is an extension to RFC4627. Likewise, C<decode> will accept those JSON 453which is an extension to RFC4627. Likewise, C<decode> will accept those JSON
439values instead of croaking. 454values instead of croaking.
440 455
441If C<$enable> is false, then the C<encode> method will croak if it isn't 456If C<$enable> is false, then the C<encode> method will croak if it isn't
442passed an arrayref or hashref, as JSON texts must either be an object 457passed an arrayref or hashref, as JSON texts must either be an object
443or array. Likewise, C<decode> will croak if given something that is not a 458or array. Likewise, C<decode> will croak if given something that is not a
444JSON object or array. 459JSON object or array.
445 460
446Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>, 461Example, encode a Perl scalar as JSON value without enabled C<allow_nonref>,
447resulting in an invalid JSON text: 462resulting in an error:
448 463
449 JSON::XS->new->allow_nonref->encode ("Hello, World!") 464 JSON::XS->new->allow_nonref (0)->encode ("Hello, World!")
450 => "Hello, World!" 465 => hash- or arrayref expected...
451 466
452=item $json = $json->allow_unknown ([$enable]) 467=item $json = $json->allow_unknown ([$enable])
453 468
454=item $enabled = $json->get_allow_unknown 469=item $enabled = $json->get_allow_unknown
455 470
… …
505 520
506This setting has no effect on C<decode>. 521This setting has no effect on C<decode>.
507 522
508=item $json = $json->allow_tags ([$enable]) 523=item $json = $json->allow_tags ([$enable])
509 524
510=item $enabled = $json->allow_tags 525=item $enabled = $json->get_allow_tags
511 526
512See L<OBJECT SERIALISATION> for details. 527See L<OBJECT SERIALISATION> for details.
513 528
514If C<$enable> is true (or missing), then C<encode>, upon encountering a 529If C<$enable> is true (or missing), then C<encode>, upon encountering a
515blessed object, will check for the availability of the C<FREEZE> method on 530blessed object, will check for the availability of the C<FREEZE> method on
… …
524in C<decode>, as if tags were not part of the grammar. 539in C<decode>, as if tags were not part of the grammar.
525 540
526=item $json = $json->filter_json_object ([$coderef->($hashref)]) 541=item $json = $json->filter_json_object ([$coderef->($hashref)])
527 542
528When C<$coderef> is specified, it will be called from C<decode> each 543When C<$coderef> is specified, it will be called from C<decode> each
529time it decodes a JSON object. The only argument is a reference to the 544time it decodes a JSON object. The only argument is a reference to
530newly-created hash. If the code references returns a single scalar (which 545the newly-created hash. If the code reference returns a single scalar
531need not be a reference), this value (i.e. a copy of that scalar to avoid 546(which need not be a reference), this value (or rather a copy of it) is
532aliasing) is inserted into the deserialised data structure. If it returns 547inserted into the deserialised data structure. If it returns an empty
533an empty list (NOTE: I<not> C<undef>, which is a valid scalar), the 548list (NOTE: I<not> C<undef>, which is a valid scalar), the original
534original deserialised hash will be inserted. This setting can slow down 549deserialised hash will be inserted. This setting can slow down decoding
535decoding considerably. 550considerably.
536 551
537When C<$coderef> is omitted or undefined, any existing callback will 552When C<$coderef> is omitted or undefined, any existing callback will
538be removed and C<decode> will not change the deserialised hash in any 553be removed and C<decode> will not change the deserialised hash in any
539way. 554way.
540 555
… …
687 702
688This is useful if your JSON texts are not delimited by an outer protocol 703This is useful if your JSON texts are not delimited by an outer protocol
689and you need to know where the JSON text ends. 704and you need to know where the JSON text ends.
690 705
691 JSON::XS->new->decode_prefix ("[1] the tail") 706 JSON::XS->new->decode_prefix ("[1] the tail")
692 => ([], 3) 707 => ([1], 3)
693 708
694=back 709=back
695 710
696 711
697=head1 INCREMENTAL PARSING 712=head1 INCREMENTAL PARSING
… …
738C<incr_skip> to skip the erroneous part). This is the most common way of 753C<incr_skip> to skip the erroneous part). This is the most common way of
739using the method. 754using the method.
740 755
741And finally, in list context, it will try to extract as many objects 756And finally, in list context, it will try to extract as many objects
742from the stream as it can find and return them, or the empty list 757from the stream as it can find and return them, or the empty list
743otherwise. For this to work, there must be no separators between the JSON 758otherwise. For this to work, there must be no separators (other than
744objects or arrays, instead they must be concatenated back-to-back. If 759whitespace) between the JSON objects or arrays, instead they must be
745an error occurs, an exception will be raised as in the scalar context 760concatenated back-to-back. If an error occurs, an exception will be
746case. Note that in this case, any previously-parsed JSON texts will be 761raised as in the scalar context case. Note that in this case, any
747lost. 762previously-parsed JSON texts will be lost.
748 763
749Example: Parse some JSON arrays/objects in a given string and return 764Example: Parse some JSON arrays/objects in a given string and return
750them. 765them.
751 766
752 my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]"); 767 my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]");
… …
758C<incr_parse> in I<scalar context> successfully returned an object. Under 773C<incr_parse> in I<scalar context> successfully returned an object. Under
759all other circumstances you must not call this function (I mean it. 774all other circumstances you must not call this function (I mean it.
760although in simple tests it might actually work, it I<will> fail under 775although in simple tests it might actually work, it I<will> fail under
761real world conditions). As a special exception, you can also call this 776real world conditions). As a special exception, you can also call this
762method before having parsed anything. 777method before having parsed anything.
778
779That means you can only use this function to look at or manipulate text
780before or after complete JSON objects, not while the parser is in the
781middle of parsing a JSON object.
763 782
764This function is useful in two cases: a) finding the trailing text after a 783This function is useful in two cases: a) finding the trailing text after a
765JSON object or b) parsing multiple JSON objects separated by non-JSON text 784JSON object or b) parsing multiple JSON objects separated by non-JSON text
766(such as commas). 785(such as commas).
767 786
… …
1275expect your input strings to be encoded as UTF-8, that is, no "character" 1294expect your input strings to be encoded as UTF-8, that is, no "character"
1276of the input string must have any value > 255, as UTF-8 does not allow 1295of the input string must have any value > 255, as UTF-8 does not allow
1277that. 1296that.
1278 1297
1279The C<utf8> flag therefore switches between two modes: disabled means you 1298The C<utf8> flag therefore switches between two modes: disabled means you
1280will get a Unicode string in Perl, enabled means you get an UTF-8 encoded 1299will get a Unicode string in Perl, enabled means you get a UTF-8 encoded
1281octet/binary string in Perl. 1300octet/binary string in Perl.
1282 1301
1283=item C<latin1> or C<ascii> flags enabled 1302=item C<latin1> or C<ascii> flags enabled
1284 1303
1285With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters 1304With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters
… …
1553are browser design bugs, but it is still you who will have to deal with 1572are browser design bugs, but it is still you who will have to deal with
1554it, as major browser developers care only for features, not about getting 1573it, as major browser developers care only for features, not about getting
1555security right). 1574security right).
1556 1575
1557 1576
1577=head2 "OLD" VS. "NEW" JSON (RFC 4627 VS. RFC 7159)
1578
1579JSON originally required JSON texts to represent an array or object -
1580scalar values were explicitly not allowed. This has changed, and versions
1581of JSON::XS beginning with C<4.0> reflect this by allowing scalar values
1582by default.
1583
1584One reason why one might not want this is that this removes a fundamental
1585property of JSON texts, namely that they are self-delimited and
1586self-contained, or in other words, you could take any number of "old"
1587JSON texts and paste them together, and the result would be unambiguously
1588parseable:
1589
1590 [1,3]{"k":5}[][null] # four JSON texts, without doubt
1591
1592By allowing scalars, this property is lost: in the following example, is
1593this one JSON text (the number 12) or two JSON texts (the numbers 1 and
15942):
1595
1596 12 # could be 12, or 1 and 2
1597
1598Another lost property of "old" JSON is that no lookahead is required to
1599know the end of a JSON text, i.e. the JSON text definitely ended at the
1600last C<]> or C<}> character, there was no need to read extra characters.
1601
1602For example, a viable network protocol with "old" JSON was to simply
1603exchange JSON texts without delimiter. For "new" JSON, you have to use a
1604suitable delimiter (such as a newline) after every JSON text or ensure you
1605never encode/decode scalar values.
1606
1607Most protocols do work by only transferring arrays or objects, and the
1608easiest way to avoid problems with the "new" JSON definition is to
1609explicitly disallow scalar values in your encoder and decoder:
1610
1611 $json_coder = JSON::XS->new->allow_nonref (0)
1612
1613This is a somewhat unhappy situation, and the blame can fully be put on
1614JSON's inmventor, Douglas Crockford, who unilaterally changed the format
1615in 2006 without consulting the IETF, forcing the IETF to either fork the
1616format or go with it (as I was told, the IETF wasn't amused).
1617
1618
1558=head1 INTEROPERABILITY WITH OTHER MODULES 1619=head1 INTEROPERABILITY WITH OTHER MODULES
1559 1620
1560C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean 1621C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
1561constants. That means that the JSON true and false values will be 1622constants. That means that the JSON true and false values will be
1562comaptible to true and false values of iother modules that do the same, 1623comaptible to true and false values of other modules that do the same,
1563such as L<JSON::PP> and L<CBOR::XS>. 1624such as L<JSON::PP> and L<CBOR::XS>.
1564 1625
1565 1626
1627=head1 INTEROPERABILITY WITH OTHER JSON DECODERS
1628
1629As long as you only serialise data that can be directly expressed in JSON,
1630C<JSON::XS> is incapable of generating invalid JSON output (modulo bugs,
1631but C<JSON::XS> has found more bugs in the official JSON testsuite (1)
1632than the official JSON testsuite has found in C<JSON::XS> (0)).
1633
1634When you have trouble decoding JSON generated by this module using other
1635decoders, then it is very likely that you have an encoding mismatch or the
1636other decoder is broken.
1637
1638When decoding, C<JSON::XS> is strict by default and will likely catch all
1639errors. There are currently two settings that change this: C<relaxed>
1640makes C<JSON::XS> accept (but not generate) some non-standard extensions,
1641and C<allow_tags> will allow you to encode and decode Perl objects, at the
1642cost of not outputting valid JSON anymore.
1643
1644=head2 TAGGED VALUE SYNTAX AND STANDARD JSON EN/DECODERS
1645
1646When you use C<allow_tags> to use the extended (and also nonstandard and
1647invalid) JSON syntax for serialised objects, and you still want to decode
1648the generated When you want to serialise objects, you can run a regex
1649to replace the tagged syntax by standard JSON arrays (it only works for
1650"normal" package names without comma, newlines or single colons). First,
1651the readable Perl version:
1652
1653 # if your FREEZE methods return no values, you need this replace first:
1654 $json =~ s/\( \s* (" (?: [^\\":,]+|\\.|::)* ") \s* \) \s* \[\s*\]/[$1]/gx;
1655
1656 # this works for non-empty constructor arg lists:
1657 $json =~ s/\( \s* (" (?: [^\\":,]+|\\.|::)* ") \s* \) \s* \[/[$1,/gx;
1658
1659And here is a less readable version that is easy to adapt to other
1660languages:
1661
1662 $json =~ s/\(\s*("([^\\":,]+|\\.|::)*")\s*\)\s*\[/[$1,/g;
1663
1664Here is an ECMAScript version (same regex):
1665
1666 json = json.replace (/\(\s*("([^\\":,]+|\\.|::)*")\s*\)\s*\[/g, "[$1,");
1667
1668Since this syntax converts to standard JSON arrays, it might be hard to
1669distinguish serialised objects from normal arrays. You can prepend a
1670"magic number" as first array element to reduce chances of a collision:
1671
1672 $json =~ s/\(\s*("([^\\":,]+|\\.|::)*")\s*\)\s*\[/["XU1peReLzT4ggEllLanBYq4G9VzliwKF",$1,/g;
1673
1674And after decoding the JSON text, you could walk the data
1675structure looking for arrays with a first element of
1676C<XU1peReLzT4ggEllLanBYq4G9VzliwKF>.
1677
1678The same approach can be used to create the tagged format with another
1679encoder. First, you create an array with the magic string as first member,
1680the classname as second, and constructor arguments last, encode it as part
1681of your JSON structure, and then:
1682
1683 $json =~ s/\[\s*"XU1peReLzT4ggEllLanBYq4G9VzliwKF"\s*,\s*("([^\\":,]+|\\.|::)*")\s*,/($1)[/g;
1684
1685Again, this has some limitations - the magic string must not be encoded
1686with character escapes, and the constructor arguments must be non-empty.
1687
1688
1689=head1 RFC7159
1690
1691Since this module was written, Google has written a new JSON RFC, RFC 7159
1692(and RFC7158). Unfortunately, this RFC breaks compatibility with both the
1693original JSON specification on www.json.org and RFC4627.
1694
1695As far as I can see, you can get partial compatibility when parsing by
1696using C<< ->allow_nonref >>. However, consider the security implications
1697of doing so.
1698
1699I haven't decided yet when to break compatibility with RFC4627 by default
1700(and potentially leave applications insecure) and change the default to
1701follow RFC7159, but application authors are well advised to call C<<
1702->allow_nonref(0) >> even if this is the current default, if they cannot
1703handle non-reference values, in preparation for the day when the default
1704will change.
1705
1706
1566=head1 THREADS 1707=head1 (I-)THREADS
1567 1708
1568This module is I<not> guaranteed to be thread safe and there are no 1709This module is I<not> guaranteed to be ithread (or MULTIPLICITY-) safe
1569plans to change this until Perl gets thread support (as opposed to the 1710and there are no plans to change this. Note that perl's builtin so-called
1570horribly slow so-called "threads" which are simply slow and bloated 1711threads/ithreads are officially deprecated and should not be used.
1571process simulations - use fork, it's I<much> faster, cheaper, better).
1572
1573(It might actually work, but you have been warned).
1574 1712
1575 1713
1576=head1 THE PERILS OF SETLOCALE 1714=head1 THE PERILS OF SETLOCALE
1577 1715
1578Sometimes people avoid the Perl locale support and directly call the 1716Sometimes people avoid the Perl locale support and directly call the

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines