ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.168 by root, Thu Nov 15 20:13:03 2018 UTC vs.
Revision 1.175 by root, Wed Mar 6 07:32:06 2019 UTC

35 35
36This module converts Perl data structures to JSON and vice versa. Its 36This module converts Perl data structures to JSON and vice versa. Its
37primary goal is to be I<correct> and its secondary goal is to be 37primary goal is to be I<correct> and its secondary goal is to be
38I<fast>. To reach the latter goal it was written in C. 38I<fast>. To reach the latter goal it was written in C.
39 39
40Beginning with version 2.0 of the JSON module, when both JSON and
41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42overridden) with no overhead due to emulation (by inheriting constructor
43and methods). If JSON::XS is not available, it will fall back to the
44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45gives you a portable JSON API that can be fast when you need it and
46doesn't require a C compiler when that is a problem.
47
48As this is the n-th-something JSON module on CPAN, what was the reason
49to write yet another JSON module? While it seems there are many JSON
50modules, none of them correctly handle all corner cases, and in most cases
51their maintainers are unresponsive, gone missing, or not listening to bug
52reports for other reasons.
53
54See MAPPING, below, on how JSON::XS maps perl values to JSON values and 40See MAPPING, below, on how JSON::XS maps perl values to JSON values and
55vice versa. 41vice versa.
56 42
57=head2 FEATURES 43=head2 FEATURES
58 44
59=over 4 45=over
60 46
61=item * correct Unicode handling 47=item * correct Unicode handling
62 48
63This module knows how to handle Unicode, documents how and when it does 49This module knows how to handle Unicode, documents how and when it does
64so, and even documents what "correct" means. 50so, and even documents what "correct" means.
101 87
102package JSON::XS; 88package JSON::XS;
103 89
104use common::sense; 90use common::sense;
105 91
106our $VERSION = 3.04; 92our $VERSION = '4.02';
107our @ISA = qw(Exporter); 93our @ISA = qw(Exporter);
108 94
109our @EXPORT = qw(encode_json decode_json); 95our @EXPORT = qw(encode_json decode_json);
110 96
111use Exporter; 97use Exporter;
116=head1 FUNCTIONAL INTERFACE 102=head1 FUNCTIONAL INTERFACE
117 103
118The following convenience methods are provided by this module. They are 104The following convenience methods are provided by this module. They are
119exported by default: 105exported by default:
120 106
121=over 4 107=over
122 108
123=item $json_text = encode_json $perl_scalar 109=item $json_text = encode_json $perl_scalar
124 110
125Converts the given Perl data structure to a UTF-8 encoded, binary string 111Converts the given Perl data structure to a UTF-8 encoded, binary string
126(that is, the string contains octets only). Croaks on error. 112(that is, the string contains octets only). Croaks on error.
149=head1 A FEW NOTES ON UNICODE AND PERL 135=head1 A FEW NOTES ON UNICODE AND PERL
150 136
151Since this often leads to confusion, here are a few very clear words on 137Since this often leads to confusion, here are a few very clear words on
152how Unicode works in Perl, modulo bugs. 138how Unicode works in Perl, modulo bugs.
153 139
154=over 4 140=over
155 141
156=item 1. Perl strings can store characters with ordinal values > 255. 142=item 1. Perl strings can store characters with ordinal values > 255.
157 143
158This enables you to store Unicode characters as single characters in a 144This enables you to store Unicode characters as single characters in a
159Perl string - very natural. 145Perl string - very natural.
197=head1 OBJECT-ORIENTED INTERFACE 183=head1 OBJECT-ORIENTED INTERFACE
198 184
199The object oriented interface lets you configure your own encoding or 185The object oriented interface lets you configure your own encoding or
200decoding style, within the limits of supported formats. 186decoding style, within the limits of supported formats.
201 187
202=over 4 188=over
203 189
204=item $json = new JSON::XS 190=item $json = new JSON::XS
205 191
206Creates a new JSON::XS object that can be used to de/encode JSON 192Creates a new JSON::XS object that can be used to de/encode JSON
207strings. All boolean flags described below are by default I<disabled>. 193strings. All boolean flags described below are by default I<disabled>
194(with the exception of C<allow_nonref>, which defaults to I<enabled> since
195version C<4.0>).
208 196
209The mutators for flags all return the JSON object again and thus calls can 197The mutators for flags all return the JSON object again and thus calls can
210be chained: 198be chained:
211 199
212 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]}) 200 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]})
375If C<$enable> is false (the default), then C<decode> will only accept 363If C<$enable> is false (the default), then C<decode> will only accept
376valid JSON texts. 364valid JSON texts.
377 365
378Currently accepted extensions are: 366Currently accepted extensions are:
379 367
380=over 4 368=over
381 369
382=item * list items can have an end-comma 370=item * list items can have an end-comma
383 371
384JSON I<separates> array elements and key-value pairs with commas. This 372JSON I<separates> array elements and key-value pairs with commas. This
385can be annoying if you write JSON texts manually and want to be able to 373can be annoying if you write JSON texts manually and want to be able to
441 429
442=item $json = $json->allow_nonref ([$enable]) 430=item $json = $json->allow_nonref ([$enable])
443 431
444=item $enabled = $json->get_allow_nonref 432=item $enabled = $json->get_allow_nonref
445 433
434Unlike other boolean options, this opotion is enabled by default beginning
435with version C<4.0>. See L<SECURITY CONSIDERATIONS> for the gory details.
436
446If C<$enable> is true (or missing), then the C<encode> method can convert a 437If C<$enable> is true (or missing), then the C<encode> method can convert a
447non-reference into its corresponding string, number or null JSON value, 438non-reference into its corresponding string, number or null JSON value,
448which is an extension to RFC4627. Likewise, C<decode> will accept those JSON 439which is an extension to RFC4627. Likewise, C<decode> will accept those JSON
449values instead of croaking. 440values instead of croaking.
450 441
451If C<$enable> is false, then the C<encode> method will croak if it isn't 442If C<$enable> is false, then the C<encode> method will croak if it isn't
452passed an arrayref or hashref, as JSON texts must either be an object 443passed an arrayref or hashref, as JSON texts must either be an object
453or array. Likewise, C<decode> will croak if given something that is not a 444or array. Likewise, C<decode> will croak if given something that is not a
454JSON object or array. 445JSON object or array.
455 446
456Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>, 447Example, encode a Perl scalar as JSON value without enabled C<allow_nonref>,
457resulting in an invalid JSON text: 448resulting in an error:
458 449
459 JSON::XS->new->allow_nonref->encode ("Hello, World!") 450 JSON::XS->new->allow_nonref (0)->encode ("Hello, World!")
460 => "Hello, World!" 451 => hash- or arrayref expected...
461 452
462=item $json = $json->allow_unknown ([$enable]) 453=item $json = $json->allow_unknown ([$enable])
463 454
464=item $enabled = $json->get_allow_unknown 455=item $enabled = $json->get_allow_unknown
465 456
530them via a call to the C<THAW> method. 521them via a call to the C<THAW> method.
531 522
532If C<$enable> is false (the default), then C<encode> will not consider 523If C<$enable> is false (the default), then C<encode> will not consider
533this type of conversion, and tagged JSON values will cause a parse error 524this type of conversion, and tagged JSON values will cause a parse error
534in C<decode>, as if tags were not part of the grammar. 525in C<decode>, as if tags were not part of the grammar.
526
527=item $json->boolean_values ([$false, $true])
528
529=item ($false, $true) = $json->get_boolean_values
530
531By default, JSON booleans will be decoded as overloaded
532C<$Types::Serialiser::false> and C<$Types::Serialiser::true> objects.
533
534With this method you can specify your own boolean values for decoding -
535on decode, JSON C<false> will be decoded as a copy of C<$false>, and JSON
536C<true> will be decoded as C<$true> ("copy" here is the same thing as
537assigning a value to another variable, i.e. C<$copy = $false>).
538
539Calling this method without any arguments will reset the booleans
540to their default values.
541
542C<get_boolean_values> will return both C<$false> and C<$true> values, or
543the empty list when they are set to the default.
535 544
536=item $json = $json->filter_json_object ([$coderef->($hashref)]) 545=item $json = $json->filter_json_object ([$coderef->($hashref)])
537 546
538When C<$coderef> is specified, it will be called from C<decode> each 547When C<$coderef> is specified, it will be called from C<decode> each
539time it decodes a JSON object. The only argument is a reference to 548time it decodes a JSON object. The only argument is a reference to
724to set resource limits (e.g. C<max_size>) to ensure the parser will stop 733to set resource limits (e.g. C<max_size>) to ensure the parser will stop
725parsing in the presence if syntax errors. 734parsing in the presence if syntax errors.
726 735
727The following methods implement this incremental parser. 736The following methods implement this incremental parser.
728 737
729=over 4 738=over
730 739
731=item [void, scalar or list context] = $json->incr_parse ([$string]) 740=item [void, scalar or list context] = $json->incr_parse ([$string])
732 741
733This is the central parsing function. It can both append new text and 742This is the central parsing function. It can both append new text and
734extract objects from the stream accumulated so far (both of these 743extract objects from the stream accumulated so far (both of these
801 810
802=back 811=back
803 812
804=head2 LIMITATIONS 813=head2 LIMITATIONS
805 814
806All options that affect decoding are supported, except 815The incremental parser is a non-exact parser: it works by gathering as
807C<allow_nonref>. The reason for this is that it cannot be made to work 816much text as possible that I<could> be a valid JSON text, followed by
808sensibly: JSON objects and arrays are self-delimited, i.e. you can 817trying to decode it.
809concatenate them back to back and still decode them perfectly. This does
810not hold true for JSON numbers, however.
811 818
812For example, is the string C<1> a single JSON number, or is it simply the 819That means it sometimes needs to read more data than strictly necessary to
813start of C<12>? Or is C<12> a single JSON number, or the concatenation 820diagnose an invalid JSON text. For example, after parsing the following
814of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS 821fragment, the parser I<could> stop with an error, as this fragment
815takes the conservative route and disallows this case. 822I<cannot> be the beginning of a valid JSON text:
823
824 [,
825
826In reality, hopwever, the parser might continue to read data until a
827length limit is exceeded or it finds a closing bracket.
816 828
817=head2 EXAMPLES 829=head2 EXAMPLES
818 830
819Some examples will make all this clearer. First, a simple example that 831Some examples will make all this clearer. First, a simple example that
820works similarly to C<decode_prefix>: We want to decode the JSON object at 832works similarly to C<decode_prefix>: We want to decode the JSON object at
964refers to the abstract Perl language itself. 976refers to the abstract Perl language itself.
965 977
966 978
967=head2 JSON -> PERL 979=head2 JSON -> PERL
968 980
969=over 4 981=over
970 982
971=item object 983=item object
972 984
973A JSON object becomes a reference to a hash in Perl. No ordering of object 985A JSON object becomes a reference to a hash in Perl. No ordering of object
974keys is preserved (JSON does not preserve object key ordering itself). 986keys is preserved (JSON does not preserve object key ordering itself).
1042 1054
1043The mapping from Perl to JSON is slightly more difficult, as Perl is a 1055The mapping from Perl to JSON is slightly more difficult, as Perl is a
1044truly typeless language, so we can only guess which JSON type is meant by 1056truly typeless language, so we can only guess which JSON type is meant by
1045a Perl value. 1057a Perl value.
1046 1058
1047=over 4 1059=over
1048 1060
1049=item hash references 1061=item hash references
1050 1062
1051Perl hash references become JSON objects. As there is no inherent 1063Perl hash references become JSON objects. As there is no inherent
1052ordering in hash keys (or JSON objects), they will usually be encoded 1064ordering in hash keys (or JSON objects), they will usually be encoded
1141 1153
1142What happens when C<JSON::XS> encounters a Perl object depends on the 1154What happens when C<JSON::XS> encounters a Perl object depends on the
1143C<allow_blessed>, C<convert_blessed> and C<allow_tags> settings, which are 1155C<allow_blessed>, C<convert_blessed> and C<allow_tags> settings, which are
1144used in this order: 1156used in this order:
1145 1157
1146=over 4 1158=over
1147 1159
1148=item 1. C<allow_tags> is enabled and the object has a C<FREEZE> method. 1160=item 1. C<allow_tags> is enabled and the object has a C<FREEZE> method.
1149 1161
1150In this case, C<JSON::XS> uses the L<Types::Serialiser> object 1162In this case, C<JSON::XS> uses the L<Types::Serialiser> object
1151serialisation protocol to create a tagged JSON value, using a nonstandard 1163serialisation protocol to create a tagged JSON value, using a nonstandard
1262takes those codepoint numbers and I<encodes> them, in our case into 1274takes those codepoint numbers and I<encodes> them, in our case into
1263octets. Unicode is (among other things) a codeset, UTF-8 is an encoding, 1275octets. Unicode is (among other things) a codeset, UTF-8 is an encoding,
1264and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at 1276and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at
1265the same time, which can be confusing. 1277the same time, which can be confusing.
1266 1278
1267=over 4 1279=over
1268 1280
1269=item C<utf8> flag disabled 1281=item C<utf8> flag disabled
1270 1282
1271When C<utf8> is disabled (the default), then C<encode>/C<decode> generate 1283When C<utf8> is disabled (the default), then C<encode>/C<decode> generate
1272and expect Unicode strings, that is, characters with high ordinal Unicode 1284and expect Unicode strings, that is, characters with high ordinal Unicode
1431general you should not try to generate YAML with a JSON generator or vice 1443general you should not try to generate YAML with a JSON generator or vice
1432versa, or try to parse JSON with a YAML parser or vice versa: chances are 1444versa, or try to parse JSON with a YAML parser or vice versa: chances are
1433high that you will run into severe interoperability problems when you 1445high that you will run into severe interoperability problems when you
1434least expect it. 1446least expect it.
1435 1447
1436=over 4 1448=over
1437 1449
1438=item (*) 1450=item (*)
1439 1451
1440I have been pressured multiple times by Brian Ingerson (one of the 1452I have been pressured multiple times by Brian Ingerson (one of the
1441authors of the YAML specification) to remove this paragraph, despite him 1453authors of the YAML specification) to remove this paragraph, despite him
1567are browser design bugs, but it is still you who will have to deal with 1579are browser design bugs, but it is still you who will have to deal with
1568it, as major browser developers care only for features, not about getting 1580it, as major browser developers care only for features, not about getting
1569security right). 1581security right).
1570 1582
1571 1583
1572=head1 "OLD" VS. "NEW" JSON (RFC 4627 VS. RFC 7159) 1584=head2 "OLD" VS. "NEW" JSON (RFC4627 VS. RFC7159)
1573 1585
1574TL;DR: Due to security concerns, JSON::XS will not allow scalar data in 1586JSON originally required JSON texts to represent an array or object -
1575JSON texts by default - you need to create your own JSON::XS object and 1587scalar values were explicitly not allowed. This has changed, and versions
1576enable C<allow_nonref>: 1588of JSON::XS beginning with C<4.0> reflect this by allowing scalar values
1589by default.
1577 1590
1591One reason why one might not want this is that this removes a fundamental
1592property of JSON texts, namely that they are self-delimited and
1593self-contained, or in other words, you could take any number of "old"
1594JSON texts and paste them together, and the result would be unambiguously
1595parseable:
1578 1596
1597 [1,3]{"k":5}[][null] # four JSON texts, without doubt
1598
1599By allowing scalars, this property is lost: in the following example, is
1600this one JSON text (the number 12) or two JSON texts (the numbers 1 and
16012):
1602
1603 12 # could be 12, or 1 and 2
1604
1605Another lost property of "old" JSON is that no lookahead is required to
1606know the end of a JSON text, i.e. the JSON text definitely ended at the
1607last C<]> or C<}> character, there was no need to read extra characters.
1608
1609For example, a viable network protocol with "old" JSON was to simply
1610exchange JSON texts without delimiter. For "new" JSON, you have to use a
1611suitable delimiter (such as a newline) after every JSON text or ensure you
1612never encode/decode scalar values.
1613
1614Most protocols do work by only transferring arrays or objects, and the
1615easiest way to avoid problems with the "new" JSON definition is to
1616explicitly disallow scalar values in your encoder and decoder:
1617
1579 my $json = JSON::XS->new->allow_nonref; 1618 $json_coder = JSON::XS->new->allow_nonref (0)
1580 1619
1581 $text = $json->encode ($data); 1620This is a somewhat unhappy situation, and the blame can fully be put on
1582 $data = $json->decode ($text); 1621JSON's inmventor, Douglas Crockford, who unilaterally changed the format
1622in 2006 without consulting the IETF, forcing the IETF to either fork the
1623format or go with it (as I was told, the IETF wasn't amused).
1583 1624
1584The long version: JSON being an important and supposedly stable format,
1585the IETF standardised it as RFC 4627 in 2006. Unfortunately, the inventor
1586of JSON, Dougles Crockford, unilaterally changed the definition of JSON in
1587javascript. Rather than create a fork, the IETF decided to standardise the
1588new syntax (apparently, so Iw as told, without finding it very amusing).
1589 1625
1590The biggest difference between thed original JSON and the new JSON is that 1626=head1 RELATIONSHIP WITH I-JSON
1591the new JSON supports scalars (anything other than arrays and objects) at
1592the toplevel of a JSON text. While this is strictly backwards compatible
1593to older versions, it breaks a number of protocols that relied on sending
1594JSON back-to-back, and is a minor security concern.
1595 1627
1596For example, imagine you have two banks communicating, and on one side, 1628JSON is a somewhat sloppily-defined format - it carries around obvious
1597trhe JSON coder gets upgraded. Two messages, such as C<10> and C<1000> 1629Javascript baggage, such as not really defining number range, probably
1598might then be confused to mean C<101000>, something that couldn't happen 1630because Javascript only has one type of numbers: IEEE 64 bit floats
1599in the original JSON, because niether of these messages would be valid 1631("binary64").
1600JSON.
1601 1632
1602If one side accepts these messages, then an upgrade in the coder on either 1633For this reaosn, RFC7493 defines "Internet JSON", which is a restricted
1603side could result in this becoming exploitable. 1634subset of JSON that is supposedly more interoperable on the internet.
1604 1635
1605This module has always allowed these messages as an optional extension, by 1636While C<JSON::XS> does not offer specific support for I-JSON, it of course
1606default disabled. The security concerns are the reason why the default is 1637accepts valid I-JSON and by default implements some of the limitations
1607still disabled, but future versions might/will likely upgrade to the newer 1638of I-JSON, such as parsing numbers as perl numbers, which are usually a
1608RFC as default format, so you are advised to check your implementation 1639superset of binary64 numbers.
1609and/or override the default with C<< ->allow_nonref (0) >> to ensure that 1640
1610future versions are safe. 1641To generate I-JSON, follow these rules:
1642
1643=over
1644
1645=item * always generate UTF-8
1646
1647I-JSON must be encoded in UTF-8, the default for C<encode_json>.
1648
1649=item * numbers should be within IEEE 754 binary64 range
1650
1651Basically all existing perl installations use binary64 to represent
1652floating point numbers, so all you need to do is to avoid large integers.
1653
1654=item * objects must not have duplicate keys
1655
1656This is trivially done, as C<JSON::XS> does not allow duplicate keys.
1657
1658=item * do not generate scalar JSON texts, use C<< ->allow_nonref (0) >>
1659
1660I-JSON strongly requests you to only encode arrays and objects into JSON.
1661
1662=item * times should be strings in ISO 8601 format
1663
1664There are a myriad of modules on CPAN dealing with ISO 8601 - search for
1665C<ISO8601> on CPAN and use one.
1666
1667=item * encode binary data as base64
1668
1669While it's tempting to just dump binary data as a string (and let
1670C<JSON::XS> do the escaping), for I-JSON, it's I<recommended> to encode
1671binary data as base64.
1672
1673=back
1674
1675There are some other considerations - read RFC7493 for the details if
1676interested.
1611 1677
1612 1678
1613=head1 INTEROPERABILITY WITH OTHER MODULES 1679=head1 INTEROPERABILITY WITH OTHER MODULES
1614 1680
1615C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean 1681C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
1678 1744
1679Again, this has some limitations - the magic string must not be encoded 1745Again, this has some limitations - the magic string must not be encoded
1680with character escapes, and the constructor arguments must be non-empty. 1746with character escapes, and the constructor arguments must be non-empty.
1681 1747
1682 1748
1683=head1 RFC7159
1684
1685Since this module was written, Google has written a new JSON RFC, RFC 7159
1686(and RFC7158). Unfortunately, this RFC breaks compatibility with both the
1687original JSON specification on www.json.org and RFC4627.
1688
1689As far as I can see, you can get partial compatibility when parsing by
1690using C<< ->allow_nonref >>. However, consider the security implications
1691of doing so.
1692
1693I haven't decided yet when to break compatibility with RFC4627 by default
1694(and potentially leave applications insecure) and change the default to
1695follow RFC7159, but application authors are well advised to call C<<
1696->allow_nonref(0) >> even if this is the current default, if they cannot
1697handle non-reference values, in preparation for the day when the default
1698will change.
1699
1700
1701=head1 (I-)THREADS 1749=head1 (I-)THREADS
1702 1750
1703This module is I<not> guaranteed to be ithread (or MULTIPLICITY-) safe 1751This module is I<not> guaranteed to be ithread (or MULTIPLICITY-) safe
1704and there are no plans to change this. Note that perl's builtin so-called 1752and there are no plans to change this. Note that perl's builtin so-called
1705threads/ithreads are officially deprecated and should not be used. 1753threads/ithreads are officially deprecated and should not be used.
1719categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>. 1767categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>.
1720 1768
1721If you need C<LC_NUMERIC>, you should enable it only around the code that 1769If you need C<LC_NUMERIC>, you should enable it only around the code that
1722actually needs it (avoiding stringification of numbers), and restore it 1770actually needs it (avoiding stringification of numbers), and restore it
1723afterwards. 1771afterwards.
1772
1773
1774=head1 SOME HISTORY
1775
1776At the time this module was created there already were a number of JSON
1777modules available on CPAN, so what was the reason to write yet another
1778JSON module? While it seems there are many JSON modules, none of them
1779correctly handled all corner cases, and in most cases their maintainers
1780are unresponsive, gone missing, or not listening to bug reports for other
1781reasons.
1782
1783Beginning with version 2.0 of the JSON module, when both JSON and
1784JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
1785overridden) with no overhead due to emulation (by inheriting constructor
1786and methods). If JSON::XS is not available, it will fall back to the
1787compatible JSON::PP module as backend, so using JSON instead of JSON::XS
1788gives you a portable JSON API that can be fast when you need it and
1789doesn't require a C compiler when that is a problem.
1790
1791Somewhere around version 3, this module was forked into
1792C<Cpanel::JSON::XS>, because its maintainer had serious trouble
1793understanding JSON and insisted on a fork with many bugs "fixed" that
1794weren't actually bugs, while spreading FUD about this module without
1795actually giving any details on his accusations. You be the judge, but
1796in my personal opinion, if you want quality, you will stay away from
1797dangerous forks like that.
1724 1798
1725 1799
1726=head1 BUGS 1800=head1 BUGS
1727 1801
1728While the goal of this module is to be correct, that unfortunately does 1802While the goal of this module is to be correct, that unfortunately does

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines