ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.169 by root, Thu Nov 15 20:49:12 2018 UTC vs.
Revision 1.175 by root, Wed Mar 6 07:32:06 2019 UTC

… …
35 35
36This module converts Perl data structures to JSON and vice versa. Its 36This module converts Perl data structures to JSON and vice versa. Its
37primary goal is to be I<correct> and its secondary goal is to be 37primary goal is to be I<correct> and its secondary goal is to be
38I<fast>. To reach the latter goal it was written in C. 38I<fast>. To reach the latter goal it was written in C.
39 39
40Beginning with version 2.0 of the JSON module, when both JSON and
41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42overridden) with no overhead due to emulation (by inheriting constructor
43and methods). If JSON::XS is not available, it will fall back to the
44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45gives you a portable JSON API that can be fast when you need it and
46doesn't require a C compiler when that is a problem.
47
48As this is the n-th-something JSON module on CPAN, what was the reason
49to write yet another JSON module? While it seems there are many JSON
50modules, none of them correctly handle all corner cases, and in most cases
51their maintainers are unresponsive, gone missing, or not listening to bug
52reports for other reasons.
53
54See MAPPING, below, on how JSON::XS maps perl values to JSON values and 40See MAPPING, below, on how JSON::XS maps perl values to JSON values and
55vice versa. 41vice versa.
56 42
57=head2 FEATURES 43=head2 FEATURES
58 44
59=over 4 45=over
60 46
61=item * correct Unicode handling 47=item * correct Unicode handling
62 48
63This module knows how to handle Unicode, documents how and when it does 49This module knows how to handle Unicode, documents how and when it does
64so, and even documents what "correct" means. 50so, and even documents what "correct" means.
… …
101 87
102package JSON::XS; 88package JSON::XS;
103 89
104use common::sense; 90use common::sense;
105 91
106our $VERSION = 3.04; 92our $VERSION = '4.02';
107our @ISA = qw(Exporter); 93our @ISA = qw(Exporter);
108 94
109our @EXPORT = qw(encode_json decode_json); 95our @EXPORT = qw(encode_json decode_json);
110 96
111use Exporter; 97use Exporter;
… …
116=head1 FUNCTIONAL INTERFACE 102=head1 FUNCTIONAL INTERFACE
117 103
118The following convenience methods are provided by this module. They are 104The following convenience methods are provided by this module. They are
119exported by default: 105exported by default:
120 106
121=over 4 107=over
122 108
123=item $json_text = encode_json $perl_scalar 109=item $json_text = encode_json $perl_scalar
124 110
125Converts the given Perl data structure to a UTF-8 encoded, binary string 111Converts the given Perl data structure to a UTF-8 encoded, binary string
126(that is, the string contains octets only). Croaks on error. 112(that is, the string contains octets only). Croaks on error.
… …
149=head1 A FEW NOTES ON UNICODE AND PERL 135=head1 A FEW NOTES ON UNICODE AND PERL
150 136
151Since this often leads to confusion, here are a few very clear words on 137Since this often leads to confusion, here are a few very clear words on
152how Unicode works in Perl, modulo bugs. 138how Unicode works in Perl, modulo bugs.
153 139
154=over 4 140=over
155 141
156=item 1. Perl strings can store characters with ordinal values > 255. 142=item 1. Perl strings can store characters with ordinal values > 255.
157 143
158This enables you to store Unicode characters as single characters in a 144This enables you to store Unicode characters as single characters in a
159Perl string - very natural. 145Perl string - very natural.
… …
197=head1 OBJECT-ORIENTED INTERFACE 183=head1 OBJECT-ORIENTED INTERFACE
198 184
199The object oriented interface lets you configure your own encoding or 185The object oriented interface lets you configure your own encoding or
200decoding style, within the limits of supported formats. 186decoding style, within the limits of supported formats.
201 187
202=over 4 188=over
203 189
204=item $json = new JSON::XS 190=item $json = new JSON::XS
205 191
206Creates a new JSON::XS object that can be used to de/encode JSON 192Creates a new JSON::XS object that can be used to de/encode JSON
207strings. All boolean flags described below are by default I<disabled> 193strings. All boolean flags described below are by default I<disabled>
… …
377If C<$enable> is false (the default), then C<decode> will only accept 363If C<$enable> is false (the default), then C<decode> will only accept
378valid JSON texts. 364valid JSON texts.
379 365
380Currently accepted extensions are: 366Currently accepted extensions are:
381 367
382=over 4 368=over
383 369
384=item * list items can have an end-comma 370=item * list items can have an end-comma
385 371
386JSON I<separates> array elements and key-value pairs with commas. This 372JSON I<separates> array elements and key-value pairs with commas. This
387can be annoying if you write JSON texts manually and want to be able to 373can be annoying if you write JSON texts manually and want to be able to
… …
535them via a call to the C<THAW> method. 521them via a call to the C<THAW> method.
536 522
537If C<$enable> is false (the default), then C<encode> will not consider 523If C<$enable> is false (the default), then C<encode> will not consider
538this type of conversion, and tagged JSON values will cause a parse error 524this type of conversion, and tagged JSON values will cause a parse error
539in C<decode>, as if tags were not part of the grammar. 525in C<decode>, as if tags were not part of the grammar.
526
527=item $json->boolean_values ([$false, $true])
528
529=item ($false, $true) = $json->get_boolean_values
530
531By default, JSON booleans will be decoded as overloaded
532C<$Types::Serialiser::false> and C<$Types::Serialiser::true> objects.
533
534With this method you can specify your own boolean values for decoding -
535on decode, JSON C<false> will be decoded as a copy of C<$false>, and JSON
536C<true> will be decoded as C<$true> ("copy" here is the same thing as
537assigning a value to another variable, i.e. C<$copy = $false>).
538
539Calling this method without any arguments will reset the booleans
540to their default values.
541
542C<get_boolean_values> will return both C<$false> and C<$true> values, or
543the empty list when they are set to the default.
540 544
541=item $json = $json->filter_json_object ([$coderef->($hashref)]) 545=item $json = $json->filter_json_object ([$coderef->($hashref)])
542 546
543When C<$coderef> is specified, it will be called from C<decode> each 547When C<$coderef> is specified, it will be called from C<decode> each
544time it decodes a JSON object. The only argument is a reference to 548time it decodes a JSON object. The only argument is a reference to
… …
729to set resource limits (e.g. C<max_size>) to ensure the parser will stop 733to set resource limits (e.g. C<max_size>) to ensure the parser will stop
730parsing in the presence if syntax errors. 734parsing in the presence if syntax errors.
731 735
732The following methods implement this incremental parser. 736The following methods implement this incremental parser.
733 737
734=over 4 738=over
735 739
736=item [void, scalar or list context] = $json->incr_parse ([$string]) 740=item [void, scalar or list context] = $json->incr_parse ([$string])
737 741
738This is the central parsing function. It can both append new text and 742This is the central parsing function. It can both append new text and
739extract objects from the stream accumulated so far (both of these 743extract objects from the stream accumulated so far (both of these
… …
806 810
807=back 811=back
808 812
809=head2 LIMITATIONS 813=head2 LIMITATIONS
810 814
811All options that affect decoding are supported, except 815The incremental parser is a non-exact parser: it works by gathering as
812C<allow_nonref>. The reason for this is that it cannot be made to work 816much text as possible that I<could> be a valid JSON text, followed by
813sensibly: JSON objects and arrays are self-delimited, i.e. you can 817trying to decode it.
814concatenate them back to back and still decode them perfectly. This does
815not hold true for JSON numbers, however.
816 818
817For example, is the string C<1> a single JSON number, or is it simply the 819That means it sometimes needs to read more data than strictly necessary to
818start of C<12>? Or is C<12> a single JSON number, or the concatenation 820diagnose an invalid JSON text. For example, after parsing the following
819of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS 821fragment, the parser I<could> stop with an error, as this fragment
820takes the conservative route and disallows this case. 822I<cannot> be the beginning of a valid JSON text:
823
824 [,
825
826In reality, hopwever, the parser might continue to read data until a
827length limit is exceeded or it finds a closing bracket.
821 828
822=head2 EXAMPLES 829=head2 EXAMPLES
823 830
824Some examples will make all this clearer. First, a simple example that 831Some examples will make all this clearer. First, a simple example that
825works similarly to C<decode_prefix>: We want to decode the JSON object at 832works similarly to C<decode_prefix>: We want to decode the JSON object at
… …
969refers to the abstract Perl language itself. 976refers to the abstract Perl language itself.
970 977
971 978
972=head2 JSON -> PERL 979=head2 JSON -> PERL
973 980
974=over 4 981=over
975 982
976=item object 983=item object
977 984
978A JSON object becomes a reference to a hash in Perl. No ordering of object 985A JSON object becomes a reference to a hash in Perl. No ordering of object
979keys is preserved (JSON does not preserve object key ordering itself). 986keys is preserved (JSON does not preserve object key ordering itself).
… …
1047 1054
1048The mapping from Perl to JSON is slightly more difficult, as Perl is a 1055The mapping from Perl to JSON is slightly more difficult, as Perl is a
1049truly typeless language, so we can only guess which JSON type is meant by 1056truly typeless language, so we can only guess which JSON type is meant by
1050a Perl value. 1057a Perl value.
1051 1058
1052=over 4 1059=over
1053 1060
1054=item hash references 1061=item hash references
1055 1062
1056Perl hash references become JSON objects. As there is no inherent 1063Perl hash references become JSON objects. As there is no inherent
1057ordering in hash keys (or JSON objects), they will usually be encoded 1064ordering in hash keys (or JSON objects), they will usually be encoded
… …
1146 1153
1147What happens when C<JSON::XS> encounters a Perl object depends on the 1154What happens when C<JSON::XS> encounters a Perl object depends on the
1148C<allow_blessed>, C<convert_blessed> and C<allow_tags> settings, which are 1155C<allow_blessed>, C<convert_blessed> and C<allow_tags> settings, which are
1149used in this order: 1156used in this order:
1150 1157
1151=over 4 1158=over
1152 1159
1153=item 1. C<allow_tags> is enabled and the object has a C<FREEZE> method. 1160=item 1. C<allow_tags> is enabled and the object has a C<FREEZE> method.
1154 1161
1155In this case, C<JSON::XS> uses the L<Types::Serialiser> object 1162In this case, C<JSON::XS> uses the L<Types::Serialiser> object
1156serialisation protocol to create a tagged JSON value, using a nonstandard 1163serialisation protocol to create a tagged JSON value, using a nonstandard
… …
1267takes those codepoint numbers and I<encodes> them, in our case into 1274takes those codepoint numbers and I<encodes> them, in our case into
1268octets. Unicode is (among other things) a codeset, UTF-8 is an encoding, 1275octets. Unicode is (among other things) a codeset, UTF-8 is an encoding,
1269and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at 1276and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at
1270the same time, which can be confusing. 1277the same time, which can be confusing.
1271 1278
1272=over 4 1279=over
1273 1280
1274=item C<utf8> flag disabled 1281=item C<utf8> flag disabled
1275 1282
1276When C<utf8> is disabled (the default), then C<encode>/C<decode> generate 1283When C<utf8> is disabled (the default), then C<encode>/C<decode> generate
1277and expect Unicode strings, that is, characters with high ordinal Unicode 1284and expect Unicode strings, that is, characters with high ordinal Unicode
… …
1436general you should not try to generate YAML with a JSON generator or vice 1443general you should not try to generate YAML with a JSON generator or vice
1437versa, or try to parse JSON with a YAML parser or vice versa: chances are 1444versa, or try to parse JSON with a YAML parser or vice versa: chances are
1438high that you will run into severe interoperability problems when you 1445high that you will run into severe interoperability problems when you
1439least expect it. 1446least expect it.
1440 1447
1441=over 4 1448=over
1442 1449
1443=item (*) 1450=item (*)
1444 1451
1445I have been pressured multiple times by Brian Ingerson (one of the 1452I have been pressured multiple times by Brian Ingerson (one of the
1446authors of the YAML specification) to remove this paragraph, despite him 1453authors of the YAML specification) to remove this paragraph, despite him
… …
1572are browser design bugs, but it is still you who will have to deal with 1579are browser design bugs, but it is still you who will have to deal with
1573it, as major browser developers care only for features, not about getting 1580it, as major browser developers care only for features, not about getting
1574security right). 1581security right).
1575 1582
1576 1583
1577=head2 "OLD" VS. "NEW" JSON (RFC 4627 VS. RFC 7159) 1584=head2 "OLD" VS. "NEW" JSON (RFC4627 VS. RFC7159)
1578 1585
1579JSON originally required JSON texts to represent an array or object - 1586JSON originally required JSON texts to represent an array or object -
1580scalar values were explicitly not allowed. This has changed, and versions 1587scalar values were explicitly not allowed. This has changed, and versions
1581of JSON::XS beginning with C<4.0> reflect this by allowing scalar values 1588of JSON::XS beginning with C<4.0> reflect this by allowing scalar values
1582by default. 1589by default.
… …
1612 1619
1613This is a somewhat unhappy situation, and the blame can fully be put on 1620This is a somewhat unhappy situation, and the blame can fully be put on
1614JSON's inmventor, Douglas Crockford, who unilaterally changed the format 1621JSON's inmventor, Douglas Crockford, who unilaterally changed the format
1615in 2006 without consulting the IETF, forcing the IETF to either fork the 1622in 2006 without consulting the IETF, forcing the IETF to either fork the
1616format or go with it (as I was told, the IETF wasn't amused). 1623format or go with it (as I was told, the IETF wasn't amused).
1624
1625
1626=head1 RELATIONSHIP WITH I-JSON
1627
1628JSON is a somewhat sloppily-defined format - it carries around obvious
1629Javascript baggage, such as not really defining number range, probably
1630because Javascript only has one type of numbers: IEEE 64 bit floats
1631("binary64").
1632
1633For this reaosn, RFC7493 defines "Internet JSON", which is a restricted
1634subset of JSON that is supposedly more interoperable on the internet.
1635
1636While C<JSON::XS> does not offer specific support for I-JSON, it of course
1637accepts valid I-JSON and by default implements some of the limitations
1638of I-JSON, such as parsing numbers as perl numbers, which are usually a
1639superset of binary64 numbers.
1640
1641To generate I-JSON, follow these rules:
1642
1643=over
1644
1645=item * always generate UTF-8
1646
1647I-JSON must be encoded in UTF-8, the default for C<encode_json>.
1648
1649=item * numbers should be within IEEE 754 binary64 range
1650
1651Basically all existing perl installations use binary64 to represent
1652floating point numbers, so all you need to do is to avoid large integers.
1653
1654=item * objects must not have duplicate keys
1655
1656This is trivially done, as C<JSON::XS> does not allow duplicate keys.
1657
1658=item * do not generate scalar JSON texts, use C<< ->allow_nonref (0) >>
1659
1660I-JSON strongly requests you to only encode arrays and objects into JSON.
1661
1662=item * times should be strings in ISO 8601 format
1663
1664There are a myriad of modules on CPAN dealing with ISO 8601 - search for
1665C<ISO8601> on CPAN and use one.
1666
1667=item * encode binary data as base64
1668
1669While it's tempting to just dump binary data as a string (and let
1670C<JSON::XS> do the escaping), for I-JSON, it's I<recommended> to encode
1671binary data as base64.
1672
1673=back
1674
1675There are some other considerations - read RFC7493 for the details if
1676interested.
1617 1677
1618 1678
1619=head1 INTEROPERABILITY WITH OTHER MODULES 1679=head1 INTEROPERABILITY WITH OTHER MODULES
1620 1680
1621C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean 1681C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
… …
1684 1744
1685Again, this has some limitations - the magic string must not be encoded 1745Again, this has some limitations - the magic string must not be encoded
1686with character escapes, and the constructor arguments must be non-empty. 1746with character escapes, and the constructor arguments must be non-empty.
1687 1747
1688 1748
1689=head1 RFC7159
1690
1691Since this module was written, Google has written a new JSON RFC, RFC 7159
1692(and RFC7158). Unfortunately, this RFC breaks compatibility with both the
1693original JSON specification on www.json.org and RFC4627.
1694
1695As far as I can see, you can get partial compatibility when parsing by
1696using C<< ->allow_nonref >>. However, consider the security implications
1697of doing so.
1698
1699I haven't decided yet when to break compatibility with RFC4627 by default
1700(and potentially leave applications insecure) and change the default to
1701follow RFC7159, but application authors are well advised to call C<<
1702->allow_nonref(0) >> even if this is the current default, if they cannot
1703handle non-reference values, in preparation for the day when the default
1704will change.
1705
1706
1707=head1 (I-)THREADS 1749=head1 (I-)THREADS
1708 1750
1709This module is I<not> guaranteed to be ithread (or MULTIPLICITY-) safe 1751This module is I<not> guaranteed to be ithread (or MULTIPLICITY-) safe
1710and there are no plans to change this. Note that perl's builtin so-called 1752and there are no plans to change this. Note that perl's builtin so-called
1711threads/ithreads are officially deprecated and should not be used. 1753threads/ithreads are officially deprecated and should not be used.
… …
1725categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>. 1767categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>.
1726 1768
1727If you need C<LC_NUMERIC>, you should enable it only around the code that 1769If you need C<LC_NUMERIC>, you should enable it only around the code that
1728actually needs it (avoiding stringification of numbers), and restore it 1770actually needs it (avoiding stringification of numbers), and restore it
1729afterwards. 1771afterwards.
1772
1773
1774=head1 SOME HISTORY
1775
1776At the time this module was created there already were a number of JSON
1777modules available on CPAN, so what was the reason to write yet another
1778JSON module? While it seems there are many JSON modules, none of them
1779correctly handled all corner cases, and in most cases their maintainers
1780are unresponsive, gone missing, or not listening to bug reports for other
1781reasons.
1782
1783Beginning with version 2.0 of the JSON module, when both JSON and
1784JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
1785overridden) with no overhead due to emulation (by inheriting constructor
1786and methods). If JSON::XS is not available, it will fall back to the
1787compatible JSON::PP module as backend, so using JSON instead of JSON::XS
1788gives you a portable JSON API that can be fast when you need it and
1789doesn't require a C compiler when that is a problem.
1790
1791Somewhere around version 3, this module was forked into
1792C<Cpanel::JSON::XS>, because its maintainer had serious trouble
1793understanding JSON and insisted on a fork with many bugs "fixed" that
1794weren't actually bugs, while spreading FUD about this module without
1795actually giving any details on his accusations. You be the judge, but
1796in my personal opinion, if you want quality, you will stay away from
1797dangerous forks like that.
1730 1798
1731 1799
1732=head1 BUGS 1800=head1 BUGS
1733 1801
1734While the goal of this module is to be correct, that unfortunately does 1802While the goal of this module is to be correct, that unfortunately does

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines