ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.158 by root, Thu Sep 1 11:38:39 2016 UTC vs.
Revision 1.169 by root, Thu Nov 15 20:49:12 2018 UTC

… …
40Beginning with version 2.0 of the JSON module, when both JSON and 40Beginning with version 2.0 of the JSON module, when both JSON and
41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be 41JSON::XS are installed, then JSON will fall back on JSON::XS (this can be
42overridden) with no overhead due to emulation (by inheriting constructor 42overridden) with no overhead due to emulation (by inheriting constructor
43and methods). If JSON::XS is not available, it will fall back to the 43and methods). If JSON::XS is not available, it will fall back to the
44compatible JSON::PP module as backend, so using JSON instead of JSON::XS 44compatible JSON::PP module as backend, so using JSON instead of JSON::XS
45gives you a portable JSON API that can be fast when you need and doesn't 45gives you a portable JSON API that can be fast when you need it and
46require a C compiler when that is a problem. 46doesn't require a C compiler when that is a problem.
47 47
48As this is the n-th-something JSON module on CPAN, what was the reason 48As this is the n-th-something JSON module on CPAN, what was the reason
49to write yet another JSON module? While it seems there are many JSON 49to write yet another JSON module? While it seems there are many JSON
50modules, none of them correctly handle all corner cases, and in most cases 50modules, none of them correctly handle all corner cases, and in most cases
51their maintainers are unresponsive, gone missing, or not listening to bug 51their maintainers are unresponsive, gone missing, or not listening to bug
… …
101 101
102package JSON::XS; 102package JSON::XS;
103 103
104use common::sense; 104use common::sense;
105 105
106our $VERSION = 3.02; 106our $VERSION = 3.04;
107our @ISA = qw(Exporter); 107our @ISA = qw(Exporter);
108 108
109our @EXPORT = qw(encode_json decode_json); 109our @EXPORT = qw(encode_json decode_json);
110 110
111use Exporter; 111use Exporter;
… …
131 131
132Except being faster. 132Except being faster.
133 133
134=item $perl_scalar = decode_json $json_text 134=item $perl_scalar = decode_json $json_text
135 135
136The opposite of C<encode_json>: expects an UTF-8 (binary) string and tries 136The opposite of C<encode_json>: expects a UTF-8 (binary) string and tries
137to parse that as an UTF-8 encoded JSON text, returning the resulting 137to parse that as a UTF-8 encoded JSON text, returning the resulting
138reference. Croaks on error. 138reference. Croaks on error.
139 139
140This function call is functionally identical to: 140This function call is functionally identical to:
141 141
142 $perl_scalar = JSON::XS->new->utf8->decode ($json_text) 142 $perl_scalar = JSON::XS->new->utf8->decode ($json_text)
… …
202=over 4 202=over 4
203 203
204=item $json = new JSON::XS 204=item $json = new JSON::XS
205 205
206Creates a new JSON::XS object that can be used to de/encode JSON 206Creates a new JSON::XS object that can be used to de/encode JSON
207strings. All boolean flags described below are by default I<disabled>. 207strings. All boolean flags described below are by default I<disabled>
208(with the exception of C<allow_nonref>, which defaults to I<enabled> since
209version C<4.0>).
208 210
209The mutators for flags all return the JSON object again and thus calls can 211The mutators for flags all return the JSON object again and thus calls can
210be chained: 212be chained:
211 213
212 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]}) 214 my $json = JSON::XS->new->utf8->space_after->encode ({a => [1,2]})
… …
270 272
271=item $enabled = $json->get_utf8 273=item $enabled = $json->get_utf8
272 274
273If C<$enable> is true (or missing), then the C<encode> method will encode 275If C<$enable> is true (or missing), then the C<encode> method will encode
274the JSON result into UTF-8, as required by many protocols, while the 276the JSON result into UTF-8, as required by many protocols, while the
275C<decode> method expects to be handled an UTF-8-encoded string. Please 277C<decode> method expects to be handed a UTF-8-encoded string. Please
276note that UTF-8-encoded strings do not contain any characters outside the 278note that UTF-8-encoded strings do not contain any characters outside the
277range C<0..255>, they are thus useful for bytewise/binary I/O. In future 279range C<0..255>, they are thus useful for bytewise/binary I/O. In future
278versions, enabling this option might enable autodetection of the UTF-16 280versions, enabling this option might enable autodetection of the UTF-16
279and UTF-32 encoding families, as described in RFC4627. 281and UTF-32 encoding families, as described in RFC4627.
280 282
… …
365 367
366=item $enabled = $json->get_relaxed 368=item $enabled = $json->get_relaxed
367 369
368If C<$enable> is true (or missing), then C<decode> will accept some 370If C<$enable> is true (or missing), then C<decode> will accept some
369extensions to normal JSON syntax (see below). C<encode> will not be 371extensions to normal JSON syntax (see below). C<encode> will not be
370affected in anyway. I<Be aware that this option makes you accept invalid 372affected in any way. I<Be aware that this option makes you accept invalid
371JSON texts as if they were valid!>. I suggest only to use this option to 373JSON texts as if they were valid!>. I suggest only to use this option to
372parse application-specific files written by humans (configuration files, 374parse application-specific files written by humans (configuration files,
373resource files etc.) 375resource files etc.)
374 376
375If C<$enable> is false (the default), then C<decode> will only accept 377If C<$enable> is false (the default), then C<decode> will only accept
… …
441 443
442=item $json = $json->allow_nonref ([$enable]) 444=item $json = $json->allow_nonref ([$enable])
443 445
444=item $enabled = $json->get_allow_nonref 446=item $enabled = $json->get_allow_nonref
445 447
448Unlike other boolean options, this opotion is enabled by default beginning
449with version C<4.0>. See L<SECURITY CONSIDERATIONS> for the gory details.
450
446If C<$enable> is true (or missing), then the C<encode> method can convert a 451If C<$enable> is true (or missing), then the C<encode> method can convert a
447non-reference into its corresponding string, number or null JSON value, 452non-reference into its corresponding string, number or null JSON value,
448which is an extension to RFC4627. Likewise, C<decode> will accept those JSON 453which is an extension to RFC4627. Likewise, C<decode> will accept those JSON
449values instead of croaking. 454values instead of croaking.
450 455
451If C<$enable> is false, then the C<encode> method will croak if it isn't 456If C<$enable> is false, then the C<encode> method will croak if it isn't
452passed an arrayref or hashref, as JSON texts must either be an object 457passed an arrayref or hashref, as JSON texts must either be an object
453or array. Likewise, C<decode> will croak if given something that is not a 458or array. Likewise, C<decode> will croak if given something that is not a
454JSON object or array. 459JSON object or array.
455 460
456Example, encode a Perl scalar as JSON value with enabled C<allow_nonref>, 461Example, encode a Perl scalar as JSON value without enabled C<allow_nonref>,
457resulting in an invalid JSON text: 462resulting in an error:
458 463
459 JSON::XS->new->allow_nonref->encode ("Hello, World!") 464 JSON::XS->new->allow_nonref (0)->encode ("Hello, World!")
460 => "Hello, World!" 465 => hash- or arrayref expected...
461 466
462=item $json = $json->allow_unknown ([$enable]) 467=item $json = $json->allow_unknown ([$enable])
463 468
464=item $enabled = $json->get_allow_unknown 469=item $enabled = $json->get_allow_unknown
465 470
… …
515 520
516This setting has no effect on C<decode>. 521This setting has no effect on C<decode>.
517 522
518=item $json = $json->allow_tags ([$enable]) 523=item $json = $json->allow_tags ([$enable])
519 524
520=item $enabled = $json->allow_tags 525=item $enabled = $json->get_allow_tags
521 526
522See L<OBJECT SERIALISATION> for details. 527See L<OBJECT SERIALISATION> for details.
523 528
524If C<$enable> is true (or missing), then C<encode>, upon encountering a 529If C<$enable> is true (or missing), then C<encode>, upon encountering a
525blessed object, will check for the availability of the C<FREEZE> method on 530blessed object, will check for the availability of the C<FREEZE> method on
… …
534in C<decode>, as if tags were not part of the grammar. 539in C<decode>, as if tags were not part of the grammar.
535 540
536=item $json = $json->filter_json_object ([$coderef->($hashref)]) 541=item $json = $json->filter_json_object ([$coderef->($hashref)])
537 542
538When C<$coderef> is specified, it will be called from C<decode> each 543When C<$coderef> is specified, it will be called from C<decode> each
539time it decodes a JSON object. The only argument is a reference to the 544time it decodes a JSON object. The only argument is a reference to
540newly-created hash. If the code references returns a single scalar (which 545the newly-created hash. If the code reference returns a single scalar
541need not be a reference), this value (i.e. a copy of that scalar to avoid 546(which need not be a reference), this value (or rather a copy of it) is
542aliasing) is inserted into the deserialised data structure. If it returns 547inserted into the deserialised data structure. If it returns an empty
543an empty list (NOTE: I<not> C<undef>, which is a valid scalar), the 548list (NOTE: I<not> C<undef>, which is a valid scalar), the original
544original deserialised hash will be inserted. This setting can slow down 549deserialised hash will be inserted. This setting can slow down decoding
545decoding considerably. 550considerably.
546 551
547When C<$coderef> is omitted or undefined, any existing callback will 552When C<$coderef> is omitted or undefined, any existing callback will
548be removed and C<decode> will not change the deserialised hash in any 553be removed and C<decode> will not change the deserialised hash in any
549way. 554way.
550 555
… …
769all other circumstances you must not call this function (I mean it. 774all other circumstances you must not call this function (I mean it.
770although in simple tests it might actually work, it I<will> fail under 775although in simple tests it might actually work, it I<will> fail under
771real world conditions). As a special exception, you can also call this 776real world conditions). As a special exception, you can also call this
772method before having parsed anything. 777method before having parsed anything.
773 778
779That means you can only use this function to look at or manipulate text
780before or after complete JSON objects, not while the parser is in the
781middle of parsing a JSON object.
782
774This function is useful in two cases: a) finding the trailing text after a 783This function is useful in two cases: a) finding the trailing text after a
775JSON object or b) parsing multiple JSON objects separated by non-JSON text 784JSON object or b) parsing multiple JSON objects separated by non-JSON text
776(such as commas). 785(such as commas).
777 786
778=item $json->incr_skip 787=item $json->incr_skip
… …
1285expect your input strings to be encoded as UTF-8, that is, no "character" 1294expect your input strings to be encoded as UTF-8, that is, no "character"
1286of the input string must have any value > 255, as UTF-8 does not allow 1295of the input string must have any value > 255, as UTF-8 does not allow
1287that. 1296that.
1288 1297
1289The C<utf8> flag therefore switches between two modes: disabled means you 1298The C<utf8> flag therefore switches between two modes: disabled means you
1290will get a Unicode string in Perl, enabled means you get an UTF-8 encoded 1299will get a Unicode string in Perl, enabled means you get a UTF-8 encoded
1291octet/binary string in Perl. 1300octet/binary string in Perl.
1292 1301
1293=item C<latin1> or C<ascii> flags enabled 1302=item C<latin1> or C<ascii> flags enabled
1294 1303
1295With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters 1304With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters
… …
1563are browser design bugs, but it is still you who will have to deal with 1572are browser design bugs, but it is still you who will have to deal with
1564it, as major browser developers care only for features, not about getting 1573it, as major browser developers care only for features, not about getting
1565security right). 1574security right).
1566 1575
1567 1576
1568=head1 "OLD" VS. "NEW" JSON (RFC 4627 VS. RFC 7159) 1577=head2 "OLD" VS. "NEW" JSON (RFC 4627 VS. RFC 7159)
1569 1578
1570TL;DR: Due to security concerns, JSON::XS will not allow scalar data in 1579JSON originally required JSON texts to represent an array or object -
1571JSON texts by default - you need to create your own JSON::XS object and 1580scalar values were explicitly not allowed. This has changed, and versions
1572enable C<allow_nonref>: 1581of JSON::XS beginning with C<4.0> reflect this by allowing scalar values
1582by default.
1573 1583
1584One reason why one might not want this is that this removes a fundamental
1585property of JSON texts, namely that they are self-delimited and
1586self-contained, or in other words, you could take any number of "old"
1587JSON texts and paste them together, and the result would be unambiguously
1588parseable:
1574 1589
1590 [1,3]{"k":5}[][null] # four JSON texts, without doubt
1591
1592By allowing scalars, this property is lost: in the following example, is
1593this one JSON text (the number 12) or two JSON texts (the numbers 1 and
15942):
1595
1596 12 # could be 12, or 1 and 2
1597
1598Another lost property of "old" JSON is that no lookahead is required to
1599know the end of a JSON text, i.e. the JSON text definitely ended at the
1600last C<]> or C<}> character, there was no need to read extra characters.
1601
1602For example, a viable network protocol with "old" JSON was to simply
1603exchange JSON texts without delimiter. For "new" JSON, you have to use a
1604suitable delimiter (such as a newline) after every JSON text or ensure you
1605never encode/decode scalar values.
1606
1607Most protocols do work by only transferring arrays or objects, and the
1608easiest way to avoid problems with the "new" JSON definition is to
1609explicitly disallow scalar values in your encoder and decoder:
1610
1575 my $json = JSON::XS->new->allow_nonref; 1611 $json_coder = JSON::XS->new->allow_nonref (0)
1576 1612
1577 $text = $json->encode ($data); 1613This is a somewhat unhappy situation, and the blame can fully be put on
1578 $data = $json->decode ($text); 1614JSON's inmventor, Douglas Crockford, who unilaterally changed the format
1579 1615in 2006 without consulting the IETF, forcing the IETF to either fork the
1580The long version: JSON being an important and supposedly stable format, 1616format or go with it (as I was told, the IETF wasn't amused).
1581the IETF standardised it as RFC 4627 in 2006. Unfortunately, the inventor
1582of JSON, Dougles Crockford, unilaterally changed the definition of JSON in
1583javascript. Rather than create a fork, the IETF decided to standardise the
1584new syntax (apparently, so Iw as told, without finding it very amusing).
1585
1586The biggest difference between thed original JSON and the new JSON is that
1587the new JSON supports scalars (anything other than arrays and objects) at
1588the toplevel of a JSON text. While this is strictly backwards compatible
1589to older versions, it breaks a number of protocols that relied on sending
1590JSON back-to-back, and is a minor security concern.
1591
1592For example, imagine you have two banks communicating, and on one side,
1593trhe JSON coder gets upgraded. Two messages, such as C<10> and C<1000>
1594might then be confused to mean C<101000>, something that couldn't happen
1595in the original JSON, because niether of these messages would be valid
1596JSON.
1597
1598If one side accepts these messages, then an upgrade in the coder on either
1599side could result in this becoming exploitable.
1600
1601This module has always allowed these messages as an optional extension, by
1602default disabled. The security concerns are the reason why the default is
1603still disabled, but future versions might/will likely upgrade to the newer
1604RFC as default format, so you are advised to check your implementation
1605and/or override the default with C<< ->allow_nonref (0) >> to ensure that
1606future versions are safe.
1607 1617
1608 1618
1609=head1 INTEROPERABILITY WITH OTHER MODULES 1619=head1 INTEROPERABILITY WITH OTHER MODULES
1610 1620
1611C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean 1621C<JSON::XS> uses the L<Types::Serialiser> module to provide boolean
1612constants. That means that the JSON true and false values will be 1622constants. That means that the JSON true and false values will be
1613comaptible to true and false values of iother modules that do the same, 1623comaptible to true and false values of other modules that do the same,
1614such as L<JSON::PP> and L<CBOR::XS>. 1624such as L<JSON::PP> and L<CBOR::XS>.
1615 1625
1616 1626
1617=head1 INTEROPERABILITY WITH OTHER JSON DECODERS 1627=head1 INTEROPERABILITY WITH OTHER JSON DECODERS
1618 1628
… …
1635 1645
1636When you use C<allow_tags> to use the extended (and also nonstandard and 1646When you use C<allow_tags> to use the extended (and also nonstandard and
1637invalid) JSON syntax for serialised objects, and you still want to decode 1647invalid) JSON syntax for serialised objects, and you still want to decode
1638the generated When you want to serialise objects, you can run a regex 1648the generated When you want to serialise objects, you can run a regex
1639to replace the tagged syntax by standard JSON arrays (it only works for 1649to replace the tagged syntax by standard JSON arrays (it only works for
1640"normal" packagesnames without comma, newlines or single colons). First, 1650"normal" package names without comma, newlines or single colons). First,
1641the readable Perl version: 1651the readable Perl version:
1642 1652
1643 # if your FREEZE methods return no values, you need this replace first: 1653 # if your FREEZE methods return no values, you need this replace first:
1644 $json =~ s/\( \s* (" (?: [^\\":,]+|\\.|::)* ") \s* \) \s* \[\s*\]/[$1]/gx; 1654 $json =~ s/\( \s* (" (?: [^\\":,]+|\\.|::)* ") \s* \) \s* \[\s*\]/[$1]/gx;
1645 1655
… …
1681Since this module was written, Google has written a new JSON RFC, RFC 7159 1691Since this module was written, Google has written a new JSON RFC, RFC 7159
1682(and RFC7158). Unfortunately, this RFC breaks compatibility with both the 1692(and RFC7158). Unfortunately, this RFC breaks compatibility with both the
1683original JSON specification on www.json.org and RFC4627. 1693original JSON specification on www.json.org and RFC4627.
1684 1694
1685As far as I can see, you can get partial compatibility when parsing by 1695As far as I can see, you can get partial compatibility when parsing by
1686using C<< ->allow_nonref >>. However, consider thew security implications 1696using C<< ->allow_nonref >>. However, consider the security implications
1687of doing so. 1697of doing so.
1688 1698
1689I haven't decided yet when to break compatibility with RFC4627 by default 1699I haven't decided yet when to break compatibility with RFC4627 by default
1690(and potentially leave applications insecure) and change the default to 1700(and potentially leave applications insecure) and change the default to
1691follow RFC7159, but application authors are well advised to call C<< 1701follow RFC7159, but application authors are well advised to call C<<
1692->allow_nonref(0) >> even if this is the current default, if they cannot 1702->allow_nonref(0) >> even if this is the current default, if they cannot
1693handle non-reference values, in preparation for the day when the4 default 1703handle non-reference values, in preparation for the day when the default
1694will change. 1704will change.
1695 1705
1696 1706
1697=head1 THREADS 1707=head1 (I-)THREADS
1698 1708
1699This module is I<not> guaranteed to be thread safe and there are no 1709This module is I<not> guaranteed to be ithread (or MULTIPLICITY-) safe
1700plans to change this until Perl gets thread support (as opposed to the 1710and there are no plans to change this. Note that perl's builtin so-called
1701horribly slow so-called "threads" which are simply slow and bloated 1711threads/ithreads are officially deprecated and should not be used.
1702process simulations - use fork, it's I<much> faster, cheaper, better).
1703
1704(It might actually work, but you have been warned).
1705 1712
1706 1713
1707=head1 THE PERILS OF SETLOCALE 1714=head1 THE PERILS OF SETLOCALE
1708 1715
1709Sometimes people avoid the Perl locale support and directly call the 1716Sometimes people avoid the Perl locale support and directly call the

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines