ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.113 by root, Thu Nov 20 03:59:53 2008 UTC vs.
Revision 1.139 by root, Thu May 23 09:31:32 2013 UTC

… …
64so, and even documents what "correct" means. 64so, and even documents what "correct" means.
65 65
66=item * round-trip integrity 66=item * round-trip integrity
67 67
68When you serialise a perl data structure using only data types supported 68When you serialise a perl data structure using only data types supported
69by JSON, the deserialised data structure is identical on the Perl level. 69by JSON and Perl, the deserialised data structure is identical on the Perl
70(e.g. the string "2.0" doesn't suddenly become "2" just because it looks 70level. (e.g. the string "2.0" doesn't suddenly become "2" just because
71like a number). There minor I<are> exceptions to this, read the MAPPING 71it looks like a number). There I<are> minor exceptions to this, read the
72section below to learn about those. 72MAPPING section below to learn about those.
73 73
74=item * strict checking of JSON correctness 74=item * strict checking of JSON correctness
75 75
76There is no guessing, no generating of illegal JSON texts by default, 76There is no guessing, no generating of illegal JSON texts by default,
77and only JSON is accepted as input by default (the latter is a security 77and only JSON is accepted as input by default (the latter is a security
… …
99 99
100=cut 100=cut
101 101
102package JSON::XS; 102package JSON::XS;
103 103
104no warnings; 104use common::sense;
105use strict;
106 105
107our $VERSION = '2.231'; 106our $VERSION = 2.34;
108our @ISA = qw(Exporter); 107our @ISA = qw(Exporter);
109 108
110our @EXPORT = qw(encode_json decode_json to_json from_json); 109our @EXPORT = qw(encode_json decode_json to_json from_json);
111 110
112sub to_json($) { 111sub to_json($) {
… …
433If C<$enable> is true (or missing), then the C<encode> method will output JSON objects 432If C<$enable> is true (or missing), then the C<encode> method will output JSON objects
434by sorting their keys. This is adding a comparatively high overhead. 433by sorting their keys. This is adding a comparatively high overhead.
435 434
436If C<$enable> is false, then the C<encode> method will output key-value 435If C<$enable> is false, then the C<encode> method will output key-value
437pairs in the order Perl stores them (which will likely change between runs 436pairs in the order Perl stores them (which will likely change between runs
438of the same script). 437of the same script, and can change even within the same run from 5.18
438onwards).
439 439
440This option is useful if you want the same data structure to be encoded as 440This option is useful if you want the same data structure to be encoded as
441the same JSON text (given the same overall settings). If it is disabled, 441the same JSON text (given the same overall settings). If it is disabled,
442the same hash might be encoded differently even if contains the same data, 442the same hash might be encoded differently even if contains the same data,
443as key-value pairs have no inherent ordering in Perl. 443as key-value pairs have no inherent ordering in Perl.
444 444
445This setting has no effect when decoding JSON texts. 445This setting has no effect when decoding JSON texts.
446
447This setting has currently no effect on tied hashes.
446 448
447=item $json = $json->allow_nonref ([$enable]) 449=item $json = $json->allow_nonref ([$enable])
448 450
449=item $enabled = $json->get_allow_nonref 451=item $enabled = $json->get_allow_nonref
450 452
… …
712calls). 714calls).
713 715
714JSON::XS will only attempt to parse the JSON text once it is sure it 716JSON::XS will only attempt to parse the JSON text once it is sure it
715has enough text to get a decisive result, using a very simple but 717has enough text to get a decisive result, using a very simple but
716truly incremental parser. This means that it sometimes won't stop as 718truly incremental parser. This means that it sometimes won't stop as
717early as the full parser, for example, it doesn't detect parenthese 719early as the full parser, for example, it doesn't detect mismatched
718mismatches. The only thing it guarantees is that it starts decoding as 720parentheses. The only thing it guarantees is that it starts decoding as
719soon as a syntactically valid JSON text has been seen. This means you need 721soon as a syntactically valid JSON text has been seen. This means you need
720to set resource limits (e.g. C<max_size>) to ensure the parser will stop 722to set resource limits (e.g. C<max_size>) to ensure the parser will stop
721parsing in the presence if syntax errors. 723parsing in the presence if syntax errors.
722 724
723The following methods implement this incremental parser. 725The following methods implement this incremental parser.
… …
750objects or arrays, instead they must be concatenated back-to-back. If 752objects or arrays, instead they must be concatenated back-to-back. If
751an error occurs, an exception will be raised as in the scalar context 753an error occurs, an exception will be raised as in the scalar context
752case. Note that in this case, any previously-parsed JSON texts will be 754case. Note that in this case, any previously-parsed JSON texts will be
753lost. 755lost.
754 756
757Example: Parse some JSON arrays/objects in a given string and return
758them.
759
760 my @objs = JSON::XS->new->incr_parse ("[5][7][1,2]");
761
755=item $lvalue_string = $json->incr_text 762=item $lvalue_string = $json->incr_text
756 763
757This method returns the currently stored JSON fragment as an lvalue, that 764This method returns the currently stored JSON fragment as an lvalue, that
758is, you can manipulate it. This I<only> works when a preceding call to 765is, you can manipulate it. This I<only> works when a preceding call to
759C<incr_parse> in I<scalar context> successfully returned an object. Under 766C<incr_parse> in I<scalar context> successfully returned an object. Under
… …
766JSON object or b) parsing multiple JSON objects separated by non-JSON text 773JSON object or b) parsing multiple JSON objects separated by non-JSON text
767(such as commas). 774(such as commas).
768 775
769=item $json->incr_skip 776=item $json->incr_skip
770 777
771This will reset the state of the incremental parser and will remove the 778This will reset the state of the incremental parser and will remove
772parsed text from the input buffer. This is useful after C<incr_parse> 779the parsed text from the input buffer so far. This is useful after
773died, in which case the input buffer and incremental parser state is left 780C<incr_parse> died, in which case the input buffer and incremental parser
774unchanged, to skip the text parsed so far and to reset the parse state. 781state is left unchanged, to skip the text parsed so far and to reset the
782parse state.
783
784The difference to C<incr_reset> is that only text until the parse error
785occured is removed.
775 786
776=item $json->incr_reset 787=item $json->incr_reset
777 788
778This completely resets the incremental parser, that is, after this call, 789This completely resets the incremental parser, that is, after this call,
779it will be as if the parser had never parsed anything. 790it will be as if the parser had never parsed anything.
780 791
781This is useful if you want ot repeatedly parse JSON objects and want to 792This is useful if you want to repeatedly parse JSON objects and want to
782ignore any trailing data, which means you have to reset the parser after 793ignore any trailing data, which means you have to reset the parser after
783each successful decode. 794each successful decode.
784 795
785=back 796=back
786 797
… …
984Numbers containing a fractional or exponential part will always be 995Numbers containing a fractional or exponential part will always be
985represented as numeric (floating point) values, possibly at a loss of 996represented as numeric (floating point) values, possibly at a loss of
986precision (in which case you might lose perfect roundtripping ability, but 997precision (in which case you might lose perfect roundtripping ability, but
987the JSON number will still be re-encoded as a JSON number). 998the JSON number will still be re-encoded as a JSON number).
988 999
1000Note that precision is not accuracy - binary floating point values cannot
1001represent most decimal fractions exactly, and when converting from and to
1002floating point, JSON::XS only guarantees precision up to but not including
1003the leats significant bit.
1004
989=item true, false 1005=item true, false
990 1006
991These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>, 1007These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>,
992respectively. They are overloaded to act almost exactly like the numbers 1008respectively. They are overloaded to act almost exactly like the numbers
993C<1> and C<0>. You can check whether a scalar is a JSON boolean by using 1009C<1> and C<0>. You can check whether a scalar is a JSON boolean by using
… …
1080 1096
1081You can not currently force the type in other, less obscure, ways. Tell me 1097You can not currently force the type in other, less obscure, ways. Tell me
1082if you need this capability (but don't forget to explain why it's needed 1098if you need this capability (but don't forget to explain why it's needed
1083:). 1099:).
1084 1100
1101Note that numerical precision has the same meaning as under Perl (so
1102binary to decimal conversion follows the same rules as in Perl, which
1103can differ to other languages). Also, your perl interpreter might expose
1104extensions to the floating point numbers of your platform, such as
1105infinities or NaN's - these cannot be represented in JSON, and it is an
1106error to pass those in.
1107
1085=back 1108=back
1086 1109
1087 1110
1088=head1 ENCODING/CODESET FLAG NOTES 1111=head1 ENCODING/CODESET FLAG NOTES
1089 1112
… …
1181proper subset of most 8-bit and multibyte encodings in use in the world. 1204proper subset of most 8-bit and multibyte encodings in use in the world.
1182 1205
1183=back 1206=back
1184 1207
1185 1208
1209=head2 JSON and ECMAscript
1210
1211JSON syntax is based on how literals are represented in javascript (the
1212not-standardised predecessor of ECMAscript) which is presumably why it is
1213called "JavaScript Object Notation".
1214
1215However, JSON is not a subset (and also not a superset of course) of
1216ECMAscript (the standard) or javascript (whatever browsers actually
1217implement).
1218
1219If you want to use javascript's C<eval> function to "parse" JSON, you
1220might run into parse errors for valid JSON texts, or the resulting data
1221structure might not be queryable:
1222
1223One of the problems is that U+2028 and U+2029 are valid characters inside
1224JSON strings, but are not allowed in ECMAscript string literals, so the
1225following Perl fragment will not output something that can be guaranteed
1226to be parsable by javascript's C<eval>:
1227
1228 use JSON::XS;
1229
1230 print encode_json [chr 0x2028];
1231
1232The right fix for this is to use a proper JSON parser in your javascript
1233programs, and not rely on C<eval> (see for example Douglas Crockford's
1234F<json2.js> parser).
1235
1236If this is not an option, you can, as a stop-gap measure, simply encode to
1237ASCII-only JSON:
1238
1239 use JSON::XS;
1240
1241 print JSON::XS->new->ascii->encode ([chr 0x2028]);
1242
1243Note that this will enlarge the resulting JSON text quite a bit if you
1244have many non-ASCII characters. You might be tempted to run some regexes
1245to only escape U+2028 and U+2029, e.g.:
1246
1247 # DO NOT USE THIS!
1248 my $json = JSON::XS->new->utf8->encode ([chr 0x2028]);
1249 $json =~ s/\xe2\x80\xa8/\\u2028/g; # escape U+2028
1250 $json =~ s/\xe2\x80\xa9/\\u2029/g; # escape U+2029
1251 print $json;
1252
1253Note that I<this is a bad idea>: the above only works for U+2028 and
1254U+2029 and thus only for fully ECMAscript-compliant parsers. Many existing
1255javascript implementations, however, have issues with other characters as
1256well - using C<eval> naively simply I<will> cause problems.
1257
1258Another problem is that some javascript implementations reserve
1259some property names for their own purposes (which probably makes
1260them non-ECMAscript-compliant). For example, Iceweasel reserves the
1261C<__proto__> property name for its own purposes.
1262
1263If that is a problem, you could parse try to filter the resulting JSON
1264output for these property strings, e.g.:
1265
1266 $json =~ s/"__proto__"\s*:/"__proto__renamed":/g;
1267
1268This works because C<__proto__> is not valid outside of strings, so every
1269occurence of C<"__proto__"\s*:> must be a string used as property name.
1270
1271If you know of other incompatibilities, please let me know.
1272
1273
1186=head2 JSON and YAML 1274=head2 JSON and YAML
1187 1275
1188You often hear that JSON is a subset of YAML. This is, however, a mass 1276You often hear that JSON is a subset of YAML. This is, however, a mass
1189hysteria(*) and very far from the truth (as of the time of this writing), 1277hysteria(*) and very far from the truth (as of the time of this writing),
1190so let me state it clearly: I<in general, there is no way to configure 1278so let me state it clearly: I<in general, there is no way to configure
… …
1198 my $yaml = $to_yaml->encode ($ref) . "\n"; 1286 my $yaml = $to_yaml->encode ($ref) . "\n";
1199 1287
1200This will I<usually> generate JSON texts that also parse as valid 1288This will I<usually> generate JSON texts that also parse as valid
1201YAML. Please note that YAML has hardcoded limits on (simple) object key 1289YAML. Please note that YAML has hardcoded limits on (simple) object key
1202lengths that JSON doesn't have and also has different and incompatible 1290lengths that JSON doesn't have and also has different and incompatible
1203unicode handling, so you should make sure that your hash keys are 1291unicode character escape syntax, so you should make sure that your hash
1204noticeably shorter than the 1024 "stream characters" YAML allows and that 1292keys are noticeably shorter than the 1024 "stream characters" YAML allows
1205you do not have characters with codepoint values outside the Unicode BMP 1293and that you do not have characters with codepoint values outside the
1206(basic multilingual page). YAML also does not allow C<\/> sequences in 1294Unicode BMP (basic multilingual page). YAML also does not allow C<\/>
1207strings (which JSON::XS does not I<currently> generate, but other JSON 1295sequences in strings (which JSON::XS does not I<currently> generate, but
1208generators might). 1296other JSON generators might).
1209 1297
1210There might be other incompatibilities that I am not aware of (or the YAML 1298There might be other incompatibilities that I am not aware of (or the YAML
1211specification has been changed yet again - it does so quite often). In 1299specification has been changed yet again - it does so quite often). In
1212general you should not try to generate YAML with a JSON generator or vice 1300general you should not try to generate YAML with a JSON generator or vice
1213versa, or try to parse JSON with a YAML parser or vice versa: chances are 1301versa, or try to parse JSON with a YAML parser or vice versa: chances are
… …
1232that difficult or long) and finally make YAML compatible to it, and 1320that difficult or long) and finally make YAML compatible to it, and
1233educating users about the changes, instead of spreading lies about the 1321educating users about the changes, instead of spreading lies about the
1234real compatibility for many I<years> and trying to silence people who 1322real compatibility for many I<years> and trying to silence people who
1235point out that it isn't true. 1323point out that it isn't true.
1236 1324
1325Addendum/2009: the YAML 1.2 spec is still incompatible with JSON, even
1326though the incompatibilities have been documented (and are known to Brian)
1327for many years and the spec makes explicit claims that YAML is a superset
1328of JSON. It would be so easy to fix, but apparently, bullying people and
1329corrupting userdata is so much easier.
1330
1237=back 1331=back
1238 1332
1239 1333
1240=head2 SPEED 1334=head2 SPEED
1241 1335
… …
1248a very short single-line JSON string (also available at 1342a very short single-line JSON string (also available at
1249L<http://dist.schmorp.de/misc/json/short.json>). 1343L<http://dist.schmorp.de/misc/json/short.json>).
1250 1344
1251 {"method": "handleMessage", "params": ["user1", 1345 {"method": "handleMessage", "params": ["user1",
1252 "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7, 1346 "we were just talking"], "id": null, "array":[1,11,234,-5,1e5,1e7,
1253 true, false]} 1347 1, 0]}
1254 1348
1255It shows the number of encodes/decodes per second (JSON::XS uses 1349It shows the number of encodes/decodes per second (JSON::XS uses
1256the functional interface, while JSON::XS/2 uses the OO interface 1350the functional interface, while JSON::XS/2 uses the OO interface
1257with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables 1351with pretty-printing and hashkey sorting enabled, JSON::XS/3 enables
1258shrink). Higher is better: 1352shrink. JSON::DWIW/DS uses the deserialise function, while JSON::DWIW::FJ
1353uses the from_json method). Higher is better:
1259 1354
1260 module | encode | decode | 1355 module | encode | decode |
1261 -----------|------------|------------| 1356 --------------|------------|------------|
1262 JSON 1.x | 4990.842 | 4088.813 | 1357 JSON::DWIW/DS | 86302.551 | 102300.098 |
1263 JSON::DWIW | 51653.990 | 71575.154 | 1358 JSON::DWIW/FJ | 86302.551 | 75983.768 |
1264 JSON::PC | 65948.176 | 74631.744 | 1359 JSON::PP | 15827.562 | 6638.658 |
1265 JSON::PP | 8931.652 | 3817.168 | 1360 JSON::Syck | 63358.066 | 47662.545 |
1266 JSON::Syck | 24877.248 | 27776.848 | 1361 JSON::XS | 511500.488 | 511500.488 |
1267 JSON::XS | 388361.481 | 227951.304 | 1362 JSON::XS/2 | 291271.111 | 388361.481 |
1268 JSON::XS/2 | 227951.304 | 218453.333 | 1363 JSON::XS/3 | 361577.931 | 361577.931 |
1269 JSON::XS/3 | 338250.323 | 218453.333 | 1364 Storable | 66788.280 | 265462.278 |
1270 Storable | 16500.016 | 135300.129 |
1271 -----------+------------+------------+ 1365 --------------+------------+------------+
1272 1366
1273That is, JSON::XS is about five times faster than JSON::DWIW on encoding, 1367That is, JSON::XS is almost six times faster than JSON::DWIW on encoding,
1274about three times faster on decoding, and over forty times faster 1368about five times faster on decoding, and over thirty to seventy times
1275than JSON, even with pretty-printing and key sorting. It also compares 1369faster than JSON's pure perl implementation. It also compares favourably
1276favourably to Storable for small amounts of data. 1370to Storable for small amounts of data.
1277 1371
1278Using a longer test string (roughly 18KB, generated from Yahoo! Locals 1372Using a longer test string (roughly 18KB, generated from Yahoo! Locals
1279search API (L<http://dist.schmorp.de/misc/json/long.json>). 1373search API (L<http://dist.schmorp.de/misc/json/long.json>).
1280 1374
1281 module | encode | decode | 1375 module | encode | decode |
1282 -----------|------------|------------| 1376 --------------|------------|------------|
1283 JSON 1.x | 55.260 | 34.971 | 1377 JSON::DWIW/DS | 1647.927 | 2673.916 |
1284 JSON::DWIW | 825.228 | 1082.513 | 1378 JSON::DWIW/FJ | 1630.249 | 2596.128 |
1285 JSON::PC | 3571.444 | 2394.829 |
1286 JSON::PP | 210.987 | 32.574 | 1379 JSON::PP | 400.640 | 62.311 |
1287 JSON::Syck | 552.551 | 787.544 | 1380 JSON::Syck | 1481.040 | 1524.869 |
1288 JSON::XS | 5780.463 | 4854.519 | 1381 JSON::XS | 20661.596 | 9541.183 |
1289 JSON::XS/2 | 3869.998 | 4798.975 | 1382 JSON::XS/2 | 10683.403 | 9416.938 |
1290 JSON::XS/3 | 5862.880 | 4798.975 | 1383 JSON::XS/3 | 20661.596 | 9400.054 |
1291 Storable | 4445.002 | 5235.027 | 1384 Storable | 19765.806 | 10000.725 |
1292 -----------+------------+------------+ 1385 --------------+------------+------------+
1293 1386
1294Again, JSON::XS leads by far (except for Storable which non-surprisingly 1387Again, JSON::XS leads by far (except for Storable which non-surprisingly
1295decodes faster). 1388decodes a bit faster).
1296 1389
1297On large strings containing lots of high Unicode characters, some modules 1390On large strings containing lots of high Unicode characters, some modules
1298(such as JSON::PC) seem to decode faster than JSON::XS, but the result 1391(such as JSON::PC) seem to decode faster than JSON::XS, but the result
1299will be broken due to missing (or wrong) Unicode handling. Others refuse 1392will be broken due to missing (or wrong) Unicode handling. Others refuse
1300to decode or encode properly, so it was impossible to prepare a fair 1393to decode or encode properly, so it was impossible to prepare a fair
… …
1336information you might want to make sure that exceptions thrown by JSON::XS 1429information you might want to make sure that exceptions thrown by JSON::XS
1337will not end up in front of untrusted eyes. 1430will not end up in front of untrusted eyes.
1338 1431
1339If you are using JSON::XS to return packets to consumption 1432If you are using JSON::XS to return packets to consumption
1340by JavaScript scripts in a browser you should have a look at 1433by JavaScript scripts in a browser you should have a look at
1341L<http://jpsykes.com/47/practical-csrf-and-json-security> to see whether 1434L<http://blog.archive.jpsykes.com/47/practical-csrf-and-json-security/> to
1342you are vulnerable to some common attack vectors (which really are browser 1435see whether you are vulnerable to some common attack vectors (which really
1343design bugs, but it is still you who will have to deal with it, as major 1436are browser design bugs, but it is still you who will have to deal with
1344browser developers care only for features, not about getting security 1437it, as major browser developers care only for features, not about getting
1345right). 1438security right).
1346 1439
1347 1440
1348=head1 THREADS 1441=head1 THREADS
1349 1442
1350This module is I<not> guaranteed to be thread safe and there are no 1443This module is I<not> guaranteed to be thread safe and there are no
1351plans to change this until Perl gets thread support (as opposed to the 1444plans to change this until Perl gets thread support (as opposed to the
1352horribly slow so-called "threads" which are simply slow and bloated 1445horribly slow so-called "threads" which are simply slow and bloated
1353process simulations - use fork, it's I<much> faster, cheaper, better). 1446process simulations - use fork, it's I<much> faster, cheaper, better).
1354 1447
1355(It might actually work, but you have been warned). 1448(It might actually work, but you have been warned).
1449
1450
1451=head1 THE PERILS OF SETLOCALE
1452
1453Sometimes people avoid the Perl locale support and directly call the
1454system's setlocale function with C<LC_ALL>.
1455
1456This breaks both perl and modules such as JSON::XS, as stringification of
1457numbers no longer works correcly (e.g. C<$x = 0.1; print "$x"+1> might
1458print C<1>, and JSON::XS might output illegal JSON as JSON::XS relies on
1459perl to stringify numbers).
1460
1461The solution is simple: don't call C<setlocale>, or use it for only those
1462categories you need, such as C<LC_MESSAGES> or C<LC_CTYPE>.
1463
1464If you need C<LC_NUMERIC>, you should enable it only around the code that
1465actually needs it (avoiding stringification of numbers), and restore it
1466afterwards.
1356 1467
1357 1468
1358=head1 BUGS 1469=head1 BUGS
1359 1470
1360While the goal of this module is to be correct, that unfortunately does 1471While the goal of this module is to be correct, that unfortunately does

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines