ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/JSON-XS/XS.pm
(Generate patch)

Comparing JSON-XS/XS.pm (file contents):
Revision 1.83 by root, Sun Jan 20 19:19:07 2008 UTC vs.
Revision 1.90 by root, Wed Mar 19 22:28:43 2008 UTC

1=encoding utf-8
2
1=head1 NAME 3=head1 NAME
2 4
3JSON::XS - JSON serialising/deserialising, done correctly and fast 5JSON::XS - JSON serialising/deserialising, done correctly and fast
4 6
5JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ 7JSON::XS - 正しくて高速な JSON シリアライザ/デシリアライザ
58 60
59=over 4 61=over 4
60 62
61=item * correct Unicode handling 63=item * correct Unicode handling
62 64
63This module knows how to handle Unicode, and even documents how and when 65This module knows how to handle Unicode, documents how and when it does
64it does so. 66so, and even documents what "correct" means.
65 67
66=item * round-trip integrity 68=item * round-trip integrity
67 69
68When you serialise a perl data structure using only datatypes supported 70When you serialise a perl data structure using only datatypes supported
69by JSON, the deserialised data structure is identical on the Perl level. 71by JSON, the deserialised data structure is identical on the Perl level.
70(e.g. the string "2.0" doesn't suddenly become "2" just because it looks 72(e.g. the string "2.0" doesn't suddenly become "2" just because it looks
71like a number). 73like a number). There minor I<are> exceptions to this, read the MAPPING
74section below to learn about those.
72 75
73=item * strict checking of JSON correctness 76=item * strict checking of JSON correctness
74 77
75There is no guessing, no generating of illegal JSON texts by default, 78There is no guessing, no generating of illegal JSON texts by default,
76and only JSON is accepted as input by default (the latter is a security 79and only JSON is accepted as input by default (the latter is a security
77feature). 80feature).
78 81
79=item * fast 82=item * fast
80 83
81Compared to other JSON modules, this module compares favourably in terms 84Compared to other JSON modules and other serialisers such as Storable,
82of speed, too. 85this module usually compares favourably in terms of speed, too.
83 86
84=item * simple to use 87=item * simple to use
85 88
86This module has both a simple functional interface as well as an OO 89This module has both a simple functional interface as well as an objetc
87interface. 90oriented interface interface.
88 91
89=item * reasonably versatile output formats 92=item * reasonably versatile output formats
90 93
91You can choose between the most compact guaranteed single-line format 94You can choose between the most compact guaranteed-single-line format
92possible (nice for simple line-based protocols), a pure-ascii format 95possible (nice for simple line-based protocols), a pure-ascii format
93(for when your transport is not 8-bit clean, still supports the whole 96(for when your transport is not 8-bit clean, still supports the whole
94Unicode range), or a pretty-printed format (for when you want to read that 97Unicode range), or a pretty-printed format (for when you want to read that
95stuff). Or you can combine those features in whatever way you like. 98stuff). Or you can combine those features in whatever way you like.
96 99
100 103
101package JSON::XS; 104package JSON::XS;
102 105
103use strict; 106use strict;
104 107
105our $VERSION = '2.01'; 108our $VERSION = '2.1';
106our @ISA = qw(Exporter); 109our @ISA = qw(Exporter);
107 110
108our @EXPORT = qw(encode_json decode_json to_json from_json); 111our @EXPORT = qw(encode_json decode_json to_json from_json);
109 112
110sub to_json($) { 113sub to_json($) {
174This enables you to store Unicode characters as single characters in a 177This enables you to store Unicode characters as single characters in a
175Perl string - very natural. 178Perl string - very natural.
176 179
177=item 2. Perl does I<not> associate an encoding with your strings. 180=item 2. Perl does I<not> associate an encoding with your strings.
178 181
179Unless you force it to, e.g. when matching it against a regex, or printing 182... until you force it to, e.g. when matching it against a regex, or
180the scalar to a file, in which case Perl either interprets your string as 183printing the scalar to a file, in which case Perl either interprets your
181locale-encoded text, octets/binary, or as Unicode, depending on various 184string as locale-encoded text, octets/binary, or as Unicode, depending
182settings. In no case is an encoding stored together with your data, it is 185on various settings. In no case is an encoding stored together with your
183I<use> that decides encoding, not any magical metadata. 186data, it is I<use> that decides encoding, not any magical meta data.
184 187
185=item 3. The internal utf-8 flag has no meaning with regards to the 188=item 3. The internal utf-8 flag has no meaning with regards to the
186encoding of your string. 189encoding of your string.
187 190
188Just ignore that flag unless you debug a Perl bug, a module written in 191Just ignore that flag unless you debug a Perl bug, a module written in
242 245
243If C<$enable> is false, then the C<encode> method will not escape Unicode 246If C<$enable> is false, then the C<encode> method will not escape Unicode
244characters unless required by the JSON syntax or other flags. This results 247characters unless required by the JSON syntax or other flags. This results
245in a faster and more compact format. 248in a faster and more compact format.
246 249
250See also the section I<ENCODING/CODESET FLAG NOTES> later in this
251document.
252
247The main use for this flag is to produce JSON texts that can be 253The main use for this flag is to produce JSON texts that can be
248transmitted over a 7-bit channel, as the encoded JSON texts will not 254transmitted over a 7-bit channel, as the encoded JSON texts will not
249contain any 8 bit characters. 255contain any 8 bit characters.
250 256
251 JSON::XS->new->ascii (1)->encode ([chr 0x10401]) 257 JSON::XS->new->ascii (1)->encode ([chr 0x10401])
262will not be affected in any way by this flag, as C<decode> by default 268will not be affected in any way by this flag, as C<decode> by default
263expects Unicode, which is a strict superset of latin1. 269expects Unicode, which is a strict superset of latin1.
264 270
265If C<$enable> is false, then the C<encode> method will not escape Unicode 271If C<$enable> is false, then the C<encode> method will not escape Unicode
266characters unless required by the JSON syntax or other flags. 272characters unless required by the JSON syntax or other flags.
273
274See also the section I<ENCODING/CODESET FLAG NOTES> later in this
275document.
267 276
268The main use for this flag is efficiently encoding binary data as JSON 277The main use for this flag is efficiently encoding binary data as JSON
269text, as most octets will not be escaped, resulting in a smaller encoded 278text, as most octets will not be escaped, resulting in a smaller encoded
270size. The disadvantage is that the resulting JSON text is encoded 279size. The disadvantage is that the resulting JSON text is encoded
271in latin1 (and must correctly be treated as such when storing and 280in latin1 (and must correctly be treated as such when storing and
290 299
291If C<$enable> is false, then the C<encode> method will return the JSON 300If C<$enable> is false, then the C<encode> method will return the JSON
292string as a (non-encoded) Unicode string, while C<decode> expects thus a 301string as a (non-encoded) Unicode string, while C<decode> expects thus a
293Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs 302Unicode string. Any decoding or encoding (e.g. to UTF-8 or UTF-16) needs
294to be done yourself, e.g. using the Encode module. 303to be done yourself, e.g. using the Encode module.
304
305See also the section I<ENCODING/CODESET FLAG NOTES> later in this
306document.
295 307
296Example, output UTF-16BE-encoded JSON: 308Example, output UTF-16BE-encoded JSON:
297 309
298 use Encode; 310 use Encode;
299 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object); 311 $jsontext = encode "UTF-16BE", JSON::XS->new->encode ($object);
706 718
707A JSON number becomes either an integer, numeric (floating point) or 719A JSON number becomes either an integer, numeric (floating point) or
708string scalar in perl, depending on its range and any fractional parts. On 720string scalar in perl, depending on its range and any fractional parts. On
709the Perl level, there is no difference between those as Perl handles all 721the Perl level, there is no difference between those as Perl handles all
710the conversion details, but an integer may take slightly less memory and 722the conversion details, but an integer may take slightly less memory and
711might represent more values exactly than (floating point) numbers. 723might represent more values exactly than floating point numbers.
712 724
713If the number consists of digits only, JSON::XS will try to represent 725If the number consists of digits only, JSON::XS will try to represent
714it as an integer value. If that fails, it will try to represent it as 726it as an integer value. If that fails, it will try to represent it as
715a numeric (floating point) value if that is possible without loss of 727a numeric (floating point) value if that is possible without loss of
716precision. Otherwise it will preserve the number as a string value. 728precision. Otherwise it will preserve the number as a string value (in
729which case you lose roundtripping ability, as the JSON number will be
730re-encoded toa JSON string).
717 731
718Numbers containing a fractional or exponential part will always be 732Numbers containing a fractional or exponential part will always be
719represented as numeric (floating point) values, possibly at a loss of 733represented as numeric (floating point) values, possibly at a loss of
720precision. 734precision (in which case you might lose perfect roundtripping ability, but
721 735the JSON number will still be re-encoded as a JSON number).
722This might create round-tripping problems as numbers might become strings,
723but as Perl is typeless there is no other way to do it.
724 736
725=item true, false 737=item true, false
726 738
727These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>, 739These JSON atoms become C<JSON::XS::true> and C<JSON::XS::false>,
728respectively. They are overloaded to act almost exactly like the numbers 740respectively. They are overloaded to act almost exactly like the numbers
819:). 831:).
820 832
821=back 833=back
822 834
823 835
836=head1 ENCODING/CODESET FLAG NOTES
837
838The interested reader might have seen a number of flags that signify
839encodings or codesets - C<utf8>, C<latin1> and C<ascii>. There seems to be
840some confusion on what these do, so here is a short comparison:
841
842C<utf8> controls wether the JSON text created by C<encode> (and expected
843by C<decode>) is UTF-8 encoded or not, while C<latin1> and C<ascii> only
844control wether C<encode> escapes character values outside their respective
845codeset range. Neither of these flags conflict with each other, although
846some combinations make less sense than others.
847
848Care has been taken to make all flags symmetrical with respect to
849C<encode> and C<decode>, that is, texts encoded with any combination of
850these flag values will be correctly decoded when the same flags are used
851- in general, if you use different flag settings while encoding vs. when
852decoding you likely have a bug somewhere.
853
854Below comes a verbose discussion of these flags. Note that a "codeset" is
855simply an abstract set of character-codepoint pairs, while an encoding
856takes those codepoint numbers and I<encodes> them, in our case into
857octets. Unicode is (among other things) a codeset, UTF-8 is an encoding,
858and ISO-8859-1 (= latin 1) and ASCII are both codesets I<and> encodings at
859the same time, which can be confusing.
860
861=over 4
862
863=item C<utf8> flag disabled
864
865When C<utf8> is disabled (the default), then C<encode>/C<decode> generate
866and expect Unicode strings, that is, characters with high ordinal Unicode
867values (> 255) will be encoded as such characters, and likewise such
868characters are decoded as-is, no canges to them will be done, except
869"(re-)interpreting" them as Unicode codepoints or Unicode characters,
870respectively (to Perl, these are the same thing in strings unless you do
871funny/weird/dumb stuff).
872
873This is useful when you want to do the encoding yourself (e.g. when you
874want to have UTF-16 encoded JSON texts) or when some other layer does
875the encoding for you (for example, when printing to a terminal using a
876filehandle that transparently encodes to UTF-8 you certainly do NOT want
877to UTF-8 encode your data first and have Perl encode it another time).
878
879=item C<utf8> flag enabled
880
881If the C<utf8>-flag is enabled, C<encode>/C<decode> will encode all
882characters using the corresponding UTF-8 multi-byte sequence, and will
883expect your input strings to be encoded as UTF-8, that is, no "character"
884of the input string must have any value > 255, as UTF-8 does not allow
885that.
886
887The C<utf8> flag therefore switches between two modes: disabled means you
888will get a Unicode string in Perl, enabled means you get an UTF-8 encoded
889octet/binary string in Perl.
890
891=item C<latin1> or C<ascii> flags enabled
892
893With C<latin1> (or C<ascii>) enabled, C<encode> will escape characters
894with ordinal values > 255 (> 127 with C<ascii>) and encode the remaining
895characters as specified by the C<utf8> flag.
896
897If C<utf8> is disabled, then the result is also correctly encoded in those
898character sets (as both are proper subsets of Unicode, meaning that a
899Unicode string with all character values < 256 is the same thing as a
900ISO-8859-1 string, and a Unicode string with all character values < 128 is
901the same thing as an ASCII string in Perl).
902
903If C<utf8> is enabled, you still get a correct UTF-8-encoded string,
904regardless of these flags, just some more characters will be escaped using
905C<\uXXXX> then before.
906
907Note that ISO-8859-1-I<encoded> strings are not compatible with UTF-8
908encoding, while ASCII-encoded strings are. That is because the ISO-8859-1
909encoding is NOT a subset of UTF-8 (despite the ISO-8859-1 I<codeset> being
910a subset of Unicode), while ASCII is.
911
912Surprisingly, C<decode> will ignore these flags and so treat all input
913values as governed by the C<utf8> flag. If it is disabled, this allows you
914to decode ISO-8859-1- and ASCII-encoded strings, as both strict subsets of
915Unicode. If it is enabled, you can correctly decode UTF-8 encoded strings.
916
917So neither C<latin1> nor C<ascii> are incompatible with the C<utf8> flag -
918they only govern when the JSON output engine escapes a character or not.
919
920The main use for C<latin1> is to relatively efficiently store binary data
921as JSON, at the expense of breaking compatibility with most JSON decoders.
922
923The main use for C<ascii> is to force the output to not contain characters
924with values > 127, which means you can interpret the resulting string
925as UTF-8, ISO-8859-1, ASCII, KOI8-R or most about any character set and
9268-bit-encoding, and still get the same data structure back. This is useful
927when your channel for JSON transfer is not 8-bit clean or the encoding
928might be mangled in between (e.g. in mail), and works because ASCII is a
929proper subset of most 8-bit and multibyte encodings in use in the world.
930
931=back
932
933
824=head1 COMPARISON 934=head1 COMPARISON
825 935
826As already mentioned, this module was created because none of the existing 936As already mentioned, this module was created because none of the existing
827JSON modules could be made to work correctly. First I will describe the 937JSON modules could be made to work correctly. First I will describe the
828problems (or pleasures) I encountered with various existing JSON modules, 938problems (or pleasures) I encountered with various existing JSON modules,
829followed by some benchmark values. JSON::XS was designed not to suffer 939followed by some benchmark values. JSON::XS was designed not to suffer
830from any of these problems or limitations. 940from any of these problems or limitations.
831 941
832=over 4 942=over 4
943
944=item JSON 2.xx
945
946A marvellous piece of engineering, this module either uses JSON::XS
947directly when available (so will be 100% compatible with it, including
948speed), or it uses JSON::PP, which is basically JSON::XS translated to
949Pure Perl, which should be 100% compatible with JSON::XS, just a bit
950slower.
951
952You cannot really lose by using this module, especially as it tries very
953hard to work even with ancient Perl versions, while JSON::XS does not.
833 954
834=item JSON 1.07 955=item JSON 1.07
835 956
836Slow (but very portable, as it is written in pure Perl). 957Slow (but very portable, as it is written in pure Perl).
837 958
909 1030
910 1031
911=head2 JSON and YAML 1032=head2 JSON and YAML
912 1033
913You often hear that JSON is a subset of YAML. This is, however, a mass 1034You often hear that JSON is a subset of YAML. This is, however, a mass
914hysteria(*) and very far from the truth. In general, there is no way to 1035hysteria(*) and very far from the truth (as of the time of this writing),
1036so let me state it clearly: I<in general, there is no way to configure
915configure JSON::XS to output a data structure as valid YAML that works for 1037JSON::XS to output a data structure as valid YAML> that works in all
916all cases. 1038cases.
917 1039
918If you really must use JSON::XS to generate YAML, you should use this 1040If you really must use JSON::XS to generate YAML, you should use this
919algorithm (subject to change in future versions): 1041algorithm (subject to change in future versions):
920 1042
921 my $to_yaml = JSON::XS->new->utf8->space_after (1); 1043 my $to_yaml = JSON::XS->new->utf8->space_after (1);
924This will I<usually> generate JSON texts that also parse as valid 1046This will I<usually> generate JSON texts that also parse as valid
925YAML. Please note that YAML has hardcoded limits on (simple) object key 1047YAML. Please note that YAML has hardcoded limits on (simple) object key
926lengths that JSON doesn't have and also has different and incompatible 1048lengths that JSON doesn't have and also has different and incompatible
927unicode handling, so you should make sure that your hash keys are 1049unicode handling, so you should make sure that your hash keys are
928noticeably shorter than the 1024 "stream characters" YAML allows and that 1050noticeably shorter than the 1024 "stream characters" YAML allows and that
929you do not have codepoints with values outside the Unicode BMP (basic 1051you do not have characters with codepoint values outside the Unicode BMP
930multilingual page). YAML also does not allow C<\/> sequences in strings 1052(basic multilingual page). YAML also does not allow C<\/> sequences in
931(which JSON::XS does not I<currently> generate). 1053strings (which JSON::XS does not I<currently> generate, but other JSON
1054generators might).
932 1055
933There might be other incompatibilities that I am not aware of (or the YAML 1056There might be other incompatibilities that I am not aware of (or the YAML
934specification has been changed yet again - it does so quite often). In 1057specification has been changed yet again - it does so quite often). In
935general you should not try to generate YAML with a JSON generator or vice 1058general you should not try to generate YAML with a JSON generator or vice
936versa, or try to parse JSON with a YAML parser or vice versa: chances are 1059versa, or try to parse JSON with a YAML parser or vice versa: chances are
939 1062
940=over 4 1063=over 4
941 1064
942=item (*) 1065=item (*)
943 1066
944This is spread actively by the YAML team, however. For many years now they 1067I have been pressured multiple times by Brian Ingerson (one of the
945claim YAML were a superset of JSON, even when proven otherwise. 1068authors of the YAML specification) to remove this paragraph, despite him
1069acknowledging that the actual incompatibilities exist. As I was personally
1070bitten by this "JSON is YAML" lie, I refused and said I will continue to
1071educate people about these issues, so others do not run into the same
1072problem again and again. After this, Brian called me a (quote)I<complete
1073and worthless idiot>(unquote).
946 1074
947Even the author of this manpage was at some point accused of providing 1075In my opinion, instead of pressuring and insulting people who actually
948"incorrect" information, despite the evidence presented (claims ranged 1076clarify issues with YAML and the wrong statements of some of its
949from "your documentation contains inaccurate and negative statements about 1077proponents, I would kindly suggest reading the JSON spec (which is not
950YAML" (the only negative comment is this footnote, and it didn't exist 1078that difficult or long) and finally make YAML compatible to it, and
951back then; the question on which claims were inaccurate was never answered 1079educating users about the changes, instead of spreading lies about the
952etc.) to "the YAML spec is not up-to-date" (the *real* and supposedly 1080real compatibility for many I<years> and trying to silence people who
953JSON-compatible spec is apparently not currently publicly available) 1081point out that it isn't true.
954to actual requests to replace this section by *incorrect* information,
955suppressing information about the real problem).
956
957So whenever you are told that YAML was a superset of JSON, first check
958wether it is really true (it might be when you check it, but it certainly
959was not true when this was written). I would much prefer if the YAML team
960would spent their time on actually making JSON compatibility a truth
961(JSON, after all, has a very small and simple specification) instead of
962trying to lobby/force people into reporting untruths.
963 1082
964=back 1083=back
965 1084
966 1085
967=head2 SPEED 1086=head2 SPEED
969It seems that JSON::XS is surprisingly fast, as shown in the following 1088It seems that JSON::XS is surprisingly fast, as shown in the following
970tables. They have been generated with the help of the C<eg/bench> program 1089tables. They have been generated with the help of the C<eg/bench> program
971in the JSON::XS distribution, to make it easy to compare on your own 1090in the JSON::XS distribution, to make it easy to compare on your own
972system. 1091system.
973 1092
974First comes a comparison between various modules using a very short 1093First comes a comparison between various modules using
975single-line JSON string: 1094a very short single-line JSON string (also available at
1095L<http://dist.schmorp.de/misc/json/short.json>).
976 1096
977 {"method": "handleMessage", "params": ["user1", "we were just talking"], \ 1097 {"method": "handleMessage", "params": ["user1", "we were just talking"], \
978 "id": null, "array":[1,11,234,-5,1e5,1e7, true, false]} 1098 "id": null, "array":[1,11,234,-5,1e5,1e7, true, false]}
979 1099
980It shows the number of encodes/decodes per second (JSON::XS uses 1100It shows the number of encodes/decodes per second (JSON::XS uses
999about three times faster on decoding, and over forty times faster 1119about three times faster on decoding, and over forty times faster
1000than JSON, even with pretty-printing and key sorting. It also compares 1120than JSON, even with pretty-printing and key sorting. It also compares
1001favourably to Storable for small amounts of data. 1121favourably to Storable for small amounts of data.
1002 1122
1003Using a longer test string (roughly 18KB, generated from Yahoo! Locals 1123Using a longer test string (roughly 18KB, generated from Yahoo! Locals
1004search API (http://nanoref.com/yahooapis/mgPdGg): 1124search API (L<http://dist.schmorp.de/misc/json/long.json>).
1005 1125
1006 module | encode | decode | 1126 module | encode | decode |
1007 -----------|------------|------------| 1127 -----------|------------|------------|
1008 JSON 1.x | 55.260 | 34.971 | 1128 JSON 1.x | 55.260 | 34.971 |
1009 JSON::DWIW | 825.228 | 1082.513 | 1129 JSON::DWIW | 825.228 | 1082.513 |
1051to free the temporary). If that is exceeded, the program crashes. To be 1171to free the temporary). If that is exceeded, the program crashes. To be
1052conservative, the default nesting limit is set to 512. If your process 1172conservative, the default nesting limit is set to 512. If your process
1053has a smaller stack, you should adjust this setting accordingly with the 1173has a smaller stack, you should adjust this setting accordingly with the
1054C<max_depth> method. 1174C<max_depth> method.
1055 1175
1056And last but least, something else could bomb you that I forgot to think 1176Something else could bomb you, too, that I forgot to think of. In that
1057of. In that case, you get to keep the pieces. I am always open for hints, 1177case, you get to keep the pieces. I am always open for hints, though...
1058though... 1178
1179Also keep in mind that JSON::XS might leak contents of your Perl data
1180structures in its error messages, so when you serialise sensitive
1181information you might want to make sure that exceptions thrown by JSON::XS
1182will not end up in front of untrusted eyes.
1059 1183
1060If you are using JSON::XS to return packets to consumption 1184If you are using JSON::XS to return packets to consumption
1061by JavaScript scripts in a browser you should have a look at 1185by JavaScript scripts in a browser you should have a look at
1062L<http://jpsykes.com/47/practical-csrf-and-json-security> to see whether 1186L<http://jpsykes.com/47/practical-csrf-and-json-security> to see whether
1063you are vulnerable to some common attack vectors (which really are browser 1187you are vulnerable to some common attack vectors (which really are browser

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines