ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
Revision: 1.6
Committed: Sun Oct 27 20:40:25 2013 UTC (10 years, 6 months ago) by root
Branch: MAIN
Changes since 1.5: +54 -39 lines
Log Message:
*** empty log message ***

File Contents

# Content
1 =head1 NAME
2
3 CBOR::XS - Concise Binary Object Representation (CBOR, RFC7049)
4
5 =encoding utf-8
6
7 =head1 SYNOPSIS
8
9 use CBOR::XS;
10
11 $binary_cbor_data = encode_cbor $perl_value;
12 $perl_value = decode_cbor $binary_cbor_data;
13
14 # OO-interface
15
16 $coder = CBOR::XS->new;
17 $binary_cbor_data = $coder->encode ($perl_value);
18 $perl_value = $coder->decode ($binary_cbor_data);
19
20 # prefix decoding
21
22 my $many_cbor_strings = ...;
23 while (length $many_cbor_strings) {
24 my ($data, $length) = $cbor->decode_prefix ($many_cbor_strings);
25 # data was decoded
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 }
28
29 =head1 DESCRIPTION
30
31 WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA
32 AND EAT YOUR CHILDREN! (Actually, apart from being untested and a bit
33 feature-limited, it might already be useful).
34
35 This module converts Perl data structures to the Concise Binary Object
36 Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
37 format that aims to use a superset of the JSON data model, i.e. when you
38 can represent something in JSON, you should be able to represent it in
39 CBOR.
40
41 This makes it a faster and more compact binary alternative to JSON, with
42 the added ability of supporting serialising of perl objects.
43
44 The primary goal of this module is to be I<correct> and the secondary goal
45 is to be I<fast>. To reach the latter goal it was written in C.
46
47 See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
48 vice versa.
49
50 =cut
51
52 package CBOR::XS;
53
54 use common::sense;
55
56 our $VERSION = 0.03;
57 our @ISA = qw(Exporter);
58
59 our @EXPORT = qw(encode_cbor decode_cbor);
60
61 use Exporter;
62 use XSLoader;
63
64 use Types::Serialiser;
65
66 our $MAGIC = "\xd9\xd9\xf7";
67
68 =head1 FUNCTIONAL INTERFACE
69
70 The following convenience methods are provided by this module. They are
71 exported by default:
72
73 =over 4
74
75 =item $cbor_data = encode_cbor $perl_scalar
76
77 Converts the given Perl data structure to CBOR representation. Croaks on
78 error.
79
80 =item $perl_scalar = decode_cbor $cbor_data
81
82 The opposite of C<encode_cbor>: expects a valid CBOR string to parse,
83 returning the resulting perl scalar. Croaks on error.
84
85 =back
86
87
88 =head1 OBJECT-ORIENTED INTERFACE
89
90 The object oriented interface lets you configure your own encoding or
91 decoding style, within the limits of supported formats.
92
93 =over 4
94
95 =item $cbor = new CBOR::XS
96
97 Creates a new CBOR::XS object that can be used to de/encode CBOR
98 strings. All boolean flags described below are by default I<disabled>.
99
100 The mutators for flags all return the CBOR object again and thus calls can
101 be chained:
102
103 #TODO
104 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
105
106 =item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
107
108 =item $max_depth = $cbor->get_max_depth
109
110 Sets the maximum nesting level (default C<512>) accepted while encoding
111 or decoding. If a higher nesting level is detected in CBOR data or a Perl
112 data structure, then the encoder and decoder will stop and croak at that
113 point.
114
115 Nesting level is defined by number of hash- or arrayrefs that the encoder
116 needs to traverse to reach a given point or the number of C<{> or C<[>
117 characters without their matching closing parenthesis crossed to reach a
118 given character in a string.
119
120 Setting the maximum depth to one disallows any nesting, so that ensures
121 that the object is only a single hash/object or array.
122
123 If no argument is given, the highest possible setting will be used, which
124 is rarely useful.
125
126 Note that nesting is implemented by recursion in C. The default value has
127 been chosen to be as large as typical operating systems allow without
128 crashing.
129
130 See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
131
132 =item $cbor = $cbor->max_size ([$maximum_string_size])
133
134 =item $max_size = $cbor->get_max_size
135
136 Set the maximum length a CBOR string may have (in bytes) where decoding
137 is being attempted. The default is C<0>, meaning no limit. When C<decode>
138 is called on a string that is longer then this many bytes, it will not
139 attempt to decode the string but throw an exception. This setting has no
140 effect on C<encode> (yet).
141
142 If no argument is given, the limit check will be deactivated (same as when
143 C<0> is specified).
144
145 See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
146
147 =item $cbor_data = $cbor->encode ($perl_scalar)
148
149 Converts the given Perl data structure (a scalar value) to its CBOR
150 representation.
151
152 =item $perl_scalar = $cbor->decode ($cbor_data)
153
154 The opposite of C<encode>: expects CBOR data and tries to parse it,
155 returning the resulting simple scalar or reference. Croaks on error.
156
157 =item ($perl_scalar, $octets) = $cbor->decode_prefix ($cbor_data)
158
159 This works like the C<decode> method, but instead of raising an exception
160 when there is trailing garbage after the CBOR string, it will silently
161 stop parsing there and return the number of characters consumed so far.
162
163 This is useful if your CBOR texts are not delimited by an outer protocol
164 and you need to know where the first CBOR string ends amd the next one
165 starts.
166
167 CBOR::XS->new->decode_prefix ("......")
168 => ("...", 3)
169
170 =back
171
172
173 =head1 MAPPING
174
175 This section describes how CBOR::XS maps Perl values to CBOR values and
176 vice versa. These mappings are designed to "do the right thing" in most
177 circumstances automatically, preserving round-tripping characteristics
178 (what you put in comes out as something equivalent).
179
180 For the more enlightened: note that in the following descriptions,
181 lowercase I<perl> refers to the Perl interpreter, while uppercase I<Perl>
182 refers to the abstract Perl language itself.
183
184
185 =head2 CBOR -> PERL
186
187 =over 4
188
189 =item integers
190
191 CBOR integers become (numeric) perl scalars. On perls without 64 bit
192 support, 64 bit integers will be truncated or otherwise corrupted.
193
194 =item byte strings
195
196 Byte strings will become octet strings in Perl (the byte values 0..255
197 will simply become characters of the same value in Perl).
198
199 =item UTF-8 strings
200
201 UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
202 decoded into proper Unicode code points. At the moment, the validity of
203 the UTF-8 octets will not be validated - corrupt input will result in
204 corrupted Perl strings.
205
206 =item arrays, maps
207
208 CBOR arrays and CBOR maps will be converted into references to a Perl
209 array or hash, respectively. The keys of the map will be stringified
210 during this process.
211
212 =item null
213
214 CBOR null becomes C<undef> in Perl.
215
216 =item true, false, undefined
217
218 These CBOR values become C<Types:Serialiser::true>,
219 C<Types:Serialiser::false> and C<Types::Serialiser::error>,
220 respectively. They are overloaded to act almost exactly like the numbers
221 C<1> and C<0> (for true and false) or to throw an exception on access (for
222 error). See the L<Types::Serialiser> manpage for details.
223
224 =item CBOR tag 256 (perl object)
225
226 The tag value C<256> (TODO: pending iana registration) will be used to
227 deserialise a Perl object.
228
229 TODO For this to work, the class must be loaded and must have a
230 C<FROM_CBOR> method. The decoder will then call the C<FROM_CBOR> method
231 with the constructor arguments provided by the C<TO_CBOR> method (see
232 below).
233
234 The C<FROM_CBOR> method must return a single value that will then be used
235 as the deserialised value.
236
237 =item CBOR tag 55799 (magic header)
238
239 The tag 55799 is ignored (this tag implements the magic header).
240
241 =item other CBOR tags
242
243 Tagged items consists of a numeric tag and another CBOR value. Tags not
244 handled internally are currently converted into a L<CBOR::XS::Tagged>
245 object, which is simply a blessed array reference consisting of the
246 numeric tag value followed by the (decoded) CBOR value.
247
248 In the future, support for user-supplied conversions might get added.
249
250 =item anything else
251
252 Anything else (e.g. unsupported simple values) will raise a decoding
253 error.
254
255 =back
256
257
258 =head2 PERL -> CBOR
259
260 The mapping from Perl to CBOR is slightly more difficult, as Perl is a
261 truly typeless language, so we can only guess which CBOR type is meant by
262 a Perl value.
263
264 =over 4
265
266 =item hash references
267
268 Perl hash references become CBOR maps. As there is no inherent ordering in
269 hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
270 order.
271
272 Currently, tied hashes will use the indefinite-length format, while normal
273 hashes will use the fixed-length format.
274
275 =item array references
276
277 Perl array references become fixed-length CBOR arrays.
278
279 =item other references
280
281 Other unblessed references are generally not allowed and will cause an
282 exception to be thrown, except for references to the integers C<0> and
283 C<1>, which get turned into false and true in CBOR.
284
285 =item CBOR::XS::Tagged objects
286
287 Objects of this type must be arrays consisting of a single C<[tag, value]>
288 pair. The (numerical) tag will be encoded as a CBOR tag, the value will be
289 encoded as appropriate for the value.
290
291 =item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
292
293 These special values become CBOR true, CBOR false and CBOR undefined
294 values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
295 if you want.
296
297 =item blessed objects
298
299 Other blessed objects currently need to have a C<TO_CBOR> method. It
300 will be called on every object that is being serialised, and must return
301 something that can be encoded in CBOR.
302
303 =item simple scalars
304
305 TODO
306 Simple Perl scalars (any scalar that is not a reference) are the most
307 difficult objects to encode: CBOR::XS will encode undefined scalars as
308 CBOR null values, scalars that have last been used in a string context
309 before encoding as CBOR strings, and anything else as number value:
310
311 # dump as number
312 encode_cbor [2] # yields [2]
313 encode_cbor [-3.0e17] # yields [-3e+17]
314 my $value = 5; encode_cbor [$value] # yields [5]
315
316 # used as string, so dump as string
317 print $value;
318 encode_cbor [$value] # yields ["5"]
319
320 # undef becomes null
321 encode_cbor [undef] # yields [null]
322
323 You can force the type to be a CBOR string by stringifying it:
324
325 my $x = 3.1; # some variable containing a number
326 "$x"; # stringified
327 $x .= ""; # another, more awkward way to stringify
328 print $x; # perl does it for you, too, quite often
329
330 You can force the type to be a CBOR number by numifying it:
331
332 my $x = "3"; # some variable containing a string
333 $x += 0; # numify it, ensuring it will be dumped as a number
334 $x *= 1; # same thing, the choice is yours.
335
336 You can not currently force the type in other, less obscure, ways. Tell me
337 if you need this capability (but don't forget to explain why it's needed
338 :).
339
340 Perl values that seem to be integers generally use the shortest possible
341 representation. Floating-point values will use either the IEEE single
342 format if possible without loss of precision, otherwise the IEEE double
343 format will be used. Perls that use formats other than IEEE double to
344 represent numerical values are supported, but might suffer loss of
345 precision.
346
347 =back
348
349
350 =head2 MAGIC HEADER
351
352 There is no way to distinguish CBOR from other formats
353 programmatically. To make it easier to distinguish CBOR from other
354 formats, the CBOR specification has a special "magic string" that can be
355 prepended to any CBOR string without changing it's meaning.
356
357 This string is available as C<$CBOR::XS::MAGIC>. This module does not
358 prepend this string tot he CBOR data it generates, but it will ignroe it
359 if present, so users can prepend this string as a "file type" indicator as
360 required.
361
362
363 =head2 CBOR and JSON
364
365 CBOR is supposed to implement a superset of the JSON data model, and is,
366 with some coercion, able to represent all JSON texts (something that other
367 "binary JSON" formats such as BSON generally do not support).
368
369 CBOR implements some extra hints and support for JSON interoperability,
370 and the spec offers further guidance for conversion between CBOR and
371 JSON. None of this is currently implemented in CBOR, and the guidelines
372 in the spec do not result in correct round-tripping of data. If JSON
373 interoperability is improved in the future, then the goal will be to
374 ensure that decoded JSON data will round-trip encoding and decoding to
375 CBOR intact.
376
377
378 =head1 SECURITY CONSIDERATIONS
379
380 When you are using CBOR in a protocol, talking to untrusted potentially
381 hostile creatures requires relatively few measures.
382
383 First of all, your CBOR decoder should be secure, that is, should not have
384 any buffer overflows. Obviously, this module should ensure that and I am
385 trying hard on making that true, but you never know.
386
387 Second, you need to avoid resource-starving attacks. That means you should
388 limit the size of CBOR data you accept, or make sure then when your
389 resources run out, that's just fine (e.g. by using a separate process that
390 can crash safely). The size of a CBOR string in octets is usually a good
391 indication of the size of the resources required to decode it into a Perl
392 structure. While CBOR::XS can check the size of the CBOR text, it might be
393 too late when you already have it in memory, so you might want to check
394 the size before you accept the string.
395
396 Third, CBOR::XS recurses using the C stack when decoding objects and
397 arrays. The C stack is a limited resource: for instance, on my amd64
398 machine with 8MB of stack size I can decode around 180k nested arrays but
399 only 14k nested CBOR objects (due to perl itself recursing deeply on croak
400 to free the temporary). If that is exceeded, the program crashes. To be
401 conservative, the default nesting limit is set to 512. If your process
402 has a smaller stack, you should adjust this setting accordingly with the
403 C<max_depth> method.
404
405 Something else could bomb you, too, that I forgot to think of. In that
406 case, you get to keep the pieces. I am always open for hints, though...
407
408 Also keep in mind that CBOR::XS might leak contents of your Perl data
409 structures in its error messages, so when you serialise sensitive
410 information you might want to make sure that exceptions thrown by CBOR::XS
411 will not end up in front of untrusted eyes.
412
413 =head1 CBOR IMPLEMENTATION NOTES
414
415 This section contains some random implementation notes. They do not
416 describe guaranteed behaviour, but merely behaviour as-is implemented
417 right now.
418
419 64 bit integers are only properly decoded when Perl was built with 64 bit
420 support.
421
422 Strings and arrays are encoded with a definite length. Hashes as well,
423 unless they are tied (or otherwise magical).
424
425 Only the double data type is supported for NV data types - when Perl uses
426 long double to represent floating point values, they might not be encoded
427 properly. Half precision types are accepted, but not encoded.
428
429 Strict mode and canonical mode are not implemented.
430
431
432 =head1 THREADS
433
434 This module is I<not> guaranteed to be thread safe and there are no
435 plans to change this until Perl gets thread support (as opposed to the
436 horribly slow so-called "threads" which are simply slow and bloated
437 process simulations - use fork, it's I<much> faster, cheaper, better).
438
439 (It might actually work, but you have been warned).
440
441
442 =head1 BUGS
443
444 While the goal of this module is to be correct, that unfortunately does
445 not mean it's bug-free, only that I think its design is bug-free. If you
446 keep reporting bugs they will be fixed swiftly, though.
447
448 Please refrain from using rt.cpan.org or any other bug reporting
449 service. I put the contact address into my modules for a reason.
450
451 =cut
452
453 XSLoader::load "CBOR::XS", $VERSION;
454
455 =head1 SEE ALSO
456
457 The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,
458 serialisation.
459
460 The L<Types::Serialiser> module provides the data model for true, false
461 and error values.
462
463 =head1 AUTHOR
464
465 Marc Lehmann <schmorp@schmorp.de>
466 http://home.schmorp.de/
467
468 =cut
469
470 1
471