ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/README
Revision: 1.5
Committed: Sun Oct 27 22:48:12 2013 UTC (10 years, 6 months ago) by root
Branch: MAIN
CVS Tags: rel-0_04
Changes since 1.4: +148 -27 lines
Log Message:
0.04

File Contents

# Content
1 NAME
2 CBOR::XS - Concise Binary Object Representation (CBOR, RFC7049)
3
4 SYNOPSIS
5 use CBOR::XS;
6
7 $binary_cbor_data = encode_cbor $perl_value;
8 $perl_value = decode_cbor $binary_cbor_data;
9
10 # OO-interface
11
12 $coder = CBOR::XS->new;
13 $binary_cbor_data = $coder->encode ($perl_value);
14 $perl_value = $coder->decode ($binary_cbor_data);
15
16 # prefix decoding
17
18 my $many_cbor_strings = ...;
19 while (length $many_cbor_strings) {
20 my ($data, $length) = $cbor->decode_prefix ($many_cbor_strings);
21 # data was decoded
22 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
23 }
24
25 DESCRIPTION
26 WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA
27 AND EAT YOUR CHILDREN! (Actually, apart from being untested and a bit
28 feature-limited, it might already be useful).
29
30 This module converts Perl data structures to the Concise Binary Object
31 Representation (CBOR) and vice versa. CBOR is a fast binary
32 serialisation format that aims to use a superset of the JSON data model,
33 i.e. when you can represent something in JSON, you should be able to
34 represent it in CBOR.
35
36 This makes it a faster and more compact binary alternative to JSON, with
37 the added ability of supporting serialising of perl objects.
38
39 The primary goal of this module is to be *correct* and the secondary
40 goal is to be *fast*. To reach the latter goal it was written in C.
41
42 See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
43 vice versa.
44
45 FUNCTIONAL INTERFACE
46 The following convenience methods are provided by this module. They are
47 exported by default:
48
49 $cbor_data = encode_cbor $perl_scalar
50 Converts the given Perl data structure to CBOR representation.
51 Croaks on error.
52
53 $perl_scalar = decode_cbor $cbor_data
54 The opposite of "encode_cbor": expects a valid CBOR string to parse,
55 returning the resulting perl scalar. Croaks on error.
56
57 OBJECT-ORIENTED INTERFACE
58 The object oriented interface lets you configure your own encoding or
59 decoding style, within the limits of supported formats.
60
61 $cbor = new CBOR::XS
62 Creates a new CBOR::XS object that can be used to de/encode CBOR
63 strings. All boolean flags described below are by default
64 *disabled*.
65
66 The mutators for flags all return the CBOR object again and thus
67 calls can be chained:
68
69 #TODO my $cbor = CBOR::XS->new->encode ({a => [1,2]});
70
71 $cbor = $cbor->max_depth ([$maximum_nesting_depth])
72 $max_depth = $cbor->get_max_depth
73 Sets the maximum nesting level (default 512) accepted while encoding
74 or decoding. If a higher nesting level is detected in CBOR data or a
75 Perl data structure, then the encoder and decoder will stop and
76 croak at that point.
77
78 Nesting level is defined by number of hash- or arrayrefs that the
79 encoder needs to traverse to reach a given point or the number of
80 "{" or "[" characters without their matching closing parenthesis
81 crossed to reach a given character in a string.
82
83 Setting the maximum depth to one disallows any nesting, so that
84 ensures that the object is only a single hash/object or array.
85
86 If no argument is given, the highest possible setting will be used,
87 which is rarely useful.
88
89 Note that nesting is implemented by recursion in C. The default
90 value has been chosen to be as large as typical operating systems
91 allow without crashing.
92
93 See SECURITY CONSIDERATIONS, below, for more info on why this is
94 useful.
95
96 $cbor = $cbor->max_size ([$maximum_string_size])
97 $max_size = $cbor->get_max_size
98 Set the maximum length a CBOR string may have (in bytes) where
99 decoding is being attempted. The default is 0, meaning no limit.
100 When "decode" is called on a string that is longer then this many
101 bytes, it will not attempt to decode the string but throw an
102 exception. This setting has no effect on "encode" (yet).
103
104 If no argument is given, the limit check will be deactivated (same
105 as when 0 is specified).
106
107 See SECURITY CONSIDERATIONS, below, for more info on why this is
108 useful.
109
110 $cbor_data = $cbor->encode ($perl_scalar)
111 Converts the given Perl data structure (a scalar value) to its CBOR
112 representation.
113
114 $perl_scalar = $cbor->decode ($cbor_data)
115 The opposite of "encode": expects CBOR data and tries to parse it,
116 returning the resulting simple scalar or reference. Croaks on error.
117
118 ($perl_scalar, $octets) = $cbor->decode_prefix ($cbor_data)
119 This works like the "decode" method, but instead of raising an
120 exception when there is trailing garbage after the CBOR string, it
121 will silently stop parsing there and return the number of characters
122 consumed so far.
123
124 This is useful if your CBOR texts are not delimited by an outer
125 protocol and you need to know where the first CBOR string ends amd
126 the next one starts.
127
128 CBOR::XS->new->decode_prefix ("......")
129 => ("...", 3)
130
131 MAPPING
132 This section describes how CBOR::XS maps Perl values to CBOR values and
133 vice versa. These mappings are designed to "do the right thing" in most
134 circumstances automatically, preserving round-tripping characteristics
135 (what you put in comes out as something equivalent).
136
137 For the more enlightened: note that in the following descriptions,
138 lowercase *perl* refers to the Perl interpreter, while uppercase *Perl*
139 refers to the abstract Perl language itself.
140
141 CBOR -> PERL
142 integers
143 CBOR integers become (numeric) perl scalars. On perls without 64 bit
144 support, 64 bit integers will be truncated or otherwise corrupted.
145
146 byte strings
147 Byte strings will become octet strings in Perl (the byte values
148 0..255 will simply become characters of the same value in Perl).
149
150 UTF-8 strings
151 UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
152 decoded into proper Unicode code points. At the moment, the validity
153 of the UTF-8 octets will not be validated - corrupt input will
154 result in corrupted Perl strings.
155
156 arrays, maps
157 CBOR arrays and CBOR maps will be converted into references to a
158 Perl array or hash, respectively. The keys of the map will be
159 stringified during this process.
160
161 null
162 CBOR null becomes "undef" in Perl.
163
164 true, false, undefined
165 These CBOR values become "Types:Serialiser::true",
166 "Types:Serialiser::false" and "Types::Serialiser::error",
167 respectively. They are overloaded to act almost exactly like the
168 numbers 1 and 0 (for true and false) or to throw an exception on
169 access (for error). See the Types::Serialiser manpage for details.
170
171 CBOR tag 256 (perl object)
172 The tag value 256 (TODO: pending iana registration) will be used to
173 deserialise a Perl object serialised with "FREEZE". See "OBJECT
174 SERIALISATION", below, for details.
175
176 CBOR tag 55799 (magic header)
177 The tag 55799 is ignored (this tag implements the magic header).
178
179 other CBOR tags
180 Tagged items consists of a numeric tag and another CBOR value. Tags
181 not handled internally are currently converted into a
182 CBOR::XS::Tagged object, which is simply a blessed array reference
183 consisting of the numeric tag value followed by the (decoded) CBOR
184 value.
185
186 In the future, support for user-supplied conversions might get
187 added.
188
189 anything else
190 Anything else (e.g. unsupported simple values) will raise a decoding
191 error.
192
193 PERL -> CBOR
194 The mapping from Perl to CBOR is slightly more difficult, as Perl is a
195 truly typeless language, so we can only guess which CBOR type is meant
196 by a Perl value.
197
198 hash references
199 Perl hash references become CBOR maps. As there is no inherent
200 ordering in hash keys (or CBOR maps), they will usually be encoded
201 in a pseudo-random order.
202
203 Currently, tied hashes will use the indefinite-length format, while
204 normal hashes will use the fixed-length format.
205
206 array references
207 Perl array references become fixed-length CBOR arrays.
208
209 other references
210 Other unblessed references are generally not allowed and will cause
211 an exception to be thrown, except for references to the integers 0
212 and 1, which get turned into false and true in CBOR.
213
214 CBOR::XS::Tagged objects
215 Objects of this type must be arrays consisting of a single "[tag,
216 value]" pair. The (numerical) tag will be encoded as a CBOR tag, the
217 value will be encoded as appropriate for the value.
218
219 Types::Serialiser::true, Types::Serialiser::false,
220 Types::Serialiser::error
221 These special values become CBOR true, CBOR false and CBOR undefined
222 values, respectively. You can also use "\1", "\0" and "\undef"
223 directly if you want.
224
225 other blessed objects
226 Other blessed objects are serialised via "TO_CBOR" or "FREEZE". See
227 "OBJECT SERIALISATION", below, for details.
228
229 simple scalars
230 TODO Simple Perl scalars (any scalar that is not a reference) are
231 the most difficult objects to encode: CBOR::XS will encode undefined
232 scalars as CBOR null values, scalars that have last been used in a
233 string context before encoding as CBOR strings, and anything else as
234 number value:
235
236 # dump as number
237 encode_cbor [2] # yields [2]
238 encode_cbor [-3.0e17] # yields [-3e+17]
239 my $value = 5; encode_cbor [$value] # yields [5]
240
241 # used as string, so dump as string
242 print $value;
243 encode_cbor [$value] # yields ["5"]
244
245 # undef becomes null
246 encode_cbor [undef] # yields [null]
247
248 You can force the type to be a CBOR string by stringifying it:
249
250 my $x = 3.1; # some variable containing a number
251 "$x"; # stringified
252 $x .= ""; # another, more awkward way to stringify
253 print $x; # perl does it for you, too, quite often
254
255 You can force the type to be a CBOR number by numifying it:
256
257 my $x = "3"; # some variable containing a string
258 $x += 0; # numify it, ensuring it will be dumped as a number
259 $x *= 1; # same thing, the choice is yours.
260
261 You can not currently force the type in other, less obscure, ways.
262 Tell me if you need this capability (but don't forget to explain why
263 it's needed :).
264
265 Perl values that seem to be integers generally use the shortest
266 possible representation. Floating-point values will use either the
267 IEEE single format if possible without loss of precision, otherwise
268 the IEEE double format will be used. Perls that use formats other
269 than IEEE double to represent numerical values are supported, but
270 might suffer loss of precision.
271
272 OBJECT SERIALISATION
273 This module knows two way to serialise a Perl object: The CBOR-specific
274 way, and the generic way.
275
276 Whenever the encoder encounters a Perl object that it cnanot serialise
277 directly (most of them), it will first look up the "TO_CBOR" method on
278 it.
279
280 If it has a "TO_CBOR" method, it will call it with the object as only
281 argument, and expects exactly one return value, which it will then
282 substitute and encode it in the place of the object.
283
284 Otherwise, it will look up the "FREEZE" method. If it exists, it will
285 call it with the object as first argument, and the constant string
286 "CBOR" as the second argument, to distinguish it from other serialisers.
287
288 The "FREEZE" method can return any number of values (i.e. zero or more).
289 These will be encoded as CBOR perl object, together with the classname.
290
291 If an object supports neither "TO_CBOR" nor "FREEZE", encoding will fail
292 with an error.
293
294 Objects encoded via "TO_CBOR" cannot be automatically decoded, but
295 objects encoded via "FREEZE" can be decoded using the following
296 protocol:
297
298 When an encoded CBOR perl object is encountered by the decoder, it will
299 look up the "THAW" method, by using the stored classname, and will fail
300 if the method cannot be found.
301
302 After the lookup it will call the "THAW" method with the stored
303 classname as first argument, the constant string "CBOR" as second
304 argument, and all values returned by "FREEZE" as remaining arguments.
305
306 EXAMPLES
307 Here is an example "TO_CBOR" method:
308
309 sub My::Object::TO_CBOR {
310 my ($obj) = @_;
311
312 ["this is a serialised My::Object object", $obj->{id}]
313 }
314
315 When a "My::Object" is encoded to CBOR, it will instead encode a simple
316 array with two members: a string, and the "object id". Decoding this
317 CBOR string will yield a normal perl array reference in place of the
318 object.
319
320 A more useful and practical example would be a serialisation method for
321 the URI module. CBOR has a custom tag value for URIs, namely 32:
322
323 sub URI::TO_CBOR {
324 my ($self) = @_;
325 my $uri = "$self"; # stringify uri
326 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
327 CBOR::XS::tagged 32, "$_[0]"
328 }
329
330 This will encode URIs as a UTF-8 string with tag 32, which indicates an
331 URI.
332
333 Decoding such an URI will not (currently) give you an URI object, but
334 instead a CBOR::XS::Tagged object with tag number 32 and the string -
335 exactly what was returned by "TO_CBOR".
336
337 To serialise an object so it can automatically be deserialised, you need
338 to use "FREEZE" and "THAW". To take the URI module as example, this
339 would be a possible implementation:
340
341 sub URI::FREEZE {
342 my ($self, $serialiser) = @_;
343 "$self" # encode url string
344 }
345
346 sub URI::THAW {
347 my ($class, $serialiser, $uri) = @_;
348
349 $class->new ($uri)
350 }
351
352 Unlike "TO_CBOR", multiple values can be returned by "FREEZE". For
353 example, a "FREEZE" method that returns "type", "id" and "variant"
354 values would cause an invocation of "THAW" with 5 arguments:
355
356 sub My::Object::FREEZE {
357 my ($self, $serialiser) = @_;
358
359 ($self->{type}, $self->{id}, $self->{variant})
360 }
361
362 sub My::Object::THAW {
363 my ($class, $serialiser, $type, $id, $variant) = @_;
364
365 $class-<new (type => $type, id => $id, variant => $variant)
366 }
367
368 MAGIC HEADER
369 There is no way to distinguish CBOR from other formats programmatically.
370 To make it easier to distinguish CBOR from other formats, the CBOR
371 specification has a special "magic string" that can be prepended to any
372 CBOR string without changing it's meaning.
373
374 This string is available as $CBOR::XS::MAGIC. This module does not
375 prepend this string tot he CBOR data it generates, but it will ignroe it
376 if present, so users can prepend this string as a "file type" indicator
377 as required.
378
379 CBOR and JSON
380 CBOR is supposed to implement a superset of the JSON data model, and is,
381 with some coercion, able to represent all JSON texts (something that
382 other "binary JSON" formats such as BSON generally do not support).
383
384 CBOR implements some extra hints and support for JSON interoperability,
385 and the spec offers further guidance for conversion between CBOR and
386 JSON. None of this is currently implemented in CBOR, and the guidelines
387 in the spec do not result in correct round-tripping of data. If JSON
388 interoperability is improved in the future, then the goal will be to
389 ensure that decoded JSON data will round-trip encoding and decoding to
390 CBOR intact.
391
392 SECURITY CONSIDERATIONS
393 When you are using CBOR in a protocol, talking to untrusted potentially
394 hostile creatures requires relatively few measures.
395
396 First of all, your CBOR decoder should be secure, that is, should not
397 have any buffer overflows. Obviously, this module should ensure that and
398 I am trying hard on making that true, but you never know.
399
400 Second, you need to avoid resource-starving attacks. That means you
401 should limit the size of CBOR data you accept, or make sure then when
402 your resources run out, that's just fine (e.g. by using a separate
403 process that can crash safely). The size of a CBOR string in octets is
404 usually a good indication of the size of the resources required to
405 decode it into a Perl structure. While CBOR::XS can check the size of
406 the CBOR text, it might be too late when you already have it in memory,
407 so you might want to check the size before you accept the string.
408
409 Third, CBOR::XS recurses using the C stack when decoding objects and
410 arrays. The C stack is a limited resource: for instance, on my amd64
411 machine with 8MB of stack size I can decode around 180k nested arrays
412 but only 14k nested CBOR objects (due to perl itself recursing deeply on
413 croak to free the temporary). If that is exceeded, the program crashes.
414 To be conservative, the default nesting limit is set to 512. If your
415 process has a smaller stack, you should adjust this setting accordingly
416 with the "max_depth" method.
417
418 Something else could bomb you, too, that I forgot to think of. In that
419 case, you get to keep the pieces. I am always open for hints, though...
420
421 Also keep in mind that CBOR::XS might leak contents of your Perl data
422 structures in its error messages, so when you serialise sensitive
423 information you might want to make sure that exceptions thrown by
424 CBOR::XS will not end up in front of untrusted eyes.
425
426 CBOR IMPLEMENTATION NOTES
427 This section contains some random implementation notes. They do not
428 describe guaranteed behaviour, but merely behaviour as-is implemented
429 right now.
430
431 64 bit integers are only properly decoded when Perl was built with 64
432 bit support.
433
434 Strings and arrays are encoded with a definite length. Hashes as well,
435 unless they are tied (or otherwise magical).
436
437 Only the double data type is supported for NV data types - when Perl
438 uses long double to represent floating point values, they might not be
439 encoded properly. Half precision types are accepted, but not encoded.
440
441 Strict mode and canonical mode are not implemented.
442
443 THREADS
444 This module is *not* guaranteed to be thread safe and there are no plans
445 to change this until Perl gets thread support (as opposed to the
446 horribly slow so-called "threads" which are simply slow and bloated
447 process simulations - use fork, it's *much* faster, cheaper, better).
448
449 (It might actually work, but you have been warned).
450
451 BUGS
452 While the goal of this module is to be correct, that unfortunately does
453 not mean it's bug-free, only that I think its design is bug-free. If you
454 keep reporting bugs they will be fixed swiftly, though.
455
456 Please refrain from using rt.cpan.org or any other bug reporting
457 service. I put the contact address into my modules for a reason.
458
459 SEE ALSO
460 The JSON and JSON::XS modules that do similar, but human-readable,
461 serialisation.
462
463 The Types::Serialiser module provides the data model for true, false and
464 error values.
465
466 AUTHOR
467 Marc Lehmann <schmorp@schmorp.de>
468 http://home.schmorp.de/
469