ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.9 by root, Mon Oct 28 21:28:14 2013 UTC vs.
Revision 1.63 by root, Fri Nov 25 12:16:12 2016 UTC

26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up to
32you to do). Furthermore, details of the implementation might change freely
33before version 1.0. And lastly, the object serialisation protocol depends
34on a pending IANA assignment, and until that assignment is official, this
35implementation is not interoperable with other implementations (even
36future versions of this module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. 38with the added ability of supporting serialisation of Perl objects. (JSON
39often compresses better than CBOR though, so if you plan to compress the
40data later and speed is less important you might want to compare both
41formats first).
42
43To give you a general idea about speed, with texts in the megabyte range,
44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
46data, the worse L<Storable> performs in comparison.
47
48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
51
52In addition to the core CBOR data format, this module implements a
53number of extensions, to support cyclic and shared data structures
54(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55C<pack_strings>) and scalar references (always enabled).
48 56
49The primary goal of this module is to be I<correct> and the secondary goal 57The primary goal of this module is to be I<correct> and the secondary goal
50is to be I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
51 59
52See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
56 64
57package CBOR::XS; 65package CBOR::XS;
58 66
59use common::sense; 67use common::sense;
60 68
61our $VERSION = 0.05; 69our $VERSION = 1.51;
62our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
63 71
64our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
65 73
66use Exporter; 74use Exporter;
103strings. All boolean flags described below are by default I<disabled>. 111strings. All boolean flags described below are by default I<disabled>.
104 112
105The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
106be chained: 114be chained:
107 115
108#TODO
109 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
110 117
111=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
112 119
113=item $max_depth = $cbor->get_max_depth 120=item $max_depth = $cbor->get_max_depth
147If no argument is given, the limit check will be deactivated (same as when 154If no argument is given, the limit check will be deactivated (same as when
148C<0> is specified). 155C<0> is specified).
149 156
150See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 157See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
151 158
159=item $cbor = $cbor->allow_unknown ([$enable])
160
161=item $enabled = $cbor->get_allow_unknown
162
163If C<$enable> is true (or missing), then C<encode> will I<not> throw an
164exception when it encounters values it cannot represent in CBOR (for
165example, filehandles) but instead will encode a CBOR C<error> value.
166
167If C<$enable> is false (the default), then C<encode> will throw an
168exception when it encounters anything it cannot encode as CBOR.
169
170This option does not affect C<decode> in any way, and it is recommended to
171leave it off unless you know your communications partner.
172
173=item $cbor = $cbor->allow_sharing ([$enable])
174
175=item $enabled = $cbor->get_allow_sharing
176
177If C<$enable> is true (or missing), then C<encode> will not double-encode
178values that have been referenced before (e.g. when the same object, such
179as an array, is referenced multiple times), but instead will emit a
180reference to the earlier value.
181
182This means that such values will only be encoded once, and will not result
183in a deep cloning of the value on decode, in decoders supporting the value
184sharing extension. This also makes it possible to encode cyclic data
185structures (which need C<allow_cycles> to be enabled to be decoded by this
186module).
187
188It is recommended to leave it off unless you know your
189communication partner supports the value sharing extensions to CBOR
190(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
191resulting data structure might be unusable.
192
193Detecting shared values incurs a runtime overhead when values are encoded
194that have a reference counter large than one, and might unnecessarily
195increase the encoded size, as potentially shared values are encode as
196shareable whether or not they are actually shared.
197
198At the moment, only targets of references can be shared (e.g. scalars,
199arrays or hashes pointed to by a reference). Weirder constructs, such as
200an array with multiple "copies" of the I<same> string, which are hard but
201not impossible to create in Perl, are not supported (this is the same as
202with L<Storable>).
203
204If C<$enable> is false (the default), then C<encode> will encode shared
205data structures repeatedly, unsharing them in the process. Cyclic data
206structures cannot be encoded in this mode.
207
208This option does not affect C<decode> in any way - shared values and
209references will always be decoded properly if present.
210
211=item $cbor = $cbor->allow_cycles ([$enable])
212
213=item $enabled = $cbor->get_allow_cycles
214
215If C<$enable> is true (or missing), then C<decode> will happily decode
216self-referential (cyclic) data structures. By default these will not be
217decoded, as they need manual cleanup to avoid memory leaks, so code that
218isn't prepared for this will not leak memory.
219
220If C<$enable> is false (the default), then C<decode> will throw an error
221when it encounters a self-referential/cyclic data structure.
222
223FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
224cycles - future versions of this module might chose to decode cyclic data
225structures using weak references when this option is off, instead of
226throwing an error.
227
228This option does not affect C<encode> in any way - shared values and
229references will always be encoded properly if present.
230
231=item $cbor = $cbor->pack_strings ([$enable])
232
233=item $enabled = $cbor->get_pack_strings
234
235If C<$enable> is true (or missing), then C<encode> will try not to encode
236the same string twice, but will instead encode a reference to the string
237instead. Depending on your data format, this can save a lot of space, but
238also results in a very large runtime overhead (expect encoding times to be
2392-4 times as high as without).
240
241It is recommended to leave it off unless you know your
242communications partner supports the stringref extension to CBOR
243(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
244resulting data structure might not be usable.
245
246If C<$enable> is false (the default), then C<encode> will encode strings
247the standard CBOR way.
248
249This option does not affect C<decode> in any way - string references will
250always be decoded properly if present.
251
252=item $cbor = $cbor->text_keys ([$enable])
253
254=item $enabled = $cbor->get_text_keys
255
256If C<$enabled> is true (or missing), then C<encode> will encode all
257perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
258
259If C<$enable> is false (the default), then C<encode> will encode hash keys
260normally - upgraded perl strings (strings internally encoded as UTF-8) as
261CBOR text strings, and downgraded perl strings as CBOR byte strings.
262
263This option does not affect C<decode> in any way.
264
265This option is useful for interoperability with CBOR decoders that don't
266treat byte strings as a form of text. It is especially useful as Perl
267gives very little control over hash keys.
268
269Enabling this option can be slow, as all downgraded hash keys that are
270encoded need to be scanned and converted to UTF-8.
271
272=item $cbor = $cbor->text_strings ([$enable])
273
274=item $enabled = $cbor->get_text_strings
275
276This option works similar to C<text_keys>, above, but works on all strings
277(including hash keys), so C<text_keys> has no further effect after
278enabling C<text_strings>.
279
280If C<$enabled> is true (or missing), then C<encode> will encode all perl
281strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
282
283If C<$enable> is false (the default), then C<encode> will encode strings
284normally (but see C<text_keys>) - upgraded perl strings (strings
285internally encoded as UTF-8) as CBOR text strings, and downgraded perl
286strings as CBOR byte strings.
287
288This option does not affect C<decode> in any way.
289
290This option has similar advantages and disadvantages as C<text_keys>. In
291addition, this option effectively removes the ability to encode byte
292strings, which might break some C<FREEZE> and C<TO_CBOR> methods that rely
293on this, such as bignum encoding, so this option is mainly useful for very
294simple data.
295
296=item $cbor = $cbor->validate_utf8 ([$enable])
297
298=item $enabled = $cbor->get_validate_utf8
299
300If C<$enable> is true (or missing), then C<decode> will validate that
301elements (text strings) containing UTF-8 data in fact contain valid UTF-8
302data (instead of blindly accepting it). This validation obviously takes
303extra time during decoding.
304
305The concept of "valid UTF-8" used is perl's concept, which is a superset
306of the official UTF-8.
307
308If C<$enable> is false (the default), then C<decode> will blindly accept
309UTF-8 data, marking them as valid UTF-8 in the resulting data structure
310regardless of whether that's true or not.
311
312Perl isn't too happy about corrupted UTF-8 in strings, but should
313generally not crash or do similarly evil things. Extensions might be not
314so forgiving, so it's recommended to turn on this setting if you receive
315untrusted CBOR.
316
317This option does not affect C<encode> in any way - strings that are
318supposedly valid UTF-8 will simply be dumped into the resulting CBOR
319string without checking whether that is, in fact, true or not.
320
321=item $cbor = $cbor->filter ([$cb->($tag, $value)])
322
323=item $cb_or_undef = $cbor->get_filter
324
325Sets or replaces the tagged value decoding filter (when C<$cb> is
326specified) or clears the filter (if no argument or C<undef> is provided).
327
328The filter callback is called only during decoding, when a non-enforced
329tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
330list of enforced tags). For specific tags, it's often better to provide a
331default converter using the C<%CBOR::XS::FILTER> hash (see below).
332
333The first argument is the numerical tag, the second is the (decoded) value
334that has been tagged.
335
336The filter function should return either exactly one value, which will
337replace the tagged value in the decoded data structure, or no values,
338which will result in default handling, which currently means the decoder
339creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
340
341When the filter is cleared (the default state), the default filter
342function, C<CBOR::XS::default_filter>, is used. This function simply looks
343up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
344a code reference that is called with tag and value, and is responsible for
345decoding the value. If no entry exists, it returns no values.
346
347Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
348objects, with no other special handling (useful when working with
349potentially "unsafe" CBOR data).
350
351 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
352
353Example: provide a global filter for tag 1347375694, converting the value
354into some string form.
355
356 $CBOR::XS::FILTER{1347375694} = sub {
357 my ($tag, $value);
358
359 "tag 1347375694 value $value"
360 };
361
152=item $cbor_data = $cbor->encode ($perl_scalar) 362=item $cbor_data = $cbor->encode ($perl_scalar)
153 363
154Converts the given Perl data structure (a scalar value) to its CBOR 364Converts the given Perl data structure (a scalar value) to its CBOR
155representation. 365representation.
156 366
169and you need to know where the first CBOR string ends amd the next one 379and you need to know where the first CBOR string ends amd the next one
170starts. 380starts.
171 381
172 CBOR::XS->new->decode_prefix ("......") 382 CBOR::XS->new->decode_prefix ("......")
173 => ("...", 3) 383 => ("...", 3)
384
385=back
386
387=head2 INCREMENTAL PARSING
388
389In some cases, there is the need for incremental parsing of JSON
390texts. While this module always has to keep both CBOR text and resulting
391Perl data structure in memory at one time, it does allow you to parse a
392CBOR stream incrementally, using a similar to using "decode_prefix" to see
393if a full CBOR object is available, but is much more efficient.
394
395It basically works by parsing as much of a CBOR string as possible - if
396the CBOR data is not complete yet, the pasrer will remember where it was,
397to be able to restart when more data has been accumulated. Once enough
398data is available to either decode a complete CBOR value or raise an
399error, a real decode will be attempted.
400
401A typical use case would be a network protocol that consists of sending
402and receiving CBOR-encoded messages. The solution that works with CBOR and
403about anything else is by prepending a length to every CBOR value, so the
404receiver knows how many octets to read. More compact (and slightly slower)
405would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
406a CBOR value ends, and doesn't need an explicit length.
407
408The following methods help with this:
409
410=over 4
411
412=item @decoded = $cbor->incr_parse ($buffer)
413
414This method attempts to decode exactly one CBOR value from the beginning
415of the given C<$buffer>. The value is removed from the C<$buffer> on
416success. When C<$buffer> doesn't contain a complete value yet, it returns
417nothing. Finally, when the C<$buffer> doesn't start with something
418that could ever be a valid CBOR value, it raises an exception, just as
419C<decode> would. In the latter case the decoder state is undefined and
420must be reset before being able to parse further.
421
422This method modifies the C<$buffer> in place. When no CBOR value can be
423decoded, the decoder stores the current string offset. On the next call,
424continues decoding at the place where it stopped before. For this to make
425sense, the C<$buffer> must begin with the same octets as on previous
426unsuccessful calls.
427
428You can call this method in scalar context, in which case it either
429returns a decoded value or C<undef>. This makes it impossible to
430distinguish between CBOR null values (which decode to C<undef>) and an
431unsuccessful decode, which is often acceptable.
432
433=item @decoded = $cbor->incr_parse_multiple ($buffer)
434
435Same as C<incr_parse>, but attempts to decode as many CBOR values as
436possible in one go, instead of at most one. Calls to C<incr_parse> and
437C<incr_parse_multiple> can be interleaved.
438
439=item $cbor->incr_reset
440
441Resets the incremental decoder. This throws away any saved state, so that
442subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
443a new CBOR value from the beginning of the C<$buffer> again.
444
445This method can be called at any time, but it I<must> be called if you want
446to change your C<$buffer> or there was a decoding error and you want to
447reuse the C<$cbor> object for future incremental parsings.
174 448
175=back 449=back
176 450
177 451
178=head1 MAPPING 452=head1 MAPPING
196CBOR integers become (numeric) perl scalars. On perls without 64 bit 470CBOR integers become (numeric) perl scalars. On perls without 64 bit
197support, 64 bit integers will be truncated or otherwise corrupted. 471support, 64 bit integers will be truncated or otherwise corrupted.
198 472
199=item byte strings 473=item byte strings
200 474
201Byte strings will become octet strings in Perl (the byte values 0..255 475Byte strings will become octet strings in Perl (the Byte values 0..255
202will simply become characters of the same value in Perl). 476will simply become characters of the same value in Perl).
203 477
204=item UTF-8 strings 478=item UTF-8 strings
205 479
206UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 480UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
224C<Types:Serialiser::false> and C<Types::Serialiser::error>, 498C<Types:Serialiser::false> and C<Types::Serialiser::error>,
225respectively. They are overloaded to act almost exactly like the numbers 499respectively. They are overloaded to act almost exactly like the numbers
226C<1> and C<0> (for true and false) or to throw an exception on access (for 500C<1> and C<0> (for true and false) or to throw an exception on access (for
227error). See the L<Types::Serialiser> manpage for details. 501error). See the L<Types::Serialiser> manpage for details.
228 502
229=item CBOR tag 256 (perl object) 503=item tagged values
230 504
231The tag value C<256> (TODO: pending iana registration) will be used
232to deserialise a Perl object serialised with C<FREEZE>. See "OBJECT
233SERIALISATION", below, for details.
234
235=item CBOR tag 55799 (magic header)
236
237The tag 55799 is ignored (this tag implements the magic header).
238
239=item other CBOR tags
240
241Tagged items consists of a numeric tag and another CBOR value. Tags not 505Tagged items consists of a numeric tag and another CBOR value.
242handled internally are currently converted into a L<CBOR::XS::Tagged>
243object, which is simply a blessed array reference consisting of the
244numeric tag value followed by the (decoded) CBOR value.
245 506
246In the future, support for user-supplied conversions might get added. 507See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
508for details on which tags are handled how.
247 509
248=item anything else 510=item anything else
249 511
250Anything else (e.g. unsupported simple values) will raise a decoding 512Anything else (e.g. unsupported simple values) will raise a decoding
251error. 513error.
254 516
255 517
256=head2 PERL -> CBOR 518=head2 PERL -> CBOR
257 519
258The mapping from Perl to CBOR is slightly more difficult, as Perl is a 520The mapping from Perl to CBOR is slightly more difficult, as Perl is a
259truly typeless language, so we can only guess which CBOR type is meant by 521typeless language. That means this module can only guess which CBOR type
260a Perl value. 522is meant by a perl value.
261 523
262=over 4 524=over 4
263 525
264=item hash references 526=item hash references
265 527
266Perl hash references become CBOR maps. As there is no inherent ordering in 528Perl hash references become CBOR maps. As there is no inherent ordering in
267hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 529hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
268order. 530order. This order can be different each time a hash is encoded.
269 531
270Currently, tied hashes will use the indefinite-length format, while normal 532Currently, tied hashes will use the indefinite-length format, while normal
271hashes will use the fixed-length format. 533hashes will use the fixed-length format.
272 534
273=item array references 535=item array references
274 536
275Perl array references become fixed-length CBOR arrays. 537Perl array references become fixed-length CBOR arrays.
276 538
277=item other references 539=item other references
278 540
279Other unblessed references are generally not allowed and will cause an 541Other unblessed references will be represented using
280exception to be thrown, except for references to the integers C<0> and 542the indirection tag extension (tag value C<22098>,
281C<1>, which get turned into false and true in CBOR. 543L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
544to be able to decode these values somehow, by either "doing the right
545thing", decoding into a generic tagged object, simply ignoring the tag, or
546something else.
282 547
283=item CBOR::XS::Tagged objects 548=item CBOR::XS::Tagged objects
284 549
285Objects of this type must be arrays consisting of a single C<[tag, value]> 550Objects of this type must be arrays consisting of a single C<[tag, value]>
286pair. The (numerical) tag will be encoded as a CBOR tag, the value will be 551pair. The (numerical) tag will be encoded as a CBOR tag, the value will
287encoded as appropriate for the value. 552be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
553create such objects.
288 554
289=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 555=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
290 556
291These special values become CBOR true, CBOR false and CBOR undefined 557These special values become CBOR true, CBOR false and CBOR undefined
292values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly 558values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
293if you want. 559if you want.
294 560
295=item other blessed objects 561=item other blessed objects
296 562
297Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See 563Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
298"OBJECT SERIALISATION", below, for details. 564L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
565module, and L<OBJECT SERIALISATION> for generic object serialisation.
299 566
300=item simple scalars 567=item simple scalars
301 568
302TODO
303Simple Perl scalars (any scalar that is not a reference) are the most 569Simple Perl scalars (any scalar that is not a reference) are the most
304difficult objects to encode: CBOR::XS will encode undefined scalars as 570difficult objects to encode: CBOR::XS will encode undefined scalars as
305CBOR null values, scalars that have last been used in a string context 571CBOR null values, scalars that have last been used in a string context
306before encoding as CBOR strings, and anything else as number value: 572before encoding as CBOR strings, and anything else as number value:
307 573
308 # dump as number 574 # dump as number
309 encode_cbor [2] # yields [2] 575 encode_cbor [2] # yields [2]
310 encode_cbor [-3.0e17] # yields [-3e+17] 576 encode_cbor [-3.0e17] # yields [-3e+17]
311 my $value = 5; encode_cbor [$value] # yields [5] 577 my $value = 5; encode_cbor [$value] # yields [5]
312 578
313 # used as string, so dump as string 579 # used as string, so dump as string (either byte or text)
314 print $value; 580 print $value;
315 encode_cbor [$value] # yields ["5"] 581 encode_cbor [$value] # yields ["5"]
316 582
317 # undef becomes null 583 # undef becomes null
318 encode_cbor [undef] # yields [null] 584 encode_cbor [undef] # yields [null]
321 587
322 my $x = 3.1; # some variable containing a number 588 my $x = 3.1; # some variable containing a number
323 "$x"; # stringified 589 "$x"; # stringified
324 $x .= ""; # another, more awkward way to stringify 590 $x .= ""; # another, more awkward way to stringify
325 print $x; # perl does it for you, too, quite often 591 print $x; # perl does it for you, too, quite often
592
593You can force whether a string is encoded as byte or text string by using
594C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled):
595
596 utf8::upgrade $x; # encode $x as text string
597 utf8::downgrade $x; # encode $x as byte string
598
599Perl doesn't define what operations up- and downgrade strings, so if the
600difference between byte and text is important, you should up- or downgrade
601your string as late as possible before encoding. You can also force the
602use of CBOR text strings by using C<text_keys> or C<text_strings>.
326 603
327You can force the type to be a CBOR number by numifying it: 604You can force the type to be a CBOR number by numifying it:
328 605
329 my $x = "3"; # some variable containing a string 606 my $x = "3"; # some variable containing a string
330 $x += 0; # numify it, ensuring it will be dumped as a number 607 $x += 0; # numify it, ensuring it will be dumped as a number
343 620
344=back 621=back
345 622
346=head2 OBJECT SERIALISATION 623=head2 OBJECT SERIALISATION
347 624
625This module implements both a CBOR-specific and the generic
626L<Types::Serialier> object serialisation protocol. The following
627subsections explain both methods.
628
629=head3 ENCODING
630
348This module knows two way to serialise a Perl object: The CBOR-specific 631This module knows two way to serialise a Perl object: The CBOR-specific
349way, and the generic way. 632way, and the generic way.
350 633
351Whenever the encoder encounters a Perl object that it cnanot serialise 634Whenever the encoder encounters a Perl object that it cannot serialise
352directly (most of them), it will first look up the C<TO_CBOR> method on 635directly (most of them), it will first look up the C<TO_CBOR> method on
353it. 636it.
354 637
355If it has a C<TO_CBOR> method, it will call it with the object as only 638If it has a C<TO_CBOR> method, it will call it with the object as only
356argument, and expects exactly one return value, which it will then 639argument, and expects exactly one return value, which it will then
362 645
363The C<FREEZE> method can return any number of values (i.e. zero or 646The C<FREEZE> method can return any number of values (i.e. zero or
364more). These will be encoded as CBOR perl object, together with the 647more). These will be encoded as CBOR perl object, together with the
365classname. 648classname.
366 649
650These methods I<MUST NOT> change the data structure that is being
651serialised. Failure to comply to this can result in memory corruption -
652and worse.
653
367If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail 654If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
368with an error. 655with an error.
369 656
657=head3 DECODING
658
370Objects encoded via C<TO_CBOR> cannot be automatically decoded, but 659Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
371objects encoded via C<FREEZE> can be decoded using the following protocol: 660but objects encoded via C<FREEZE> can be decoded using the following
661protocol:
372 662
373When an encoded CBOR perl object is encountered by the decoder, it will 663When an encoded CBOR perl object is encountered by the decoder, it will
374look up the C<THAW> method, by using the stored classname, and will fail 664look up the C<THAW> method, by using the stored classname, and will fail
375if the method cannot be found. 665if the method cannot be found.
376 666
377After the lookup it will call the C<THAW> method with the stored classname 667After the lookup it will call the C<THAW> method with the stored classname
378as first argument, the constant string C<CBOR> as second argument, and all 668as first argument, the constant string C<CBOR> as second argument, and all
379values returned by C<FREEZE> as remaining arguments. 669values returned by C<FREEZE> as remaining arguments.
380 670
381=head4 EXAMPLES 671=head3 EXAMPLES
382 672
383Here is an example C<TO_CBOR> method: 673Here is an example C<TO_CBOR> method:
384 674
385 sub My::Object::TO_CBOR { 675 sub My::Object::TO_CBOR {
386 my ($obj) = @_; 676 my ($obj) = @_;
397 687
398 sub URI::TO_CBOR { 688 sub URI::TO_CBOR {
399 my ($self) = @_; 689 my ($self) = @_;
400 my $uri = "$self"; # stringify uri 690 my $uri = "$self"; # stringify uri
401 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 691 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
402 CBOR::XS::tagged 32, "$_[0]" 692 CBOR::XS::tag 32, "$_[0]"
403 } 693 }
404 694
405This will encode URIs as a UTF-8 string with tag 32, which indicates an 695This will encode URIs as a UTF-8 string with tag 32, which indicates an
406URI. 696URI.
407 697
418 "$self" # encode url string 708 "$self" # encode url string
419 } 709 }
420 710
421 sub URI::THAW { 711 sub URI::THAW {
422 my ($class, $serialiser, $uri) = @_; 712 my ($class, $serialiser, $uri) = @_;
423
424 $class->new ($uri) 713 $class->new ($uri)
425 } 714 }
426 715
427Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For 716Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
428example, a C<FREEZE> method that returns "type", "id" and "variant" values 717example, a C<FREEZE> method that returns "type", "id" and "variant" values
444=head1 MAGIC HEADER 733=head1 MAGIC HEADER
445 734
446There is no way to distinguish CBOR from other formats 735There is no way to distinguish CBOR from other formats
447programmatically. To make it easier to distinguish CBOR from other 736programmatically. To make it easier to distinguish CBOR from other
448formats, the CBOR specification has a special "magic string" that can be 737formats, the CBOR specification has a special "magic string" that can be
449prepended to any CBOR string without changing it's meaning. 738prepended to any CBOR string without changing its meaning.
450 739
451This string is available as C<$CBOR::XS::MAGIC>. This module does not 740This string is available as C<$CBOR::XS::MAGIC>. This module does not
452prepend this string tot he CBOR data it generates, but it will ignroe it 741prepend this string to the CBOR data it generates, but it will ignore it
453if present, so users can prepend this string as a "file type" indicator as 742if present, so users can prepend this string as a "file type" indicator as
454required. 743required.
455 744
745
746=head1 THE CBOR::XS::Tagged CLASS
747
748CBOR has the concept of tagged values - any CBOR value can be tagged with
749a numeric 64 bit number, which are centrally administered.
750
751C<CBOR::XS> handles a few tags internally when en- or decoding. You can
752also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
753decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
754unknown tag.
755
756These objects are simply blessed array references - the first member of
757the array being the numerical tag, the second being the value.
758
759You can interact with C<CBOR::XS::Tagged> objects in the following ways:
760
761=over 4
762
763=item $tagged = CBOR::XS::tag $tag, $value
764
765This function(!) creates a new C<CBOR::XS::Tagged> object using the given
766C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
767value that can be encoded in CBOR, including serialisable Perl objects and
768C<CBOR::XS::Tagged> objects).
769
770=item $tagged->[0]
771
772=item $tagged->[0] = $new_tag
773
774=item $tag = $tagged->tag
775
776=item $new_tag = $tagged->tag ($new_tag)
777
778Access/mutate the tag.
779
780=item $tagged->[1]
781
782=item $tagged->[1] = $new_value
783
784=item $value = $tagged->value
785
786=item $new_value = $tagged->value ($new_value)
787
788Access/mutate the tagged value.
789
790=back
791
792=cut
793
794sub tag($$) {
795 bless [@_], CBOR::XS::Tagged::;
796}
797
798sub CBOR::XS::Tagged::tag {
799 $_[0][0] = $_[1] if $#_;
800 $_[0][0]
801}
802
803sub CBOR::XS::Tagged::value {
804 $_[0][1] = $_[1] if $#_;
805 $_[0][1]
806}
807
808=head2 EXAMPLES
809
810Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
811
812You can look up CBOR tag value and emanings in the IANA registry at
813L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
814
815Prepend a magic header (C<$CBOR::XS::MAGIC>):
816
817 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
818 # same as:
819 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
820
821Serialise some URIs and a regex in an array:
822
823 my $cbor = encode_cbor [
824 (CBOR::XS::tag 32, "http://www.nethype.de/"),
825 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
826 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
827 ];
828
829Wrap CBOR data in CBOR:
830
831 my $cbor_cbor = encode_cbor
832 CBOR::XS::tag 24,
833 encode_cbor [1, 2, 3];
834
835=head1 TAG HANDLING AND EXTENSIONS
836
837This section describes how this module handles specific tagged values
838and extensions. If a tag is not mentioned here and no additional filters
839are provided for it, then the default handling applies (creating a
840CBOR::XS::Tagged object on decoding, and only encoding the tag when
841explicitly requested).
842
843Tags not handled specifically are currently converted into a
844L<CBOR::XS::Tagged> object, which is simply a blessed array reference
845consisting of the numeric tag value followed by the (decoded) CBOR value.
846
847Future versions of this module reserve the right to special case
848additional tags (such as base64url).
849
850=head2 ENFORCED TAGS
851
852These tags are always handled when decoding, and their handling cannot be
853overridden by the user.
854
855=over 4
856
857=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
858
859These tags are automatically created (and decoded) for serialisable
860objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
861serialisation protocol). See L<OBJECT SERIALISATION> for details.
862
863=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
864
865These tags are automatically decoded when encountered (and they do not
866result in a cyclic data structure, see C<allow_cycles>), resulting in
867shared values in the decoded object. They are only encoded, however, when
868C<allow_sharing> is enabled.
869
870Not all shared values can be successfully decoded: values that reference
871themselves will I<currently> decode as C<undef> (this is not the same
872as a reference pointing to itself, which will be represented as a value
873that contains an indirect reference to itself - these will be decoded
874properly).
875
876Note that considerably more shared value data structures can be decoded
877than will be encoded - currently, only values pointed to by references
878will be shared, others will not. While non-reference shared values can be
879generated in Perl with some effort, they were considered too unimportant
880to be supported in the encoder. The decoder, however, will decode these
881values as shared values.
882
883=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
884
885These tags are automatically decoded when encountered. They are only
886encoded, however, when C<pack_strings> is enabled.
887
888=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
889
890This tag is automatically generated when a reference are encountered (with
891the exception of hash and array references). It is converted to a reference
892when decoding.
893
894=item 55799 (self-describe CBOR, RFC 7049)
895
896This value is not generated on encoding (unless explicitly requested by
897the user), and is simply ignored when decoding.
898
899=back
900
901=head2 NON-ENFORCED TAGS
902
903These tags have default filters provided when decoding. Their handling can
904be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
905providing a custom C<filter> callback when decoding.
906
907When they result in decoding into a specific Perl class, the module
908usually provides a corresponding C<TO_CBOR> method as well.
909
910When any of these need to load additional modules that are not part of the
911perl core distribution (e.g. L<URI>), it is (currently) up to the user to
912provide these modules. The decoding usually fails with an exception if the
913required module cannot be loaded.
914
915=over 4
916
917=item 0, 1 (date/time string, seconds since the epoch)
918
919These tags are decoded into L<Time::Piece> objects. The corresponding
920C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
921
922The L<Time::Piece> API is generally surprisingly bad, and fractional
923seconds are only accidentally kept intact, so watch out. On the plus side,
924the module comes with perl since 5.10, which has to count for something.
925
926=item 2, 3 (positive/negative bignum)
927
928These tags are decoded into L<Math::BigInt> objects. The corresponding
929C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
930integers, and others into positive/negative CBOR bignums.
931
932=item 4, 5, 264, 265 (decimal fraction/bigfloat)
933
934Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
935objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
936encodes into a decimal fraction (either tag 4 or 264).
937
938NaN and infinities are not encoded properly, as they cannot be represented
939in CBOR.
940
941See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
942
943=item 30 (rational numbers)
944
945These tags are decoded into L<Math::BigRat> objects. The corresponding
946C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
947C<1> via their numerator only, i.e., they become normal integers or
948C<bignums>.
949
950See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
951
952=item 21, 22, 23 (expected later JSON conversion)
953
954CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
955tags.
956
957=item 32 (URI)
958
959These objects decode into L<URI> objects. The corresponding
960C<URI::TO_CBOR> method again results in a CBOR URI value.
961
962=back
963
964=cut
456 965
457=head1 CBOR and JSON 966=head1 CBOR and JSON
458 967
459CBOR is supposed to implement a superset of the JSON data model, and is, 968CBOR is supposed to implement a superset of the JSON data model, and is,
460with some coercion, able to represent all JSON texts (something that other 969with some coercion, able to represent all JSON texts (something that other
476 985
477First of all, your CBOR decoder should be secure, that is, should not have 986First of all, your CBOR decoder should be secure, that is, should not have
478any buffer overflows. Obviously, this module should ensure that and I am 987any buffer overflows. Obviously, this module should ensure that and I am
479trying hard on making that true, but you never know. 988trying hard on making that true, but you never know.
480 989
990Second, CBOR::XS supports object serialisation - decoding CBOR can cause
991calls to I<any> C<THAW> method in I<any> package that exists in your
992process (that is, CBOR::XS will not try to load modules, but any existing
993C<THAW> method or function can be called, so they all have to be secure).
994
481Second, you need to avoid resource-starving attacks. That means you should 995Third, you need to avoid resource-starving attacks. That means you should
482limit the size of CBOR data you accept, or make sure then when your 996limit the size of CBOR data you accept, or make sure then when your
483resources run out, that's just fine (e.g. by using a separate process that 997resources run out, that's just fine (e.g. by using a separate process that
484can crash safely). The size of a CBOR string in octets is usually a good 998can crash safely). The size of a CBOR string in octets is usually a good
485indication of the size of the resources required to decode it into a Perl 999indication of the size of the resources required to decode it into a Perl
486structure. While CBOR::XS can check the size of the CBOR text, it might be 1000structure. While CBOR::XS can check the size of the CBOR text, it might be
487too late when you already have it in memory, so you might want to check 1001too late when you already have it in memory, so you might want to check
488the size before you accept the string. 1002the size before you accept the string.
489 1003
490Third, CBOR::XS recurses using the C stack when decoding objects and 1004Fourth, CBOR::XS recurses using the C stack when decoding objects and
491arrays. The C stack is a limited resource: for instance, on my amd64 1005arrays. The C stack is a limited resource: for instance, on my amd64
492machine with 8MB of stack size I can decode around 180k nested arrays but 1006machine with 8MB of stack size I can decode around 180k nested arrays but
493only 14k nested CBOR objects (due to perl itself recursing deeply on croak 1007only 14k nested CBOR objects (due to perl itself recursing deeply on croak
494to free the temporary). If that is exceeded, the program crashes. To be 1008to free the temporary). If that is exceeded, the program crashes. To be
495conservative, the default nesting limit is set to 512. If your process 1009conservative, the default nesting limit is set to 512. If your process
502Also keep in mind that CBOR::XS might leak contents of your Perl data 1016Also keep in mind that CBOR::XS might leak contents of your Perl data
503structures in its error messages, so when you serialise sensitive 1017structures in its error messages, so when you serialise sensitive
504information you might want to make sure that exceptions thrown by CBOR::XS 1018information you might want to make sure that exceptions thrown by CBOR::XS
505will not end up in front of untrusted eyes. 1019will not end up in front of untrusted eyes.
506 1020
1021
1022=head1 BIGNUM SECURITY CONSIDERATIONS
1023
1024CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1025L<Math::BigFloat> that tries to encode the number in the simplest possible
1026way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
10274) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1028(L<Math::BigRat>, tag 30) can also contain bignums as members.
1029
1030CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1031bigfloats (tags 5 and 265), but it will never generate these on its own.
1032
1033Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1034decimal fractions is generally fast. Decoding bigints can be slow for very
1035big numbers (tens of thousands of digits, something that could potentially
1036be caught by limiting the size of CBOR texts), and decoding bigfloats or
1037arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1038for large exponents (roughly 40 bit and longer).
1039
1040Additionally, L<Math::BigInt> can take advantage of other bignum
1041libraries, such as L<Math::GMP>, which cannot handle big floats with large
1042exponents, and might simply abort or crash your program, due to their code
1043quality.
1044
1045This can be a concern if you want to parse untrusted CBOR. If it is, you
1046might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1047types. You should also disable types 5 and 265, as these can be slow even
1048without bigints.
1049
1050Disabling bigints will also partially or fully disable types that rely on
1051them, e.g. rational numbers that use bignums.
1052
1053
507=head1 CBOR IMPLEMENTATION NOTES 1054=head1 CBOR IMPLEMENTATION NOTES
508 1055
509This section contains some random implementation notes. They do not 1056This section contains some random implementation notes. They do not
510describe guaranteed behaviour, but merely behaviour as-is implemented 1057describe guaranteed behaviour, but merely behaviour as-is implemented
511right now. 1058right now.
519Only the double data type is supported for NV data types - when Perl uses 1066Only the double data type is supported for NV data types - when Perl uses
520long double to represent floating point values, they might not be encoded 1067long double to represent floating point values, they might not be encoded
521properly. Half precision types are accepted, but not encoded. 1068properly. Half precision types are accepted, but not encoded.
522 1069
523Strict mode and canonical mode are not implemented. 1070Strict mode and canonical mode are not implemented.
1071
1072
1073=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
1074
1075On perls that were built without 64 bit integer support (these are rare
1076nowadays, even on 32 bit architectures, as all major Perl distributions
1077are built with 64 bit integer support), support for any kind of 64 bit
1078integer in CBOR is very limited - most likely, these 64 bit values will
1079be truncated, corrupted, or otherwise not decoded correctly. This also
1080includes string, array and map sizes that are stored as 64 bit integers.
524 1081
525 1082
526=head1 THREADS 1083=head1 THREADS
527 1084
528This module is I<not> guaranteed to be thread safe and there are no 1085This module is I<not> guaranteed to be thread safe and there are no
542Please refrain from using rt.cpan.org or any other bug reporting 1099Please refrain from using rt.cpan.org or any other bug reporting
543service. I put the contact address into my modules for a reason. 1100service. I put the contact address into my modules for a reason.
544 1101
545=cut 1102=cut
546 1103
1104our %FILTER = (
1105 0 => sub { # rfc4287 datetime, utf-8
1106 require Time::Piece;
1107 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1108 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1109 # else either. Whats incredibe over standard strptime totally escapes me.
1110 # doesn't do fractional times, either. sigh.
1111 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1112 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1113 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1114 scalar eval {
1115 my $s = $_[1];
1116
1117 $s =~ s/Z$/+00:00/;
1118 $s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
1119 or die;
1120
1121 my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
1122 my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
1123
1124 Time::Piece::gmtime ($d->epoch + $b)
1125 } || die "corrupted CBOR date/time string ($_[0])";
1126 },
1127
1128 1 => sub { # seconds since the epoch, possibly fractional
1129 require Time::Piece;
1130 scalar Time::Piece::gmtime (pop)
1131 },
1132
1133 2 => sub { # pos bigint
1134 require Math::BigInt;
1135 Math::BigInt->new ("0x" . unpack "H*", pop)
1136 },
1137
1138 3 => sub { # neg bigint
1139 require Math::BigInt;
1140 -Math::BigInt->new ("0x" . unpack "H*", pop)
1141 },
1142
1143 4 => sub { # decimal fraction, array
1144 require Math::BigFloat;
1145 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1146 },
1147
1148 264 => sub { # decimal fraction with arbitrary exponent
1149 require Math::BigFloat;
1150 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1151 },
1152
1153 5 => sub { # bigfloat, array
1154 require Math::BigFloat;
1155 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1156 },
1157
1158 265 => sub { # bigfloat with arbitrary exponent
1159 require Math::BigFloat;
1160 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1161 },
1162
1163 30 => sub { # rational number
1164 require Math::BigRat;
1165 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
1166 },
1167
1168 21 => sub { pop }, # expected conversion to base64url encoding
1169 22 => sub { pop }, # expected conversion to base64 encoding
1170 23 => sub { pop }, # expected conversion to base16 encoding
1171
1172 # 24 # embedded cbor, byte string
1173
1174 32 => sub {
1175 require URI;
1176 URI->new (pop)
1177 },
1178
1179 # 33 # base64url rfc4648, utf-8
1180 # 34 # base64 rfc46484, utf-8
1181 # 35 # regex pcre/ecma262, utf-8
1182 # 36 # mime message rfc2045, utf-8
1183);
1184
1185sub CBOR::XS::default_filter {
1186 &{ $FILTER{$_[0]} or return }
1187}
1188
1189sub URI::TO_CBOR {
1190 my $uri = $_[0]->as_string;
1191 utf8::upgrade $uri;
1192 tag 32, $uri
1193}
1194
1195sub Math::BigInt::TO_CBOR {
1196 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
1197 $_[0]->numify
1198 } else {
1199 my $hex = substr $_[0]->as_hex, 2;
1200 $hex = "0$hex" if 1 & length $hex; # sigh
1201 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
1202 }
1203}
1204
1205sub Math::BigFloat::TO_CBOR {
1206 my ($m, $e) = $_[0]->parts;
1207
1208 -9223372036854775808 <= $e && $e <= 18446744073709551615
1209 ? tag 4, [$e->numify, $m]
1210 : tag 264, [$e, $m]
1211}
1212
1213sub Math::BigRat::TO_CBOR {
1214 my ($n, $d) = $_[0]->parts;
1215
1216 # older versions of BigRat need *1, as they not always return numbers
1217
1218 $d*1 == 1
1219 ? $n*1
1220 : tag 30, [$n*1, $d*1]
1221}
1222
1223sub Time::Piece::TO_CBOR {
1224 tag 1, 0 + $_[0]->epoch
1225}
1226
547XSLoader::load "CBOR::XS", $VERSION; 1227XSLoader::load "CBOR::XS", $VERSION;
548 1228
549=head1 SEE ALSO 1229=head1 SEE ALSO
550 1230
551The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 1231The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines