ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.3 by root, Sat Oct 26 11:08:34 2013 UTC vs.
Revision 1.62 by root, Fri Nov 25 06:13:16 2016 UTC

12 $perl_value = decode_cbor $binary_cbor_data; 12 $perl_value = decode_cbor $binary_cbor_data;
13 13
14 # OO-interface 14 # OO-interface
15 15
16 $coder = CBOR::XS->new; 16 $coder = CBOR::XS->new;
17 #TODO 17 $binary_cbor_data = $coder->encode ($perl_value);
18 $perl_value = $coder->decode ($binary_cbor_data);
19
20 # prefix decoding
21
22 my $many_cbor_strings = ...;
23 while (length $many_cbor_strings) {
24 my ($data, $length) = $cbor->decode_prefix ($many_cbor_strings);
25 # data was decoded
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 }
18 28
19=head1 DESCRIPTION 29=head1 DESCRIPTION
20 30
21WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA AND 31This module converts Perl data structures to the Concise Binary Object
22EAT YOUR CHILDREN! 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
33format that aims to use an (almost) superset of the JSON data model, i.e.
34when you can represent something useful in JSON, you should be able to
35represent it in CBOR.
23 36
24This module converts Perl data structures to CBOR and vice versa. Its 37In short, CBOR is a faster and quite compact binary alternative to JSON,
38with the added ability of supporting serialisation of Perl objects. (JSON
39often compresses better than CBOR though, so if you plan to compress the
40data later and speed is less important you might want to compare both
41formats first).
42
43To give you a general idea about speed, with texts in the megabyte range,
44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
46data, the worse L<Storable> performs in comparison.
47
48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
51
52In addition to the core CBOR data format, this module implements a
53number of extensions, to support cyclic and shared data structures
54(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55C<pack_strings>) and scalar references (always enabled).
56
25primary goal is to be I<correct> and its secondary goal is to be 57The primary goal of this module is to be I<correct> and the secondary goal
26I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
27 59
28See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
29vice versa. 61vice versa.
30 62
31=cut 63=cut
32 64
33package CBOR::XS; 65package CBOR::XS;
34 66
35use common::sense; 67use common::sense;
36 68
37our $VERSION = 0.02; 69our $VERSION = 1.51;
38our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
39 71
40our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
41 73
42use Exporter; 74use Exporter;
43use XSLoader; 75use XSLoader;
44 76
77use Types::Serialiser;
78
45our $MAGIC = "\xd9\xd9\xf7"; 79our $MAGIC = "\xd9\xd9\xf7";
46 80
47=head1 FUNCTIONAL INTERFACE 81=head1 FUNCTIONAL INTERFACE
48 82
49The following convenience methods are provided by this module. They are 83The following convenience methods are provided by this module. They are
77strings. All boolean flags described below are by default I<disabled>. 111strings. All boolean flags described below are by default I<disabled>.
78 112
79The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
80be chained: 114be chained:
81 115
82#TODO
83 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
84 117
85=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
86 119
87=item $max_depth = $cbor->get_max_depth 120=item $max_depth = $cbor->get_max_depth
121If no argument is given, the limit check will be deactivated (same as when 154If no argument is given, the limit check will be deactivated (same as when
122C<0> is specified). 155C<0> is specified).
123 156
124See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 157See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
125 158
159=item $cbor = $cbor->allow_unknown ([$enable])
160
161=item $enabled = $cbor->get_allow_unknown
162
163If C<$enable> is true (or missing), then C<encode> will I<not> throw an
164exception when it encounters values it cannot represent in CBOR (for
165example, filehandles) but instead will encode a CBOR C<error> value.
166
167If C<$enable> is false (the default), then C<encode> will throw an
168exception when it encounters anything it cannot encode as CBOR.
169
170This option does not affect C<decode> in any way, and it is recommended to
171leave it off unless you know your communications partner.
172
173=item $cbor = $cbor->allow_sharing ([$enable])
174
175=item $enabled = $cbor->get_allow_sharing
176
177If C<$enable> is true (or missing), then C<encode> will not double-encode
178values that have been referenced before (e.g. when the same object, such
179as an array, is referenced multiple times), but instead will emit a
180reference to the earlier value.
181
182This means that such values will only be encoded once, and will not result
183in a deep cloning of the value on decode, in decoders supporting the value
184sharing extension. This also makes it possible to encode cyclic data
185structures (which need C<allow_cycles> to be enabled to be decoded by this
186module).
187
188It is recommended to leave it off unless you know your
189communication partner supports the value sharing extensions to CBOR
190(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
191resulting data structure might be unusable.
192
193Detecting shared values incurs a runtime overhead when values are encoded
194that have a reference counter large than one, and might unnecessarily
195increase the encoded size, as potentially shared values are encode as
196shareable whether or not they are actually shared.
197
198At the moment, only targets of references can be shared (e.g. scalars,
199arrays or hashes pointed to by a reference). Weirder constructs, such as
200an array with multiple "copies" of the I<same> string, which are hard but
201not impossible to create in Perl, are not supported (this is the same as
202with L<Storable>).
203
204If C<$enable> is false (the default), then C<encode> will encode shared
205data structures repeatedly, unsharing them in the process. Cyclic data
206structures cannot be encoded in this mode.
207
208This option does not affect C<decode> in any way - shared values and
209references will always be decoded properly if present.
210
211=item $cbor = $cbor->allow_cycles ([$enable])
212
213=item $enabled = $cbor->get_allow_cycles
214
215If C<$enable> is true (or missing), then C<decode> will happily decode
216self-referential (cyclic) data structures. By default these will not be
217decoded, as they need manual cleanup to avoid memory leaks, so code that
218isn't prepared for this will not leak memory.
219
220If C<$enable> is false (the default), then C<decode> will throw an error
221when it encounters a self-referential/cyclic data structure.
222
223FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
224cycles - future versions of this module might chose to decode cyclic data
225structures using weak references when this option is off, instead of
226throwing an error.
227
228This option does not affect C<encode> in any way - shared values and
229references will always be encoded properly if present.
230
231=item $cbor = $cbor->pack_strings ([$enable])
232
233=item $enabled = $cbor->get_pack_strings
234
235If C<$enable> is true (or missing), then C<encode> will try not to encode
236the same string twice, but will instead encode a reference to the string
237instead. Depending on your data format, this can save a lot of space, but
238also results in a very large runtime overhead (expect encoding times to be
2392-4 times as high as without).
240
241It is recommended to leave it off unless you know your
242communications partner supports the stringref extension to CBOR
243(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
244resulting data structure might not be usable.
245
246If C<$enable> is false (the default), then C<encode> will encode strings
247the standard CBOR way.
248
249This option does not affect C<decode> in any way - string references will
250always be decoded properly if present.
251
252=item $cbor = $cbor->text_keys ([$enable])
253
254=item $enabled = $cbor->get_text_keys
255
256If C<$enabled> is true (or missing), then C<encode> will encode all
257perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
258
259If C<$enable> is false (the default), then C<encode> will encode hash keys
260normally - upgraded perl strings (strings internally encoded as UTF-8) as
261CBOR text strings, and downgraded perl strings as CBOR byte strings.
262
263This option does not affect C<decode> in any way.
264
265This option is useful for interoperability with CBOR decoders that don't
266treat byte strings as a form of text. It is especially useful as Perl
267gives very little control over hash keys.
268
269Enabling this option can be slow, as all downgraded hash keys that are
270encoded need to be scanned and converted to UTF-8.
271
272=item $cbor = $cbor->text_strings ([$enable])
273
274=item $enabled = $cbor->get_text_strings
275
276This option works similar to C<text_keys>, above, but works on all strings
277(including hash keys), so C<text_keys> has no further effect after
278enabling C<text_strings>.
279
280If C<$enabled> is true (or missing), then C<encode> will encode all perl
281strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
282
283If C<$enable> is false (the default), then C<encode> will encode strings
284normally (but see C<text_keys>) - upgraded perl strings (strings
285internally encoded as UTF-8) as CBOR text strings, and downgraded perl
286strings as CBOR byte strings.
287
288This option does not affect C<decode> in any way.
289
290This option has similar advantages and disadvantages as C<text_keys>. In
291addition, this option effectively removes the ability to encode byte
292strings, which might break some C<FREEZE> and C<TO_CBOR> methods that rely
293on this, such as bignum encoding, so this option is mainly useful for very
294simple data.
295
296=item $cbor = $cbor->validate_utf8 ([$enable])
297
298=item $enabled = $cbor->get_validate_utf8
299
300If C<$enable> is true (or missing), then C<decode> will validate that
301elements (text strings) containing UTF-8 data in fact contain valid UTF-8
302data (instead of blindly accepting it). This validation obviously takes
303extra time during decoding.
304
305The concept of "valid UTF-8" used is perl's concept, which is a superset
306of the official UTF-8.
307
308If C<$enable> is false (the default), then C<decode> will blindly accept
309UTF-8 data, marking them as valid UTF-8 in the resulting data structure
310regardless of whether that's true or not.
311
312Perl isn't too happy about corrupted UTF-8 in strings, but should
313generally not crash or do similarly evil things. Extensions might be not
314so forgiving, so it's recommended to turn on this setting if you receive
315untrusted CBOR.
316
317This option does not affect C<encode> in any way - strings that are
318supposedly valid UTF-8 will simply be dumped into the resulting CBOR
319string without checking whether that is, in fact, true or not.
320
321=item $cbor = $cbor->filter ([$cb->($tag, $value)])
322
323=item $cb_or_undef = $cbor->get_filter
324
325Sets or replaces the tagged value decoding filter (when C<$cb> is
326specified) or clears the filter (if no argument or C<undef> is provided).
327
328The filter callback is called only during decoding, when a non-enforced
329tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
330list of enforced tags). For specific tags, it's often better to provide a
331default converter using the C<%CBOR::XS::FILTER> hash (see below).
332
333The first argument is the numerical tag, the second is the (decoded) value
334that has been tagged.
335
336The filter function should return either exactly one value, which will
337replace the tagged value in the decoded data structure, or no values,
338which will result in default handling, which currently means the decoder
339creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
340
341When the filter is cleared (the default state), the default filter
342function, C<CBOR::XS::default_filter>, is used. This function simply looks
343up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
344a code reference that is called with tag and value, and is responsible for
345decoding the value. If no entry exists, it returns no values.
346
347Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
348objects, with no other special handling (useful when working with
349potentially "unsafe" CBOR data).
350
351 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
352
353Example: provide a global filter for tag 1347375694, converting the value
354into some string form.
355
356 $CBOR::XS::FILTER{1347375694} = sub {
357 my ($tag, $value);
358
359 "tag 1347375694 value $value"
360 };
361
126=item $cbor_data = $cbor->encode ($perl_scalar) 362=item $cbor_data = $cbor->encode ($perl_scalar)
127 363
128Converts the given Perl data structure (a scalar value) to its CBOR 364Converts the given Perl data structure (a scalar value) to its CBOR
129representation. 365representation.
130 366
143and you need to know where the first CBOR string ends amd the next one 379and you need to know where the first CBOR string ends amd the next one
144starts. 380starts.
145 381
146 CBOR::XS->new->decode_prefix ("......") 382 CBOR::XS->new->decode_prefix ("......")
147 => ("...", 3) 383 => ("...", 3)
384
385=back
386
387=head2 INCREMENTAL PARSING
388
389In some cases, there is the need for incremental parsing of JSON
390texts. While this module always has to keep both CBOR text and resulting
391Perl data structure in memory at one time, it does allow you to parse a
392CBOR stream incrementally, using a similar to using "decode_prefix" to see
393if a full CBOR object is available, but is much more efficient.
394
395It basically works by parsing as much of a CBOR string as possible - if
396the CBOR data is not complete yet, the pasrer will remember where it was,
397to be able to restart when more data has been accumulated. Once enough
398data is available to either decode a complete CBOR value or raise an
399error, a real decode will be attempted.
400
401A typical use case would be a network protocol that consists of sending
402and receiving CBOR-encoded messages. The solution that works with CBOR and
403about anything else is by prepending a length to every CBOR value, so the
404receiver knows how many octets to read. More compact (and slightly slower)
405would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
406a CBOR value ends, and doesn't need an explicit length.
407
408The following methods help with this:
409
410=over 4
411
412=item @decoded = $cbor->incr_parse ($buffer)
413
414This method attempts to decode exactly one CBOR value from the beginning
415of the given C<$buffer>. The value is removed from the C<$buffer> on
416success. When C<$buffer> doesn't contain a complete value yet, it returns
417nothing. Finally, when the C<$buffer> doesn't start with something
418that could ever be a valid CBOR value, it raises an exception, just as
419C<decode> would. In the latter case the decoder state is undefined and
420must be reset before being able to parse further.
421
422This method modifies the C<$buffer> in place. When no CBOR value can be
423decoded, the decoder stores the current string offset. On the next call,
424continues decoding at the place where it stopped before. For this to make
425sense, the C<$buffer> must begin with the same octets as on previous
426unsuccessful calls.
427
428You can call this method in scalar context, in which case it either
429returns a decoded value or C<undef>. This makes it impossible to
430distinguish between CBOR null values (which decode to C<undef>) and an
431unsuccessful decode, which is often acceptable.
432
433=item @decoded = $cbor->incr_parse_multiple ($buffer)
434
435Same as C<incr_parse>, but attempts to decode as many CBOR values as
436possible in one go, instead of at most one. Calls to C<incr_parse> and
437C<incr_parse_multiple> can be interleaved.
438
439=item $cbor->incr_reset
440
441Resets the incremental decoder. This throws away any saved state, so that
442subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
443a new CBOR value from the beginning of the C<$buffer> again.
444
445This method can be called at any time, but it I<must> be called if you want
446to change your C<$buffer> or there was a decoding error and you want to
447reuse the C<$cbor> object for future incremental parsings.
148 448
149=back 449=back
150 450
151 451
152=head1 MAPPING 452=head1 MAPPING
163 463
164=head2 CBOR -> PERL 464=head2 CBOR -> PERL
165 465
166=over 4 466=over 4
167 467
168=item True, False 468=item integers
169 469
170These CBOR values become C<CBOR::XS::true> and C<CBOR::XS::false>, 470CBOR integers become (numeric) perl scalars. On perls without 64 bit
471support, 64 bit integers will be truncated or otherwise corrupted.
472
473=item byte strings
474
475Byte strings will become octet strings in Perl (the Byte values 0..255
476will simply become characters of the same value in Perl).
477
478=item UTF-8 strings
479
480UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
481decoded into proper Unicode code points. At the moment, the validity of
482the UTF-8 octets will not be validated - corrupt input will result in
483corrupted Perl strings.
484
485=item arrays, maps
486
487CBOR arrays and CBOR maps will be converted into references to a Perl
488array or hash, respectively. The keys of the map will be stringified
489during this process.
490
491=item null
492
493CBOR null becomes C<undef> in Perl.
494
495=item true, false, undefined
496
497These CBOR values become C<Types:Serialiser::true>,
498C<Types:Serialiser::false> and C<Types::Serialiser::error>,
171respectively. They are overloaded to act almost exactly like the numbers 499respectively. They are overloaded to act almost exactly like the numbers
172C<1> and C<0>. You can check whether a scalar is a CBOR boolean by using 500C<1> and C<0> (for true and false) or to throw an exception on access (for
173the C<CBOR::XS::is_bool> function. 501error). See the L<Types::Serialiser> manpage for details.
174 502
175=item Null, Undefined 503=item tagged values
176 504
177CBOR Null and Undefined values becomes C<undef> in Perl (in the future, 505Tagged items consists of a numeric tag and another CBOR value.
178Undefined may raise an exception). 506
507See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
508for details on which tags are handled how.
509
510=item anything else
511
512Anything else (e.g. unsupported simple values) will raise a decoding
513error.
179 514
180=back 515=back
181 516
182 517
183=head2 PERL -> CBOR 518=head2 PERL -> CBOR
184 519
185The mapping from Perl to CBOR is slightly more difficult, as Perl is a 520The mapping from Perl to CBOR is slightly more difficult, as Perl is a
186truly typeless language, so we can only guess which CBOR type is meant by 521typeless language. That means this module can only guess which CBOR type
187a Perl value. 522is meant by a perl value.
188 523
189=over 4 524=over 4
190 525
191=item hash references 526=item hash references
192 527
193Perl hash references become CBOR maps. As there is no inherent ordering 528Perl hash references become CBOR maps. As there is no inherent ordering in
194in hash keys (or CBOR maps), they will usually be encoded in a 529hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
195pseudo-random order. 530order. This order can be different each time a hash is encoded.
531
532Currently, tied hashes will use the indefinite-length format, while normal
533hashes will use the fixed-length format.
196 534
197=item array references 535=item array references
198 536
199Perl array references become CBOR arrays. 537Perl array references become fixed-length CBOR arrays.
200 538
201=item other references 539=item other references
202 540
203Other unblessed references are generally not allowed and will cause an 541Other unblessed references will be represented using
204exception to be thrown, except for references to the integers C<0> and 542the indirection tag extension (tag value C<22098>,
205C<1>, which get turned into C<False> and C<True> in CBOR. 543L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
544to be able to decode these values somehow, by either "doing the right
545thing", decoding into a generic tagged object, simply ignoring the tag, or
546something else.
206 547
207=item CBOR::XS::true, CBOR::XS::false 548=item CBOR::XS::Tagged objects
208 549
550Objects of this type must be arrays consisting of a single C<[tag, value]>
551pair. The (numerical) tag will be encoded as a CBOR tag, the value will
552be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
553create such objects.
554
555=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
556
209These special values become CBOR True and CBOR False values, 557These special values become CBOR true, CBOR false and CBOR undefined
210respectively. You can also use C<\1> and C<\0> directly if you want. 558values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
559if you want.
211 560
212=item blessed objects 561=item other blessed objects
213 562
214Blessed objects are not directly representable in CBOR. TODO 563Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
215See the 564L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
216C<allow_blessed> and C<convert_blessed> methods on various options on 565module, and L<OBJECT SERIALISATION> for generic object serialisation.
217how to deal with this: basically, you can choose between throwing an
218exception, encoding the reference as if it weren't blessed, or provide
219your own serialiser method.
220 566
221=item simple scalars 567=item simple scalars
222 568
223TODO
224Simple Perl scalars (any scalar that is not a reference) are the most 569Simple Perl scalars (any scalar that is not a reference) are the most
225difficult objects to encode: CBOR::XS will encode undefined scalars as 570difficult objects to encode: CBOR::XS will encode undefined scalars as
226CBOR C<Null> values, scalars that have last been used in a string context 571CBOR null values, scalars that have last been used in a string context
227before encoding as CBOR strings, and anything else as number value: 572before encoding as CBOR strings, and anything else as number value:
228 573
229 # dump as number 574 # dump as number
230 encode_cbor [2] # yields [2] 575 encode_cbor [2] # yields [2]
231 encode_cbor [-3.0e17] # yields [-3e+17] 576 encode_cbor [-3.0e17] # yields [-3e+17]
232 my $value = 5; encode_cbor [$value] # yields [5] 577 my $value = 5; encode_cbor [$value] # yields [5]
233 578
234 # used as string, so dump as string 579 # used as string, so dump as string (either byte or text)
235 print $value; 580 print $value;
236 encode_cbor [$value] # yields ["5"] 581 encode_cbor [$value] # yields ["5"]
237 582
238 # undef becomes null 583 # undef becomes null
239 encode_cbor [undef] # yields [null] 584 encode_cbor [undef] # yields [null]
243 my $x = 3.1; # some variable containing a number 588 my $x = 3.1; # some variable containing a number
244 "$x"; # stringified 589 "$x"; # stringified
245 $x .= ""; # another, more awkward way to stringify 590 $x .= ""; # another, more awkward way to stringify
246 print $x; # perl does it for you, too, quite often 591 print $x; # perl does it for you, too, quite often
247 592
593You can force whether a string is encoded as byte or text string by using
594C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled):
595
596 utf8::upgrade $x; # encode $x as text string
597 utf8::downgrade $x; # encode $x as byte string
598
599Perl doesn't define what operations up- and downgrade strings, so if the
600difference between byte and text is important, you should up- or downgrade
601your string as late as possible before encoding. You can also force the
602use of CBOR text strings by using C<text_keys> or C<text_strings>.
603
248You can force the type to be a CBOR number by numifying it: 604You can force the type to be a CBOR number by numifying it:
249 605
250 my $x = "3"; # some variable containing a string 606 my $x = "3"; # some variable containing a string
251 $x += 0; # numify it, ensuring it will be dumped as a number 607 $x += 0; # numify it, ensuring it will be dumped as a number
252 $x *= 1; # same thing, the choice is yours. 608 $x *= 1; # same thing, the choice is yours.
253 609
254You can not currently force the type in other, less obscure, ways. Tell me 610You can not currently force the type in other, less obscure, ways. Tell me
255if you need this capability (but don't forget to explain why it's needed 611if you need this capability (but don't forget to explain why it's needed
256:). 612:).
257 613
258Note that numerical precision has the same meaning as under Perl (so 614Perl values that seem to be integers generally use the shortest possible
259binary to decimal conversion follows the same rules as in Perl, which 615representation. Floating-point values will use either the IEEE single
260can differ to other languages). Also, your perl interpreter might expose 616format if possible without loss of precision, otherwise the IEEE double
261extensions to the floating point numbers of your platform, such as 617format will be used. Perls that use formats other than IEEE double to
262infinities or NaN's - these cannot be represented in CBOR, and it is an 618represent numerical values are supported, but might suffer loss of
263error to pass those in. 619precision.
264 620
265=back 621=back
266 622
623=head2 OBJECT SERIALISATION
267 624
625This module implements both a CBOR-specific and the generic
626L<Types::Serialier> object serialisation protocol. The following
627subsections explain both methods.
628
629=head3 ENCODING
630
631This module knows two way to serialise a Perl object: The CBOR-specific
632way, and the generic way.
633
634Whenever the encoder encounters a Perl object that it cannot serialise
635directly (most of them), it will first look up the C<TO_CBOR> method on
636it.
637
638If it has a C<TO_CBOR> method, it will call it with the object as only
639argument, and expects exactly one return value, which it will then
640substitute and encode it in the place of the object.
641
642Otherwise, it will look up the C<FREEZE> method. If it exists, it will
643call it with the object as first argument, and the constant string C<CBOR>
644as the second argument, to distinguish it from other serialisers.
645
646The C<FREEZE> method can return any number of values (i.e. zero or
647more). These will be encoded as CBOR perl object, together with the
648classname.
649
650These methods I<MUST NOT> change the data structure that is being
651serialised. Failure to comply to this can result in memory corruption -
652and worse.
653
654If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
655with an error.
656
657=head3 DECODING
658
659Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
660but objects encoded via C<FREEZE> can be decoded using the following
661protocol:
662
663When an encoded CBOR perl object is encountered by the decoder, it will
664look up the C<THAW> method, by using the stored classname, and will fail
665if the method cannot be found.
666
667After the lookup it will call the C<THAW> method with the stored classname
668as first argument, the constant string C<CBOR> as second argument, and all
669values returned by C<FREEZE> as remaining arguments.
670
671=head3 EXAMPLES
672
673Here is an example C<TO_CBOR> method:
674
675 sub My::Object::TO_CBOR {
676 my ($obj) = @_;
677
678 ["this is a serialised My::Object object", $obj->{id}]
679 }
680
681When a C<My::Object> is encoded to CBOR, it will instead encode a simple
682array with two members: a string, and the "object id". Decoding this CBOR
683string will yield a normal perl array reference in place of the object.
684
685A more useful and practical example would be a serialisation method for
686the URI module. CBOR has a custom tag value for URIs, namely 32:
687
688 sub URI::TO_CBOR {
689 my ($self) = @_;
690 my $uri = "$self"; # stringify uri
691 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
692 CBOR::XS::tag 32, "$_[0]"
693 }
694
695This will encode URIs as a UTF-8 string with tag 32, which indicates an
696URI.
697
698Decoding such an URI will not (currently) give you an URI object, but
699instead a CBOR::XS::Tagged object with tag number 32 and the string -
700exactly what was returned by C<TO_CBOR>.
701
702To serialise an object so it can automatically be deserialised, you need
703to use C<FREEZE> and C<THAW>. To take the URI module as example, this
704would be a possible implementation:
705
706 sub URI::FREEZE {
707 my ($self, $serialiser) = @_;
708 "$self" # encode url string
709 }
710
711 sub URI::THAW {
712 my ($class, $serialiser, $uri) = @_;
713 $class->new ($uri)
714 }
715
716Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
717example, a C<FREEZE> method that returns "type", "id" and "variant" values
718would cause an invocation of C<THAW> with 5 arguments:
719
720 sub My::Object::FREEZE {
721 my ($self, $serialiser) = @_;
722
723 ($self->{type}, $self->{id}, $self->{variant})
724 }
725
726 sub My::Object::THAW {
727 my ($class, $serialiser, $type, $id, $variant) = @_;
728
729 $class-<new (type => $type, id => $id, variant => $variant)
730 }
731
732
268=head2 MAGIC HEADER 733=head1 MAGIC HEADER
269 734
270There is no way to distinguish CBOR from other formats 735There is no way to distinguish CBOR from other formats
271programmatically. To make it easier to distinguish CBOR from other 736programmatically. To make it easier to distinguish CBOR from other
272formats, the CBOR specification has a special "magic string" that can be 737formats, the CBOR specification has a special "magic string" that can be
273prepended to any CBOR string without changing it's meaning. 738prepended to any CBOR string without changing its meaning.
274 739
275This string is available as C<$CBOR::XS::MAGIC>. This module does not 740This string is available as C<$CBOR::XS::MAGIC>. This module does not
276prepend this string tot he CBOR data it generates, but it will ignroe it 741prepend this string to the CBOR data it generates, but it will ignore it
277if present, so users can prepend this string as a "file type" indicator as 742if present, so users can prepend this string as a "file type" indicator as
278required. 743required.
279 744
280 745
746=head1 THE CBOR::XS::Tagged CLASS
747
748CBOR has the concept of tagged values - any CBOR value can be tagged with
749a numeric 64 bit number, which are centrally administered.
750
751C<CBOR::XS> handles a few tags internally when en- or decoding. You can
752also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
753decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
754unknown tag.
755
756These objects are simply blessed array references - the first member of
757the array being the numerical tag, the second being the value.
758
759You can interact with C<CBOR::XS::Tagged> objects in the following ways:
760
761=over 4
762
763=item $tagged = CBOR::XS::tag $tag, $value
764
765This function(!) creates a new C<CBOR::XS::Tagged> object using the given
766C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
767value that can be encoded in CBOR, including serialisable Perl objects and
768C<CBOR::XS::Tagged> objects).
769
770=item $tagged->[0]
771
772=item $tagged->[0] = $new_tag
773
774=item $tag = $tagged->tag
775
776=item $new_tag = $tagged->tag ($new_tag)
777
778Access/mutate the tag.
779
780=item $tagged->[1]
781
782=item $tagged->[1] = $new_value
783
784=item $value = $tagged->value
785
786=item $new_value = $tagged->value ($new_value)
787
788Access/mutate the tagged value.
789
790=back
791
792=cut
793
794sub tag($$) {
795 bless [@_], CBOR::XS::Tagged::;
796}
797
798sub CBOR::XS::Tagged::tag {
799 $_[0][0] = $_[1] if $#_;
800 $_[0][0]
801}
802
803sub CBOR::XS::Tagged::value {
804 $_[0][1] = $_[1] if $#_;
805 $_[0][1]
806}
807
808=head2 EXAMPLES
809
810Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
811
812You can look up CBOR tag value and emanings in the IANA registry at
813L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
814
815Prepend a magic header (C<$CBOR::XS::MAGIC>):
816
817 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
818 # same as:
819 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
820
821Serialise some URIs and a regex in an array:
822
823 my $cbor = encode_cbor [
824 (CBOR::XS::tag 32, "http://www.nethype.de/"),
825 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
826 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
827 ];
828
829Wrap CBOR data in CBOR:
830
831 my $cbor_cbor = encode_cbor
832 CBOR::XS::tag 24,
833 encode_cbor [1, 2, 3];
834
835=head1 TAG HANDLING AND EXTENSIONS
836
837This section describes how this module handles specific tagged values
838and extensions. If a tag is not mentioned here and no additional filters
839are provided for it, then the default handling applies (creating a
840CBOR::XS::Tagged object on decoding, and only encoding the tag when
841explicitly requested).
842
843Tags not handled specifically are currently converted into a
844L<CBOR::XS::Tagged> object, which is simply a blessed array reference
845consisting of the numeric tag value followed by the (decoded) CBOR value.
846
847Future versions of this module reserve the right to special case
848additional tags (such as base64url).
849
850=head2 ENFORCED TAGS
851
852These tags are always handled when decoding, and their handling cannot be
853overridden by the user.
854
855=over 4
856
857=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
858
859These tags are automatically created (and decoded) for serialisable
860objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
861serialisation protocol). See L<OBJECT SERIALISATION> for details.
862
863=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
864
865These tags are automatically decoded when encountered (and they do not
866result in a cyclic data structure, see C<allow_cycles>), resulting in
867shared values in the decoded object. They are only encoded, however, when
868C<allow_sharing> is enabled.
869
870Not all shared values can be successfully decoded: values that reference
871themselves will I<currently> decode as C<undef> (this is not the same
872as a reference pointing to itself, which will be represented as a value
873that contains an indirect reference to itself - these will be decoded
874properly).
875
876Note that considerably more shared value data structures can be decoded
877than will be encoded - currently, only values pointed to by references
878will be shared, others will not. While non-reference shared values can be
879generated in Perl with some effort, they were considered too unimportant
880to be supported in the encoder. The decoder, however, will decode these
881values as shared values.
882
883=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
884
885These tags are automatically decoded when encountered. They are only
886encoded, however, when C<pack_strings> is enabled.
887
888=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
889
890This tag is automatically generated when a reference are encountered (with
891the exception of hash and array references). It is converted to a reference
892when decoding.
893
894=item 55799 (self-describe CBOR, RFC 7049)
895
896This value is not generated on encoding (unless explicitly requested by
897the user), and is simply ignored when decoding.
898
899=back
900
901=head2 NON-ENFORCED TAGS
902
903These tags have default filters provided when decoding. Their handling can
904be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
905providing a custom C<filter> callback when decoding.
906
907When they result in decoding into a specific Perl class, the module
908usually provides a corresponding C<TO_CBOR> method as well.
909
910When any of these need to load additional modules that are not part of the
911perl core distribution (e.g. L<URI>), it is (currently) up to the user to
912provide these modules. The decoding usually fails with an exception if the
913required module cannot be loaded.
914
915=over 4
916
917=item 0, 1 (date/time string, seconds since the epoch)
918
919These tags are decoded into L<Time::Piece> objects. The corresponding
920C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
921
922The L<Time::Piece> API is generally surprisingly bad, and fractional
923seconds are only accidentally kept intact, so watch out. On the plus side,
924the module comes with perl since 5.10, which has to count for something.
925
926=item 2, 3 (positive/negative bignum)
927
928These tags are decoded into L<Math::BigInt> objects. The corresponding
929C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
930integers, and others into positive/negative CBOR bignums.
931
932=item 4, 5, 264, 265 (decimal fraction/bigfloat)
933
934Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
935objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
936encodes into a decimal fraction (either tag 4 or 264).
937
938NaN and infinities are not encoded properly, as they cannot be represented
939in CBOR.
940
941See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
942
943=item 30 (rational numbers)
944
945These tags are decoded into L<Math::BigRat> objects. The corresponding
946C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
947C<1> via their numerator only, i.e., they become normal integers or
948C<bignums>.
949
950See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
951
952=item 21, 22, 23 (expected later JSON conversion)
953
954CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
955tags.
956
957=item 32 (URI)
958
959These objects decode into L<URI> objects. The corresponding
960C<URI::TO_CBOR> method again results in a CBOR URI value.
961
962=back
963
964=cut
965
281=head2 CBOR and JSON 966=head1 CBOR and JSON
282 967
283TODO 968CBOR is supposed to implement a superset of the JSON data model, and is,
969with some coercion, able to represent all JSON texts (something that other
970"binary JSON" formats such as BSON generally do not support).
971
972CBOR implements some extra hints and support for JSON interoperability,
973and the spec offers further guidance for conversion between CBOR and
974JSON. None of this is currently implemented in CBOR, and the guidelines
975in the spec do not result in correct round-tripping of data. If JSON
976interoperability is improved in the future, then the goal will be to
977ensure that decoded JSON data will round-trip encoding and decoding to
978CBOR intact.
284 979
285 980
286=head1 SECURITY CONSIDERATIONS 981=head1 SECURITY CONSIDERATIONS
287 982
288When you are using CBOR in a protocol, talking to untrusted potentially 983When you are using CBOR in a protocol, talking to untrusted potentially
316Also keep in mind that CBOR::XS might leak contents of your Perl data 1011Also keep in mind that CBOR::XS might leak contents of your Perl data
317structures in its error messages, so when you serialise sensitive 1012structures in its error messages, so when you serialise sensitive
318information you might want to make sure that exceptions thrown by CBOR::XS 1013information you might want to make sure that exceptions thrown by CBOR::XS
319will not end up in front of untrusted eyes. 1014will not end up in front of untrusted eyes.
320 1015
1016
1017=head1 BIGNUM SECURITY CONSIDERATIONS
1018
1019CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1020L<Math::BigFloat> that tries to encode the number in the simplest possible
1021way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
10224) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1023(L<Math::BigRat>, tag 30) can also contain bignums as members.
1024
1025CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1026bigfloats (tags 5 and 265), but it will never generate these on its own.
1027
1028Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1029decimal fractions is generally fast. Decoding bigints can be slow for very
1030big numbers (tens of thousands of digits, something that could potentially
1031be caught by limiting the size of CBOR texts), and decoding bigfloats or
1032arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1033for large exponents (roughly 40 bit and longer).
1034
1035Additionally, L<Math::BigInt> can take advantage of other bignum
1036libraries, such as L<Math::GMP>, which cannot handle big floats with large
1037exponents, and might simply abort or crash your program, due to their code
1038quality.
1039
1040This can be a concern if you want to parse untrusted CBOR. If it is, you
1041might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1042types. You should also disable types 5 and 265, as these can be slow even
1043without bigints.
1044
1045Disabling bigints will also partially or fully disable types that rely on
1046them, e.g. rational numbers that use bignums.
1047
1048
321=head1 CBOR IMPLEMENTATION NOTES 1049=head1 CBOR IMPLEMENTATION NOTES
322 1050
323This section contains some random implementation notes. They do not 1051This section contains some random implementation notes. They do not
324describe guaranteed behaviour, but merely behaviour as-is implemented 1052describe guaranteed behaviour, but merely behaviour as-is implemented
325right now. 1053right now.
333Only the double data type is supported for NV data types - when Perl uses 1061Only the double data type is supported for NV data types - when Perl uses
334long double to represent floating point values, they might not be encoded 1062long double to represent floating point values, they might not be encoded
335properly. Half precision types are accepted, but not encoded. 1063properly. Half precision types are accepted, but not encoded.
336 1064
337Strict mode and canonical mode are not implemented. 1065Strict mode and canonical mode are not implemented.
1066
1067
1068=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
1069
1070On perls that were built without 64 bit integer support (these are rare
1071nowadays, even on 32 bit architectures, as all major Perl distributions
1072are built with 64 bit integer support), support for any kind of 64 bit
1073integer in CBOR is very limited - most likely, these 64 bit values will
1074be truncated, corrupted, or otherwise not decoded correctly. This also
1075includes string, array and map sizes that are stored as 64 bit integers.
338 1076
339 1077
340=head1 THREADS 1078=head1 THREADS
341 1079
342This module is I<not> guaranteed to be thread safe and there are no 1080This module is I<not> guaranteed to be thread safe and there are no
356Please refrain from using rt.cpan.org or any other bug reporting 1094Please refrain from using rt.cpan.org or any other bug reporting
357service. I put the contact address into my modules for a reason. 1095service. I put the contact address into my modules for a reason.
358 1096
359=cut 1097=cut
360 1098
361our $true = do { bless \(my $dummy = 1), "CBOR::XS::Boolean" }; 1099our %FILTER = (
362our $false = do { bless \(my $dummy = 0), "CBOR::XS::Boolean" }; 1100 0 => sub { # rfc4287 datetime, utf-8
1101 require Time::Piece;
1102 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1103 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1104 # else either. Whats incredibe over standard strptime totally escapes me.
1105 # doesn't do fractional times, either. sigh.
1106 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1107 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1108 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1109 scalar eval {
1110 my $s = $_[1];
363 1111
364sub true() { $true } 1112 $s =~ s/Z$/+00:00/;
365sub false() { $false } 1113 $s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
1114 or die;
366 1115
367sub is_bool($) { 1116 my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
368 UNIVERSAL::isa $_[0], "CBOR::XS::Boolean" 1117 my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
369# or UNIVERSAL::isa $_[0], "CBOR::Literal" 1118
1119 Time::Piece::gmtime ($d->epoch + $b)
1120 } || die "corrupted CBOR date/time string ($_[0])";
1121 },
1122
1123 1 => sub { # seconds since the epoch, possibly fractional
1124 require Time::Piece;
1125 scalar Time::Piece::gmtime (pop)
1126 },
1127
1128 2 => sub { # pos bigint
1129 require Math::BigInt;
1130 Math::BigInt->new ("0x" . unpack "H*", pop)
1131 },
1132
1133 3 => sub { # neg bigint
1134 require Math::BigInt;
1135 -Math::BigInt->new ("0x" . unpack "H*", pop)
1136 },
1137
1138 4 => sub { # decimal fraction, array
1139 require Math::BigFloat;
1140 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1141 },
1142
1143 264 => sub { # decimal fraction with arbitrary exponent
1144 require Math::BigFloat;
1145 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1146 },
1147
1148 5 => sub { # bigfloat, array
1149 require Math::BigFloat;
1150 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1151 },
1152
1153 265 => sub { # bigfloat with arbitrary exponent
1154 require Math::BigFloat;
1155 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1156 },
1157
1158 30 => sub { # rational number
1159 require Math::BigRat;
1160 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
1161 },
1162
1163 21 => sub { pop }, # expected conversion to base64url encoding
1164 22 => sub { pop }, # expected conversion to base64 encoding
1165 23 => sub { pop }, # expected conversion to base16 encoding
1166
1167 # 24 # embedded cbor, byte string
1168
1169 32 => sub {
1170 require URI;
1171 URI->new (pop)
1172 },
1173
1174 # 33 # base64url rfc4648, utf-8
1175 # 34 # base64 rfc46484, utf-8
1176 # 35 # regex pcre/ecma262, utf-8
1177 # 36 # mime message rfc2045, utf-8
1178);
1179
1180sub CBOR::XS::default_filter {
1181 &{ $FILTER{$_[0]} or return }
370} 1182}
371 1183
1184sub URI::TO_CBOR {
1185 my $uri = $_[0]->as_string;
1186 utf8::upgrade $uri;
1187 tag 32, $uri
1188}
1189
1190sub Math::BigInt::TO_CBOR {
1191 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
1192 $_[0]->numify
1193 } else {
1194 my $hex = substr $_[0]->as_hex, 2;
1195 $hex = "0$hex" if 1 & length $hex; # sigh
1196 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
1197 }
1198}
1199
1200sub Math::BigFloat::TO_CBOR {
1201 my ($m, $e) = $_[0]->parts;
1202
1203 -9223372036854775808 <= $e && $e <= 18446744073709551615
1204 ? tag 4, [$e->numify, $m]
1205 : tag 264, [$e, $m]
1206}
1207
1208sub Math::BigRat::TO_CBOR {
1209 my ($n, $d) = $_[0]->parts;
1210
1211 # older versions of BigRat need *1, as they not always return numbers
1212
1213 $d*1 == 1
1214 ? $n*1
1215 : tag 30, [$n*1, $d*1]
1216}
1217
1218sub Time::Piece::TO_CBOR {
1219 tag 1, 0 + $_[0]->epoch
1220}
1221
372XSLoader::load "CBOR::XS", $VERSION; 1222XSLoader::load "CBOR::XS", $VERSION;
373
374package CBOR::XS::Boolean;
375
376use overload
377 "0+" => sub { ${$_[0]} },
378 "++" => sub { $_[0] = ${$_[0]} + 1 },
379 "--" => sub { $_[0] = ${$_[0]} - 1 },
380 fallback => 1;
381
3821;
383 1223
384=head1 SEE ALSO 1224=head1 SEE ALSO
385 1225
386The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 1226The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,
387serialisation. 1227serialisation.
388 1228
1229The L<Types::Serialiser> module provides the data model for true, false
1230and error values.
1231
389=head1 AUTHOR 1232=head1 AUTHOR
390 1233
391 Marc Lehmann <schmorp@schmorp.de> 1234 Marc Lehmann <schmorp@schmorp.de>
392 http://home.schmorp.de/ 1235 http://home.schmorp.de/
393 1236
394=cut 1237=cut
395 1238
12391
1240

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines