ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.23 by root, Fri Nov 22 16:00:30 2013 UTC vs.
Revision 1.30 by root, Sat Nov 30 16:19:59 2013 UTC

… …
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up
32to you to do). Furthermore, details of the implementation might change
33freely before version 1.0. And lastly, most extensions depend on an IANA
34assignment, and until that assignment is official, this implementation is
35not interoperable with other implementations (even future versions of this
36module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 40data later and speed is less important you might want to compare both
41formats first).
50 42
51To give you a general idea about speed, with texts in the megabyte range, 43To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or 44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the 45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison. 46data, the worse L<Storable> performs in comparison.
55 47
56As for compactness, C<CBOR::XS> encoded data structures are usually about 48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
5720% smaller than the same data encoded as (compact) JSON or L<Storable>. 49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
58 51
59In addition to the core CBOR data format, this module implements a number 52In addition to the core CBOR data format, this module implements a
60of extensions, to support cyclic and self-referencing data structures 53number of extensions, to support cyclic and shared data structures (see
61(see C<allow_sharing>), string deduplication (see C<allow_stringref>) and 54C<allow_sharing>), string deduplication (see C<pack_strings>) and scalar
62scalar references (always enabled). 55references (always enabled).
63 56
64The primary goal of this module is to be I<correct> and the secondary goal 57The primary goal of this module is to be I<correct> and the secondary goal
65is to be I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
66 59
67See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
… …
71 64
72package CBOR::XS; 65package CBOR::XS;
73 66
74use common::sense; 67use common::sense;
75 68
76our $VERSION = 0.08; 69our $VERSION = '1.0';
77our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
78 71
79our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
80 73
81use Exporter; 74use Exporter;
… …
186as an array, is referenced multiple times), but instead will emit a 179as an array, is referenced multiple times), but instead will emit a
187reference to the earlier value. 180reference to the earlier value.
188 181
189This means that such values will only be encoded once, and will not result 182This means that such values will only be encoded once, and will not result
190in a deep cloning of the value on decode, in decoders supporting the value 183in a deep cloning of the value on decode, in decoders supporting the value
191sharing extension. 184sharing extension. This also makes it possible to encode cyclic data
185structures.
192 186
193It is recommended to leave it off unless you know your 187It is recommended to leave it off unless you know your
194communication partner supports the value sharing extensions to CBOR 188communication partner supports the value sharing extensions to CBOR
195(http://cbor.schmorp.de/value-sharing). 189(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
190resulting data structure might be unusable.
196 191
197Detecting shared values incurs a runtime overhead when values are encoded 192Detecting shared values incurs a runtime overhead when values are encoded
198that have a reference counter large than one, and might unnecessarily 193that have a reference counter large than one, and might unnecessarily
199increase the encoded size, as potentially shared values are encode as 194increase the encoded size, as potentially shared values are encode as
200sharable whether or not they are actually shared. 195sharable whether or not they are actually shared.
201 196
202At the moment, only targets of references can be shared (e.g. scalars, 197At the moment, only targets of references can be shared (e.g. scalars,
203arrays or hashes pointed to by a reference). Weirder constructs, such as 198arrays or hashes pointed to by a reference). Weirder constructs, such as
204an array with multiple "copies" of the I<same> string, which are hard but 199an array with multiple "copies" of the I<same> string, which are hard but
205not impossible to create in Perl, are not supported (this is the same as 200not impossible to create in Perl, are not supported (this is the same as
206for L<Storable>). 201with L<Storable>).
207 202
208If C<$enable> is false (the default), then C<encode> will encode 203If C<$enable> is false (the default), then C<encode> will encode shared
209exception when it encounters anything it cannot encode as CBOR. 204data structures repeatedly, unsharing them in the process. Cyclic data
205structures cannot be encoded in this mode.
210 206
211This option does not affect C<decode> in any way - shared values and 207This option does not affect C<decode> in any way - shared values and
212references will always be decoded properly if present. 208references will always be decoded properly if present.
213 209
214=item $cbor = $cbor->allow_stringref ([$enable]) 210=item $cbor = $cbor->pack_strings ([$enable])
215 211
216=item $enabled = $cbor->get_allow_stringref 212=item $enabled = $cbor->get_pack_strings
217 213
218If C<$enable> is true (or missing), then C<encode> will try not to encode 214If C<$enable> is true (or missing), then C<encode> will try not to encode
219the same string twice, but will instead encode a reference to the string 215the same string twice, but will instead encode a reference to the string
220instead. Depending on your data format. this can save a lot of space, but 216instead. Depending on your data format, this can save a lot of space, but
221also results in a very large runtime overhead (expect encoding times to be 217also results in a very large runtime overhead (expect encoding times to be
2222-4 times as high as without). 2182-4 times as high as without).
223 219
224It is recommended to leave it off unless you know your 220It is recommended to leave it off unless you know your
225communications partner supports the stringref extension to CBOR 221communications partner supports the stringref extension to CBOR
226(http://cbor.schmorp.de/stringref). 222(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
223resulting data structure might not be usable.
227 224
228If C<$enable> is false (the default), then C<encode> will encode 225If C<$enable> is false (the default), then C<encode> will encode strings
229exception when it encounters anything it cannot encode as CBOR. 226the standard CBOR way.
230 227
231This option does not affect C<decode> in any way - string references will 228This option does not affect C<decode> in any way - string references will
232always be decoded properly if present. 229always be decoded properly if present.
233 230
234=item $cbor = $cbor->filter ([$cb->($tag, $value)]) 231=item $cbor = $cbor->filter ([$cb->($tag, $value)])
235 232
236=item $cb_or_undef = $cbor->get_filter 233=item $cb_or_undef = $cbor->get_filter
237 234
238TODO 235Sets or replaces the tagged value decoding filter (when C<$cb> is
236specified) or clears the filter (if no argument or C<undef> is provided).
237
238The filter callback is called only during decoding, when a non-enforced
239tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
240list of enforced tags). For specific tags, it's often better to provide a
241default converter using the C<%CBOR::XS::FILTER> hash (see below).
242
243The first argument is the numerical tag, the second is the (decoded) value
244that has been tagged.
245
246The filter function should return either exactly one value, which will
247replace the tagged value in the decoded data structure, or no values,
248which will result in default handling, which currently means the decoder
249creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
250
251When the filter is cleared (the default state), the default filter
252function, C<CBOR::XS::default_filter>, is used. This function simply looks
253up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
254a code reference that is called with tag and value, and is responsible for
255decoding the value. If no entry exists, it returns no values.
256
257Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
258objects, with no other special handling (useful when working with
259potentially "unsafe" CBOR data).
260
261 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
262
263Example: provide a global filter for tag 1347375694, converting the value
264into some string form.
265
266 $CBOR::XS::FILTER{1347375694} = sub {
267 my ($tag, $value);
268
269 "tag 1347375694 value $value"
270 };
239 271
240=item $cbor_data = $cbor->encode ($perl_scalar) 272=item $cbor_data = $cbor->encode ($perl_scalar)
241 273
242Converts the given Perl data structure (a scalar value) to its CBOR 274Converts the given Perl data structure (a scalar value) to its CBOR
243representation. 275representation.
… …
284CBOR integers become (numeric) perl scalars. On perls without 64 bit 316CBOR integers become (numeric) perl scalars. On perls without 64 bit
285support, 64 bit integers will be truncated or otherwise corrupted. 317support, 64 bit integers will be truncated or otherwise corrupted.
286 318
287=item byte strings 319=item byte strings
288 320
289Byte strings will become octet strings in Perl (the byte values 0..255 321Byte strings will become octet strings in Perl (the Byte values 0..255
290will simply become characters of the same value in Perl). 322will simply become characters of the same value in Perl).
291 323
292=item UTF-8 strings 324=item UTF-8 strings
293 325
294UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 326UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
… …
317=item tagged values 349=item tagged values
318 350
319Tagged items consists of a numeric tag and another CBOR value. 351Tagged items consists of a numeric tag and another CBOR value.
320 352
321See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >> 353See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
322for details. 354for details on which tags are handled how.
323 355
324=item anything else 356=item anything else
325 357
326Anything else (e.g. unsupported simple values) will raise a decoding 358Anything else (e.g. unsupported simple values) will raise a decoding
327error. 359error.
… …
330 362
331 363
332=head2 PERL -> CBOR 364=head2 PERL -> CBOR
333 365
334The mapping from Perl to CBOR is slightly more difficult, as Perl is a 366The mapping from Perl to CBOR is slightly more difficult, as Perl is a
335truly typeless language, so we can only guess which CBOR type is meant by 367typeless language. That means this module can only guess which CBOR type
336a Perl value. 368is meant by a perl value.
337 369
338=over 4 370=over 4
339 371
340=item hash references 372=item hash references
341 373
342Perl hash references become CBOR maps. As there is no inherent ordering in 374Perl hash references become CBOR maps. As there is no inherent ordering in
343hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 375hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
344order. 376order. This order can be different each time a hahs is encoded.
345 377
346Currently, tied hashes will use the indefinite-length format, while normal 378Currently, tied hashes will use the indefinite-length format, while normal
347hashes will use the fixed-length format. 379hashes will use the fixed-length format.
348 380
349=item array references 381=item array references
350 382
351Perl array references become fixed-length CBOR arrays. 383Perl array references become fixed-length CBOR arrays.
352 384
353=item other references 385=item other references
354 386
355Other unblessed references are generally not allowed and will cause an 387Other unblessed references will be represented using
356exception to be thrown, except for references to the integers C<0> and 388the indirection tag extension (tag value C<22098>,
357C<1>, which get turned into false and true in CBOR. 389L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
390to be able to decode these values somehow, by either "doing the right
391thing", decoding into a generic tagged object, simply ignoring the tag, or
392something else.
358 393
359=item CBOR::XS::Tagged objects 394=item CBOR::XS::Tagged objects
360 395
361Objects of this type must be arrays consisting of a single C<[tag, value]> 396Objects of this type must be arrays consisting of a single C<[tag, value]>
362pair. The (numerical) tag will be encoded as a CBOR tag, the value will 397pair. The (numerical) tag will be encoded as a CBOR tag, the value will
363be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to 398be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
364create such objects. 399create such objects.
365 400
366=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 401=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
367 402
368These special values become CBOR true, CBOR false and CBOR undefined 403These special values become CBOR true, CBOR false and CBOR undefined
… …
385 # dump as number 420 # dump as number
386 encode_cbor [2] # yields [2] 421 encode_cbor [2] # yields [2]
387 encode_cbor [-3.0e17] # yields [-3e+17] 422 encode_cbor [-3.0e17] # yields [-3e+17]
388 my $value = 5; encode_cbor [$value] # yields [5] 423 my $value = 5; encode_cbor [$value] # yields [5]
389 424
390 # used as string, so dump as string 425 # used as string, so dump as string (either byte or text)
391 print $value; 426 print $value;
392 encode_cbor [$value] # yields ["5"] 427 encode_cbor [$value] # yields ["5"]
393 428
394 # undef becomes null 429 # undef becomes null
395 encode_cbor [undef] # yields [null] 430 encode_cbor [undef] # yields [null]
… …
398 433
399 my $x = 3.1; # some variable containing a number 434 my $x = 3.1; # some variable containing a number
400 "$x"; # stringified 435 "$x"; # stringified
401 $x .= ""; # another, more awkward way to stringify 436 $x .= ""; # another, more awkward way to stringify
402 print $x; # perl does it for you, too, quite often 437 print $x; # perl does it for you, too, quite often
438
439You can force whether a string ie encoded as byte or text string by using
440C<utf8::upgrade> and C<utf8::downgrade>):
441
442 utf8::upgrade $x; # encode $x as text string
443 utf8::downgrade $x; # encode $x as byte string
444
445Perl doesn't define what operations up- and downgrade strings, so if the
446difference between byte and text is important, you should up- or downgrade
447your string as late as possible before encoding.
403 448
404You can force the type to be a CBOR number by numifying it: 449You can force the type to be a CBOR number by numifying it:
405 450
406 my $x = "3"; # some variable containing a string 451 my $x = "3"; # some variable containing a string
407 $x += 0; # numify it, ensuring it will be dumped as a number 452 $x += 0; # numify it, ensuring it will be dumped as a number
… …
420 465
421=back 466=back
422 467
423=head2 OBJECT SERIALISATION 468=head2 OBJECT SERIALISATION
424 469
470This module implements both a CBOR-specific and the generic
471L<Types::Serialier> object serialisation protocol. The following
472subsections explain both methods.
473
474=head3 ENCODING
475
425This module knows two way to serialise a Perl object: The CBOR-specific 476This module knows two way to serialise a Perl object: The CBOR-specific
426way, and the generic way. 477way, and the generic way.
427 478
428Whenever the encoder encounters a Perl object that it cnanot serialise 479Whenever the encoder encounters a Perl object that it cannot serialise
429directly (most of them), it will first look up the C<TO_CBOR> method on 480directly (most of them), it will first look up the C<TO_CBOR> method on
430it. 481it.
431 482
432If it has a C<TO_CBOR> method, it will call it with the object as only 483If it has a C<TO_CBOR> method, it will call it with the object as only
433argument, and expects exactly one return value, which it will then 484argument, and expects exactly one return value, which it will then
… …
439 490
440The C<FREEZE> method can return any number of values (i.e. zero or 491The C<FREEZE> method can return any number of values (i.e. zero or
441more). These will be encoded as CBOR perl object, together with the 492more). These will be encoded as CBOR perl object, together with the
442classname. 493classname.
443 494
495These methods I<MUST NOT> change the data structure that is being
496serialised. Failure to comply to this can result in memory corruption -
497and worse.
498
444If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail 499If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
445with an error. 500with an error.
446 501
502=head3 DECODING
503
447Objects encoded via C<TO_CBOR> cannot be automatically decoded, but 504Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
448objects encoded via C<FREEZE> can be decoded using the following protocol: 505but objects encoded via C<FREEZE> can be decoded using the following
506protocol:
449 507
450When an encoded CBOR perl object is encountered by the decoder, it will 508When an encoded CBOR perl object is encountered by the decoder, it will
451look up the C<THAW> method, by using the stored classname, and will fail 509look up the C<THAW> method, by using the stored classname, and will fail
452if the method cannot be found. 510if the method cannot be found.
453 511
454After the lookup it will call the C<THAW> method with the stored classname 512After the lookup it will call the C<THAW> method with the stored classname
455as first argument, the constant string C<CBOR> as second argument, and all 513as first argument, the constant string C<CBOR> as second argument, and all
456values returned by C<FREEZE> as remaining arguments. 514values returned by C<FREEZE> as remaining arguments.
457 515
458=head4 EXAMPLES 516=head3 EXAMPLES
459 517
460Here is an example C<TO_CBOR> method: 518Here is an example C<TO_CBOR> method:
461 519
462 sub My::Object::TO_CBOR { 520 sub My::Object::TO_CBOR {
463 my ($obj) = @_; 521 my ($obj) = @_;
… …
474 532
475 sub URI::TO_CBOR { 533 sub URI::TO_CBOR {
476 my ($self) = @_; 534 my ($self) = @_;
477 my $uri = "$self"; # stringify uri 535 my $uri = "$self"; # stringify uri
478 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 536 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
479 CBOR::XS::tagged 32, "$_[0]" 537 CBOR::XS::tag 32, "$_[0]"
480 } 538 }
481 539
482This will encode URIs as a UTF-8 string with tag 32, which indicates an 540This will encode URIs as a UTF-8 string with tag 32, which indicates an
483URI. 541URI.
484 542
… …
640These tags are always handled when decoding, and their handling cannot be 698These tags are always handled when decoding, and their handling cannot be
641overriden by the user. 699overriden by the user.
642 700
643=over 4 701=over 4
644 702
645=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>) 703=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
646 704
647These tags are automatically created (and decoded) for serialisable 705These tags are automatically created (and decoded) for serialisable
648objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object 706objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
649serialisation protocol). See L<OBJECT SERIALISATION> for details. 707serialisation protocol). See L<OBJECT SERIALISATION> for details.
650 708
651=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>) 709=item 28, 29 (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
652 710
653These tags are automatically decoded when encountered, resulting in 711These tags are automatically decoded when encountered, resulting in
654shared values in the decoded object. They are only encoded, however, when 712shared values in the decoded object. They are only encoded, however, when
655C<allow_sharable> is enabled. 713C<allow_sharable> is enabled.
656 714
657=item <unassigned>, <unassigned> (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>) 715=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)
658 716
659These tags are automatically decoded when encountered. They are only 717These tags are automatically decoded when encountered. They are only
660encoded, however, when C<allow_stringref> is enabled. 718encoded, however, when C<pack_strings> is enabled.
661 719
662=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>) 720=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
663 721
664This tag is automatically generated when a reference are encountered (with 722This tag is automatically generated when a reference are encountered (with
665the exception of hash and array refernces). It is converted to a reference 723the exception of hash and array refernces). It is converted to a reference
… …
670This value is not generated on encoding (unless explicitly requested by 728This value is not generated on encoding (unless explicitly requested by
671the user), and is simply ignored when decoding. 729the user), and is simply ignored when decoding.
672 730
673=back 731=back
674 732
675=head2 OPTIONAL TAGS 733=head2 NON-ENFORCED TAGS
676 734
677These tags have default filters provided when decoding. Their handling can 735These tags have default filters provided when decoding. Their handling can
678be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by 736be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
679providing a custom C<filter> function when decoding. 737providing a custom C<filter> callback when decoding.
680 738
681When they result in decoding into a specific Perl class, the module 739When they result in decoding into a specific Perl class, the module
682usually provides a corresponding C<TO_CBOR> method as well. 740usually provides a corresponding C<TO_CBOR> method as well.
683 741
684When any of these need to load additional modules that are not part of the 742When any of these need to load additional modules that are not part of the
… …
826Only the double data type is supported for NV data types - when Perl uses 884Only the double data type is supported for NV data types - when Perl uses
827long double to represent floating point values, they might not be encoded 885long double to represent floating point values, they might not be encoded
828properly. Half precision types are accepted, but not encoded. 886properly. Half precision types are accepted, but not encoded.
829 887
830Strict mode and canonical mode are not implemented. 888Strict mode and canonical mode are not implemented.
889
890
891=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
892
893On perls that were built without 64 bit integer support (these are rare
894nowadays, even on 32 bit architectures), support for any kind of 64 bit
895integer in CBOR is very limited - most likely, these 64 bit values will
896be truncated, corrupted, or otherwise not decoded correctly. This also
897includes string, array and map sizes that are stored as 64 bit integers.
831 898
832 899
833=head1 THREADS 900=head1 THREADS
834 901
835This module is I<not> guaranteed to be thread safe and there are no 902This module is I<not> guaranteed to be thread safe and there are no

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines