ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.26 by root, Thu Nov 28 12:11:06 2013 UTC vs.
Revision 1.32 by root, Sat Nov 30 18:42:27 2013 UTC

… …
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up
32to you to do). Furthermore, details of the implementation might change
33freely before version 1.0. And lastly, most extensions depend on an IANA
34assignment, and until that assignment is official, this implementation is
35not interoperable with other implementations (even future versions of this
36module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 40data later and speed is less important you might want to compare both
41formats first).
50 42
51To give you a general idea about speed, with texts in the megabyte range, 43To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or 44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the 45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison. 46data, the worse L<Storable> performs in comparison.
55 47
56As for compactness, C<CBOR::XS> encoded data structures are usually about 48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
5720% smaller than the same data encoded as (compact) JSON or L<Storable>. 49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
58 51
59In addition to the core CBOR data format, this module implements a number 52In addition to the core CBOR data format, this module implements a
60of extensions, to support cyclic and self-referencing data structures 53number of extensions, to support cyclic and shared data structures
61(see C<allow_sharing>), string deduplication (see C<pack_strings>) and 54(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
62scalar references (always enabled). 55C<pack_strings>) and scalar references (always enabled).
63 56
64The primary goal of this module is to be I<correct> and the secondary goal 57The primary goal of this module is to be I<correct> and the secondary goal
65is to be I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
66 59
67See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
… …
71 64
72package CBOR::XS; 65package CBOR::XS;
73 66
74use common::sense; 67use common::sense;
75 68
76our $VERSION = 0.09; 69our $VERSION = 1.1;
77our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
78 71
79our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
80 73
81use Exporter; 74use Exporter;
… …
187reference to the earlier value. 180reference to the earlier value.
188 181
189This means that such values will only be encoded once, and will not result 182This means that such values will only be encoded once, and will not result
190in a deep cloning of the value on decode, in decoders supporting the value 183in a deep cloning of the value on decode, in decoders supporting the value
191sharing extension. This also makes it possible to encode cyclic data 184sharing extension. This also makes it possible to encode cyclic data
192structures. 185structures (which need C<allow_cycles> to ne enabled to be decoded by this
186module).
193 187
194It is recommended to leave it off unless you know your 188It is recommended to leave it off unless you know your
195communication partner supports the value sharing extensions to CBOR 189communication partner supports the value sharing extensions to CBOR
196(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the 190(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
197resulting data structure might be unusable. 191resulting data structure might be unusable.
198 192
199Detecting shared values incurs a runtime overhead when values are encoded 193Detecting shared values incurs a runtime overhead when values are encoded
200that have a reference counter large than one, and might unnecessarily 194that have a reference counter large than one, and might unnecessarily
201increase the encoded size, as potentially shared values are encode as 195increase the encoded size, as potentially shared values are encode as
202sharable whether or not they are actually shared. 196shareable whether or not they are actually shared.
203 197
204At the moment, only targets of references can be shared (e.g. scalars, 198At the moment, only targets of references can be shared (e.g. scalars,
205arrays or hashes pointed to by a reference). Weirder constructs, such as 199arrays or hashes pointed to by a reference). Weirder constructs, such as
206an array with multiple "copies" of the I<same> string, which are hard but 200an array with multiple "copies" of the I<same> string, which are hard but
207not impossible to create in Perl, are not supported (this is the same as 201not impossible to create in Perl, are not supported (this is the same as
… …
212structures cannot be encoded in this mode. 206structures cannot be encoded in this mode.
213 207
214This option does not affect C<decode> in any way - shared values and 208This option does not affect C<decode> in any way - shared values and
215references will always be decoded properly if present. 209references will always be decoded properly if present.
216 210
211=item $cbor = $cbor->allow_cycles ([$enable])
212
213=item $enabled = $cbor->get_allow_cycles
214
215If C<$enable> is true (or missing), then C<decode> will happily decode
216self-referential (cyclic) data structures. By default these will not be
217decoded, as they need manual cleanup to avoid memory leaks, so code that
218isn't prepared for this will not leak memory.
219
220If C<$enable> is false (the default), then C<decode> will throw an error
221when it encounters a self-referential/cyclic data structure.
222
223This option does not affect C<encode> in any way - shared values and
224references will always be decoded properly if present.
225
217=item $cbor = $cbor->pack_strings ([$enable]) 226=item $cbor = $cbor->pack_strings ([$enable])
218 227
219=item $enabled = $cbor->get_pack_strings 228=item $enabled = $cbor->get_pack_strings
220 229
221If C<$enable> is true (or missing), then C<encode> will try not to encode 230If C<$enable> is true (or missing), then C<encode> will try not to encode
… …
259function, C<CBOR::XS::default_filter>, is used. This function simply looks 268function, C<CBOR::XS::default_filter>, is used. This function simply looks
260up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be 269up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
261a code reference that is called with tag and value, and is responsible for 270a code reference that is called with tag and value, and is responsible for
262decoding the value. If no entry exists, it returns no values. 271decoding the value. If no entry exists, it returns no values.
263 272
264Example: decode all tags not handled internally into CBOR::XS::Tagged 273Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
265objects, with no other special handling (useful when working with 274objects, with no other special handling (useful when working with
266potentially "unsafe" CBOR data). 275potentially "unsafe" CBOR data).
267 276
268 CBOR::XS->new->filter (sub { })->decode ($cbor_data); 277 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
269 278
… …
323CBOR integers become (numeric) perl scalars. On perls without 64 bit 332CBOR integers become (numeric) perl scalars. On perls without 64 bit
324support, 64 bit integers will be truncated or otherwise corrupted. 333support, 64 bit integers will be truncated or otherwise corrupted.
325 334
326=item byte strings 335=item byte strings
327 336
328Byte strings will become octet strings in Perl (the byte values 0..255 337Byte strings will become octet strings in Perl (the Byte values 0..255
329will simply become characters of the same value in Perl). 338will simply become characters of the same value in Perl).
330 339
331=item UTF-8 strings 340=item UTF-8 strings
332 341
333UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 342UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
… …
356=item tagged values 365=item tagged values
357 366
358Tagged items consists of a numeric tag and another CBOR value. 367Tagged items consists of a numeric tag and another CBOR value.
359 368
360See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >> 369See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
361for details. 370for details on which tags are handled how.
362 371
363=item anything else 372=item anything else
364 373
365Anything else (e.g. unsupported simple values) will raise a decoding 374Anything else (e.g. unsupported simple values) will raise a decoding
366error. 375error.
… …
369 378
370 379
371=head2 PERL -> CBOR 380=head2 PERL -> CBOR
372 381
373The mapping from Perl to CBOR is slightly more difficult, as Perl is a 382The mapping from Perl to CBOR is slightly more difficult, as Perl is a
374truly typeless language, so we can only guess which CBOR type is meant by 383typeless language. That means this module can only guess which CBOR type
375a Perl value. 384is meant by a perl value.
376 385
377=over 4 386=over 4
378 387
379=item hash references 388=item hash references
380 389
381Perl hash references become CBOR maps. As there is no inherent ordering in 390Perl hash references become CBOR maps. As there is no inherent ordering in
382hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 391hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
383order. 392order. This order can be different each time a hahs is encoded.
384 393
385Currently, tied hashes will use the indefinite-length format, while normal 394Currently, tied hashes will use the indefinite-length format, while normal
386hashes will use the fixed-length format. 395hashes will use the fixed-length format.
387 396
388=item array references 397=item array references
389 398
390Perl array references become fixed-length CBOR arrays. 399Perl array references become fixed-length CBOR arrays.
391 400
392=item other references 401=item other references
393 402
394Other unblessed references are generally not allowed and will cause an 403Other unblessed references will be represented using
395exception to be thrown, except for references to the integers C<0> and 404the indirection tag extension (tag value C<22098>,
396C<1>, which get turned into false and true in CBOR. 405L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
406to be able to decode these values somehow, by either "doing the right
407thing", decoding into a generic tagged object, simply ignoring the tag, or
408something else.
397 409
398=item CBOR::XS::Tagged objects 410=item CBOR::XS::Tagged objects
399 411
400Objects of this type must be arrays consisting of a single C<[tag, value]> 412Objects of this type must be arrays consisting of a single C<[tag, value]>
401pair. The (numerical) tag will be encoded as a CBOR tag, the value will 413pair. The (numerical) tag will be encoded as a CBOR tag, the value will
402be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to 414be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
403create such objects. 415create such objects.
404 416
405=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 417=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
406 418
407These special values become CBOR true, CBOR false and CBOR undefined 419These special values become CBOR true, CBOR false and CBOR undefined
… …
424 # dump as number 436 # dump as number
425 encode_cbor [2] # yields [2] 437 encode_cbor [2] # yields [2]
426 encode_cbor [-3.0e17] # yields [-3e+17] 438 encode_cbor [-3.0e17] # yields [-3e+17]
427 my $value = 5; encode_cbor [$value] # yields [5] 439 my $value = 5; encode_cbor [$value] # yields [5]
428 440
429 # used as string, so dump as string 441 # used as string, so dump as string (either byte or text)
430 print $value; 442 print $value;
431 encode_cbor [$value] # yields ["5"] 443 encode_cbor [$value] # yields ["5"]
432 444
433 # undef becomes null 445 # undef becomes null
434 encode_cbor [undef] # yields [null] 446 encode_cbor [undef] # yields [null]
… …
437 449
438 my $x = 3.1; # some variable containing a number 450 my $x = 3.1; # some variable containing a number
439 "$x"; # stringified 451 "$x"; # stringified
440 $x .= ""; # another, more awkward way to stringify 452 $x .= ""; # another, more awkward way to stringify
441 print $x; # perl does it for you, too, quite often 453 print $x; # perl does it for you, too, quite often
454
455You can force whether a string ie encoded as byte or text string by using
456C<utf8::upgrade> and C<utf8::downgrade>):
457
458 utf8::upgrade $x; # encode $x as text string
459 utf8::downgrade $x; # encode $x as byte string
460
461Perl doesn't define what operations up- and downgrade strings, so if the
462difference between byte and text is important, you should up- or downgrade
463your string as late as possible before encoding.
442 464
443You can force the type to be a CBOR number by numifying it: 465You can force the type to be a CBOR number by numifying it:
444 466
445 my $x = "3"; # some variable containing a string 467 my $x = "3"; # some variable containing a string
446 $x += 0; # numify it, ensuring it will be dumped as a number 468 $x += 0; # numify it, ensuring it will be dumped as a number
… …
459 481
460=back 482=back
461 483
462=head2 OBJECT SERIALISATION 484=head2 OBJECT SERIALISATION
463 485
486This module implements both a CBOR-specific and the generic
487L<Types::Serialier> object serialisation protocol. The following
488subsections explain both methods.
489
490=head3 ENCODING
491
464This module knows two way to serialise a Perl object: The CBOR-specific 492This module knows two way to serialise a Perl object: The CBOR-specific
465way, and the generic way. 493way, and the generic way.
466 494
467Whenever the encoder encounters a Perl object that it cnanot serialise 495Whenever the encoder encounters a Perl object that it cannot serialise
468directly (most of them), it will first look up the C<TO_CBOR> method on 496directly (most of them), it will first look up the C<TO_CBOR> method on
469it. 497it.
470 498
471If it has a C<TO_CBOR> method, it will call it with the object as only 499If it has a C<TO_CBOR> method, it will call it with the object as only
472argument, and expects exactly one return value, which it will then 500argument, and expects exactly one return value, which it will then
… …
478 506
479The C<FREEZE> method can return any number of values (i.e. zero or 507The C<FREEZE> method can return any number of values (i.e. zero or
480more). These will be encoded as CBOR perl object, together with the 508more). These will be encoded as CBOR perl object, together with the
481classname. 509classname.
482 510
511These methods I<MUST NOT> change the data structure that is being
512serialised. Failure to comply to this can result in memory corruption -
513and worse.
514
483If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail 515If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
484with an error. 516with an error.
485 517
518=head3 DECODING
519
486Objects encoded via C<TO_CBOR> cannot be automatically decoded, but 520Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
487objects encoded via C<FREEZE> can be decoded using the following protocol: 521but objects encoded via C<FREEZE> can be decoded using the following
522protocol:
488 523
489When an encoded CBOR perl object is encountered by the decoder, it will 524When an encoded CBOR perl object is encountered by the decoder, it will
490look up the C<THAW> method, by using the stored classname, and will fail 525look up the C<THAW> method, by using the stored classname, and will fail
491if the method cannot be found. 526if the method cannot be found.
492 527
493After the lookup it will call the C<THAW> method with the stored classname 528After the lookup it will call the C<THAW> method with the stored classname
494as first argument, the constant string C<CBOR> as second argument, and all 529as first argument, the constant string C<CBOR> as second argument, and all
495values returned by C<FREEZE> as remaining arguments. 530values returned by C<FREEZE> as remaining arguments.
496 531
497=head4 EXAMPLES 532=head3 EXAMPLES
498 533
499Here is an example C<TO_CBOR> method: 534Here is an example C<TO_CBOR> method:
500 535
501 sub My::Object::TO_CBOR { 536 sub My::Object::TO_CBOR {
502 my ($obj) = @_; 537 my ($obj) = @_;
… …
513 548
514 sub URI::TO_CBOR { 549 sub URI::TO_CBOR {
515 my ($self) = @_; 550 my ($self) = @_;
516 my $uri = "$self"; # stringify uri 551 my $uri = "$self"; # stringify uri
517 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 552 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
518 CBOR::XS::tagged 32, "$_[0]" 553 CBOR::XS::tag 32, "$_[0]"
519 } 554 }
520 555
521This will encode URIs as a UTF-8 string with tag 32, which indicates an 556This will encode URIs as a UTF-8 string with tag 32, which indicates an
522URI. 557URI.
523 558
… …
685 720
686These tags are automatically created (and decoded) for serialisable 721These tags are automatically created (and decoded) for serialisable
687objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object 722objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
688serialisation protocol). See L<OBJECT SERIALISATION> for details. 723serialisation protocol). See L<OBJECT SERIALISATION> for details.
689 724
690=item 28, 29 (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>) 725=item 28, 29 (shareable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
691 726
692These tags are automatically decoded when encountered, resulting in 727These tags are automatically decoded when encountered (and they do not
728result in a cyclic data structure, see C<allow_cycles>), resulting in
693shared values in the decoded object. They are only encoded, however, when 729shared values in the decoded object. They are only encoded, however, when
694C<allow_sharable> is enabled. 730C<allow_sharing> is enabled.
731
732Not all shared values can be successfully decoded: values that reference
733themselves will I<currently> decode as C<undef> (this is not the same
734as a reference pointing to itself, which will be represented as a value
735that contains an indirect reference to itself - these will be decoded
736properly).
737
738Note that considerably more shared value data structures can be decoded
739than will be encoded - currently, only values pointed to by references
740will be shared, others will not. While non-reference shared values can be
741generated in Perl with some effort, they were considered too unimportant
742to be supported in the encoder. The decoder, however, will decode these
743values as shared values.
695 744
696=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>) 745=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)
697 746
698These tags are automatically decoded when encountered. They are only 747These tags are automatically decoded when encountered. They are only
699encoded, however, when C<pack_strings> is enabled. 748encoded, however, when C<pack_strings> is enabled.
… …
865Only the double data type is supported for NV data types - when Perl uses 914Only the double data type is supported for NV data types - when Perl uses
866long double to represent floating point values, they might not be encoded 915long double to represent floating point values, they might not be encoded
867properly. Half precision types are accepted, but not encoded. 916properly. Half precision types are accepted, but not encoded.
868 917
869Strict mode and canonical mode are not implemented. 918Strict mode and canonical mode are not implemented.
919
920
921=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
922
923On perls that were built without 64 bit integer support (these are rare
924nowadays, even on 32 bit architectures), support for any kind of 64 bit
925integer in CBOR is very limited - most likely, these 64 bit values will
926be truncated, corrupted, or otherwise not decoded correctly. This also
927includes string, array and map sizes that are stored as 64 bit integers.
870 928
871 929
872=head1 THREADS 930=head1 THREADS
873 931
874This module is I<not> guaranteed to be thread safe and there are no 932This module is I<not> guaranteed to be thread safe and there are no

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines