ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.22 by root, Fri Nov 22 15:28:38 2013 UTC vs.
Revision 1.28 by root, Thu Nov 28 16:09:04 2013 UTC

… …
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up
32to you to do). Furthermore, details of the implementation might change
33freely before version 1.0. And lastly, most extensions depend on an IANA
34assignment, and until that assignment is official, this implementation is
35not interoperable with other implementations (even future versions of this
36module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 40data later and speed is less important you might want to compare both
41formats first).
50 42
51To give you a general idea about speed, with texts in the megabyte range, 43To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or 44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the 45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison. 46data, the worse L<Storable> performs in comparison.
55 47
56As for compactness, C<CBOR::XS> encoded data structures are usually about 48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
5720% smaller than the same data encoded as (compact) JSON or L<Storable>. 49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
58 51
59In addition to the core CBOR data format, this module implements a number 52In addition to the core CBOR data format, this module implements a
60of extensions, to support cyclic and self-referencing data structures 53number of extensions, to support cyclic and shared data structures (see
61(see C<allow_sharing>), string deduplication (see C<allow_stringref>) and 54C<allow_sharing>), string deduplication (see C<pack_strings>) and scalar
62scalar references (always enabled). 55references (always enabled).
63 56
64The primary goal of this module is to be I<correct> and the secondary goal 57The primary goal of this module is to be I<correct> and the secondary goal
65is to be I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
66 59
67See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
… …
71 64
72package CBOR::XS; 65package CBOR::XS;
73 66
74use common::sense; 67use common::sense;
75 68
76our $VERSION = 0.08; 69our $VERSION = '1.0';
77our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
78 71
79our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
80 73
81use Exporter; 74use Exporter;
… …
118strings. All boolean flags described below are by default I<disabled>. 111strings. All boolean flags described below are by default I<disabled>.
119 112
120The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
121be chained: 114be chained:
122 115
123#TODO
124 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
125 117
126=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
127 119
128=item $max_depth = $cbor->get_max_depth 120=item $max_depth = $cbor->get_max_depth
… …
187as an array, is referenced multiple times), but instead will emit a 179as an array, is referenced multiple times), but instead will emit a
188reference to the earlier value. 180reference to the earlier value.
189 181
190This means that such values will only be encoded once, and will not result 182This means that such values will only be encoded once, and will not result
191in a deep cloning of the value on decode, in decoders supporting the value 183in a deep cloning of the value on decode, in decoders supporting the value
192sharing extension. 184sharing extension. This also makes it possible to encode cyclic data
185structures.
193 186
194It is recommended to leave it off unless you know your 187It is recommended to leave it off unless you know your
195communication partner supports the value sharing extensions to CBOR 188communication partner supports the value sharing extensions to CBOR
196(http://cbor.schmorp.de/value-sharing). 189(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
190resulting data structure might be unusable.
197 191
198Detecting shared values incurs a runtime overhead when values are encoded 192Detecting shared values incurs a runtime overhead when values are encoded
199that have a reference counter large than one, and might unnecessarily 193that have a reference counter large than one, and might unnecessarily
200increase the encoded size, as potentially shared values are encode as 194increase the encoded size, as potentially shared values are encode as
201sharable whether or not they are actually shared. 195sharable whether or not they are actually shared.
202 196
203At the moment, only targets of references can be shared (e.g. scalars, 197At the moment, only targets of references can be shared (e.g. scalars,
204arrays or hashes pointed to by a reference). Weirder constructs, such as 198arrays or hashes pointed to by a reference). Weirder constructs, such as
205an array with multiple "copies" of the I<same> string, which are hard but 199an array with multiple "copies" of the I<same> string, which are hard but
206not impossible to create in Perl, are not supported (this is the same as 200not impossible to create in Perl, are not supported (this is the same as
207for L<Storable>). 201with L<Storable>).
208 202
209If C<$enable> is false (the default), then C<encode> will encode 203If C<$enable> is false (the default), then C<encode> will encode shared
210exception when it encounters anything it cannot encode as CBOR. 204data structures repeatedly, unsharing them in the process. Cyclic data
205structures cannot be encoded in this mode.
211 206
212This option does not affect C<decode> in any way - shared values and 207This option does not affect C<decode> in any way - shared values and
213references will always be decoded properly if present. 208references will always be decoded properly if present.
214 209
215=item $cbor = $cbor->allow_stringref ([$enable]) 210=item $cbor = $cbor->pack_strings ([$enable])
216 211
217=item $enabled = $cbor->get_allow_stringref 212=item $enabled = $cbor->get_pack_strings
218 213
219If C<$enable> is true (or missing), then C<encode> will try not to encode 214If C<$enable> is true (or missing), then C<encode> will try not to encode
220the same string twice, but will instead encode a reference to the string 215the same string twice, but will instead encode a reference to the string
221instead. Depending on your data format. this can save a lot of space, but 216instead. Depending on your data format, this can save a lot of space, but
222also results in a very large runtime overhead (expect encoding times to be 217also results in a very large runtime overhead (expect encoding times to be
2232-4 times as high as without). 2182-4 times as high as without).
224 219
225It is recommended to leave it off unless you know your 220It is recommended to leave it off unless you know your
226communications partner supports the stringref extension to CBOR 221communications partner supports the stringref extension to CBOR
227(http://cbor.schmorp.de/stringref). 222(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
223resulting data structure might not be usable.
228 224
229If C<$enable> is false (the default), then C<encode> will encode 225If C<$enable> is false (the default), then C<encode> will encode strings
230exception when it encounters anything it cannot encode as CBOR. 226the standard CBOR way.
231 227
232This option does not affect C<decode> in any way - string references will 228This option does not affect C<decode> in any way - string references will
233always be decoded properly if present. 229always be decoded properly if present.
230
231=item $cbor = $cbor->filter ([$cb->($tag, $value)])
232
233=item $cb_or_undef = $cbor->get_filter
234
235Sets or replaces the tagged value decoding filter (when C<$cb> is
236specified) or clears the filter (if no argument or C<undef> is provided).
237
238The filter callback is called only during decoding, when a non-enforced
239tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
240list of enforced tags). For specific tags, it's often better to provide a
241default converter using the C<%CBOR::XS::FILTER> hash (see below).
242
243The first argument is the numerical tag, the second is the (decoded) value
244that has been tagged.
245
246The filter function should return either exactly one value, which will
247replace the tagged value in the decoded data structure, or no values,
248which will result in default handling, which currently means the decoder
249creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
250
251When the filter is cleared (the default state), the default filter
252function, C<CBOR::XS::default_filter>, is used. This function simply looks
253up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
254a code reference that is called with tag and value, and is responsible for
255decoding the value. If no entry exists, it returns no values.
256
257Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
258objects, with no other special handling (useful when working with
259potentially "unsafe" CBOR data).
260
261 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
262
263Example: provide a global filter for tag 1347375694, converting the value
264into some string form.
265
266 $CBOR::XS::FILTER{1347375694} = sub {
267 my ($tag, $value);
268
269 "tag 1347375694 value $value"
270 };
234 271
235=item $cbor_data = $cbor->encode ($perl_scalar) 272=item $cbor_data = $cbor->encode ($perl_scalar)
236 273
237Converts the given Perl data structure (a scalar value) to its CBOR 274Converts the given Perl data structure (a scalar value) to its CBOR
238representation. 275representation.
… …
279CBOR integers become (numeric) perl scalars. On perls without 64 bit 316CBOR integers become (numeric) perl scalars. On perls without 64 bit
280support, 64 bit integers will be truncated or otherwise corrupted. 317support, 64 bit integers will be truncated or otherwise corrupted.
281 318
282=item byte strings 319=item byte strings
283 320
284Byte strings will become octet strings in Perl (the byte values 0..255 321Byte strings will become octet strings in Perl (the Byte values 0..255
285will simply become characters of the same value in Perl). 322will simply become characters of the same value in Perl).
286 323
287=item UTF-8 strings 324=item UTF-8 strings
288 325
289UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 326UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
… …
307C<Types:Serialiser::false> and C<Types::Serialiser::error>, 344C<Types:Serialiser::false> and C<Types::Serialiser::error>,
308respectively. They are overloaded to act almost exactly like the numbers 345respectively. They are overloaded to act almost exactly like the numbers
309C<1> and C<0> (for true and false) or to throw an exception on access (for 346C<1> and C<0> (for true and false) or to throw an exception on access (for
310error). See the L<Types::Serialiser> manpage for details. 347error). See the L<Types::Serialiser> manpage for details.
311 348
312=item CBOR tag 256 (perl object) 349=item tagged values
313 350
314The tag value C<256> (TODO: pending iana registration) will be used
315to deserialise a Perl object serialised with C<FREEZE>. See L<OBJECT
316SERIALISATION>, below, for details.
317
318=item CBOR tag 55799 (magic header)
319
320The tag 55799 is ignored (this tag implements the magic header).
321
322=item other CBOR tags
323
324Tagged items consists of a numeric tag and another CBOR value. Tags not 351Tagged items consists of a numeric tag and another CBOR value.
325handled internally are currently converted into a L<CBOR::XS::Tagged>
326object, which is simply a blessed array reference consisting of the
327numeric tag value followed by the (decoded) CBOR value.
328 352
329In the future, support for user-supplied conversions might get added. 353See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
354for details on which tags are handled how.
330 355
331=item anything else 356=item anything else
332 357
333Anything else (e.g. unsupported simple values) will raise a decoding 358Anything else (e.g. unsupported simple values) will raise a decoding
334error. 359error.
… …
337 362
338 363
339=head2 PERL -> CBOR 364=head2 PERL -> CBOR
340 365
341The mapping from Perl to CBOR is slightly more difficult, as Perl is a 366The mapping from Perl to CBOR is slightly more difficult, as Perl is a
342truly typeless language, so we can only guess which CBOR type is meant by 367typeless language. That means this module can only guess which CBOR type
343a Perl value. 368is meant by a perl value.
344 369
345=over 4 370=over 4
346 371
347=item hash references 372=item hash references
348 373
349Perl hash references become CBOR maps. As there is no inherent ordering in 374Perl hash references become CBOR maps. As there is no inherent ordering in
350hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 375hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
351order. 376order. This order can be different each time a hahs is encoded.
352 377
353Currently, tied hashes will use the indefinite-length format, while normal 378Currently, tied hashes will use the indefinite-length format, while normal
354hashes will use the fixed-length format. 379hashes will use the fixed-length format.
355 380
356=item array references 381=item array references
357 382
358Perl array references become fixed-length CBOR arrays. 383Perl array references become fixed-length CBOR arrays.
359 384
360=item other references 385=item other references
361 386
362Other unblessed references are generally not allowed and will cause an 387Other unblessed references will be represented using
363exception to be thrown, except for references to the integers C<0> and 388the indirection tag extension (tag value C<22098>,
364C<1>, which get turned into false and true in CBOR. 389L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
390to be able to decode these values somehow, by either "doing the right
391thing", decoding into a generic tagged object, simply ignoring the tag, or
392something else.
365 393
366=item CBOR::XS::Tagged objects 394=item CBOR::XS::Tagged objects
367 395
368Objects of this type must be arrays consisting of a single C<[tag, value]> 396Objects of this type must be arrays consisting of a single C<[tag, value]>
369pair. The (numerical) tag will be encoded as a CBOR tag, the value will 397pair. The (numerical) tag will be encoded as a CBOR tag, the value will
370be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to 398be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
371create such objects. 399create such objects.
372 400
373=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 401=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
374 402
375These special values become CBOR true, CBOR false and CBOR undefined 403These special values become CBOR true, CBOR false and CBOR undefined
… …
377if you want. 405if you want.
378 406
379=item other blessed objects 407=item other blessed objects
380 408
381Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See 409Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
382L<OBJECT SERIALISATION>, below, for details. 410L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
411module, and L<OBJECT SERIALISATION> for generic object serialisation.
383 412
384=item simple scalars 413=item simple scalars
385 414
386TODO
387Simple Perl scalars (any scalar that is not a reference) are the most 415Simple Perl scalars (any scalar that is not a reference) are the most
388difficult objects to encode: CBOR::XS will encode undefined scalars as 416difficult objects to encode: CBOR::XS will encode undefined scalars as
389CBOR null values, scalars that have last been used in a string context 417CBOR null values, scalars that have last been used in a string context
390before encoding as CBOR strings, and anything else as number value: 418before encoding as CBOR strings, and anything else as number value:
391 419
392 # dump as number 420 # dump as number
393 encode_cbor [2] # yields [2] 421 encode_cbor [2] # yields [2]
394 encode_cbor [-3.0e17] # yields [-3e+17] 422 encode_cbor [-3.0e17] # yields [-3e+17]
395 my $value = 5; encode_cbor [$value] # yields [5] 423 my $value = 5; encode_cbor [$value] # yields [5]
396 424
397 # used as string, so dump as string 425 # used as string, so dump as string (either byte or text)
398 print $value; 426 print $value;
399 encode_cbor [$value] # yields ["5"] 427 encode_cbor [$value] # yields ["5"]
400 428
401 # undef becomes null 429 # undef becomes null
402 encode_cbor [undef] # yields [null] 430 encode_cbor [undef] # yields [null]
… …
405 433
406 my $x = 3.1; # some variable containing a number 434 my $x = 3.1; # some variable containing a number
407 "$x"; # stringified 435 "$x"; # stringified
408 $x .= ""; # another, more awkward way to stringify 436 $x .= ""; # another, more awkward way to stringify
409 print $x; # perl does it for you, too, quite often 437 print $x; # perl does it for you, too, quite often
438
439You can force whether a string ie encoded as byte or text string by using
440C<utf8::upgrade> and C<utf8::downgrade>):
441
442 utf8::upgrade $x; # encode $x as text string
443 utf8::downgrade $x; # encode $x as byte string
444
445Perl doesn't define what operations up- and downgrade strings, so if the
446difference between byte and text is important, you should up- or downgrade
447your string as late as possible before encoding.
410 448
411You can force the type to be a CBOR number by numifying it: 449You can force the type to be a CBOR number by numifying it:
412 450
413 my $x = "3"; # some variable containing a string 451 my $x = "3"; # some variable containing a string
414 $x += 0; # numify it, ensuring it will be dumped as a number 452 $x += 0; # numify it, ensuring it will be dumped as a number
… …
481 519
482 sub URI::TO_CBOR { 520 sub URI::TO_CBOR {
483 my ($self) = @_; 521 my ($self) = @_;
484 my $uri = "$self"; # stringify uri 522 my $uri = "$self"; # stringify uri
485 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 523 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
486 CBOR::XS::tagged 32, "$_[0]" 524 CBOR::XS::tag 32, "$_[0]"
487 } 525 }
488 526
489This will encode URIs as a UTF-8 string with tag 32, which indicates an 527This will encode URIs as a UTF-8 string with tag 32, which indicates an
490URI. 528URI.
491 529
… …
633and extensions. If a tag is not mentioned here and no additional filters 671and extensions. If a tag is not mentioned here and no additional filters
634are provided for it, then the default handling applies (creating a 672are provided for it, then the default handling applies (creating a
635CBOR::XS::Tagged object on decoding, and only encoding the tag when 673CBOR::XS::Tagged object on decoding, and only encoding the tag when
636explicitly requested). 674explicitly requested).
637 675
676Tags not handled specifically are currently converted into a
677L<CBOR::XS::Tagged> object, which is simply a blessed array reference
678consisting of the numeric tag value followed by the (decoded) CBOR value.
679
638Future versions of this module reserve the right to special case 680Future versions of this module reserve the right to special case
639additional tags (such as base64url). 681additional tags (such as base64url).
640 682
641=head2 ENFORCED TAGS 683=head2 ENFORCED TAGS
642 684
643These tags are always handled when decoding, and their handling cannot be 685These tags are always handled when decoding, and their handling cannot be
644overriden by the user. 686overriden by the user.
645 687
646=over 4 688=over 4
647 689
648=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>) 690=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
649 691
650These tags are automatically created for serialisable objects using the 692These tags are automatically created (and decoded) for serialisable
651C<FREEZE/THAW> methods (the L<Types::Serialier> object serialisation 693objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
652protocol). 694serialisation protocol). See L<OBJECT SERIALISATION> for details.
653 695
654=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>) 696=item 28, 29 (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
655 697
656These tags are automatically decoded when encountered, resulting in 698These tags are automatically decoded when encountered, resulting in
657shared values in the decoded object. They are only encoded, however, when 699shared values in the decoded object. They are only encoded, however, when
658C<allow_sharable> is enabled. 700C<allow_sharable> is enabled.
659 701
660=item <unassigned>, <unassigned> (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>) 702=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)
661 703
662These tags are automatically decoded when encountered. They are only 704These tags are automatically decoded when encountered. They are only
663encoded, however, when C<allow_stringref> is enabled. 705encoded, however, when C<pack_strings> is enabled.
664 706
665=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>) 707=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
666 708
667This tag is automatically generated when a reference are encountered (with 709This tag is automatically generated when a reference are encountered (with
668the exception of hash and array refernces). It is converted to a reference 710the exception of hash and array refernces). It is converted to a reference
… …
673This value is not generated on encoding (unless explicitly requested by 715This value is not generated on encoding (unless explicitly requested by
674the user), and is simply ignored when decoding. 716the user), and is simply ignored when decoding.
675 717
676=back 718=back
677 719
678=head2 OPTIONAL TAGS 720=head2 NON-ENFORCED TAGS
679 721
680These tags have default filters provided when decoding. Their handling can 722These tags have default filters provided when decoding. Their handling can
681be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by 723be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
682providing a custom C<filter> function when decoding. 724providing a custom C<filter> callback when decoding.
683 725
684When they result in decoding into a specific Perl class, the module 726When they result in decoding into a specific Perl class, the module
685usually provides a corresponding C<TO_CBOR> method as well. 727usually provides a corresponding C<TO_CBOR> method as well.
686 728
687When any of these need to load additional modules that are not part of the 729When any of these need to load additional modules that are not part of the

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines