ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.3 by root, Sat Oct 26 11:08:34 2013 UTC vs.
Revision 1.19 by root, Wed Nov 20 01:09:46 2013 UTC

12 $perl_value = decode_cbor $binary_cbor_data; 12 $perl_value = decode_cbor $binary_cbor_data;
13 13
14 # OO-interface 14 # OO-interface
15 15
16 $coder = CBOR::XS->new; 16 $coder = CBOR::XS->new;
17 #TODO 17 $binary_cbor_data = $coder->encode ($perl_value);
18 $perl_value = $coder->decode ($binary_cbor_data);
19
20 # prefix decoding
21
22 my $many_cbor_strings = ...;
23 while (length $many_cbor_strings) {
24 my ($data, $length) = $cbor->decode_prefix ($many_cbor_strings);
25 # data was decoded
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 }
18 28
19=head1 DESCRIPTION 29=head1 DESCRIPTION
20 30
21WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA AND 31WARNING! This module is very new, and not very well tested (that's up to
22EAT YOUR CHILDREN! 32you to do). Furthermore, details of the implementation might change freely
33before version 1.0. And lastly, the object serialisation protocol depends
34on a pending IANA assignment, and until that assignment is official, this
35implementation is not interoperable with other implementations (even
36future versions of this module) until the assignment is done.
23 37
24This module converts Perl data structures to CBOR and vice versa. Its 38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you
43can represent something in JSON, you should be able to represent it in
44CBOR.
45
46In short, CBOR is a faster and very compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first).
50
51To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison.
55
56As for compactness, C<CBOR::XS> encoded data structures are usually about
5720% smaller than the same data encoded as (compact) JSON or L<Storable>.
58
25primary goal is to be I<correct> and its secondary goal is to be 59The primary goal of this module is to be I<correct> and the secondary goal
26I<fast>. To reach the latter goal it was written in C. 60is to be I<fast>. To reach the latter goal it was written in C.
27 61
28See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 62See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
29vice versa. 63vice versa.
30 64
31=cut 65=cut
32 66
33package CBOR::XS; 67package CBOR::XS;
34 68
35use common::sense; 69use common::sense;
36 70
37our $VERSION = 0.02; 71our $VERSION = 0.08;
38our @ISA = qw(Exporter); 72our @ISA = qw(Exporter);
39 73
40our @EXPORT = qw(encode_cbor decode_cbor); 74our @EXPORT = qw(encode_cbor decode_cbor);
41 75
42use Exporter; 76use Exporter;
43use XSLoader; 77use XSLoader;
78
79use Types::Serialiser;
44 80
45our $MAGIC = "\xd9\xd9\xf7"; 81our $MAGIC = "\xd9\xd9\xf7";
46 82
47=head1 FUNCTIONAL INTERFACE 83=head1 FUNCTIONAL INTERFACE
48 84
121If no argument is given, the limit check will be deactivated (same as when 157If no argument is given, the limit check will be deactivated (same as when
122C<0> is specified). 158C<0> is specified).
123 159
124See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 160See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
125 161
162=item $cbor = $cbor->allow_unknown ([$enable])
163
164=item $enabled = $cbor->get_allow_unknown
165
166If C<$enable> is true (or missing), then C<encode> will I<not> throw an
167exception when it encounters values it cannot represent in CBOR (for
168example, filehandles) but instead will encode a CBOR C<error> value.
169
170If C<$enable> is false (the default), then C<encode> will throw an
171exception when it encounters anything it cannot encode as CBOR.
172
173This option does not affect C<decode> in any way, and it is recommended to
174leave it off unless you know your communications partner.
175
176=item $cbor = $cbor->allow_sharable ([$enable])
177
178=item $enabled = $cbor->get_allow_sharable
179
180If C<$enable> is true (or missing), then C<encode> will not double-encode
181values that have been seen before (e.g. when the same object, such as an
182array, is referenced multiple times), but instead will emit a reference to
183the earlier value.
184
185This means that such values will only be encoded once, and will not result
186in a deep cloning of the value on decode, in decoders supporting the value
187sharing extension.
188
189Detecting shared values incurs a runtime overhead when values are encoded
190that have a reference counter large than one, and might unnecessarily
191increase the encoded size, as potentially shared values are encode as
192sharable whether or not they are actually shared.
193
194At the moment, all shared values will be detected, even weird and unusual
195cases, such as an array with multiple "copies" of the I<same> scalar,
196which are hard but not impossible to create in Perl (L<Storable> for
197example doesn't handle these cases). If this turns out ot be a performance
198issue then future versions might limit the shared value detection to
199references only.
200
201If C<$enable> is false (the default), then C<encode> will encode
202exception when it encounters anything it cannot encode as CBOR.
203
204This option does not affect C<decode> in any way - shared values and
205references will always be decoded properly if present. It is recommended
206to leave it off unless you know your communications partner supports the
207value sharing extensions to CBOR (http://cbor.schmorp.de/value-sharing).
208
126=item $cbor_data = $cbor->encode ($perl_scalar) 209=item $cbor_data = $cbor->encode ($perl_scalar)
127 210
128Converts the given Perl data structure (a scalar value) to its CBOR 211Converts the given Perl data structure (a scalar value) to its CBOR
129representation. 212representation.
130 213
163 246
164=head2 CBOR -> PERL 247=head2 CBOR -> PERL
165 248
166=over 4 249=over 4
167 250
168=item True, False 251=item integers
169 252
170These CBOR values become C<CBOR::XS::true> and C<CBOR::XS::false>, 253CBOR integers become (numeric) perl scalars. On perls without 64 bit
254support, 64 bit integers will be truncated or otherwise corrupted.
255
256=item byte strings
257
258Byte strings will become octet strings in Perl (the byte values 0..255
259will simply become characters of the same value in Perl).
260
261=item UTF-8 strings
262
263UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
264decoded into proper Unicode code points. At the moment, the validity of
265the UTF-8 octets will not be validated - corrupt input will result in
266corrupted Perl strings.
267
268=item arrays, maps
269
270CBOR arrays and CBOR maps will be converted into references to a Perl
271array or hash, respectively. The keys of the map will be stringified
272during this process.
273
274=item null
275
276CBOR null becomes C<undef> in Perl.
277
278=item true, false, undefined
279
280These CBOR values become C<Types:Serialiser::true>,
281C<Types:Serialiser::false> and C<Types::Serialiser::error>,
171respectively. They are overloaded to act almost exactly like the numbers 282respectively. They are overloaded to act almost exactly like the numbers
172C<1> and C<0>. You can check whether a scalar is a CBOR boolean by using 283C<1> and C<0> (for true and false) or to throw an exception on access (for
173the C<CBOR::XS::is_bool> function. 284error). See the L<Types::Serialiser> manpage for details.
174 285
175=item Null, Undefined 286=item CBOR tag 256 (perl object)
176 287
177CBOR Null and Undefined values becomes C<undef> in Perl (in the future, 288The tag value C<256> (TODO: pending iana registration) will be used
178Undefined may raise an exception). 289to deserialise a Perl object serialised with C<FREEZE>. See L<OBJECT
290SERIALISATION>, below, for details.
291
292=item CBOR tag 55799 (magic header)
293
294The tag 55799 is ignored (this tag implements the magic header).
295
296=item other CBOR tags
297
298Tagged items consists of a numeric tag and another CBOR value. Tags not
299handled internally are currently converted into a L<CBOR::XS::Tagged>
300object, which is simply a blessed array reference consisting of the
301numeric tag value followed by the (decoded) CBOR value.
302
303In the future, support for user-supplied conversions might get added.
304
305=item anything else
306
307Anything else (e.g. unsupported simple values) will raise a decoding
308error.
179 309
180=back 310=back
181 311
182 312
183=head2 PERL -> CBOR 313=head2 PERL -> CBOR
188 318
189=over 4 319=over 4
190 320
191=item hash references 321=item hash references
192 322
193Perl hash references become CBOR maps. As there is no inherent ordering 323Perl hash references become CBOR maps. As there is no inherent ordering in
194in hash keys (or CBOR maps), they will usually be encoded in a 324hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
195pseudo-random order. 325order.
326
327Currently, tied hashes will use the indefinite-length format, while normal
328hashes will use the fixed-length format.
196 329
197=item array references 330=item array references
198 331
199Perl array references become CBOR arrays. 332Perl array references become fixed-length CBOR arrays.
200 333
201=item other references 334=item other references
202 335
203Other unblessed references are generally not allowed and will cause an 336Other unblessed references are generally not allowed and will cause an
204exception to be thrown, except for references to the integers C<0> and 337exception to be thrown, except for references to the integers C<0> and
205C<1>, which get turned into C<False> and C<True> in CBOR. 338C<1>, which get turned into false and true in CBOR.
206 339
207=item CBOR::XS::true, CBOR::XS::false 340=item CBOR::XS::Tagged objects
208 341
342Objects of this type must be arrays consisting of a single C<[tag, value]>
343pair. The (numerical) tag will be encoded as a CBOR tag, the value will
344be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to
345create such objects.
346
347=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
348
209These special values become CBOR True and CBOR False values, 349These special values become CBOR true, CBOR false and CBOR undefined
210respectively. You can also use C<\1> and C<\0> directly if you want. 350values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
351if you want.
211 352
212=item blessed objects 353=item other blessed objects
213 354
214Blessed objects are not directly representable in CBOR. TODO 355Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
215See the 356L<OBJECT SERIALISATION>, below, for details.
216C<allow_blessed> and C<convert_blessed> methods on various options on
217how to deal with this: basically, you can choose between throwing an
218exception, encoding the reference as if it weren't blessed, or provide
219your own serialiser method.
220 357
221=item simple scalars 358=item simple scalars
222 359
223TODO 360TODO
224Simple Perl scalars (any scalar that is not a reference) are the most 361Simple Perl scalars (any scalar that is not a reference) are the most
225difficult objects to encode: CBOR::XS will encode undefined scalars as 362difficult objects to encode: CBOR::XS will encode undefined scalars as
226CBOR C<Null> values, scalars that have last been used in a string context 363CBOR null values, scalars that have last been used in a string context
227before encoding as CBOR strings, and anything else as number value: 364before encoding as CBOR strings, and anything else as number value:
228 365
229 # dump as number 366 # dump as number
230 encode_cbor [2] # yields [2] 367 encode_cbor [2] # yields [2]
231 encode_cbor [-3.0e17] # yields [-3e+17] 368 encode_cbor [-3.0e17] # yields [-3e+17]
253 390
254You can not currently force the type in other, less obscure, ways. Tell me 391You can not currently force the type in other, less obscure, ways. Tell me
255if you need this capability (but don't forget to explain why it's needed 392if you need this capability (but don't forget to explain why it's needed
256:). 393:).
257 394
258Note that numerical precision has the same meaning as under Perl (so 395Perl values that seem to be integers generally use the shortest possible
259binary to decimal conversion follows the same rules as in Perl, which 396representation. Floating-point values will use either the IEEE single
260can differ to other languages). Also, your perl interpreter might expose 397format if possible without loss of precision, otherwise the IEEE double
261extensions to the floating point numbers of your platform, such as 398format will be used. Perls that use formats other than IEEE double to
262infinities or NaN's - these cannot be represented in CBOR, and it is an 399represent numerical values are supported, but might suffer loss of
263error to pass those in. 400precision.
264 401
265=back 402=back
266 403
404=head2 OBJECT SERIALISATION
267 405
406This module knows two way to serialise a Perl object: The CBOR-specific
407way, and the generic way.
408
409Whenever the encoder encounters a Perl object that it cnanot serialise
410directly (most of them), it will first look up the C<TO_CBOR> method on
411it.
412
413If it has a C<TO_CBOR> method, it will call it with the object as only
414argument, and expects exactly one return value, which it will then
415substitute and encode it in the place of the object.
416
417Otherwise, it will look up the C<FREEZE> method. If it exists, it will
418call it with the object as first argument, and the constant string C<CBOR>
419as the second argument, to distinguish it from other serialisers.
420
421The C<FREEZE> method can return any number of values (i.e. zero or
422more). These will be encoded as CBOR perl object, together with the
423classname.
424
425If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
426with an error.
427
428Objects encoded via C<TO_CBOR> cannot be automatically decoded, but
429objects encoded via C<FREEZE> can be decoded using the following protocol:
430
431When an encoded CBOR perl object is encountered by the decoder, it will
432look up the C<THAW> method, by using the stored classname, and will fail
433if the method cannot be found.
434
435After the lookup it will call the C<THAW> method with the stored classname
436as first argument, the constant string C<CBOR> as second argument, and all
437values returned by C<FREEZE> as remaining arguments.
438
439=head4 EXAMPLES
440
441Here is an example C<TO_CBOR> method:
442
443 sub My::Object::TO_CBOR {
444 my ($obj) = @_;
445
446 ["this is a serialised My::Object object", $obj->{id}]
447 }
448
449When a C<My::Object> is encoded to CBOR, it will instead encode a simple
450array with two members: a string, and the "object id". Decoding this CBOR
451string will yield a normal perl array reference in place of the object.
452
453A more useful and practical example would be a serialisation method for
454the URI module. CBOR has a custom tag value for URIs, namely 32:
455
456 sub URI::TO_CBOR {
457 my ($self) = @_;
458 my $uri = "$self"; # stringify uri
459 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
460 CBOR::XS::tagged 32, "$_[0]"
461 }
462
463This will encode URIs as a UTF-8 string with tag 32, which indicates an
464URI.
465
466Decoding such an URI will not (currently) give you an URI object, but
467instead a CBOR::XS::Tagged object with tag number 32 and the string -
468exactly what was returned by C<TO_CBOR>.
469
470To serialise an object so it can automatically be deserialised, you need
471to use C<FREEZE> and C<THAW>. To take the URI module as example, this
472would be a possible implementation:
473
474 sub URI::FREEZE {
475 my ($self, $serialiser) = @_;
476 "$self" # encode url string
477 }
478
479 sub URI::THAW {
480 my ($class, $serialiser, $uri) = @_;
481
482 $class->new ($uri)
483 }
484
485Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
486example, a C<FREEZE> method that returns "type", "id" and "variant" values
487would cause an invocation of C<THAW> with 5 arguments:
488
489 sub My::Object::FREEZE {
490 my ($self, $serialiser) = @_;
491
492 ($self->{type}, $self->{id}, $self->{variant})
493 }
494
495 sub My::Object::THAW {
496 my ($class, $serialiser, $type, $id, $variant) = @_;
497
498 $class-<new (type => $type, id => $id, variant => $variant)
499 }
500
501
268=head2 MAGIC HEADER 502=head1 MAGIC HEADER
269 503
270There is no way to distinguish CBOR from other formats 504There is no way to distinguish CBOR from other formats
271programmatically. To make it easier to distinguish CBOR from other 505programmatically. To make it easier to distinguish CBOR from other
272formats, the CBOR specification has a special "magic string" that can be 506formats, the CBOR specification has a special "magic string" that can be
273prepended to any CBOR string without changing it's meaning. 507prepended to any CBOR string without changing its meaning.
274 508
275This string is available as C<$CBOR::XS::MAGIC>. This module does not 509This string is available as C<$CBOR::XS::MAGIC>. This module does not
276prepend this string tot he CBOR data it generates, but it will ignroe it 510prepend this string to the CBOR data it generates, but it will ignore it
277if present, so users can prepend this string as a "file type" indicator as 511if present, so users can prepend this string as a "file type" indicator as
278required. 512required.
279 513
280 514
515=head1 THE CBOR::XS::Tagged CLASS
516
517CBOR has the concept of tagged values - any CBOR value can be tagged with
518a numeric 64 bit number, which are centrally administered.
519
520C<CBOR::XS> handles a few tags internally when en- or decoding. You can
521also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
522decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
523unknown tag.
524
525These objects are simply blessed array references - the first member of
526the array being the numerical tag, the second being the value.
527
528You can interact with C<CBOR::XS::Tagged> objects in the following ways:
529
530=over 4
531
532=item $tagged = CBOR::XS::tag $tag, $value
533
534This function(!) creates a new C<CBOR::XS::Tagged> object using the given
535C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
536value that can be encoded in CBOR, including serialisable Perl objects and
537C<CBOR::XS::Tagged> objects).
538
539=item $tagged->[0]
540
541=item $tagged->[0] = $new_tag
542
543=item $tag = $tagged->tag
544
545=item $new_tag = $tagged->tag ($new_tag)
546
547Access/mutate the tag.
548
549=item $tagged->[1]
550
551=item $tagged->[1] = $new_value
552
553=item $value = $tagged->value
554
555=item $new_value = $tagged->value ($new_value)
556
557Access/mutate the tagged value.
558
559=back
560
561=cut
562
563sub tag($$) {
564 bless [@_], CBOR::XS::Tagged::;
565}
566
567sub CBOR::XS::Tagged::tag {
568 $_[0][0] = $_[1] if $#_;
569 $_[0][0]
570}
571
572sub CBOR::XS::Tagged::value {
573 $_[0][1] = $_[1] if $#_;
574 $_[0][1]
575}
576
577=head2 EXAMPLES
578
579Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
580
581You can look up CBOR tag value and emanings in the IANA registry at
582L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
583
584Prepend a magic header (C<$CBOR::XS::MAGIC>):
585
586 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
587 # same as:
588 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
589
590Serialise some URIs and a regex in an array:
591
592 my $cbor = encode_cbor [
593 (CBOR::XS::tag 32, "http://www.nethype.de/"),
594 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
595 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
596 ];
597
598Wrap CBOR data in CBOR:
599
600 my $cbor_cbor = encode_cbor
601 CBOR::XS::tag 24,
602 encode_cbor [1, 2, 3];
603
604=head1 TAG HANDLING AND EXTENSIONS
605
606This section describes how this module handles specific tagged values and
607extensions. If a tag is not mentioned here, then the default handling
608applies (creating a CBOR::XS::Tagged object on decoding, and only encoding
609the tag when explicitly requested).
610
611Future versions of this module reserve the right to special case
612additional tags (such as bigfloat or base64url).
613
614=over 4
615
616=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>)
617
618These tags are automatically created for serialisable objects using the
619C<FREEZE/THAW> methods (the L<Types::Serialier> object serialisation
620protocol).
621
622=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
623
624These tags are automatically decoded when encountered, resulting in
625shared values in the decoded object. They are only encoded, however, when
626C<allow_sharable> is enabled.
627
628=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
629
630This tag is automatically generated when a reference are encountered (with
631the exception of hash and array refernces). It is converted to a reference
632when decoding.
633
634=item 55799 (self-describe CBOR, RFC 7049)
635
636This value is not generated on encoding (unless explicitly requested by
637the user), and is simply ignored when decoding.
638
639=back
640
641
281=head2 CBOR and JSON 642=head1 CBOR and JSON
282 643
283TODO 644CBOR is supposed to implement a superset of the JSON data model, and is,
645with some coercion, able to represent all JSON texts (something that other
646"binary JSON" formats such as BSON generally do not support).
647
648CBOR implements some extra hints and support for JSON interoperability,
649and the spec offers further guidance for conversion between CBOR and
650JSON. None of this is currently implemented in CBOR, and the guidelines
651in the spec do not result in correct round-tripping of data. If JSON
652interoperability is improved in the future, then the goal will be to
653ensure that decoded JSON data will round-trip encoding and decoding to
654CBOR intact.
284 655
285 656
286=head1 SECURITY CONSIDERATIONS 657=head1 SECURITY CONSIDERATIONS
287 658
288When you are using CBOR in a protocol, talking to untrusted potentially 659When you are using CBOR in a protocol, talking to untrusted potentially
356Please refrain from using rt.cpan.org or any other bug reporting 727Please refrain from using rt.cpan.org or any other bug reporting
357service. I put the contact address into my modules for a reason. 728service. I put the contact address into my modules for a reason.
358 729
359=cut 730=cut
360 731
361our $true = do { bless \(my $dummy = 1), "CBOR::XS::Boolean" };
362our $false = do { bless \(my $dummy = 0), "CBOR::XS::Boolean" };
363
364sub true() { $true }
365sub false() { $false }
366
367sub is_bool($) {
368 UNIVERSAL::isa $_[0], "CBOR::XS::Boolean"
369# or UNIVERSAL::isa $_[0], "CBOR::Literal"
370}
371
372XSLoader::load "CBOR::XS", $VERSION; 732XSLoader::load "CBOR::XS", $VERSION;
373
374package CBOR::XS::Boolean;
375
376use overload
377 "0+" => sub { ${$_[0]} },
378 "++" => sub { $_[0] = ${$_[0]} + 1 },
379 "--" => sub { $_[0] = ${$_[0]} - 1 },
380 fallback => 1;
381
3821;
383 733
384=head1 SEE ALSO 734=head1 SEE ALSO
385 735
386The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 736The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,
387serialisation. 737serialisation.
388 738
739The L<Types::Serialiser> module provides the data model for true, false
740and error values.
741
389=head1 AUTHOR 742=head1 AUTHOR
390 743
391 Marc Lehmann <schmorp@schmorp.de> 744 Marc Lehmann <schmorp@schmorp.de>
392 http://home.schmorp.de/ 745 http://home.schmorp.de/
393 746
394=cut 747=cut
395 748
7491
750

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines