ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.2 by root, Sat Oct 26 10:41:12 2013 UTC vs.
Revision 1.20 by root, Wed Nov 20 02:03:08 2013 UTC

12 $perl_value = decode_cbor $binary_cbor_data; 12 $perl_value = decode_cbor $binary_cbor_data;
13 13
14 # OO-interface 14 # OO-interface
15 15
16 $coder = CBOR::XS->new; 16 $coder = CBOR::XS->new;
17 #TODO 17 $binary_cbor_data = $coder->encode ($perl_value);
18 $perl_value = $coder->decode ($binary_cbor_data);
19
20 # prefix decoding
21
22 my $many_cbor_strings = ...;
23 while (length $many_cbor_strings) {
24 my ($data, $length) = $cbor->decode_prefix ($many_cbor_strings);
25 # data was decoded
26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 }
18 28
19=head1 DESCRIPTION 29=head1 DESCRIPTION
20 30
21WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA AND 31WARNING! This module is very new, and not very well tested (that's up to
22EAT YOUR CHILDREN! 32you to do). Furthermore, details of the implementation might change freely
33before version 1.0. And lastly, the object serialisation protocol depends
34on a pending IANA assignment, and until that assignment is official, this
35implementation is not interoperable with other implementations (even
36future versions of this module) until the assignment is done.
23 37
24This module converts Perl data structures to CBOR and vice versa. Its 38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you
43can represent something in JSON, you should be able to represent it in
44CBOR.
45
46In short, CBOR is a faster and very compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first).
50
51To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison.
55
56As for compactness, C<CBOR::XS> encoded data structures are usually about
5720% smaller than the same data encoded as (compact) JSON or L<Storable>.
58
25primary goal is to be I<correct> and its secondary goal is to be 59The primary goal of this module is to be I<correct> and the secondary goal
26I<fast>. To reach the latter goal it was written in C. 60is to be I<fast>. To reach the latter goal it was written in C.
27 61
28See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 62See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
29vice versa. 63vice versa.
30 64
31=cut 65=cut
32 66
33package CBOR::XS; 67package CBOR::XS;
34 68
35use common::sense; 69use common::sense;
36 70
37our $VERSION = 0.01; 71our $VERSION = 0.08;
38our @ISA = qw(Exporter); 72our @ISA = qw(Exporter);
39 73
40our @EXPORT = qw(encode_cbor decode_cbor); 74our @EXPORT = qw(encode_cbor decode_cbor);
41 75
42use Exporter; 76use Exporter;
43use XSLoader; 77use XSLoader;
78
79use Types::Serialiser;
80
81our $MAGIC = "\xd9\xd9\xf7";
44 82
45=head1 FUNCTIONAL INTERFACE 83=head1 FUNCTIONAL INTERFACE
46 84
47The following convenience methods are provided by this module. They are 85The following convenience methods are provided by this module. They are
48exported by default: 86exported by default:
119If no argument is given, the limit check will be deactivated (same as when 157If no argument is given, the limit check will be deactivated (same as when
120C<0> is specified). 158C<0> is specified).
121 159
122See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 160See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
123 161
162=item $cbor = $cbor->allow_unknown ([$enable])
163
164=item $enabled = $cbor->get_allow_unknown
165
166If C<$enable> is true (or missing), then C<encode> will I<not> throw an
167exception when it encounters values it cannot represent in CBOR (for
168example, filehandles) but instead will encode a CBOR C<error> value.
169
170If C<$enable> is false (the default), then C<encode> will throw an
171exception when it encounters anything it cannot encode as CBOR.
172
173This option does not affect C<decode> in any way, and it is recommended to
174leave it off unless you know your communications partner.
175
176=item $cbor = $cbor->allow_sharing ([$enable])
177
178=item $enabled = $cbor->get_allow_sharing
179
180If C<$enable> is true (or missing), then C<encode> will not double-encode
181values that have been referenced before (e.g. when the same object, such
182as an array, is referenced multiple times), but instead will emit a
183reference to the earlier value.
184
185This means that such values will only be encoded once, and will not result
186in a deep cloning of the value on decode, in decoders supporting the value
187sharing extension.
188
189Detecting shared values incurs a runtime overhead when values are encoded
190that have a reference counter large than one, and might unnecessarily
191increase the encoded size, as potentially shared values are encode as
192sharable whether or not they are actually shared.
193
194At the moment, only targets of references can be shared (e.g. scalars,
195arrays or hashes pointed to by a reference). Weirder constructs, such as
196an array with multiple "copies" of the I<same> string, which are hard but
197not impossible to create in Perl, are not supported (this is the same as
198for L<Storable>).
199
200If C<$enable> is false (the default), then C<encode> will encode
201exception when it encounters anything it cannot encode as CBOR.
202
203This option does not affect C<decode> in any way - shared values and
204references will always be decoded properly if present. It is recommended
205to leave it off unless you know your communications partner supports the
206value sharing extensions to CBOR (http://cbor.schmorp.de/value-sharing).
207
124=item $cbor_data = $cbor->encode ($perl_scalar) 208=item $cbor_data = $cbor->encode ($perl_scalar)
125 209
126Converts the given Perl data structure (a scalar value) to its CBOR 210Converts the given Perl data structure (a scalar value) to its CBOR
127representation. 211representation.
128 212
161 245
162=head2 CBOR -> PERL 246=head2 CBOR -> PERL
163 247
164=over 4 248=over 4
165 249
166=item True, False 250=item integers
167 251
168These CBOR values become C<CBOR::XS::true> and C<CBOR::XS::false>, 252CBOR integers become (numeric) perl scalars. On perls without 64 bit
253support, 64 bit integers will be truncated or otherwise corrupted.
254
255=item byte strings
256
257Byte strings will become octet strings in Perl (the byte values 0..255
258will simply become characters of the same value in Perl).
259
260=item UTF-8 strings
261
262UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
263decoded into proper Unicode code points. At the moment, the validity of
264the UTF-8 octets will not be validated - corrupt input will result in
265corrupted Perl strings.
266
267=item arrays, maps
268
269CBOR arrays and CBOR maps will be converted into references to a Perl
270array or hash, respectively. The keys of the map will be stringified
271during this process.
272
273=item null
274
275CBOR null becomes C<undef> in Perl.
276
277=item true, false, undefined
278
279These CBOR values become C<Types:Serialiser::true>,
280C<Types:Serialiser::false> and C<Types::Serialiser::error>,
169respectively. They are overloaded to act almost exactly like the numbers 281respectively. They are overloaded to act almost exactly like the numbers
170C<1> and C<0>. You can check whether a scalar is a CBOR boolean by using 282C<1> and C<0> (for true and false) or to throw an exception on access (for
171the C<CBOR::XS::is_bool> function. 283error). See the L<Types::Serialiser> manpage for details.
172 284
173=item Null, Undefined 285=item CBOR tag 256 (perl object)
174 286
175CBOR Null and Undefined values becomes C<undef> in Perl (in the future, 287The tag value C<256> (TODO: pending iana registration) will be used
176Undefined may raise an exception). 288to deserialise a Perl object serialised with C<FREEZE>. See L<OBJECT
289SERIALISATION>, below, for details.
290
291=item CBOR tag 55799 (magic header)
292
293The tag 55799 is ignored (this tag implements the magic header).
294
295=item other CBOR tags
296
297Tagged items consists of a numeric tag and another CBOR value. Tags not
298handled internally are currently converted into a L<CBOR::XS::Tagged>
299object, which is simply a blessed array reference consisting of the
300numeric tag value followed by the (decoded) CBOR value.
301
302In the future, support for user-supplied conversions might get added.
303
304=item anything else
305
306Anything else (e.g. unsupported simple values) will raise a decoding
307error.
177 308
178=back 309=back
179 310
180 311
181=head2 PERL -> CBOR 312=head2 PERL -> CBOR
186 317
187=over 4 318=over 4
188 319
189=item hash references 320=item hash references
190 321
191Perl hash references become CBOR maps. As there is no inherent ordering 322Perl hash references become CBOR maps. As there is no inherent ordering in
192in hash keys (or CBOR maps), they will usually be encoded in a 323hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
193pseudo-random order. 324order.
325
326Currently, tied hashes will use the indefinite-length format, while normal
327hashes will use the fixed-length format.
194 328
195=item array references 329=item array references
196 330
197Perl array references become CBOR arrays. 331Perl array references become fixed-length CBOR arrays.
198 332
199=item other references 333=item other references
200 334
201Other unblessed references are generally not allowed and will cause an 335Other unblessed references are generally not allowed and will cause an
202exception to be thrown, except for references to the integers C<0> and 336exception to be thrown, except for references to the integers C<0> and
203C<1>, which get turned into C<False> and C<True> in CBOR. 337C<1>, which get turned into false and true in CBOR.
204 338
205=item CBOR::XS::true, CBOR::XS::false 339=item CBOR::XS::Tagged objects
206 340
341Objects of this type must be arrays consisting of a single C<[tag, value]>
342pair. The (numerical) tag will be encoded as a CBOR tag, the value will
343be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to
344create such objects.
345
346=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
347
207These special values become CBOR True and CBOR False values, 348These special values become CBOR true, CBOR false and CBOR undefined
208respectively. You can also use C<\1> and C<\0> directly if you want. 349values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
350if you want.
209 351
210=item blessed objects 352=item other blessed objects
211 353
212Blessed objects are not directly representable in CBOR. TODO 354Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
213See the 355L<OBJECT SERIALISATION>, below, for details.
214C<allow_blessed> and C<convert_blessed> methods on various options on
215how to deal with this: basically, you can choose between throwing an
216exception, encoding the reference as if it weren't blessed, or provide
217your own serialiser method.
218 356
219=item simple scalars 357=item simple scalars
220 358
221TODO 359TODO
222Simple Perl scalars (any scalar that is not a reference) are the most 360Simple Perl scalars (any scalar that is not a reference) are the most
223difficult objects to encode: CBOR::XS will encode undefined scalars as 361difficult objects to encode: CBOR::XS will encode undefined scalars as
224CBOR C<Null> values, scalars that have last been used in a string context 362CBOR null values, scalars that have last been used in a string context
225before encoding as CBOR strings, and anything else as number value: 363before encoding as CBOR strings, and anything else as number value:
226 364
227 # dump as number 365 # dump as number
228 encode_cbor [2] # yields [2] 366 encode_cbor [2] # yields [2]
229 encode_cbor [-3.0e17] # yields [-3e+17] 367 encode_cbor [-3.0e17] # yields [-3e+17]
251 389
252You can not currently force the type in other, less obscure, ways. Tell me 390You can not currently force the type in other, less obscure, ways. Tell me
253if you need this capability (but don't forget to explain why it's needed 391if you need this capability (but don't forget to explain why it's needed
254:). 392:).
255 393
256Note that numerical precision has the same meaning as under Perl (so 394Perl values that seem to be integers generally use the shortest possible
257binary to decimal conversion follows the same rules as in Perl, which 395representation. Floating-point values will use either the IEEE single
258can differ to other languages). Also, your perl interpreter might expose 396format if possible without loss of precision, otherwise the IEEE double
259extensions to the floating point numbers of your platform, such as 397format will be used. Perls that use formats other than IEEE double to
260infinities or NaN's - these cannot be represented in CBOR, and it is an 398represent numerical values are supported, but might suffer loss of
261error to pass those in. 399precision.
262 400
263=back 401=back
264 402
403=head2 OBJECT SERIALISATION
265 404
405This module knows two way to serialise a Perl object: The CBOR-specific
406way, and the generic way.
407
408Whenever the encoder encounters a Perl object that it cnanot serialise
409directly (most of them), it will first look up the C<TO_CBOR> method on
410it.
411
412If it has a C<TO_CBOR> method, it will call it with the object as only
413argument, and expects exactly one return value, which it will then
414substitute and encode it in the place of the object.
415
416Otherwise, it will look up the C<FREEZE> method. If it exists, it will
417call it with the object as first argument, and the constant string C<CBOR>
418as the second argument, to distinguish it from other serialisers.
419
420The C<FREEZE> method can return any number of values (i.e. zero or
421more). These will be encoded as CBOR perl object, together with the
422classname.
423
424If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
425with an error.
426
427Objects encoded via C<TO_CBOR> cannot be automatically decoded, but
428objects encoded via C<FREEZE> can be decoded using the following protocol:
429
430When an encoded CBOR perl object is encountered by the decoder, it will
431look up the C<THAW> method, by using the stored classname, and will fail
432if the method cannot be found.
433
434After the lookup it will call the C<THAW> method with the stored classname
435as first argument, the constant string C<CBOR> as second argument, and all
436values returned by C<FREEZE> as remaining arguments.
437
438=head4 EXAMPLES
439
440Here is an example C<TO_CBOR> method:
441
442 sub My::Object::TO_CBOR {
443 my ($obj) = @_;
444
445 ["this is a serialised My::Object object", $obj->{id}]
446 }
447
448When a C<My::Object> is encoded to CBOR, it will instead encode a simple
449array with two members: a string, and the "object id". Decoding this CBOR
450string will yield a normal perl array reference in place of the object.
451
452A more useful and practical example would be a serialisation method for
453the URI module. CBOR has a custom tag value for URIs, namely 32:
454
455 sub URI::TO_CBOR {
456 my ($self) = @_;
457 my $uri = "$self"; # stringify uri
458 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
459 CBOR::XS::tagged 32, "$_[0]"
460 }
461
462This will encode URIs as a UTF-8 string with tag 32, which indicates an
463URI.
464
465Decoding such an URI will not (currently) give you an URI object, but
466instead a CBOR::XS::Tagged object with tag number 32 and the string -
467exactly what was returned by C<TO_CBOR>.
468
469To serialise an object so it can automatically be deserialised, you need
470to use C<FREEZE> and C<THAW>. To take the URI module as example, this
471would be a possible implementation:
472
473 sub URI::FREEZE {
474 my ($self, $serialiser) = @_;
475 "$self" # encode url string
476 }
477
478 sub URI::THAW {
479 my ($class, $serialiser, $uri) = @_;
480
481 $class->new ($uri)
482 }
483
484Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
485example, a C<FREEZE> method that returns "type", "id" and "variant" values
486would cause an invocation of C<THAW> with 5 arguments:
487
488 sub My::Object::FREEZE {
489 my ($self, $serialiser) = @_;
490
491 ($self->{type}, $self->{id}, $self->{variant})
492 }
493
494 sub My::Object::THAW {
495 my ($class, $serialiser, $type, $id, $variant) = @_;
496
497 $class-<new (type => $type, id => $id, variant => $variant)
498 }
499
500
501=head1 MAGIC HEADER
502
503There is no way to distinguish CBOR from other formats
504programmatically. To make it easier to distinguish CBOR from other
505formats, the CBOR specification has a special "magic string" that can be
506prepended to any CBOR string without changing its meaning.
507
508This string is available as C<$CBOR::XS::MAGIC>. This module does not
509prepend this string to the CBOR data it generates, but it will ignore it
510if present, so users can prepend this string as a "file type" indicator as
511required.
512
513
514=head1 THE CBOR::XS::Tagged CLASS
515
516CBOR has the concept of tagged values - any CBOR value can be tagged with
517a numeric 64 bit number, which are centrally administered.
518
519C<CBOR::XS> handles a few tags internally when en- or decoding. You can
520also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
521decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
522unknown tag.
523
524These objects are simply blessed array references - the first member of
525the array being the numerical tag, the second being the value.
526
527You can interact with C<CBOR::XS::Tagged> objects in the following ways:
528
529=over 4
530
531=item $tagged = CBOR::XS::tag $tag, $value
532
533This function(!) creates a new C<CBOR::XS::Tagged> object using the given
534C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
535value that can be encoded in CBOR, including serialisable Perl objects and
536C<CBOR::XS::Tagged> objects).
537
538=item $tagged->[0]
539
540=item $tagged->[0] = $new_tag
541
542=item $tag = $tagged->tag
543
544=item $new_tag = $tagged->tag ($new_tag)
545
546Access/mutate the tag.
547
548=item $tagged->[1]
549
550=item $tagged->[1] = $new_value
551
552=item $value = $tagged->value
553
554=item $new_value = $tagged->value ($new_value)
555
556Access/mutate the tagged value.
557
558=back
559
560=cut
561
562sub tag($$) {
563 bless [@_], CBOR::XS::Tagged::;
564}
565
566sub CBOR::XS::Tagged::tag {
567 $_[0][0] = $_[1] if $#_;
568 $_[0][0]
569}
570
571sub CBOR::XS::Tagged::value {
572 $_[0][1] = $_[1] if $#_;
573 $_[0][1]
574}
575
576=head2 EXAMPLES
577
578Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
579
580You can look up CBOR tag value and emanings in the IANA registry at
581L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
582
583Prepend a magic header (C<$CBOR::XS::MAGIC>):
584
585 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
586 # same as:
587 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
588
589Serialise some URIs and a regex in an array:
590
591 my $cbor = encode_cbor [
592 (CBOR::XS::tag 32, "http://www.nethype.de/"),
593 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
594 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
595 ];
596
597Wrap CBOR data in CBOR:
598
599 my $cbor_cbor = encode_cbor
600 CBOR::XS::tag 24,
601 encode_cbor [1, 2, 3];
602
603=head1 TAG HANDLING AND EXTENSIONS
604
605This section describes how this module handles specific tagged values and
606extensions. If a tag is not mentioned here, then the default handling
607applies (creating a CBOR::XS::Tagged object on decoding, and only encoding
608the tag when explicitly requested).
609
610Future versions of this module reserve the right to special case
611additional tags (such as bigfloat or base64url).
612
613=over 4
614
615=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>)
616
617These tags are automatically created for serialisable objects using the
618C<FREEZE/THAW> methods (the L<Types::Serialier> object serialisation
619protocol).
620
621=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
622
623These tags are automatically decoded when encountered, resulting in
624shared values in the decoded object. They are only encoded, however, when
625C<allow_sharable> is enabled.
626
627=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
628
629This tag is automatically generated when a reference are encountered (with
630the exception of hash and array refernces). It is converted to a reference
631when decoding.
632
633=item 55799 (self-describe CBOR, RFC 7049)
634
635This value is not generated on encoding (unless explicitly requested by
636the user), and is simply ignored when decoding.
637
638=back
639
640
266=head2 CBOR and JSON 641=head1 CBOR and JSON
267 642
268TODO 643CBOR is supposed to implement a superset of the JSON data model, and is,
644with some coercion, able to represent all JSON texts (something that other
645"binary JSON" formats such as BSON generally do not support).
646
647CBOR implements some extra hints and support for JSON interoperability,
648and the spec offers further guidance for conversion between CBOR and
649JSON. None of this is currently implemented in CBOR, and the guidelines
650in the spec do not result in correct round-tripping of data. If JSON
651interoperability is improved in the future, then the goal will be to
652ensure that decoded JSON data will round-trip encoding and decoding to
653CBOR intact.
269 654
270 655
271=head1 SECURITY CONSIDERATIONS 656=head1 SECURITY CONSIDERATIONS
272 657
273When you are using CBOR in a protocol, talking to untrusted potentially 658When you are using CBOR in a protocol, talking to untrusted potentially
341Please refrain from using rt.cpan.org or any other bug reporting 726Please refrain from using rt.cpan.org or any other bug reporting
342service. I put the contact address into my modules for a reason. 727service. I put the contact address into my modules for a reason.
343 728
344=cut 729=cut
345 730
346our $true = do { bless \(my $dummy = 1), "CBOR::XS::Boolean" };
347our $false = do { bless \(my $dummy = 0), "CBOR::XS::Boolean" };
348
349sub true() { $true }
350sub false() { $false }
351
352sub is_bool($) {
353 UNIVERSAL::isa $_[0], "CBOR::XS::Boolean"
354# or UNIVERSAL::isa $_[0], "CBOR::Literal"
355}
356
357XSLoader::load "CBOR::XS", $VERSION; 731XSLoader::load "CBOR::XS", $VERSION;
358
359package CBOR::XS::Boolean;
360
361use overload
362 "0+" => sub { ${$_[0]} },
363 "++" => sub { $_[0] = ${$_[0]} + 1 },
364 "--" => sub { $_[0] = ${$_[0]} - 1 },
365 fallback => 1;
366
3671;
368 732
369=head1 SEE ALSO 733=head1 SEE ALSO
370 734
371The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 735The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,
372serialisation. 736serialisation.
373 737
738The L<Types::Serialiser> module provides the data model for true, false
739and error values.
740
374=head1 AUTHOR 741=head1 AUTHOR
375 742
376 Marc Lehmann <schmorp@schmorp.de> 743 Marc Lehmann <schmorp@schmorp.de>
377 http://home.schmorp.de/ 744 http://home.schmorp.de/
378 745
379=cut 746=cut
380 747
7481
749

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines