ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.10 by root, Mon Oct 28 22:03:20 2013 UTC vs.
Revision 1.23 by root, Fri Nov 22 16:00:30 2013 UTC

26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up to 31WARNING! This module is very new, and not very well tested (that's up
32you to do). Furthermore, details of the implementation might change freely 32to you to do). Furthermore, details of the implementation might change
33before version 1.0. And lastly, the object serialisation protocol depends 33freely before version 1.0. And lastly, most extensions depend on an IANA
34on a pending IANA assignment, and until that assignment is official, this 34assignment, and until that assignment is official, this implementation is
35implementation is not interoperable with other implementations (even 35not interoperable with other implementations (even future versions of this
36future versions of this module) until the assignment is done. 36module) until the assignment is done.
37 37
38You are still invited to try out CBOR, and this module. 38You are still invited to try out CBOR, and this module.
39 39
40This module converts Perl data structures to the Concise Binary Object 40This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
46In short, CBOR is a faster and very compact binary alternative to JSON, 46In short, CBOR is a faster and very compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 47with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 48often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 49data later you might want to compare both formats first).
50 50
51To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison.
55
56As for compactness, C<CBOR::XS> encoded data structures are usually about
5720% smaller than the same data encoded as (compact) JSON or L<Storable>.
58
59In addition to the core CBOR data format, this module implements a number
60of extensions, to support cyclic and self-referencing data structures
61(see C<allow_sharing>), string deduplication (see C<allow_stringref>) and
62scalar references (always enabled).
63
51The primary goal of this module is to be I<correct> and the secondary goal 64The primary goal of this module is to be I<correct> and the secondary goal
52is to be I<fast>. To reach the latter goal it was written in C. 65is to be I<fast>. To reach the latter goal it was written in C.
53 66
54See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 67See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
55vice versa. 68vice versa.
58 71
59package CBOR::XS; 72package CBOR::XS;
60 73
61use common::sense; 74use common::sense;
62 75
63our $VERSION = 0.05; 76our $VERSION = 0.08;
64our @ISA = qw(Exporter); 77our @ISA = qw(Exporter);
65 78
66our @EXPORT = qw(encode_cbor decode_cbor); 79our @EXPORT = qw(encode_cbor decode_cbor);
67 80
68use Exporter; 81use Exporter;
105strings. All boolean flags described below are by default I<disabled>. 118strings. All boolean flags described below are by default I<disabled>.
106 119
107The mutators for flags all return the CBOR object again and thus calls can 120The mutators for flags all return the CBOR object again and thus calls can
108be chained: 121be chained:
109 122
110#TODO
111 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 123 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
112 124
113=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 125=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
114 126
115=item $max_depth = $cbor->get_max_depth 127=item $max_depth = $cbor->get_max_depth
149If no argument is given, the limit check will be deactivated (same as when 161If no argument is given, the limit check will be deactivated (same as when
150C<0> is specified). 162C<0> is specified).
151 163
152See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 164See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
153 165
166=item $cbor = $cbor->allow_unknown ([$enable])
167
168=item $enabled = $cbor->get_allow_unknown
169
170If C<$enable> is true (or missing), then C<encode> will I<not> throw an
171exception when it encounters values it cannot represent in CBOR (for
172example, filehandles) but instead will encode a CBOR C<error> value.
173
174If C<$enable> is false (the default), then C<encode> will throw an
175exception when it encounters anything it cannot encode as CBOR.
176
177This option does not affect C<decode> in any way, and it is recommended to
178leave it off unless you know your communications partner.
179
180=item $cbor = $cbor->allow_sharing ([$enable])
181
182=item $enabled = $cbor->get_allow_sharing
183
184If C<$enable> is true (or missing), then C<encode> will not double-encode
185values that have been referenced before (e.g. when the same object, such
186as an array, is referenced multiple times), but instead will emit a
187reference to the earlier value.
188
189This means that such values will only be encoded once, and will not result
190in a deep cloning of the value on decode, in decoders supporting the value
191sharing extension.
192
193It is recommended to leave it off unless you know your
194communication partner supports the value sharing extensions to CBOR
195(http://cbor.schmorp.de/value-sharing).
196
197Detecting shared values incurs a runtime overhead when values are encoded
198that have a reference counter large than one, and might unnecessarily
199increase the encoded size, as potentially shared values are encode as
200sharable whether or not they are actually shared.
201
202At the moment, only targets of references can be shared (e.g. scalars,
203arrays or hashes pointed to by a reference). Weirder constructs, such as
204an array with multiple "copies" of the I<same> string, which are hard but
205not impossible to create in Perl, are not supported (this is the same as
206for L<Storable>).
207
208If C<$enable> is false (the default), then C<encode> will encode
209exception when it encounters anything it cannot encode as CBOR.
210
211This option does not affect C<decode> in any way - shared values and
212references will always be decoded properly if present.
213
214=item $cbor = $cbor->allow_stringref ([$enable])
215
216=item $enabled = $cbor->get_allow_stringref
217
218If C<$enable> is true (or missing), then C<encode> will try not to encode
219the same string twice, but will instead encode a reference to the string
220instead. Depending on your data format. this can save a lot of space, but
221also results in a very large runtime overhead (expect encoding times to be
2222-4 times as high as without).
223
224It is recommended to leave it off unless you know your
225communications partner supports the stringref extension to CBOR
226(http://cbor.schmorp.de/stringref).
227
228If C<$enable> is false (the default), then C<encode> will encode
229exception when it encounters anything it cannot encode as CBOR.
230
231This option does not affect C<decode> in any way - string references will
232always be decoded properly if present.
233
234=item $cbor = $cbor->filter ([$cb->($tag, $value)])
235
236=item $cb_or_undef = $cbor->get_filter
237
238TODO
239
154=item $cbor_data = $cbor->encode ($perl_scalar) 240=item $cbor_data = $cbor->encode ($perl_scalar)
155 241
156Converts the given Perl data structure (a scalar value) to its CBOR 242Converts the given Perl data structure (a scalar value) to its CBOR
157representation. 243representation.
158 244
226C<Types:Serialiser::false> and C<Types::Serialiser::error>, 312C<Types:Serialiser::false> and C<Types::Serialiser::error>,
227respectively. They are overloaded to act almost exactly like the numbers 313respectively. They are overloaded to act almost exactly like the numbers
228C<1> and C<0> (for true and false) or to throw an exception on access (for 314C<1> and C<0> (for true and false) or to throw an exception on access (for
229error). See the L<Types::Serialiser> manpage for details. 315error). See the L<Types::Serialiser> manpage for details.
230 316
231=item CBOR tag 256 (perl object) 317=item tagged values
232 318
233The tag value C<256> (TODO: pending iana registration) will be used
234to deserialise a Perl object serialised with C<FREEZE>. See "OBJECT
235SERIALISATION", below, for details.
236
237=item CBOR tag 55799 (magic header)
238
239The tag 55799 is ignored (this tag implements the magic header).
240
241=item other CBOR tags
242
243Tagged items consists of a numeric tag and another CBOR value. Tags not 319Tagged items consists of a numeric tag and another CBOR value.
244handled internally are currently converted into a L<CBOR::XS::Tagged>
245object, which is simply a blessed array reference consisting of the
246numeric tag value followed by the (decoded) CBOR value.
247 320
248In the future, support for user-supplied conversions might get added. 321See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
322for details.
249 323
250=item anything else 324=item anything else
251 325
252Anything else (e.g. unsupported simple values) will raise a decoding 326Anything else (e.g. unsupported simple values) will raise a decoding
253error. 327error.
283C<1>, which get turned into false and true in CBOR. 357C<1>, which get turned into false and true in CBOR.
284 358
285=item CBOR::XS::Tagged objects 359=item CBOR::XS::Tagged objects
286 360
287Objects of this type must be arrays consisting of a single C<[tag, value]> 361Objects of this type must be arrays consisting of a single C<[tag, value]>
288pair. The (numerical) tag will be encoded as a CBOR tag, the value will be 362pair. The (numerical) tag will be encoded as a CBOR tag, the value will
289encoded as appropriate for the value. 363be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to
364create such objects.
290 365
291=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 366=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
292 367
293These special values become CBOR true, CBOR false and CBOR undefined 368These special values become CBOR true, CBOR false and CBOR undefined
294values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly 369values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
295if you want. 370if you want.
296 371
297=item other blessed objects 372=item other blessed objects
298 373
299Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See 374Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
300"OBJECT SERIALISATION", below, for details. 375L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
376module, and L<OBJECT SERIALISATION> for generic object serialisation.
301 377
302=item simple scalars 378=item simple scalars
303 379
304TODO
305Simple Perl scalars (any scalar that is not a reference) are the most 380Simple Perl scalars (any scalar that is not a reference) are the most
306difficult objects to encode: CBOR::XS will encode undefined scalars as 381difficult objects to encode: CBOR::XS will encode undefined scalars as
307CBOR null values, scalars that have last been used in a string context 382CBOR null values, scalars that have last been used in a string context
308before encoding as CBOR strings, and anything else as number value: 383before encoding as CBOR strings, and anything else as number value:
309 384
446=head1 MAGIC HEADER 521=head1 MAGIC HEADER
447 522
448There is no way to distinguish CBOR from other formats 523There is no way to distinguish CBOR from other formats
449programmatically. To make it easier to distinguish CBOR from other 524programmatically. To make it easier to distinguish CBOR from other
450formats, the CBOR specification has a special "magic string" that can be 525formats, the CBOR specification has a special "magic string" that can be
451prepended to any CBOR string without changing it's meaning. 526prepended to any CBOR string without changing its meaning.
452 527
453This string is available as C<$CBOR::XS::MAGIC>. This module does not 528This string is available as C<$CBOR::XS::MAGIC>. This module does not
454prepend this string tot he CBOR data it generates, but it will ignroe it 529prepend this string to the CBOR data it generates, but it will ignore it
455if present, so users can prepend this string as a "file type" indicator as 530if present, so users can prepend this string as a "file type" indicator as
456required. 531required.
532
533
534=head1 THE CBOR::XS::Tagged CLASS
535
536CBOR has the concept of tagged values - any CBOR value can be tagged with
537a numeric 64 bit number, which are centrally administered.
538
539C<CBOR::XS> handles a few tags internally when en- or decoding. You can
540also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
541decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
542unknown tag.
543
544These objects are simply blessed array references - the first member of
545the array being the numerical tag, the second being the value.
546
547You can interact with C<CBOR::XS::Tagged> objects in the following ways:
548
549=over 4
550
551=item $tagged = CBOR::XS::tag $tag, $value
552
553This function(!) creates a new C<CBOR::XS::Tagged> object using the given
554C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
555value that can be encoded in CBOR, including serialisable Perl objects and
556C<CBOR::XS::Tagged> objects).
557
558=item $tagged->[0]
559
560=item $tagged->[0] = $new_tag
561
562=item $tag = $tagged->tag
563
564=item $new_tag = $tagged->tag ($new_tag)
565
566Access/mutate the tag.
567
568=item $tagged->[1]
569
570=item $tagged->[1] = $new_value
571
572=item $value = $tagged->value
573
574=item $new_value = $tagged->value ($new_value)
575
576Access/mutate the tagged value.
577
578=back
579
580=cut
581
582sub tag($$) {
583 bless [@_], CBOR::XS::Tagged::;
584}
585
586sub CBOR::XS::Tagged::tag {
587 $_[0][0] = $_[1] if $#_;
588 $_[0][0]
589}
590
591sub CBOR::XS::Tagged::value {
592 $_[0][1] = $_[1] if $#_;
593 $_[0][1]
594}
595
596=head2 EXAMPLES
597
598Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
599
600You can look up CBOR tag value and emanings in the IANA registry at
601L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
602
603Prepend a magic header (C<$CBOR::XS::MAGIC>):
604
605 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
606 # same as:
607 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
608
609Serialise some URIs and a regex in an array:
610
611 my $cbor = encode_cbor [
612 (CBOR::XS::tag 32, "http://www.nethype.de/"),
613 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
614 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
615 ];
616
617Wrap CBOR data in CBOR:
618
619 my $cbor_cbor = encode_cbor
620 CBOR::XS::tag 24,
621 encode_cbor [1, 2, 3];
622
623=head1 TAG HANDLING AND EXTENSIONS
624
625This section describes how this module handles specific tagged values
626and extensions. If a tag is not mentioned here and no additional filters
627are provided for it, then the default handling applies (creating a
628CBOR::XS::Tagged object on decoding, and only encoding the tag when
629explicitly requested).
630
631Tags not handled specifically are currently converted into a
632L<CBOR::XS::Tagged> object, which is simply a blessed array reference
633consisting of the numeric tag value followed by the (decoded) CBOR value.
634
635Future versions of this module reserve the right to special case
636additional tags (such as base64url).
637
638=head2 ENFORCED TAGS
639
640These tags are always handled when decoding, and their handling cannot be
641overriden by the user.
642
643=over 4
644
645=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>)
646
647These tags are automatically created (and decoded) for serialisable
648objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
649serialisation protocol). See L<OBJECT SERIALISATION> for details.
650
651=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
652
653These tags are automatically decoded when encountered, resulting in
654shared values in the decoded object. They are only encoded, however, when
655C<allow_sharable> is enabled.
656
657=item <unassigned>, <unassigned> (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)
658
659These tags are automatically decoded when encountered. They are only
660encoded, however, when C<allow_stringref> is enabled.
661
662=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
663
664This tag is automatically generated when a reference are encountered (with
665the exception of hash and array refernces). It is converted to a reference
666when decoding.
667
668=item 55799 (self-describe CBOR, RFC 7049)
669
670This value is not generated on encoding (unless explicitly requested by
671the user), and is simply ignored when decoding.
672
673=back
674
675=head2 OPTIONAL TAGS
676
677These tags have default filters provided when decoding. Their handling can
678be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
679providing a custom C<filter> function when decoding.
680
681When they result in decoding into a specific Perl class, the module
682usually provides a corresponding C<TO_CBOR> method as well.
683
684When any of these need to load additional modules that are not part of the
685perl core distribution (e.g. L<URI>), it is (currently) up to the user to
686provide these modules. The decoding usually fails with an exception if the
687required module cannot be loaded.
688
689=over 4
690
691=item 2, 3 (positive/negative bignum)
692
693These tags are decoded into L<Math::BigInt> objects. The corresponding
694C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
695integers, and others into positive/negative CBOR bignums.
696
697=item 4, 5 (decimal fraction/bigfloat)
698
699Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
700objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
701encodes into a decimal fraction.
702
703CBOR cannot represent bigfloats with I<very> large exponents - conversion
704of such big float objects is undefined.
705
706Also, NaN and infinities are not encoded properly.
707
708=item 21, 22, 23 (expected later JSON conversion)
709
710CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
711tags.
712
713=item 32 (URI)
714
715These objects decode into L<URI> objects. The corresponding
716C<URI::TO_CBOR> method again results in a CBOR URI value.
717
718=back
719
720=cut
721
722our %FILTER = (
723 # 0 # rfc4287 datetime, utf-8
724 # 1 # unix timestamp, any
725
726 2 => sub { # pos bigint
727 require Math::BigInt;
728 Math::BigInt->new ("0x" . unpack "H*", pop)
729 },
730
731 3 => sub { # neg bigint
732 require Math::BigInt;
733 -Math::BigInt->new ("0x" . unpack "H*", pop)
734 },
735
736 4 => sub { # decimal fraction, array
737 require Math::BigFloat;
738 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
739 },
740
741 5 => sub { # bigfloat, array
742 require Math::BigFloat;
743 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
744 },
745
746 21 => sub { pop }, # expected conversion to base64url encoding
747 22 => sub { pop }, # expected conversion to base64 encoding
748 23 => sub { pop }, # expected conversion to base16 encoding
749
750 # 24 # embedded cbor, byte string
751
752 32 => sub {
753 require URI;
754 URI->new (pop)
755 },
756
757 # 33 # base64url rfc4648, utf-8
758 # 34 # base64 rfc46484, utf-8
759 # 35 # regex pcre/ecma262, utf-8
760 # 36 # mime message rfc2045, utf-8
761);
457 762
458 763
459=head1 CBOR and JSON 764=head1 CBOR and JSON
460 765
461CBOR is supposed to implement a superset of the JSON data model, and is, 766CBOR is supposed to implement a superset of the JSON data model, and is,
544Please refrain from using rt.cpan.org or any other bug reporting 849Please refrain from using rt.cpan.org or any other bug reporting
545service. I put the contact address into my modules for a reason. 850service. I put the contact address into my modules for a reason.
546 851
547=cut 852=cut
548 853
854our %FILTER = (
855 # 0 # rfc4287 datetime, utf-8
856 # 1 # unix timestamp, any
857
858 2 => sub { # pos bigint
859 require Math::BigInt;
860 Math::BigInt->new ("0x" . unpack "H*", pop)
861 },
862
863 3 => sub { # neg bigint
864 require Math::BigInt;
865 -Math::BigInt->new ("0x" . unpack "H*", pop)
866 },
867
868 4 => sub { # decimal fraction, array
869 require Math::BigFloat;
870 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
871 },
872
873 5 => sub { # bigfloat, array
874 require Math::BigFloat;
875 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
876 },
877
878 21 => sub { pop }, # expected conversion to base64url encoding
879 22 => sub { pop }, # expected conversion to base64 encoding
880 23 => sub { pop }, # expected conversion to base16 encoding
881
882 # 24 # embedded cbor, byte string
883
884 32 => sub {
885 require URI;
886 URI->new (pop)
887 },
888
889 # 33 # base64url rfc4648, utf-8
890 # 34 # base64 rfc46484, utf-8
891 # 35 # regex pcre/ecma262, utf-8
892 # 36 # mime message rfc2045, utf-8
893);
894
895sub CBOR::XS::default_filter {
896 &{ $FILTER{$_[0]} or return }
897}
898
899sub URI::TO_CBOR {
900 my $uri = $_[0]->as_string;
901 utf8::upgrade $uri;
902 CBOR::XS::tag 32, $uri
903}
904
905sub Math::BigInt::TO_CBOR {
906 if ($_[0] >= -2147483648 && $_[0] <= 2147483647) {
907 $_[0]->numify
908 } else {
909 my $hex = substr $_[0]->as_hex, 2;
910 $hex = "0$hex" if 1 & length $hex; # sigh
911 CBOR::XS::tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
912 }
913}
914
915sub Math::BigFloat::TO_CBOR {
916 my ($m, $e) = $_[0]->parts;
917 CBOR::XS::tag 4, [$e->numify, $m]
918}
919
549XSLoader::load "CBOR::XS", $VERSION; 920XSLoader::load "CBOR::XS", $VERSION;
550 921
551=head1 SEE ALSO 922=head1 SEE ALSO
552 923
553The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 924The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines