ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.7 by root, Sun Oct 27 22:35:15 2013 UTC vs.
Revision 1.24 by root, Fri Nov 22 16:18:59 2013 UTC

26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA 31WARNING! This module is very new, and not very well tested (that's up
32AND EAT YOUR CHILDREN! (Actually, apart from being untested and a bit 32to you to do). Furthermore, details of the implementation might change
33feature-limited, it might already be useful). 33freely before version 1.0. And lastly, most extensions depend on an IANA
34assignment, and until that assignment is official, this implementation is
35not interoperable with other implementations (even future versions of this
36module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
34 39
35This module converts Perl data structures to the Concise Binary Object 40This module converts Perl data structures to the Concise Binary Object
36Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
37format that aims to use a superset of the JSON data model, i.e. when you 42format that aims to use a superset of the JSON data model, i.e. when you
38can represent something in JSON, you should be able to represent it in 43can represent something in JSON, you should be able to represent it in
39CBOR. 44CBOR.
40 45
41This makes it a faster and more compact binary alternative to JSON, with 46In short, CBOR is a faster and very compact binary alternative to JSON,
42the added ability of supporting serialising of perl objects. 47with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first).
50
51To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison.
55
56As for compactness, C<CBOR::XS> encoded data structures are usually about
5720% smaller than the same data encoded as (compact) JSON or L<Storable>.
58
59In addition to the core CBOR data format, this module implements a number
60of extensions, to support cyclic and self-referencing data structures
61(see C<allow_sharing>), string deduplication (see C<allow_stringref>) and
62scalar references (always enabled).
43 63
44The primary goal of this module is to be I<correct> and the secondary goal 64The primary goal of this module is to be I<correct> and the secondary goal
45is to be I<fast>. To reach the latter goal it was written in C. 65is to be I<fast>. To reach the latter goal it was written in C.
46 66
47See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 67See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
51 71
52package CBOR::XS; 72package CBOR::XS;
53 73
54use common::sense; 74use common::sense;
55 75
56our $VERSION = 0.03; 76our $VERSION = 0.09;
57our @ISA = qw(Exporter); 77our @ISA = qw(Exporter);
58 78
59our @EXPORT = qw(encode_cbor decode_cbor); 79our @EXPORT = qw(encode_cbor decode_cbor);
60 80
61use Exporter; 81use Exporter;
98strings. All boolean flags described below are by default I<disabled>. 118strings. All boolean flags described below are by default I<disabled>.
99 119
100The mutators for flags all return the CBOR object again and thus calls can 120The mutators for flags all return the CBOR object again and thus calls can
101be chained: 121be chained:
102 122
103#TODO
104 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 123 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
105 124
106=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 125=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
107 126
108=item $max_depth = $cbor->get_max_depth 127=item $max_depth = $cbor->get_max_depth
142If no argument is given, the limit check will be deactivated (same as when 161If no argument is given, the limit check will be deactivated (same as when
143C<0> is specified). 162C<0> is specified).
144 163
145See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 164See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
146 165
166=item $cbor = $cbor->allow_unknown ([$enable])
167
168=item $enabled = $cbor->get_allow_unknown
169
170If C<$enable> is true (or missing), then C<encode> will I<not> throw an
171exception when it encounters values it cannot represent in CBOR (for
172example, filehandles) but instead will encode a CBOR C<error> value.
173
174If C<$enable> is false (the default), then C<encode> will throw an
175exception when it encounters anything it cannot encode as CBOR.
176
177This option does not affect C<decode> in any way, and it is recommended to
178leave it off unless you know your communications partner.
179
180=item $cbor = $cbor->allow_sharing ([$enable])
181
182=item $enabled = $cbor->get_allow_sharing
183
184If C<$enable> is true (or missing), then C<encode> will not double-encode
185values that have been referenced before (e.g. when the same object, such
186as an array, is referenced multiple times), but instead will emit a
187reference to the earlier value.
188
189This means that such values will only be encoded once, and will not result
190in a deep cloning of the value on decode, in decoders supporting the value
191sharing extension.
192
193It is recommended to leave it off unless you know your
194communication partner supports the value sharing extensions to CBOR
195(http://cbor.schmorp.de/value-sharing).
196
197Detecting shared values incurs a runtime overhead when values are encoded
198that have a reference counter large than one, and might unnecessarily
199increase the encoded size, as potentially shared values are encode as
200sharable whether or not they are actually shared.
201
202At the moment, only targets of references can be shared (e.g. scalars,
203arrays or hashes pointed to by a reference). Weirder constructs, such as
204an array with multiple "copies" of the I<same> string, which are hard but
205not impossible to create in Perl, are not supported (this is the same as
206for L<Storable>).
207
208If C<$enable> is false (the default), then C<encode> will encode
209exception when it encounters anything it cannot encode as CBOR.
210
211This option does not affect C<decode> in any way - shared values and
212references will always be decoded properly if present.
213
214=item $cbor = $cbor->allow_stringref ([$enable])
215
216=item $enabled = $cbor->get_allow_stringref
217
218If C<$enable> is true (or missing), then C<encode> will try not to encode
219the same string twice, but will instead encode a reference to the string
220instead. Depending on your data format. this can save a lot of space, but
221also results in a very large runtime overhead (expect encoding times to be
2222-4 times as high as without).
223
224It is recommended to leave it off unless you know your
225communications partner supports the stringref extension to CBOR
226(http://cbor.schmorp.de/stringref).
227
228If C<$enable> is false (the default), then C<encode> will encode
229exception when it encounters anything it cannot encode as CBOR.
230
231This option does not affect C<decode> in any way - string references will
232always be decoded properly if present.
233
234=item $cbor = $cbor->filter ([$cb->($tag, $value)])
235
236=item $cb_or_undef = $cbor->get_filter
237
238Sets or replaces the tagged value decoding filter (when C<$cb> is
239specified) or clears the filter (if no argument or C<undef> is provided).
240
241The filter callback is called only during decoding, when a non-enforced
242tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
243list of enforced tags). For specific tags, it's often better to provide a
244default converter using the C<%CBOR::XS::FILTER> hash (see below).
245
246The first argument is the numerical tag, the second is the (decoded) value
247that has been tagged.
248
249The filter function should return either exactly one value, which will
250replace the tagged value in the decoded data structure, or no values,
251which will result in default handling, which currently means the decoder
252creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
253
254When the filter is cleared (the default state), the default filter
255function, C<CBOR::XS::default_filter>, is used. This function simply looks
256up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
257a code reference that is called with tag and value, and is responsible for
258decoding the value. If no entry exists, it returns no values.
259
260Example: decode all tags not handled internally into CBOR::XS::Tagged
261objects, with no other special handling (useful when working with
262potentially "unsafe" CBOR data).
263
264 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
265
266Example: provide a global filter for tag 1347375694, converting the value
267into some string form.
268
269 $CBOR::XS::FILTER{1347375694} = sub {
270 my ($tag, $value);
271
272 "tag 1347375694 value $value"
273 };
274
147=item $cbor_data = $cbor->encode ($perl_scalar) 275=item $cbor_data = $cbor->encode ($perl_scalar)
148 276
149Converts the given Perl data structure (a scalar value) to its CBOR 277Converts the given Perl data structure (a scalar value) to its CBOR
150representation. 278representation.
151 279
219C<Types:Serialiser::false> and C<Types::Serialiser::error>, 347C<Types:Serialiser::false> and C<Types::Serialiser::error>,
220respectively. They are overloaded to act almost exactly like the numbers 348respectively. They are overloaded to act almost exactly like the numbers
221C<1> and C<0> (for true and false) or to throw an exception on access (for 349C<1> and C<0> (for true and false) or to throw an exception on access (for
222error). See the L<Types::Serialiser> manpage for details. 350error). See the L<Types::Serialiser> manpage for details.
223 351
224=item CBOR tag 256 (perl object) 352=item tagged values
225 353
226The tag value C<256> (TODO: pending iana registration) will be used
227to deserialise a Perl object serialised with C<FREEZE>. See "OBJECT
228SERIALISATION", below, for details.
229
230=item CBOR tag 55799 (magic header)
231
232The tag 55799 is ignored (this tag implements the magic header).
233
234=item other CBOR tags
235
236Tagged items consists of a numeric tag and another CBOR value. Tags not 354Tagged items consists of a numeric tag and another CBOR value.
237handled internally are currently converted into a L<CBOR::XS::Tagged>
238object, which is simply a blessed array reference consisting of the
239numeric tag value followed by the (decoded) CBOR value.
240 355
241In the future, support for user-supplied conversions might get added. 356See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
357for details.
242 358
243=item anything else 359=item anything else
244 360
245Anything else (e.g. unsupported simple values) will raise a decoding 361Anything else (e.g. unsupported simple values) will raise a decoding
246error. 362error.
276C<1>, which get turned into false and true in CBOR. 392C<1>, which get turned into false and true in CBOR.
277 393
278=item CBOR::XS::Tagged objects 394=item CBOR::XS::Tagged objects
279 395
280Objects of this type must be arrays consisting of a single C<[tag, value]> 396Objects of this type must be arrays consisting of a single C<[tag, value]>
281pair. The (numerical) tag will be encoded as a CBOR tag, the value will be 397pair. The (numerical) tag will be encoded as a CBOR tag, the value will
282encoded as appropriate for the value. 398be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to
399create such objects.
283 400
284=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 401=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
285 402
286These special values become CBOR true, CBOR false and CBOR undefined 403These special values become CBOR true, CBOR false and CBOR undefined
287values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly 404values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
288if you want. 405if you want.
289 406
290=item other blessed objects 407=item other blessed objects
291 408
292Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See 409Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
293"OBJECT SERIALISATION", below, for details. 410L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
411module, and L<OBJECT SERIALISATION> for generic object serialisation.
294 412
295=item simple scalars 413=item simple scalars
296 414
297TODO
298Simple Perl scalars (any scalar that is not a reference) are the most 415Simple Perl scalars (any scalar that is not a reference) are the most
299difficult objects to encode: CBOR::XS will encode undefined scalars as 416difficult objects to encode: CBOR::XS will encode undefined scalars as
300CBOR null values, scalars that have last been used in a string context 417CBOR null values, scalars that have last been used in a string context
301before encoding as CBOR strings, and anything else as number value: 418before encoding as CBOR strings, and anything else as number value:
302 419
439=head1 MAGIC HEADER 556=head1 MAGIC HEADER
440 557
441There is no way to distinguish CBOR from other formats 558There is no way to distinguish CBOR from other formats
442programmatically. To make it easier to distinguish CBOR from other 559programmatically. To make it easier to distinguish CBOR from other
443formats, the CBOR specification has a special "magic string" that can be 560formats, the CBOR specification has a special "magic string" that can be
444prepended to any CBOR string without changing it's meaning. 561prepended to any CBOR string without changing its meaning.
445 562
446This string is available as C<$CBOR::XS::MAGIC>. This module does not 563This string is available as C<$CBOR::XS::MAGIC>. This module does not
447prepend this string tot he CBOR data it generates, but it will ignroe it 564prepend this string to the CBOR data it generates, but it will ignore it
448if present, so users can prepend this string as a "file type" indicator as 565if present, so users can prepend this string as a "file type" indicator as
449required. 566required.
567
568
569=head1 THE CBOR::XS::Tagged CLASS
570
571CBOR has the concept of tagged values - any CBOR value can be tagged with
572a numeric 64 bit number, which are centrally administered.
573
574C<CBOR::XS> handles a few tags internally when en- or decoding. You can
575also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
576decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
577unknown tag.
578
579These objects are simply blessed array references - the first member of
580the array being the numerical tag, the second being the value.
581
582You can interact with C<CBOR::XS::Tagged> objects in the following ways:
583
584=over 4
585
586=item $tagged = CBOR::XS::tag $tag, $value
587
588This function(!) creates a new C<CBOR::XS::Tagged> object using the given
589C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
590value that can be encoded in CBOR, including serialisable Perl objects and
591C<CBOR::XS::Tagged> objects).
592
593=item $tagged->[0]
594
595=item $tagged->[0] = $new_tag
596
597=item $tag = $tagged->tag
598
599=item $new_tag = $tagged->tag ($new_tag)
600
601Access/mutate the tag.
602
603=item $tagged->[1]
604
605=item $tagged->[1] = $new_value
606
607=item $value = $tagged->value
608
609=item $new_value = $tagged->value ($new_value)
610
611Access/mutate the tagged value.
612
613=back
614
615=cut
616
617sub tag($$) {
618 bless [@_], CBOR::XS::Tagged::;
619}
620
621sub CBOR::XS::Tagged::tag {
622 $_[0][0] = $_[1] if $#_;
623 $_[0][0]
624}
625
626sub CBOR::XS::Tagged::value {
627 $_[0][1] = $_[1] if $#_;
628 $_[0][1]
629}
630
631=head2 EXAMPLES
632
633Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
634
635You can look up CBOR tag value and emanings in the IANA registry at
636L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
637
638Prepend a magic header (C<$CBOR::XS::MAGIC>):
639
640 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
641 # same as:
642 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
643
644Serialise some URIs and a regex in an array:
645
646 my $cbor = encode_cbor [
647 (CBOR::XS::tag 32, "http://www.nethype.de/"),
648 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
649 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
650 ];
651
652Wrap CBOR data in CBOR:
653
654 my $cbor_cbor = encode_cbor
655 CBOR::XS::tag 24,
656 encode_cbor [1, 2, 3];
657
658=head1 TAG HANDLING AND EXTENSIONS
659
660This section describes how this module handles specific tagged values
661and extensions. If a tag is not mentioned here and no additional filters
662are provided for it, then the default handling applies (creating a
663CBOR::XS::Tagged object on decoding, and only encoding the tag when
664explicitly requested).
665
666Tags not handled specifically are currently converted into a
667L<CBOR::XS::Tagged> object, which is simply a blessed array reference
668consisting of the numeric tag value followed by the (decoded) CBOR value.
669
670Future versions of this module reserve the right to special case
671additional tags (such as base64url).
672
673=head2 ENFORCED TAGS
674
675These tags are always handled when decoding, and their handling cannot be
676overriden by the user.
677
678=over 4
679
680=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>)
681
682These tags are automatically created (and decoded) for serialisable
683objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
684serialisation protocol). See L<OBJECT SERIALISATION> for details.
685
686=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
687
688These tags are automatically decoded when encountered, resulting in
689shared values in the decoded object. They are only encoded, however, when
690C<allow_sharable> is enabled.
691
692=item <unassigned>, <unassigned> (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)
693
694These tags are automatically decoded when encountered. They are only
695encoded, however, when C<allow_stringref> is enabled.
696
697=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
698
699This tag is automatically generated when a reference are encountered (with
700the exception of hash and array refernces). It is converted to a reference
701when decoding.
702
703=item 55799 (self-describe CBOR, RFC 7049)
704
705This value is not generated on encoding (unless explicitly requested by
706the user), and is simply ignored when decoding.
707
708=back
709
710=head2 NON-ENFORCED TAGS
711
712These tags have default filters provided when decoding. Their handling can
713be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
714providing a custom C<filter> callback when decoding.
715
716When they result in decoding into a specific Perl class, the module
717usually provides a corresponding C<TO_CBOR> method as well.
718
719When any of these need to load additional modules that are not part of the
720perl core distribution (e.g. L<URI>), it is (currently) up to the user to
721provide these modules. The decoding usually fails with an exception if the
722required module cannot be loaded.
723
724=over 4
725
726=item 2, 3 (positive/negative bignum)
727
728These tags are decoded into L<Math::BigInt> objects. The corresponding
729C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
730integers, and others into positive/negative CBOR bignums.
731
732=item 4, 5 (decimal fraction/bigfloat)
733
734Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
735objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
736encodes into a decimal fraction.
737
738CBOR cannot represent bigfloats with I<very> large exponents - conversion
739of such big float objects is undefined.
740
741Also, NaN and infinities are not encoded properly.
742
743=item 21, 22, 23 (expected later JSON conversion)
744
745CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
746tags.
747
748=item 32 (URI)
749
750These objects decode into L<URI> objects. The corresponding
751C<URI::TO_CBOR> method again results in a CBOR URI value.
752
753=back
754
755=cut
756
757our %FILTER = (
758 # 0 # rfc4287 datetime, utf-8
759 # 1 # unix timestamp, any
760
761 2 => sub { # pos bigint
762 require Math::BigInt;
763 Math::BigInt->new ("0x" . unpack "H*", pop)
764 },
765
766 3 => sub { # neg bigint
767 require Math::BigInt;
768 -Math::BigInt->new ("0x" . unpack "H*", pop)
769 },
770
771 4 => sub { # decimal fraction, array
772 require Math::BigFloat;
773 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
774 },
775
776 5 => sub { # bigfloat, array
777 require Math::BigFloat;
778 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
779 },
780
781 21 => sub { pop }, # expected conversion to base64url encoding
782 22 => sub { pop }, # expected conversion to base64 encoding
783 23 => sub { pop }, # expected conversion to base16 encoding
784
785 # 24 # embedded cbor, byte string
786
787 32 => sub {
788 require URI;
789 URI->new (pop)
790 },
791
792 # 33 # base64url rfc4648, utf-8
793 # 34 # base64 rfc46484, utf-8
794 # 35 # regex pcre/ecma262, utf-8
795 # 36 # mime message rfc2045, utf-8
796);
450 797
451 798
452=head1 CBOR and JSON 799=head1 CBOR and JSON
453 800
454CBOR is supposed to implement a superset of the JSON data model, and is, 801CBOR is supposed to implement a superset of the JSON data model, and is,
537Please refrain from using rt.cpan.org or any other bug reporting 884Please refrain from using rt.cpan.org or any other bug reporting
538service. I put the contact address into my modules for a reason. 885service. I put the contact address into my modules for a reason.
539 886
540=cut 887=cut
541 888
889our %FILTER = (
890 # 0 # rfc4287 datetime, utf-8
891 # 1 # unix timestamp, any
892
893 2 => sub { # pos bigint
894 require Math::BigInt;
895 Math::BigInt->new ("0x" . unpack "H*", pop)
896 },
897
898 3 => sub { # neg bigint
899 require Math::BigInt;
900 -Math::BigInt->new ("0x" . unpack "H*", pop)
901 },
902
903 4 => sub { # decimal fraction, array
904 require Math::BigFloat;
905 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
906 },
907
908 5 => sub { # bigfloat, array
909 require Math::BigFloat;
910 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
911 },
912
913 21 => sub { pop }, # expected conversion to base64url encoding
914 22 => sub { pop }, # expected conversion to base64 encoding
915 23 => sub { pop }, # expected conversion to base16 encoding
916
917 # 24 # embedded cbor, byte string
918
919 32 => sub {
920 require URI;
921 URI->new (pop)
922 },
923
924 # 33 # base64url rfc4648, utf-8
925 # 34 # base64 rfc46484, utf-8
926 # 35 # regex pcre/ecma262, utf-8
927 # 36 # mime message rfc2045, utf-8
928);
929
930sub CBOR::XS::default_filter {
931 &{ $FILTER{$_[0]} or return }
932}
933
934sub URI::TO_CBOR {
935 my $uri = $_[0]->as_string;
936 utf8::upgrade $uri;
937 CBOR::XS::tag 32, $uri
938}
939
940sub Math::BigInt::TO_CBOR {
941 if ($_[0] >= -2147483648 && $_[0] <= 2147483647) {
942 $_[0]->numify
943 } else {
944 my $hex = substr $_[0]->as_hex, 2;
945 $hex = "0$hex" if 1 & length $hex; # sigh
946 CBOR::XS::tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
947 }
948}
949
950sub Math::BigFloat::TO_CBOR {
951 my ($m, $e) = $_[0]->parts;
952 CBOR::XS::tag 4, [$e->numify, $m]
953}
954
542XSLoader::load "CBOR::XS", $VERSION; 955XSLoader::load "CBOR::XS", $VERSION;
543 956
544=head1 SEE ALSO 957=head1 SEE ALSO
545 958
546The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 959The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines