ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.41 by root, Mon Jan 6 04:15:31 2014 UTC vs.
Revision 1.71 by root, Sun Nov 29 21:32:01 2020 UTC

38with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
39often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
40data later and speed is less important you might want to compare both 40data later and speed is less important you might want to compare both
41formats first). 41formats first).
42 42
43The primary goal of this module is to be I<correct> and the secondary goal
44is to be I<fast>. To reach the latter goal it was written in C.
45
43To give you a general idea about speed, with texts in the megabyte range, 46To give you a general idea about speed, with texts in the megabyte range,
44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or 47C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the 48L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
46data, the worse L<Storable> performs in comparison. 49data, the worse L<Storable> performs in comparison.
47 50
52In addition to the core CBOR data format, this module implements a 55In addition to the core CBOR data format, this module implements a
53number of extensions, to support cyclic and shared data structures 56number of extensions, to support cyclic and shared data structures
54(see C<allow_sharing> and C<allow_cycles>), string deduplication (see 57(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55C<pack_strings>) and scalar references (always enabled). 58C<pack_strings>) and scalar references (always enabled).
56 59
57The primary goal of this module is to be I<correct> and the secondary goal
58is to be I<fast>. To reach the latter goal it was written in C.
59
60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
61vice versa. 61vice versa.
62 62
63=cut 63=cut
64 64
65package CBOR::XS; 65package CBOR::XS;
66 66
67use common::sense; 67use common::sense;
68 68
69our $VERSION = 1.25; 69our $VERSION = 1.71;
70our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
71 71
72our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
73 73
74use Exporter; 74use Exporter;
112 112
113The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
114be chained: 114be chained:
115 115
116 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
117
118=item $cbor = new_safe CBOR::XS
119
120Create a new, safe/secure CBOR::XS object. This is similar to C<new>,
121but configures the coder object to be safe to use with untrusted
122data. Currently, this is equivalent to:
123
124 my $cbor = CBOR::XS
125 ->new
126 ->forbid_objects
127 ->filter (\&CBOR::XS::safe_filter)
128 ->max_size (1e8);
129
130But is more future proof (it is better to crash because of a change than
131to be exploited in other ways).
132
133=cut
134
135sub new_safe {
136 CBOR::XS
137 ->new
138 ->forbid_objects
139 ->filter (\&CBOR::XS::safe_filter)
140 ->max_size (1e8)
141}
117 142
118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 143=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
119 144
120=item $max_depth = $cbor->get_max_depth 145=item $max_depth = $cbor->get_max_depth
121 146
137 162
138Note that nesting is implemented by recursion in C. The default value has 163Note that nesting is implemented by recursion in C. The default value has
139been chosen to be as large as typical operating systems allow without 164been chosen to be as large as typical operating systems allow without
140crashing. 165crashing.
141 166
142See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 167See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
143 168
144=item $cbor = $cbor->max_size ([$maximum_string_size]) 169=item $cbor = $cbor->max_size ([$maximum_string_size])
145 170
146=item $max_size = $cbor->get_max_size 171=item $max_size = $cbor->get_max_size
147 172
152effect on C<encode> (yet). 177effect on C<encode> (yet).
153 178
154If no argument is given, the limit check will be deactivated (same as when 179If no argument is given, the limit check will be deactivated (same as when
155C<0> is specified). 180C<0> is specified).
156 181
157See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 182See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
158 183
159=item $cbor = $cbor->allow_unknown ([$enable]) 184=item $cbor = $cbor->allow_unknown ([$enable])
160 185
161=item $enabled = $cbor->get_allow_unknown 186=item $enabled = $cbor->get_allow_unknown
162 187
180reference to the earlier value. 205reference to the earlier value.
181 206
182This means that such values will only be encoded once, and will not result 207This means that such values will only be encoded once, and will not result
183in a deep cloning of the value on decode, in decoders supporting the value 208in a deep cloning of the value on decode, in decoders supporting the value
184sharing extension. This also makes it possible to encode cyclic data 209sharing extension. This also makes it possible to encode cyclic data
185structures (which need C<allow_cycles> to ne enabled to be decoded by this 210structures (which need C<allow_cycles> to be enabled to be decoded by this
186module). 211module).
187 212
188It is recommended to leave it off unless you know your 213It is recommended to leave it off unless you know your
189communication partner supports the value sharing extensions to CBOR 214communication partner supports the value sharing extensions to CBOR
190(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the 215(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
191resulting data structure might be unusable. 216resulting data structure might be unusable.
192 217
193Detecting shared values incurs a runtime overhead when values are encoded 218Detecting shared values incurs a runtime overhead when values are encoded
194that have a reference counter large than one, and might unnecessarily 219that have a reference counter large than one, and might unnecessarily
195increase the encoded size, as potentially shared values are encode as 220increase the encoded size, as potentially shared values are encoded as
196shareable whether or not they are actually shared. 221shareable whether or not they are actually shared.
197 222
198At the moment, only targets of references can be shared (e.g. scalars, 223At the moment, only targets of references can be shared (e.g. scalars,
199arrays or hashes pointed to by a reference). Weirder constructs, such as 224arrays or hashes pointed to by a reference). Weirder constructs, such as
200an array with multiple "copies" of the I<same> string, which are hard but 225an array with multiple "copies" of the I<same> string, which are hard but
224cycles - future versions of this module might chose to decode cyclic data 249cycles - future versions of this module might chose to decode cyclic data
225structures using weak references when this option is off, instead of 250structures using weak references when this option is off, instead of
226throwing an error. 251throwing an error.
227 252
228This option does not affect C<encode> in any way - shared values and 253This option does not affect C<encode> in any way - shared values and
229references will always be decoded properly if present. 254references will always be encoded properly if present.
255
256=item $cbor = $cbor->forbid_objects ([$enable])
257
258=item $enabled = $cbor->get_forbid_objects
259
260Disables the use of the object serialiser protocol.
261
262If C<$enable> is true (or missing), then C<encode> will will throw an
263exception when it encounters perl objects that would be encoded using the
264perl-object tag (26). When C<decode> encounters such tags, it will fall
265back to the general filter/tagged logic as if this were an unknown tag (by
266default resulting in a C<CBOR::XC::Tagged> object).
267
268If C<$enable> is false (the default), then C<encode> will use the
269L<Types::Serialiser> object serialisation protocol to serialise objects
270into perl-object tags, and C<decode> will do the same to decode such tags.
271
272See L<SECURITY CONSIDERATIONS>, below, for more info on why forbidding this
273protocol can be useful.
230 274
231=item $cbor = $cbor->pack_strings ([$enable]) 275=item $cbor = $cbor->pack_strings ([$enable])
232 276
233=item $enabled = $cbor->get_pack_strings 277=item $enabled = $cbor->get_pack_strings
234 278
247the standard CBOR way. 291the standard CBOR way.
248 292
249This option does not affect C<decode> in any way - string references will 293This option does not affect C<decode> in any way - string references will
250always be decoded properly if present. 294always be decoded properly if present.
251 295
296=item $cbor = $cbor->text_keys ([$enable])
297
298=item $enabled = $cbor->get_text_keys
299
300If C<$enabled> is true (or missing), then C<encode> will encode all
301perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
302
303If C<$enable> is false (the default), then C<encode> will encode hash keys
304normally - upgraded perl strings (strings internally encoded as UTF-8) as
305CBOR text strings, and downgraded perl strings as CBOR byte strings.
306
307This option does not affect C<decode> in any way.
308
309This option is useful for interoperability with CBOR decoders that don't
310treat byte strings as a form of text. It is especially useful as Perl
311gives very little control over hash keys.
312
313Enabling this option can be slow, as all downgraded hash keys that are
314encoded need to be scanned and converted to UTF-8.
315
316=item $cbor = $cbor->text_strings ([$enable])
317
318=item $enabled = $cbor->get_text_strings
319
320This option works similar to C<text_keys>, above, but works on all strings
321(including hash keys), so C<text_keys> has no further effect after
322enabling C<text_strings>.
323
324If C<$enabled> is true (or missing), then C<encode> will encode all perl
325strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
326
327If C<$enable> is false (the default), then C<encode> will encode strings
328normally (but see C<text_keys>) - upgraded perl strings (strings
329internally encoded as UTF-8) as CBOR text strings, and downgraded perl
330strings as CBOR byte strings.
331
332This option does not affect C<decode> in any way.
333
334This option has similar advantages and disadvantages as C<text_keys>. In
335addition, this option effectively removes the ability to automatically
336encode byte strings, which might break some C<FREEZE> and C<TO_CBOR>
337methods that rely on this.
338
339A workaround is to use explicit type casts, which are unaffected by this option.
340
252=item $cbor = $cbor->validate_utf8 ([$enable]) 341=item $cbor = $cbor->validate_utf8 ([$enable])
253 342
254=item $enabled = $cbor->get_validate_utf8 343=item $enabled = $cbor->get_validate_utf8
255 344
256If C<$enable> is true (or missing), then C<decode> will validate that 345If C<$enable> is true (or missing), then C<decode> will validate that
261The concept of "valid UTF-8" used is perl's concept, which is a superset 350The concept of "valid UTF-8" used is perl's concept, which is a superset
262of the official UTF-8. 351of the official UTF-8.
263 352
264If C<$enable> is false (the default), then C<decode> will blindly accept 353If C<$enable> is false (the default), then C<decode> will blindly accept
265UTF-8 data, marking them as valid UTF-8 in the resulting data structure 354UTF-8 data, marking them as valid UTF-8 in the resulting data structure
266regardless of whether thats true or not. 355regardless of whether that's true or not.
267 356
268Perl isn't too happy about corrupted UTF-8 in strings, but should 357Perl isn't too happy about corrupted UTF-8 in strings, but should
269generally not crash or do similarly evil things. Extensions might be not 358generally not crash or do similarly evil things. Extensions might be not
270so forgiving, so it's recommended to turn on this setting if you receive 359so forgiving, so it's recommended to turn on this setting if you receive
271untrusted CBOR. 360untrusted CBOR.
293replace the tagged value in the decoded data structure, or no values, 382replace the tagged value in the decoded data structure, or no values,
294which will result in default handling, which currently means the decoder 383which will result in default handling, which currently means the decoder
295creates a C<CBOR::XS::Tagged> object to hold the tag and the value. 384creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
296 385
297When the filter is cleared (the default state), the default filter 386When the filter is cleared (the default state), the default filter
298function, C<CBOR::XS::default_filter>, is used. This function simply looks 387function, C<CBOR::XS::default_filter>, is used. This function simply
299up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be 388looks up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists
300a code reference that is called with tag and value, and is responsible for 389it must be a code reference that is called with tag and value, and is
301decoding the value. If no entry exists, it returns no values. 390responsible for decoding the value. If no entry exists, it returns no
391values. C<CBOR::XS> provides a number of default filter functions already,
392the the C<%CBOR::XS::FILTER> hash can be freely extended with more.
393
394C<CBOR::XS> additionally provides an alternative filter function that is
395supposed to be safe to use with untrusted data (which the default filter
396might not), called C<CBOR::XS::safe_filter>, which works the same as
397the C<default_filter> but uses the C<%CBOR::XS::SAFE_FILTER> variable
398instead. It is prepopulated with the tag decoding functions that are
399deemed safe (basically the same as C<%CBOR::XS::FILTER> without all
400the bignum tags), and can be extended by user code as wlel, although,
401obviously, one should be very careful about adding decoding functions
402here, since the expectation is that they are safe to use on untrusted
403data, after all.
302 404
303Example: decode all tags not handled internally into C<CBOR::XS::Tagged> 405Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
304objects, with no other special handling (useful when working with 406objects, with no other special handling (useful when working with
305potentially "unsafe" CBOR data). 407potentially "unsafe" CBOR data).
306 408
313 my ($tag, $value); 415 my ($tag, $value);
314 416
315 "tag 1347375694 value $value" 417 "tag 1347375694 value $value"
316 }; 418 };
317 419
420Example: provide your own filter function that looks up tags in your own
421hash:
422
423 my %my_filter = (
424 998347484 => sub {
425 my ($tag, $value);
426
427 "tag 998347484 value $value"
428 };
429 );
430
431 my $coder = CBOR::XS->new->filter (sub {
432 &{ $my_filter{$_[0]} or return }
433 });
434
435
436Example: use the safe filter function (see L<SECURITY CONSIDERATIONS> for
437more considerations on security).
438
439 CBOR::XS->new->filter (\&CBOR::XS::safe_filter)->decode ($cbor_data);
440
318=item $cbor_data = $cbor->encode ($perl_scalar) 441=item $cbor_data = $cbor->encode ($perl_scalar)
319 442
320Converts the given Perl data structure (a scalar value) to its CBOR 443Converts the given Perl data structure (a scalar value) to its CBOR
321representation. 444representation.
322 445
331when there is trailing garbage after the CBOR string, it will silently 454when there is trailing garbage after the CBOR string, it will silently
332stop parsing there and return the number of characters consumed so far. 455stop parsing there and return the number of characters consumed so far.
333 456
334This is useful if your CBOR texts are not delimited by an outer protocol 457This is useful if your CBOR texts are not delimited by an outer protocol
335and you need to know where the first CBOR string ends amd the next one 458and you need to know where the first CBOR string ends amd the next one
336starts. 459starts - CBOR strings are self-delimited, so it is possible to concatenate
460CBOR strings without any delimiters or size fields and recover their data.
337 461
338 CBOR::XS->new->decode_prefix ("......") 462 CBOR::XS->new->decode_prefix ("......")
339 => ("...", 3) 463 => ("...", 3)
340 464
341=back 465=back
396 520
397Resets the incremental decoder. This throws away any saved state, so that 521Resets the incremental decoder. This throws away any saved state, so that
398subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse 522subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
399a new CBOR value from the beginning of the C<$buffer> again. 523a new CBOR value from the beginning of the C<$buffer> again.
400 524
401This method can be caled at any time, but it I<must> be called if you want 525This method can be called at any time, but it I<must> be called if you want
402to change your C<$buffer> or there was a decoding error and you want to 526to change your C<$buffer> or there was a decoding error and you want to
403reuse the C<$cbor> object for future incremental parsings. 527reuse the C<$cbor> object for future incremental parsings.
404 528
405=back 529=back
406 530
481 605
482=item hash references 606=item hash references
483 607
484Perl hash references become CBOR maps. As there is no inherent ordering in 608Perl hash references become CBOR maps. As there is no inherent ordering in
485hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 609hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
486order. This order can be different each time a hahs is encoded. 610order. This order can be different each time a hash is encoded.
487 611
488Currently, tied hashes will use the indefinite-length format, while normal 612Currently, tied hashes will use the indefinite-length format, while normal
489hashes will use the fixed-length format. 613hashes will use the fixed-length format.
490 614
491=item array references 615=item array references
544 my $x = 3.1; # some variable containing a number 668 my $x = 3.1; # some variable containing a number
545 "$x"; # stringified 669 "$x"; # stringified
546 $x .= ""; # another, more awkward way to stringify 670 $x .= ""; # another, more awkward way to stringify
547 print $x; # perl does it for you, too, quite often 671 print $x; # perl does it for you, too, quite often
548 672
549You can force whether a string ie encoded as byte or text string by using 673You can force whether a string is encoded as byte or text string by using
550C<utf8::upgrade> and C<utf8::downgrade>): 674C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled).
551 675
552 utf8::upgrade $x; # encode $x as text string 676 utf8::upgrade $x; # encode $x as text string
553 utf8::downgrade $x; # encode $x as byte string 677 utf8::downgrade $x; # encode $x as byte string
554 678
679More options are available, see L<TYPE CASTS>, below, and the C<text_keys>
680and C<text_strings> options.
681
555Perl doesn't define what operations up- and downgrade strings, so if the 682Perl doesn't define what operations up- and downgrade strings, so if the
556difference between byte and text is important, you should up- or downgrade 683difference between byte and text is important, you should up- or downgrade
557your string as late as possible before encoding. 684your string as late as possible before encoding. You can also force the
685use of CBOR text strings by using C<text_keys> or C<text_strings>.
558 686
559You can force the type to be a CBOR number by numifying it: 687You can force the type to be a CBOR number by numifying it:
560 688
561 my $x = "3"; # some variable containing a string 689 my $x = "3"; # some variable containing a string
562 $x += 0; # numify it, ensuring it will be dumped as a number 690 $x += 0; # numify it, ensuring it will be dumped as a number
573represent numerical values are supported, but might suffer loss of 701represent numerical values are supported, but might suffer loss of
574precision. 702precision.
575 703
576=back 704=back
577 705
706=head2 TYPE CASTS
707
708B<EXPERIMENTAL>: As an experimental extension, C<CBOR::XS> allows you to
709force specific cbor types to be used when encoding. That allows you to
710encode types not normally accessible (e.g. half floats) as well as force
711string types even when C<text_strings> is in effect.
712
713Type forcing is done by calling a special "cast" function which keeps a
714copy of the value and returns a new value that can be handed over to any
715CBOR encoder function.
716
717The following casts are currently available (all of which are unary operators):
718
719=over
720
721=item CBOR::XS::as_text $value
722
723Forces the value to be encoded as (UTF-8) text values.
724
725=item CBOR::XS::as_bytes $value
726
727Forces the value to be encoded as a (binary) string value.
728
729=item CBOR::XS::as_float16 $value
730
731Forces half-float (IEEE 754 binary16) encoding of the given value.
732
733=item CBOR::XS::as_float32 $value
734
735Forces single-float (IEEE 754 binary32) encoding of the given value.
736
737=item CBOR::XS::as_float64 $value
738
739Forces double-float (IEEE 754 binary64) encoding of the given value.
740
741=item, CBOR::XS::as_cbor $cbor_text
742
743Bot a type cast per-se, this type cast forces the argument to eb encoded
744as-is. This can be used to embed pre-encoded CBOR data.
745
746Note that no checking on the validity of the C<$cbor_text> is done - it's
747the callers responsibility to correctly encode values.
748
749=back
750
751Example: encode a perl string as binary even though C<text_strings> is in
752effect.
753
754 CBOR::XS->new->text_strings->encode ([4, "text", CBOR::XS::bytes "bytevalue"]);
755
756=cut
757
758sub CBOR::XS::as_cbor ($) { bless [$_[0], 0, undef], CBOR::XS::Tagged:: }
759sub CBOR::XS::as_bytes ($) { bless [$_[0], 1, undef], CBOR::XS::Tagged:: }
760sub CBOR::XS::as_text ($) { bless [$_[0], 2, undef], CBOR::XS::Tagged:: }
761sub CBOR::XS::as_float16 ($) { bless [$_[0], 3, undef], CBOR::XS::Tagged:: }
762sub CBOR::XS::as_float32 ($) { bless [$_[0], 4, undef], CBOR::XS::Tagged:: }
763sub CBOR::XS::as_float64 ($) { bless [$_[0], 5, undef], CBOR::XS::Tagged:: }
764
578=head2 OBJECT SERIALISATION 765=head2 OBJECT SERIALISATION
579 766
580This module implements both a CBOR-specific and the generic 767This module implements both a CBOR-specific and the generic
581L<Types::Serialier> object serialisation protocol. The following 768L<Types::Serialier> object serialisation protocol. The following
582subsections explain both methods. 769subsections explain both methods.
663 "$self" # encode url string 850 "$self" # encode url string
664 } 851 }
665 852
666 sub URI::THAW { 853 sub URI::THAW {
667 my ($class, $serialiser, $uri) = @_; 854 my ($class, $serialiser, $uri) = @_;
668
669 $class->new ($uri) 855 $class->new ($uri)
670 } 856 }
671 857
672Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For 858Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
673example, a C<FREEZE> method that returns "type", "id" and "variant" values 859example, a C<FREEZE> method that returns "type", "id" and "variant" values
804additional tags (such as base64url). 990additional tags (such as base64url).
805 991
806=head2 ENFORCED TAGS 992=head2 ENFORCED TAGS
807 993
808These tags are always handled when decoding, and their handling cannot be 994These tags are always handled when decoding, and their handling cannot be
809overriden by the user. 995overridden by the user.
810 996
811=over 4 997=over 4
812 998
813=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>) 999=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
814 1000
815These tags are automatically created (and decoded) for serialisable 1001These tags are automatically created (and decoded) for serialisable
816objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object 1002objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
817serialisation protocol). See L<OBJECT SERIALISATION> for details. 1003serialisation protocol). See L<OBJECT SERIALISATION> for details.
818 1004
819=item 28, 29 (shareable, sharedref, L <http://cbor.schmorp.de/value-sharing>) 1005=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
820 1006
821These tags are automatically decoded when encountered (and they do not 1007These tags are automatically decoded when encountered (and they do not
822result in a cyclic data structure, see C<allow_cycles>), resulting in 1008result in a cyclic data structure, see C<allow_cycles>), resulting in
823shared values in the decoded object. They are only encoded, however, when 1009shared values in the decoded object. They are only encoded, however, when
824C<allow_sharing> is enabled. 1010C<allow_sharing> is enabled.
834will be shared, others will not. While non-reference shared values can be 1020will be shared, others will not. While non-reference shared values can be
835generated in Perl with some effort, they were considered too unimportant 1021generated in Perl with some effort, they were considered too unimportant
836to be supported in the encoder. The decoder, however, will decode these 1022to be supported in the encoder. The decoder, however, will decode these
837values as shared values. 1023values as shared values.
838 1024
839=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>) 1025=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
840 1026
841These tags are automatically decoded when encountered. They are only 1027These tags are automatically decoded when encountered. They are only
842encoded, however, when C<pack_strings> is enabled. 1028encoded, however, when C<pack_strings> is enabled.
843 1029
844=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>) 1030=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
845 1031
846This tag is automatically generated when a reference are encountered (with 1032This tag is automatically generated when a reference are encountered (with
847the exception of hash and array refernces). It is converted to a reference 1033the exception of hash and array references). It is converted to a reference
848when decoding. 1034when decoding.
849 1035
850=item 55799 (self-describe CBOR, RFC 7049) 1036=item 55799 (self-describe CBOR, RFC 7049)
851 1037
852This value is not generated on encoding (unless explicitly requested by 1038This value is not generated on encoding (unless explicitly requested by
855=back 1041=back
856 1042
857=head2 NON-ENFORCED TAGS 1043=head2 NON-ENFORCED TAGS
858 1044
859These tags have default filters provided when decoding. Their handling can 1045These tags have default filters provided when decoding. Their handling can
860be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by 1046be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
861providing a custom C<filter> callback when decoding. 1047providing a custom C<filter> callback when decoding.
862 1048
863When they result in decoding into a specific Perl class, the module 1049When they result in decoding into a specific Perl class, the module
864usually provides a corresponding C<TO_CBOR> method as well. 1050usually provides a corresponding C<TO_CBOR> method as well.
865 1051
883 1069
884These tags are decoded into L<Math::BigInt> objects. The corresponding 1070These tags are decoded into L<Math::BigInt> objects. The corresponding
885C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR 1071C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
886integers, and others into positive/negative CBOR bignums. 1072integers, and others into positive/negative CBOR bignums.
887 1073
888=item 4, 5 (decimal fraction/bigfloat) 1074=item 4, 5, 264, 265 (decimal fraction/bigfloat)
889 1075
890Both decimal fractions and bigfloats are decoded into L<Math::BigFloat> 1076Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
891objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always> 1077objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
892encodes into a decimal fraction. 1078encodes into a decimal fraction (either tag 4 or 264).
893 1079
894CBOR cannot represent bigfloats with I<very> large exponents - conversion 1080NaN and infinities are not encoded properly, as they cannot be represented
895of such big float objects is undefined. 1081in CBOR.
896 1082
897Also, NaN and infinities are not encoded properly. 1083See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
1084
1085=item 30 (rational numbers)
1086
1087These tags are decoded into L<Math::BigRat> objects. The corresponding
1088C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
1089C<1> via their numerator only, i.e., they become normal integers or
1090C<bignums>.
1091
1092See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
898 1093
899=item 21, 22, 23 (expected later JSON conversion) 1094=item 21, 22, 23 (expected later JSON conversion)
900 1095
901CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these 1096CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
902tags. 1097tags.
907C<URI::TO_CBOR> method again results in a CBOR URI value. 1102C<URI::TO_CBOR> method again results in a CBOR URI value.
908 1103
909=back 1104=back
910 1105
911=cut 1106=cut
912
913our %FILTER = (
914 # 0 # rfc4287 datetime, utf-8
915 # 1 # unix timestamp, any
916
917 2 => sub { # pos bigint
918 require Math::BigInt;
919 Math::BigInt->new ("0x" . unpack "H*", pop)
920 },
921
922 3 => sub { # neg bigint
923 require Math::BigInt;
924 -Math::BigInt->new ("0x" . unpack "H*", pop)
925 },
926
927 4 => sub { # decimal fraction, array
928 require Math::BigFloat;
929 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
930 },
931
932 5 => sub { # bigfloat, array
933 require Math::BigFloat;
934 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
935 },
936
937 21 => sub { pop }, # expected conversion to base64url encoding
938 22 => sub { pop }, # expected conversion to base64 encoding
939 23 => sub { pop }, # expected conversion to base16 encoding
940
941 # 24 # embedded cbor, byte string
942
943 32 => sub {
944 require URI;
945 URI->new (pop)
946 },
947
948 # 33 # base64url rfc4648, utf-8
949 # 34 # base64 rfc46484, utf-8
950 # 35 # regex pcre/ecma262, utf-8
951 # 36 # mime message rfc2045, utf-8
952);
953
954 1107
955=head1 CBOR and JSON 1108=head1 CBOR and JSON
956 1109
957CBOR is supposed to implement a superset of the JSON data model, and is, 1110CBOR is supposed to implement a superset of the JSON data model, and is,
958with some coercion, able to represent all JSON texts (something that other 1111with some coercion, able to represent all JSON texts (something that other
967CBOR intact. 1120CBOR intact.
968 1121
969 1122
970=head1 SECURITY CONSIDERATIONS 1123=head1 SECURITY CONSIDERATIONS
971 1124
972When you are using CBOR in a protocol, talking to untrusted potentially 1125Tl;dr... if you want to decode or encode CBOR from untrusted sources, you
973hostile creatures requires relatively few measures. 1126should start with a coder object created via C<new_safe> (which implements
1127the mitigations explained below):
974 1128
1129 my $coder = CBOR::XS->new_safe;
1130
1131 my $data = $coder->decode ($cbor_text);
1132 my $cbor = $coder->encode ($data);
1133
1134Longer version: When you are using CBOR in a protocol, talking to
1135untrusted potentially hostile creatures requires some thought:
1136
1137=over 4
1138
1139=item Security of the CBOR decoder itself
1140
975First of all, your CBOR decoder should be secure, that is, should not have 1141First and foremost, your CBOR decoder should be secure, that is, should
1142not have any buffer overflows or similar bugs that could potentially be
976any buffer overflows. Obviously, this module should ensure that and I am 1143exploited. Obviously, this module should ensure that and I am trying hard
977trying hard on making that true, but you never know. 1144on making that true, but you never know.
978 1145
1146=item CBOR::XS can invoke almost arbitrary callbacks during decoding
1147
1148CBOR::XS supports object serialisation - decoding CBOR can cause calls
1149to I<any> C<THAW> method in I<any> package that exists in your process
1150(that is, CBOR::XS will not try to load modules, but any existing C<THAW>
1151method or function can be called, so they all have to be secure).
1152
1153Less obviously, it will also invoke C<TO_CBOR> and C<FREEZE> methods -
1154even if all your C<THAW> methods are secure, encoding data structures from
1155untrusted sources can invoke those and trigger bugs in those.
1156
1157So, if you are not sure about the security of all the modules you
1158have loaded (you shouldn't), you should disable this part using
1159C<forbid_objects> or using C<new_safe>.
1160
1161=item CBOR can be extended with tags that call library code
1162
1163CBOR can be extended with tags, and C<CBOR::XS> has a registry of
1164conversion functions for many existing tags that can be extended via
1165third-party modules (see the C<filter> method).
1166
1167If you don't trust these, you should configure the "safe" filter function,
1168C<CBOR::XS::safe_filter> (C<new_safe> does this), which by default only
1169includes conversion functions that are considered "safe" by the author
1170(but again, they can be extended by third party modules).
1171
1172Depending on your level of paranoia, you can use the "safe" filter:
1173
1174 $cbor->filter (\&CBOR::XS::safe_filter);
1175
1176... your own filter...
1177
1178 $cbor->filter (sub { ... do your stuffs here ... });
1179
1180... or even no filter at all, disabling all tag decoding:
1181
1182 $cbor->filter (sub { });
1183
1184This is never a problem for encoding, as the tag mechanism only exists in
1185CBOR texts.
1186
1187=item Resource-starving attacks: object memory usage
1188
979Second, you need to avoid resource-starving attacks. That means you should 1189You need to avoid resource-starving attacks. That means you should limit
980limit the size of CBOR data you accept, or make sure then when your 1190the size of CBOR data you accept, or make sure then when your resources
981resources run out, that's just fine (e.g. by using a separate process that 1191run out, that's just fine (e.g. by using a separate process that can
982can crash safely). The size of a CBOR string in octets is usually a good 1192crash safely). The size of a CBOR string in octets is usually a good
983indication of the size of the resources required to decode it into a Perl 1193indication of the size of the resources required to decode it into a Perl
984structure. While CBOR::XS can check the size of the CBOR text, it might be 1194structure. While CBOR::XS can check the size of the CBOR text (using
985too late when you already have it in memory, so you might want to check 1195C<max_size> - done by C<new_safe>), it might be too late when you already
986the size before you accept the string. 1196have it in memory, so you might want to check the size before you accept
1197the string.
987 1198
1199As for encoding, it is possible to construct data structures that are
1200relatively small but result in large CBOR texts (for example by having an
1201array full of references to the same big data structure, which will all be
1202deep-cloned during encoding by default). This is rarely an actual issue
1203(and the worst case is still just running out of memory), but you can
1204reduce this risk by using C<allow_sharing>.
1205
1206=item Resource-starving attacks: stack overflows
1207
988Third, CBOR::XS recurses using the C stack when decoding objects and 1208CBOR::XS recurses using the C stack when decoding objects and arrays. The
989arrays. The C stack is a limited resource: for instance, on my amd64 1209C stack is a limited resource: for instance, on my amd64 machine with 8MB
990machine with 8MB of stack size I can decode around 180k nested arrays but 1210of stack size I can decode around 180k nested arrays but only 14k nested
991only 14k nested CBOR objects (due to perl itself recursing deeply on croak 1211CBOR objects (due to perl itself recursing deeply on croak to free the
992to free the temporary). If that is exceeded, the program crashes. To be 1212temporary). If that is exceeded, the program crashes. To be conservative,
993conservative, the default nesting limit is set to 512. If your process 1213the default nesting limit is set to 512. If your process has a smaller
994has a smaller stack, you should adjust this setting accordingly with the 1214stack, you should adjust this setting accordingly with the C<max_depth>
995C<max_depth> method. 1215method.
1216
1217=item Resource-starving attacks: CPU en-/decoding complexity
1218
1219CBOR::XS will use the L<Math::BigInt>, L<Math::BigFloat> and
1220L<Math::BigRat> libraries to represent encode/decode bignums. These can be
1221very slow (as in, centuries of CPU time) and can even crash your program
1222(and are generally not very trustworthy). See the next section on bignum
1223security for details.
1224
1225=item Data breaches: leaking information in error messages
1226
1227CBOR::XS might leak contents of your Perl data structures in its error
1228messages, so when you serialise sensitive information you might want to
1229make sure that exceptions thrown by CBOR::XS will not end up in front of
1230untrusted eyes.
1231
1232=item Something else...
996 1233
997Something else could bomb you, too, that I forgot to think of. In that 1234Something else could bomb you, too, that I forgot to think of. In that
998case, you get to keep the pieces. I am always open for hints, though... 1235case, you get to keep the pieces. I am always open for hints, though...
999 1236
1000Also keep in mind that CBOR::XS might leak contents of your Perl data 1237=back
1001structures in its error messages, so when you serialise sensitive 1238
1002information you might want to make sure that exceptions thrown by CBOR::XS 1239
1003will not end up in front of untrusted eyes. 1240=head1 BIGNUM SECURITY CONSIDERATIONS
1241
1242CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1243L<Math::BigFloat> that tries to encode the number in the simplest possible
1244way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
12454) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1246(L<Math::BigRat>, tag 30) can also contain bignums as members.
1247
1248CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1249bigfloats (tags 5 and 265), but it will never generate these on its own.
1250
1251Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1252decimal fractions is generally fast. Decoding bigints can be slow for very
1253big numbers (tens of thousands of digits, something that could potentially
1254be caught by limiting the size of CBOR texts), and decoding bigfloats or
1255arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1256for large exponents (roughly 40 bit and longer).
1257
1258Additionally, L<Math::BigInt> can take advantage of other bignum
1259libraries, such as L<Math::GMP>, which cannot handle big floats with large
1260exponents, and might simply abort or crash your program, due to their code
1261quality.
1262
1263This can be a concern if you want to parse untrusted CBOR. If it is, you
1264might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1265types. You should also disable types 5 and 265, as these can be slow even
1266without bigints.
1267
1268Disabling bigints will also partially or fully disable types that rely on
1269them, e.g. rational numbers that use bignums.
1270
1004 1271
1005=head1 CBOR IMPLEMENTATION NOTES 1272=head1 CBOR IMPLEMENTATION NOTES
1006 1273
1007This section contains some random implementation notes. They do not 1274This section contains some random implementation notes. They do not
1008describe guaranteed behaviour, but merely behaviour as-is implemented 1275describe guaranteed behaviour, but merely behaviour as-is implemented
1022 1289
1023 1290
1024=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT 1291=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
1025 1292
1026On perls that were built without 64 bit integer support (these are rare 1293On perls that were built without 64 bit integer support (these are rare
1027nowadays, even on 32 bit architectures), support for any kind of 64 bit 1294nowadays, even on 32 bit architectures, as all major Perl distributions
1295are built with 64 bit integer support), support for any kind of 64 bit
1028integer in CBOR is very limited - most likely, these 64 bit values will 1296value in CBOR is very limited - most likely, these 64 bit values will
1029be truncated, corrupted, or otherwise not decoded correctly. This also 1297be truncated, corrupted, or otherwise not decoded correctly. This also
1030includes string, array and map sizes that are stored as 64 bit integers. 1298includes string, float, array and map sizes that are stored as 64 bit
1299integers.
1031 1300
1032 1301
1033=head1 THREADS 1302=head1 THREADS
1034 1303
1035This module is I<not> guaranteed to be thread safe and there are no 1304This module is I<not> guaranteed to be thread safe and there are no
1048 1317
1049Please refrain from using rt.cpan.org or any other bug reporting 1318Please refrain from using rt.cpan.org or any other bug reporting
1050service. I put the contact address into my modules for a reason. 1319service. I put the contact address into my modules for a reason.
1051 1320
1052=cut 1321=cut
1322
1323# clumsy and slow hv_store-in-hash helper function
1324sub _hv_store {
1325 $_[0]{$_[1]} = $_[2];
1326}
1053 1327
1054our %FILTER = ( 1328our %FILTER = (
1055 0 => sub { # rfc4287 datetime, utf-8 1329 0 => sub { # rfc4287 datetime, utf-8
1056 require Time::Piece; 1330 require Time::Piece;
1057 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine" 1331 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1058 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything 1332 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1059 # else either. Whats incredibe over standard strptime totally escapes me. 1333 # else either. Whats incredibe over standard strptime totally escapes me.
1060 # doesn't do fractional times, either. sigh. 1334 # doesn't do fractional times, either. sigh.
1061 # In fact, it's all a lie, it uses whatever strptime it wants, and of course, 1335 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1062 # they are all incomptible. The openbsd one simply ignores %z (but according to the 1336 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1063 # docs, it would be much more incredibly flexible indeed. If it worked, that is.). 1337 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1064 scalar eval { 1338 scalar eval {
1065 my $s = $_[1]; 1339 my $s = $_[1];
1066 1340
1067 $s =~ s/Z$/+00:00/; 1341 $s =~ s/Z$/+00:00/;
1093 4 => sub { # decimal fraction, array 1367 4 => sub { # decimal fraction, array
1094 require Math::BigFloat; 1368 require Math::BigFloat;
1095 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0]) 1369 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1096 }, 1370 },
1097 1371
1372 264 => sub { # decimal fraction with arbitrary exponent
1373 require Math::BigFloat;
1374 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1375 },
1376
1098 5 => sub { # bigfloat, array 1377 5 => sub { # bigfloat, array
1099 require Math::BigFloat; 1378 require Math::BigFloat;
1100 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2) 1379 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1380 },
1381
1382 265 => sub { # bigfloat with arbitrary exponent
1383 require Math::BigFloat;
1384 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1385 },
1386
1387 30 => sub { # rational number
1388 require Math::BigRat;
1389 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
1101 }, 1390 },
1102 1391
1103 21 => sub { pop }, # expected conversion to base64url encoding 1392 21 => sub { pop }, # expected conversion to base64url encoding
1104 22 => sub { pop }, # expected conversion to base64 encoding 1393 22 => sub { pop }, # expected conversion to base64 encoding
1105 23 => sub { pop }, # expected conversion to base16 encoding 1394 23 => sub { pop }, # expected conversion to base16 encoding
1115 # 34 # base64 rfc46484, utf-8 1404 # 34 # base64 rfc46484, utf-8
1116 # 35 # regex pcre/ecma262, utf-8 1405 # 35 # regex pcre/ecma262, utf-8
1117 # 36 # mime message rfc2045, utf-8 1406 # 36 # mime message rfc2045, utf-8
1118); 1407);
1119 1408
1120sub CBOR::XS::default_filter { 1409sub default_filter {
1121 &{ $FILTER{$_[0]} or return } 1410 &{ $FILTER{$_[0]} or return }
1411}
1412
1413our %SAFE_FILTER = map { $_ => $FILTER{$_} } 0, 1, 21, 22, 23, 32;
1414
1415sub safe_filter {
1416 &{ $SAFE_FILTER{$_[0]} or return }
1122} 1417}
1123 1418
1124sub URI::TO_CBOR { 1419sub URI::TO_CBOR {
1125 my $uri = $_[0]->as_string; 1420 my $uri = $_[0]->as_string;
1126 utf8::upgrade $uri; 1421 utf8::upgrade $uri;
1127 tag 32, $uri 1422 tag 32, $uri
1128} 1423}
1129 1424
1130sub Math::BigInt::TO_CBOR { 1425sub Math::BigInt::TO_CBOR {
1131 if ($_[0] >= -2147483648 && $_[0] <= 2147483647) { 1426 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
1132 $_[0]->numify 1427 $_[0]->numify
1133 } else { 1428 } else {
1134 my $hex = substr $_[0]->as_hex, 2; 1429 my $hex = substr $_[0]->as_hex, 2;
1135 $hex = "0$hex" if 1 & length $hex; # sigh 1430 $hex = "0$hex" if 1 & length $hex; # sigh
1136 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex 1431 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
1137 } 1432 }
1138} 1433}
1139 1434
1140sub Math::BigFloat::TO_CBOR { 1435sub Math::BigFloat::TO_CBOR {
1141 my ($m, $e) = $_[0]->parts; 1436 my ($m, $e) = $_[0]->parts;
1437
1438 -9223372036854775808 <= $e && $e <= 18446744073709551615
1142 tag 4, [$e->numify, $m] 1439 ? tag 4, [$e->numify, $m]
1440 : tag 264, [$e, $m]
1441}
1442
1443sub Math::BigRat::TO_CBOR {
1444 my ($n, $d) = $_[0]->parts;
1445
1446 # older versions of BigRat need *1, as they not always return numbers
1447
1448 $d*1 == 1
1449 ? $n*1
1450 : tag 30, [$n*1, $d*1]
1143} 1451}
1144 1452
1145sub Time::Piece::TO_CBOR { 1453sub Time::Piece::TO_CBOR {
1146 tag 1, 0 + $_[0]->epoch 1454 tag 1, 0 + $_[0]->epoch
1147} 1455}

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines