ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.30 by root, Sat Nov 30 16:19:59 2013 UTC vs.
Revision 1.70 by root, Sat Nov 9 07:30:36 2019 UTC

38with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
39often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
40data later and speed is less important you might want to compare both 40data later and speed is less important you might want to compare both
41formats first). 41formats first).
42 42
43The primary goal of this module is to be I<correct> and the secondary goal
44is to be I<fast>. To reach the latter goal it was written in C.
45
43To give you a general idea about speed, with texts in the megabyte range, 46To give you a general idea about speed, with texts in the megabyte range,
44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or 47C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the 48L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
46data, the worse L<Storable> performs in comparison. 49data, the worse L<Storable> performs in comparison.
47 50
48Regarding compactness, C<CBOR::XS>-encoded data structures are usually 51Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49about 20% smaller than the same data encoded as (compact) JSON or 52about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>. 53L<Storable>.
51 54
52In addition to the core CBOR data format, this module implements a 55In addition to the core CBOR data format, this module implements a
53number of extensions, to support cyclic and shared data structures (see 56number of extensions, to support cyclic and shared data structures
54C<allow_sharing>), string deduplication (see C<pack_strings>) and scalar 57(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55references (always enabled). 58C<pack_strings>) and scalar references (always enabled).
56
57The primary goal of this module is to be I<correct> and the secondary goal
58is to be I<fast>. To reach the latter goal it was written in C.
59 59
60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
61vice versa. 61vice versa.
62 62
63=cut 63=cut
64 64
65package CBOR::XS; 65package CBOR::XS;
66 66
67use common::sense; 67use common::sense;
68 68
69our $VERSION = '1.0'; 69our $VERSION = 1.71;
70our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
71 71
72our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
73 73
74use Exporter; 74use Exporter;
112 112
113The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
114be chained: 114be chained:
115 115
116 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
117
118=item $cbor = new_safe CBOR::XS
119
120Create a new, safe/secure CBOR::XS object. This is similar to C<new>,
121but configures the coder object to be safe to use with untrusted
122data. Currently, this is equivalent to:
123
124 my $cbor = CBOR::XS
125 ->new
126 ->forbid_objects
127 ->filter (\&CBOR::XS::safe_filter)
128 ->max_size (1e8);
129
130But is more future proof (it is better to crash because of a change than
131to be exploited in other ways).
132
133=cut
134
135sub new_safe {
136 CBOR::XS
137 ->new
138 ->forbid_objects
139 ->filter (\&CBOR::XS::safe_filter)
140 ->max_size (1e8)
141}
117 142
118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 143=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
119 144
120=item $max_depth = $cbor->get_max_depth 145=item $max_depth = $cbor->get_max_depth
121 146
137 162
138Note that nesting is implemented by recursion in C. The default value has 163Note that nesting is implemented by recursion in C. The default value has
139been chosen to be as large as typical operating systems allow without 164been chosen to be as large as typical operating systems allow without
140crashing. 165crashing.
141 166
142See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 167See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
143 168
144=item $cbor = $cbor->max_size ([$maximum_string_size]) 169=item $cbor = $cbor->max_size ([$maximum_string_size])
145 170
146=item $max_size = $cbor->get_max_size 171=item $max_size = $cbor->get_max_size
147 172
152effect on C<encode> (yet). 177effect on C<encode> (yet).
153 178
154If no argument is given, the limit check will be deactivated (same as when 179If no argument is given, the limit check will be deactivated (same as when
155C<0> is specified). 180C<0> is specified).
156 181
157See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 182See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
158 183
159=item $cbor = $cbor->allow_unknown ([$enable]) 184=item $cbor = $cbor->allow_unknown ([$enable])
160 185
161=item $enabled = $cbor->get_allow_unknown 186=item $enabled = $cbor->get_allow_unknown
162 187
180reference to the earlier value. 205reference to the earlier value.
181 206
182This means that such values will only be encoded once, and will not result 207This means that such values will only be encoded once, and will not result
183in a deep cloning of the value on decode, in decoders supporting the value 208in a deep cloning of the value on decode, in decoders supporting the value
184sharing extension. This also makes it possible to encode cyclic data 209sharing extension. This also makes it possible to encode cyclic data
185structures. 210structures (which need C<allow_cycles> to be enabled to be decoded by this
211module).
186 212
187It is recommended to leave it off unless you know your 213It is recommended to leave it off unless you know your
188communication partner supports the value sharing extensions to CBOR 214communication partner supports the value sharing extensions to CBOR
189(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the 215(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
190resulting data structure might be unusable. 216resulting data structure might be unusable.
191 217
192Detecting shared values incurs a runtime overhead when values are encoded 218Detecting shared values incurs a runtime overhead when values are encoded
193that have a reference counter large than one, and might unnecessarily 219that have a reference counter large than one, and might unnecessarily
194increase the encoded size, as potentially shared values are encode as 220increase the encoded size, as potentially shared values are encoded as
195sharable whether or not they are actually shared. 221shareable whether or not they are actually shared.
196 222
197At the moment, only targets of references can be shared (e.g. scalars, 223At the moment, only targets of references can be shared (e.g. scalars,
198arrays or hashes pointed to by a reference). Weirder constructs, such as 224arrays or hashes pointed to by a reference). Weirder constructs, such as
199an array with multiple "copies" of the I<same> string, which are hard but 225an array with multiple "copies" of the I<same> string, which are hard but
200not impossible to create in Perl, are not supported (this is the same as 226not impossible to create in Perl, are not supported (this is the same as
205structures cannot be encoded in this mode. 231structures cannot be encoded in this mode.
206 232
207This option does not affect C<decode> in any way - shared values and 233This option does not affect C<decode> in any way - shared values and
208references will always be decoded properly if present. 234references will always be decoded properly if present.
209 235
236=item $cbor = $cbor->allow_cycles ([$enable])
237
238=item $enabled = $cbor->get_allow_cycles
239
240If C<$enable> is true (or missing), then C<decode> will happily decode
241self-referential (cyclic) data structures. By default these will not be
242decoded, as they need manual cleanup to avoid memory leaks, so code that
243isn't prepared for this will not leak memory.
244
245If C<$enable> is false (the default), then C<decode> will throw an error
246when it encounters a self-referential/cyclic data structure.
247
248FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
249cycles - future versions of this module might chose to decode cyclic data
250structures using weak references when this option is off, instead of
251throwing an error.
252
253This option does not affect C<encode> in any way - shared values and
254references will always be encoded properly if present.
255
256=item $cbor = $cbor->forbid_objects ([$enable])
257
258=item $enabled = $cbor->get_forbid_objects
259
260Disables the use of the object serialiser protocol.
261
262If C<$enable> is true (or missing), then C<encode> will will throw an
263exception when it encounters perl objects that would be encoded using the
264perl-object tag (26). When C<decode> encounters such tags, it will fall
265back to the general filter/tagged logic as if this were an unknown tag (by
266default resulting in a C<CBOR::XC::Tagged> object).
267
268If C<$enable> is false (the default), then C<encode> will use the
269L<Types::Serialiser> object serialisation protocol to serialise objects
270into perl-object tags, and C<decode> will do the same to decode such tags.
271
272See L<SECURITY CONSIDERATIONS>, below, for more info on why forbidding this
273protocol can be useful.
274
210=item $cbor = $cbor->pack_strings ([$enable]) 275=item $cbor = $cbor->pack_strings ([$enable])
211 276
212=item $enabled = $cbor->get_pack_strings 277=item $enabled = $cbor->get_pack_strings
213 278
214If C<$enable> is true (or missing), then C<encode> will try not to encode 279If C<$enable> is true (or missing), then C<encode> will try not to encode
226the standard CBOR way. 291the standard CBOR way.
227 292
228This option does not affect C<decode> in any way - string references will 293This option does not affect C<decode> in any way - string references will
229always be decoded properly if present. 294always be decoded properly if present.
230 295
296=item $cbor = $cbor->text_keys ([$enable])
297
298=item $enabled = $cbor->get_text_keys
299
300If C<$enabled> is true (or missing), then C<encode> will encode all
301perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
302
303If C<$enable> is false (the default), then C<encode> will encode hash keys
304normally - upgraded perl strings (strings internally encoded as UTF-8) as
305CBOR text strings, and downgraded perl strings as CBOR byte strings.
306
307This option does not affect C<decode> in any way.
308
309This option is useful for interoperability with CBOR decoders that don't
310treat byte strings as a form of text. It is especially useful as Perl
311gives very little control over hash keys.
312
313Enabling this option can be slow, as all downgraded hash keys that are
314encoded need to be scanned and converted to UTF-8.
315
316=item $cbor = $cbor->text_strings ([$enable])
317
318=item $enabled = $cbor->get_text_strings
319
320This option works similar to C<text_keys>, above, but works on all strings
321(including hash keys), so C<text_keys> has no further effect after
322enabling C<text_strings>.
323
324If C<$enabled> is true (or missing), then C<encode> will encode all perl
325strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
326
327If C<$enable> is false (the default), then C<encode> will encode strings
328normally (but see C<text_keys>) - upgraded perl strings (strings
329internally encoded as UTF-8) as CBOR text strings, and downgraded perl
330strings as CBOR byte strings.
331
332This option does not affect C<decode> in any way.
333
334This option has similar advantages and disadvantages as C<text_keys>. In
335addition, this option effectively removes the ability to encode byte
336strings, which might break some C<FREEZE> and C<TO_CBOR> methods that rely
337on this, such as bignum encoding, so this option is mainly useful for very
338simple data.
339
340=item $cbor = $cbor->validate_utf8 ([$enable])
341
342=item $enabled = $cbor->get_validate_utf8
343
344If C<$enable> is true (or missing), then C<decode> will validate that
345elements (text strings) containing UTF-8 data in fact contain valid UTF-8
346data (instead of blindly accepting it). This validation obviously takes
347extra time during decoding.
348
349The concept of "valid UTF-8" used is perl's concept, which is a superset
350of the official UTF-8.
351
352If C<$enable> is false (the default), then C<decode> will blindly accept
353UTF-8 data, marking them as valid UTF-8 in the resulting data structure
354regardless of whether that's true or not.
355
356Perl isn't too happy about corrupted UTF-8 in strings, but should
357generally not crash or do similarly evil things. Extensions might be not
358so forgiving, so it's recommended to turn on this setting if you receive
359untrusted CBOR.
360
361This option does not affect C<encode> in any way - strings that are
362supposedly valid UTF-8 will simply be dumped into the resulting CBOR
363string without checking whether that is, in fact, true or not.
364
231=item $cbor = $cbor->filter ([$cb->($tag, $value)]) 365=item $cbor = $cbor->filter ([$cb->($tag, $value)])
232 366
233=item $cb_or_undef = $cbor->get_filter 367=item $cb_or_undef = $cbor->get_filter
234 368
235Sets or replaces the tagged value decoding filter (when C<$cb> is 369Sets or replaces the tagged value decoding filter (when C<$cb> is
247replace the tagged value in the decoded data structure, or no values, 381replace the tagged value in the decoded data structure, or no values,
248which will result in default handling, which currently means the decoder 382which will result in default handling, which currently means the decoder
249creates a C<CBOR::XS::Tagged> object to hold the tag and the value. 383creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
250 384
251When the filter is cleared (the default state), the default filter 385When the filter is cleared (the default state), the default filter
252function, C<CBOR::XS::default_filter>, is used. This function simply looks 386function, C<CBOR::XS::default_filter>, is used. This function simply
253up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be 387looks up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists
254a code reference that is called with tag and value, and is responsible for 388it must be a code reference that is called with tag and value, and is
255decoding the value. If no entry exists, it returns no values. 389responsible for decoding the value. If no entry exists, it returns no
390values. C<CBOR::XS> provides a number of default filter functions already,
391the the C<%CBOR::XS::FILTER> hash can be freely extended with more.
392
393C<CBOR::XS> additionally provides an alternative filter function that is
394supposed to be safe to use with untrusted data (which the default filter
395might not), called C<CBOR::XS::safe_filter>, which works the same as
396the C<default_filter> but uses the C<%CBOR::XS::SAFE_FILTER> variable
397instead. It is prepopulated with the tag decoding functions that are
398deemed safe (basically the same as C<%CBOR::XS::FILTER> without all
399the bignum tags), and can be extended by user code as wlel, although,
400obviously, one should be very careful about adding decoding functions
401here, since the expectation is that they are safe to use on untrusted
402data, after all.
256 403
257Example: decode all tags not handled internally into C<CBOR::XS::Tagged> 404Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
258objects, with no other special handling (useful when working with 405objects, with no other special handling (useful when working with
259potentially "unsafe" CBOR data). 406potentially "unsafe" CBOR data).
260 407
267 my ($tag, $value); 414 my ($tag, $value);
268 415
269 "tag 1347375694 value $value" 416 "tag 1347375694 value $value"
270 }; 417 };
271 418
419Example: provide your own filter function that looks up tags in your own
420hash:
421
422 my %my_filter = (
423 998347484 => sub {
424 my ($tag, $value);
425
426 "tag 998347484 value $value"
427 };
428 );
429
430 my $coder = CBOR::XS->new->filter (sub {
431 &{ $my_filter{$_[0]} or return }
432 });
433
434
435Example: use the safe filter function (see L<SECURITY CONSIDERATIONS> for
436more considerations on security).
437
438 CBOR::XS->new->filter (\&CBOR::XS::safe_filter)->decode ($cbor_data);
439
272=item $cbor_data = $cbor->encode ($perl_scalar) 440=item $cbor_data = $cbor->encode ($perl_scalar)
273 441
274Converts the given Perl data structure (a scalar value) to its CBOR 442Converts the given Perl data structure (a scalar value) to its CBOR
275representation. 443representation.
276 444
285when there is trailing garbage after the CBOR string, it will silently 453when there is trailing garbage after the CBOR string, it will silently
286stop parsing there and return the number of characters consumed so far. 454stop parsing there and return the number of characters consumed so far.
287 455
288This is useful if your CBOR texts are not delimited by an outer protocol 456This is useful if your CBOR texts are not delimited by an outer protocol
289and you need to know where the first CBOR string ends amd the next one 457and you need to know where the first CBOR string ends amd the next one
290starts. 458starts - CBOR strings are self-delimited, so it is possible to concatenate
459CBOR strings without any delimiters or size fields and recover their data.
291 460
292 CBOR::XS->new->decode_prefix ("......") 461 CBOR::XS->new->decode_prefix ("......")
293 => ("...", 3) 462 => ("...", 3)
463
464=back
465
466=head2 INCREMENTAL PARSING
467
468In some cases, there is the need for incremental parsing of JSON
469texts. While this module always has to keep both CBOR text and resulting
470Perl data structure in memory at one time, it does allow you to parse a
471CBOR stream incrementally, using a similar to using "decode_prefix" to see
472if a full CBOR object is available, but is much more efficient.
473
474It basically works by parsing as much of a CBOR string as possible - if
475the CBOR data is not complete yet, the pasrer will remember where it was,
476to be able to restart when more data has been accumulated. Once enough
477data is available to either decode a complete CBOR value or raise an
478error, a real decode will be attempted.
479
480A typical use case would be a network protocol that consists of sending
481and receiving CBOR-encoded messages. The solution that works with CBOR and
482about anything else is by prepending a length to every CBOR value, so the
483receiver knows how many octets to read. More compact (and slightly slower)
484would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
485a CBOR value ends, and doesn't need an explicit length.
486
487The following methods help with this:
488
489=over 4
490
491=item @decoded = $cbor->incr_parse ($buffer)
492
493This method attempts to decode exactly one CBOR value from the beginning
494of the given C<$buffer>. The value is removed from the C<$buffer> on
495success. When C<$buffer> doesn't contain a complete value yet, it returns
496nothing. Finally, when the C<$buffer> doesn't start with something
497that could ever be a valid CBOR value, it raises an exception, just as
498C<decode> would. In the latter case the decoder state is undefined and
499must be reset before being able to parse further.
500
501This method modifies the C<$buffer> in place. When no CBOR value can be
502decoded, the decoder stores the current string offset. On the next call,
503continues decoding at the place where it stopped before. For this to make
504sense, the C<$buffer> must begin with the same octets as on previous
505unsuccessful calls.
506
507You can call this method in scalar context, in which case it either
508returns a decoded value or C<undef>. This makes it impossible to
509distinguish between CBOR null values (which decode to C<undef>) and an
510unsuccessful decode, which is often acceptable.
511
512=item @decoded = $cbor->incr_parse_multiple ($buffer)
513
514Same as C<incr_parse>, but attempts to decode as many CBOR values as
515possible in one go, instead of at most one. Calls to C<incr_parse> and
516C<incr_parse_multiple> can be interleaved.
517
518=item $cbor->incr_reset
519
520Resets the incremental decoder. This throws away any saved state, so that
521subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
522a new CBOR value from the beginning of the C<$buffer> again.
523
524This method can be called at any time, but it I<must> be called if you want
525to change your C<$buffer> or there was a decoding error and you want to
526reuse the C<$cbor> object for future incremental parsings.
294 527
295=back 528=back
296 529
297 530
298=head1 MAPPING 531=head1 MAPPING
371 604
372=item hash references 605=item hash references
373 606
374Perl hash references become CBOR maps. As there is no inherent ordering in 607Perl hash references become CBOR maps. As there is no inherent ordering in
375hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 608hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
376order. This order can be different each time a hahs is encoded. 609order. This order can be different each time a hash is encoded.
377 610
378Currently, tied hashes will use the indefinite-length format, while normal 611Currently, tied hashes will use the indefinite-length format, while normal
379hashes will use the fixed-length format. 612hashes will use the fixed-length format.
380 613
381=item array references 614=item array references
434 my $x = 3.1; # some variable containing a number 667 my $x = 3.1; # some variable containing a number
435 "$x"; # stringified 668 "$x"; # stringified
436 $x .= ""; # another, more awkward way to stringify 669 $x .= ""; # another, more awkward way to stringify
437 print $x; # perl does it for you, too, quite often 670 print $x; # perl does it for you, too, quite often
438 671
439You can force whether a string ie encoded as byte or text string by using 672You can force whether a string is encoded as byte or text string by using
440C<utf8::upgrade> and C<utf8::downgrade>): 673C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled):
441 674
442 utf8::upgrade $x; # encode $x as text string 675 utf8::upgrade $x; # encode $x as text string
443 utf8::downgrade $x; # encode $x as byte string 676 utf8::downgrade $x; # encode $x as byte string
444 677
445Perl doesn't define what operations up- and downgrade strings, so if the 678Perl doesn't define what operations up- and downgrade strings, so if the
446difference between byte and text is important, you should up- or downgrade 679difference between byte and text is important, you should up- or downgrade
447your string as late as possible before encoding. 680your string as late as possible before encoding. You can also force the
681use of CBOR text strings by using C<text_keys> or C<text_strings>.
448 682
449You can force the type to be a CBOR number by numifying it: 683You can force the type to be a CBOR number by numifying it:
450 684
451 my $x = "3"; # some variable containing a string 685 my $x = "3"; # some variable containing a string
452 $x += 0; # numify it, ensuring it will be dumped as a number 686 $x += 0; # numify it, ensuring it will be dumped as a number
553 "$self" # encode url string 787 "$self" # encode url string
554 } 788 }
555 789
556 sub URI::THAW { 790 sub URI::THAW {
557 my ($class, $serialiser, $uri) = @_; 791 my ($class, $serialiser, $uri) = @_;
558
559 $class->new ($uri) 792 $class->new ($uri)
560 } 793 }
561 794
562Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For 795Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
563example, a C<FREEZE> method that returns "type", "id" and "variant" values 796example, a C<FREEZE> method that returns "type", "id" and "variant" values
694additional tags (such as base64url). 927additional tags (such as base64url).
695 928
696=head2 ENFORCED TAGS 929=head2 ENFORCED TAGS
697 930
698These tags are always handled when decoding, and their handling cannot be 931These tags are always handled when decoding, and their handling cannot be
699overriden by the user. 932overridden by the user.
700 933
701=over 4 934=over 4
702 935
703=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>) 936=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
704 937
705These tags are automatically created (and decoded) for serialisable 938These tags are automatically created (and decoded) for serialisable
706objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object 939objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
707serialisation protocol). See L<OBJECT SERIALISATION> for details. 940serialisation protocol). See L<OBJECT SERIALISATION> for details.
708 941
709=item 28, 29 (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>) 942=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
710 943
711These tags are automatically decoded when encountered, resulting in 944These tags are automatically decoded when encountered (and they do not
945result in a cyclic data structure, see C<allow_cycles>), resulting in
712shared values in the decoded object. They are only encoded, however, when 946shared values in the decoded object. They are only encoded, however, when
713C<allow_sharable> is enabled. 947C<allow_sharing> is enabled.
714 948
949Not all shared values can be successfully decoded: values that reference
950themselves will I<currently> decode as C<undef> (this is not the same
951as a reference pointing to itself, which will be represented as a value
952that contains an indirect reference to itself - these will be decoded
953properly).
954
955Note that considerably more shared value data structures can be decoded
956than will be encoded - currently, only values pointed to by references
957will be shared, others will not. While non-reference shared values can be
958generated in Perl with some effort, they were considered too unimportant
959to be supported in the encoder. The decoder, however, will decode these
960values as shared values.
961
715=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>) 962=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
716 963
717These tags are automatically decoded when encountered. They are only 964These tags are automatically decoded when encountered. They are only
718encoded, however, when C<pack_strings> is enabled. 965encoded, however, when C<pack_strings> is enabled.
719 966
720=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>) 967=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
721 968
722This tag is automatically generated when a reference are encountered (with 969This tag is automatically generated when a reference are encountered (with
723the exception of hash and array refernces). It is converted to a reference 970the exception of hash and array references). It is converted to a reference
724when decoding. 971when decoding.
725 972
726=item 55799 (self-describe CBOR, RFC 7049) 973=item 55799 (self-describe CBOR, RFC 7049)
727 974
728This value is not generated on encoding (unless explicitly requested by 975This value is not generated on encoding (unless explicitly requested by
731=back 978=back
732 979
733=head2 NON-ENFORCED TAGS 980=head2 NON-ENFORCED TAGS
734 981
735These tags have default filters provided when decoding. Their handling can 982These tags have default filters provided when decoding. Their handling can
736be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by 983be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
737providing a custom C<filter> callback when decoding. 984providing a custom C<filter> callback when decoding.
738 985
739When they result in decoding into a specific Perl class, the module 986When they result in decoding into a specific Perl class, the module
740usually provides a corresponding C<TO_CBOR> method as well. 987usually provides a corresponding C<TO_CBOR> method as well.
741 988
744provide these modules. The decoding usually fails with an exception if the 991provide these modules. The decoding usually fails with an exception if the
745required module cannot be loaded. 992required module cannot be loaded.
746 993
747=over 4 994=over 4
748 995
996=item 0, 1 (date/time string, seconds since the epoch)
997
998These tags are decoded into L<Time::Piece> objects. The corresponding
999C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
1000
1001The L<Time::Piece> API is generally surprisingly bad, and fractional
1002seconds are only accidentally kept intact, so watch out. On the plus side,
1003the module comes with perl since 5.10, which has to count for something.
1004
749=item 2, 3 (positive/negative bignum) 1005=item 2, 3 (positive/negative bignum)
750 1006
751These tags are decoded into L<Math::BigInt> objects. The corresponding 1007These tags are decoded into L<Math::BigInt> objects. The corresponding
752C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR 1008C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
753integers, and others into positive/negative CBOR bignums. 1009integers, and others into positive/negative CBOR bignums.
754 1010
755=item 4, 5 (decimal fraction/bigfloat) 1011=item 4, 5, 264, 265 (decimal fraction/bigfloat)
756 1012
757Both decimal fractions and bigfloats are decoded into L<Math::BigFloat> 1013Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
758objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always> 1014objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
759encodes into a decimal fraction. 1015encodes into a decimal fraction (either tag 4 or 264).
760 1016
761CBOR cannot represent bigfloats with I<very> large exponents - conversion 1017NaN and infinities are not encoded properly, as they cannot be represented
762of such big float objects is undefined. 1018in CBOR.
763 1019
764Also, NaN and infinities are not encoded properly. 1020See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
1021
1022=item 30 (rational numbers)
1023
1024These tags are decoded into L<Math::BigRat> objects. The corresponding
1025C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
1026C<1> via their numerator only, i.e., they become normal integers or
1027C<bignums>.
1028
1029See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
765 1030
766=item 21, 22, 23 (expected later JSON conversion) 1031=item 21, 22, 23 (expected later JSON conversion)
767 1032
768CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these 1033CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
769tags. 1034tags.
774C<URI::TO_CBOR> method again results in a CBOR URI value. 1039C<URI::TO_CBOR> method again results in a CBOR URI value.
775 1040
776=back 1041=back
777 1042
778=cut 1043=cut
779
780our %FILTER = (
781 # 0 # rfc4287 datetime, utf-8
782 # 1 # unix timestamp, any
783
784 2 => sub { # pos bigint
785 require Math::BigInt;
786 Math::BigInt->new ("0x" . unpack "H*", pop)
787 },
788
789 3 => sub { # neg bigint
790 require Math::BigInt;
791 -Math::BigInt->new ("0x" . unpack "H*", pop)
792 },
793
794 4 => sub { # decimal fraction, array
795 require Math::BigFloat;
796 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
797 },
798
799 5 => sub { # bigfloat, array
800 require Math::BigFloat;
801 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
802 },
803
804 21 => sub { pop }, # expected conversion to base64url encoding
805 22 => sub { pop }, # expected conversion to base64 encoding
806 23 => sub { pop }, # expected conversion to base16 encoding
807
808 # 24 # embedded cbor, byte string
809
810 32 => sub {
811 require URI;
812 URI->new (pop)
813 },
814
815 # 33 # base64url rfc4648, utf-8
816 # 34 # base64 rfc46484, utf-8
817 # 35 # regex pcre/ecma262, utf-8
818 # 36 # mime message rfc2045, utf-8
819);
820
821 1044
822=head1 CBOR and JSON 1045=head1 CBOR and JSON
823 1046
824CBOR is supposed to implement a superset of the JSON data model, and is, 1047CBOR is supposed to implement a superset of the JSON data model, and is,
825with some coercion, able to represent all JSON texts (something that other 1048with some coercion, able to represent all JSON texts (something that other
834CBOR intact. 1057CBOR intact.
835 1058
836 1059
837=head1 SECURITY CONSIDERATIONS 1060=head1 SECURITY CONSIDERATIONS
838 1061
839When you are using CBOR in a protocol, talking to untrusted potentially 1062Tl;dr... if you want to decode or encode CBOR from untrusted sources, you
840hostile creatures requires relatively few measures. 1063should start with a coder object created via C<new_safe> (which implements
1064the mitigations explained below):
841 1065
1066 my $coder = CBOR::XS->new_safe;
1067
1068 my $data = $coder->decode ($cbor_text);
1069 my $cbor = $coder->encode ($data);
1070
1071Longer version: When you are using CBOR in a protocol, talking to
1072untrusted potentially hostile creatures requires some thought:
1073
1074=over 4
1075
1076=item Security of the CBOR decoder itself
1077
842First of all, your CBOR decoder should be secure, that is, should not have 1078First and foremost, your CBOR decoder should be secure, that is, should
1079not have any buffer overflows or similar bugs that could potentially be
843any buffer overflows. Obviously, this module should ensure that and I am 1080exploited. Obviously, this module should ensure that and I am trying hard
844trying hard on making that true, but you never know. 1081on making that true, but you never know.
845 1082
1083=item CBOR::XS can invoke almost arbitrary callbacks during decoding
1084
1085CBOR::XS supports object serialisation - decoding CBOR can cause calls
1086to I<any> C<THAW> method in I<any> package that exists in your process
1087(that is, CBOR::XS will not try to load modules, but any existing C<THAW>
1088method or function can be called, so they all have to be secure).
1089
1090Less obviously, it will also invoke C<TO_CBOR> and C<FREEZE> methods -
1091even if all your C<THAW> methods are secure, encoding data structures from
1092untrusted sources can invoke those and trigger bugs in those.
1093
1094So, if you are not sure about the security of all the modules you
1095have loaded (you shouldn't), you should disable this part using
1096C<forbid_objects> or using C<new_safe>.
1097
1098=item CBOR can be extended with tags that call library code
1099
1100CBOR can be extended with tags, and C<CBOR::XS> has a registry of
1101conversion functions for many existing tags that can be extended via
1102third-party modules (see the C<filter> method).
1103
1104If you don't trust these, you should configure the "safe" filter function,
1105C<CBOR::XS::safe_filter> (C<new_safe> does this), which by default only
1106includes conversion functions that are considered "safe" by the author
1107(but again, they can be extended by third party modules).
1108
1109Depending on your level of paranoia, you can use the "safe" filter:
1110
1111 $cbor->filter (\&CBOR::XS::safe_filter);
1112
1113... your own filter...
1114
1115 $cbor->filter (sub { ... do your stuffs here ... });
1116
1117... or even no filter at all, disabling all tag decoding:
1118
1119 $cbor->filter (sub { });
1120
1121This is never a problem for encoding, as the tag mechanism only exists in
1122CBOR texts.
1123
1124=item Resource-starving attacks: object memory usage
1125
846Second, you need to avoid resource-starving attacks. That means you should 1126You need to avoid resource-starving attacks. That means you should limit
847limit the size of CBOR data you accept, or make sure then when your 1127the size of CBOR data you accept, or make sure then when your resources
848resources run out, that's just fine (e.g. by using a separate process that 1128run out, that's just fine (e.g. by using a separate process that can
849can crash safely). The size of a CBOR string in octets is usually a good 1129crash safely). The size of a CBOR string in octets is usually a good
850indication of the size of the resources required to decode it into a Perl 1130indication of the size of the resources required to decode it into a Perl
851structure. While CBOR::XS can check the size of the CBOR text, it might be 1131structure. While CBOR::XS can check the size of the CBOR text (using
852too late when you already have it in memory, so you might want to check 1132C<max_size> - done by C<new_safe>), it might be too late when you already
853the size before you accept the string. 1133have it in memory, so you might want to check the size before you accept
1134the string.
854 1135
1136As for encoding, it is possible to construct data structures that are
1137relatively small but result in large CBOR texts (for example by having an
1138array full of references to the same big data structure, which will all be
1139deep-cloned during encoding by default). This is rarely an actual issue
1140(and the worst case is still just running out of memory), but you can
1141reduce this risk by using C<allow_sharing>.
1142
1143=item Resource-starving attacks: stack overflows
1144
855Third, CBOR::XS recurses using the C stack when decoding objects and 1145CBOR::XS recurses using the C stack when decoding objects and arrays. The
856arrays. The C stack is a limited resource: for instance, on my amd64 1146C stack is a limited resource: for instance, on my amd64 machine with 8MB
857machine with 8MB of stack size I can decode around 180k nested arrays but 1147of stack size I can decode around 180k nested arrays but only 14k nested
858only 14k nested CBOR objects (due to perl itself recursing deeply on croak 1148CBOR objects (due to perl itself recursing deeply on croak to free the
859to free the temporary). If that is exceeded, the program crashes. To be 1149temporary). If that is exceeded, the program crashes. To be conservative,
860conservative, the default nesting limit is set to 512. If your process 1150the default nesting limit is set to 512. If your process has a smaller
861has a smaller stack, you should adjust this setting accordingly with the 1151stack, you should adjust this setting accordingly with the C<max_depth>
862C<max_depth> method. 1152method.
1153
1154=item Resource-starving attacks: CPU en-/decoding complexity
1155
1156CBOR::XS will use the L<Math::BigInt>, L<Math::BigFloat> and
1157L<Math::BigRat> libraries to represent encode/decode bignums. These can be
1158very slow (as in, centuries of CPU time) and can even crash your program
1159(and are generally not very trustworthy). See the next section on bignum
1160security for details.
1161
1162=item Data breaches: leaking information in error messages
1163
1164CBOR::XS might leak contents of your Perl data structures in its error
1165messages, so when you serialise sensitive information you might want to
1166make sure that exceptions thrown by CBOR::XS will not end up in front of
1167untrusted eyes.
1168
1169=item Something else...
863 1170
864Something else could bomb you, too, that I forgot to think of. In that 1171Something else could bomb you, too, that I forgot to think of. In that
865case, you get to keep the pieces. I am always open for hints, though... 1172case, you get to keep the pieces. I am always open for hints, though...
866 1173
867Also keep in mind that CBOR::XS might leak contents of your Perl data 1174=back
868structures in its error messages, so when you serialise sensitive 1175
869information you might want to make sure that exceptions thrown by CBOR::XS 1176
870will not end up in front of untrusted eyes. 1177=head1 BIGNUM SECURITY CONSIDERATIONS
1178
1179CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1180L<Math::BigFloat> that tries to encode the number in the simplest possible
1181way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
11824) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1183(L<Math::BigRat>, tag 30) can also contain bignums as members.
1184
1185CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1186bigfloats (tags 5 and 265), but it will never generate these on its own.
1187
1188Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1189decimal fractions is generally fast. Decoding bigints can be slow for very
1190big numbers (tens of thousands of digits, something that could potentially
1191be caught by limiting the size of CBOR texts), and decoding bigfloats or
1192arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1193for large exponents (roughly 40 bit and longer).
1194
1195Additionally, L<Math::BigInt> can take advantage of other bignum
1196libraries, such as L<Math::GMP>, which cannot handle big floats with large
1197exponents, and might simply abort or crash your program, due to their code
1198quality.
1199
1200This can be a concern if you want to parse untrusted CBOR. If it is, you
1201might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1202types. You should also disable types 5 and 265, as these can be slow even
1203without bigints.
1204
1205Disabling bigints will also partially or fully disable types that rely on
1206them, e.g. rational numbers that use bignums.
1207
871 1208
872=head1 CBOR IMPLEMENTATION NOTES 1209=head1 CBOR IMPLEMENTATION NOTES
873 1210
874This section contains some random implementation notes. They do not 1211This section contains some random implementation notes. They do not
875describe guaranteed behaviour, but merely behaviour as-is implemented 1212describe guaranteed behaviour, but merely behaviour as-is implemented
889 1226
890 1227
891=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT 1228=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
892 1229
893On perls that were built without 64 bit integer support (these are rare 1230On perls that were built without 64 bit integer support (these are rare
894nowadays, even on 32 bit architectures), support for any kind of 64 bit 1231nowadays, even on 32 bit architectures, as all major Perl distributions
1232are built with 64 bit integer support), support for any kind of 64 bit
895integer in CBOR is very limited - most likely, these 64 bit values will 1233integer in CBOR is very limited - most likely, these 64 bit values will
896be truncated, corrupted, or otherwise not decoded correctly. This also 1234be truncated, corrupted, or otherwise not decoded correctly. This also
897includes string, array and map sizes that are stored as 64 bit integers. 1235includes string, array and map sizes that are stored as 64 bit integers.
898 1236
899 1237
916Please refrain from using rt.cpan.org or any other bug reporting 1254Please refrain from using rt.cpan.org or any other bug reporting
917service. I put the contact address into my modules for a reason. 1255service. I put the contact address into my modules for a reason.
918 1256
919=cut 1257=cut
920 1258
1259# clumsy and slow hv_store-in-hash helper function
1260sub _hv_store {
1261 $_[0]{$_[1]} = $_[2];
1262}
1263
921our %FILTER = ( 1264our %FILTER = (
922 # 0 # rfc4287 datetime, utf-8 1265 0 => sub { # rfc4287 datetime, utf-8
923 # 1 # unix timestamp, any 1266 require Time::Piece;
1267 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1268 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1269 # else either. Whats incredibe over standard strptime totally escapes me.
1270 # doesn't do fractional times, either. sigh.
1271 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1272 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1273 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1274 scalar eval {
1275 my $s = $_[1];
1276
1277 $s =~ s/Z$/+00:00/;
1278 $s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
1279 or die;
1280
1281 my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
1282 my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
1283
1284 Time::Piece::gmtime ($d->epoch + $b)
1285 } || die "corrupted CBOR date/time string ($_[0])";
1286 },
1287
1288 1 => sub { # seconds since the epoch, possibly fractional
1289 require Time::Piece;
1290 scalar Time::Piece::gmtime (pop)
1291 },
924 1292
925 2 => sub { # pos bigint 1293 2 => sub { # pos bigint
926 require Math::BigInt; 1294 require Math::BigInt;
927 Math::BigInt->new ("0x" . unpack "H*", pop) 1295 Math::BigInt->new ("0x" . unpack "H*", pop)
928 }, 1296 },
935 4 => sub { # decimal fraction, array 1303 4 => sub { # decimal fraction, array
936 require Math::BigFloat; 1304 require Math::BigFloat;
937 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0]) 1305 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
938 }, 1306 },
939 1307
1308 264 => sub { # decimal fraction with arbitrary exponent
1309 require Math::BigFloat;
1310 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1311 },
1312
940 5 => sub { # bigfloat, array 1313 5 => sub { # bigfloat, array
941 require Math::BigFloat; 1314 require Math::BigFloat;
942 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2) 1315 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1316 },
1317
1318 265 => sub { # bigfloat with arbitrary exponent
1319 require Math::BigFloat;
1320 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1321 },
1322
1323 30 => sub { # rational number
1324 require Math::BigRat;
1325 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
943 }, 1326 },
944 1327
945 21 => sub { pop }, # expected conversion to base64url encoding 1328 21 => sub { pop }, # expected conversion to base64url encoding
946 22 => sub { pop }, # expected conversion to base64 encoding 1329 22 => sub { pop }, # expected conversion to base64 encoding
947 23 => sub { pop }, # expected conversion to base16 encoding 1330 23 => sub { pop }, # expected conversion to base16 encoding
957 # 34 # base64 rfc46484, utf-8 1340 # 34 # base64 rfc46484, utf-8
958 # 35 # regex pcre/ecma262, utf-8 1341 # 35 # regex pcre/ecma262, utf-8
959 # 36 # mime message rfc2045, utf-8 1342 # 36 # mime message rfc2045, utf-8
960); 1343);
961 1344
962sub CBOR::XS::default_filter { 1345sub default_filter {
963 &{ $FILTER{$_[0]} or return } 1346 &{ $FILTER{$_[0]} or return }
1347}
1348
1349our %SAFE_FILTER = map { $_ => $FILTER{$_} } 0, 1, 21, 22, 23, 32;
1350
1351sub safe_filter {
1352 &{ $SAFE_FILTER{$_[0]} or return }
964} 1353}
965 1354
966sub URI::TO_CBOR { 1355sub URI::TO_CBOR {
967 my $uri = $_[0]->as_string; 1356 my $uri = $_[0]->as_string;
968 utf8::upgrade $uri; 1357 utf8::upgrade $uri;
969 CBOR::XS::tag 32, $uri 1358 tag 32, $uri
970} 1359}
971 1360
972sub Math::BigInt::TO_CBOR { 1361sub Math::BigInt::TO_CBOR {
973 if ($_[0] >= -2147483648 && $_[0] <= 2147483647) { 1362 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
974 $_[0]->numify 1363 $_[0]->numify
975 } else { 1364 } else {
976 my $hex = substr $_[0]->as_hex, 2; 1365 my $hex = substr $_[0]->as_hex, 2;
977 $hex = "0$hex" if 1 & length $hex; # sigh 1366 $hex = "0$hex" if 1 & length $hex; # sigh
978 CBOR::XS::tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex 1367 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
979 } 1368 }
980} 1369}
981 1370
982sub Math::BigFloat::TO_CBOR { 1371sub Math::BigFloat::TO_CBOR {
983 my ($m, $e) = $_[0]->parts; 1372 my ($m, $e) = $_[0]->parts;
1373
1374 -9223372036854775808 <= $e && $e <= 18446744073709551615
984 CBOR::XS::tag 4, [$e->numify, $m] 1375 ? tag 4, [$e->numify, $m]
1376 : tag 264, [$e, $m]
1377}
1378
1379sub Math::BigRat::TO_CBOR {
1380 my ($n, $d) = $_[0]->parts;
1381
1382 # older versions of BigRat need *1, as they not always return numbers
1383
1384 $d*1 == 1
1385 ? $n*1
1386 : tag 30, [$n*1, $d*1]
1387}
1388
1389sub Time::Piece::TO_CBOR {
1390 tag 1, 0 + $_[0]->epoch
985} 1391}
986 1392
987XSLoader::load "CBOR::XS", $VERSION; 1393XSLoader::load "CBOR::XS", $VERSION;
988 1394
989=head1 SEE ALSO 1395=head1 SEE ALSO

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines