ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.24 by root, Fri Nov 22 16:18:59 2013 UTC vs.
Revision 1.71 by root, Sun Nov 29 21:32:01 2020 UTC

26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up
32to you to do). Furthermore, details of the implementation might change
33freely before version 1.0. And lastly, most extensions depend on an IANA
34assignment, and until that assignment is official, this implementation is
35not interoperable with other implementations (even future versions of this
36module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 40data later and speed is less important you might want to compare both
41formats first).
42
43The primary goal of this module is to be I<correct> and the secondary goal
44is to be I<fast>. To reach the latter goal it was written in C.
50 45
51To give you a general idea about speed, with texts in the megabyte range, 46To give you a general idea about speed, with texts in the megabyte range,
52C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or 47C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the 48L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54data, the worse L<Storable> performs in comparison. 49data, the worse L<Storable> performs in comparison.
55 50
56As for compactness, C<CBOR::XS> encoded data structures are usually about 51Regarding compactness, C<CBOR::XS>-encoded data structures are usually
5720% smaller than the same data encoded as (compact) JSON or L<Storable>. 52about 20% smaller than the same data encoded as (compact) JSON or
53L<Storable>.
58 54
59In addition to the core CBOR data format, this module implements a number 55In addition to the core CBOR data format, this module implements a
60of extensions, to support cyclic and self-referencing data structures 56number of extensions, to support cyclic and shared data structures
61(see C<allow_sharing>), string deduplication (see C<allow_stringref>) and 57(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
62scalar references (always enabled). 58C<pack_strings>) and scalar references (always enabled).
63
64The primary goal of this module is to be I<correct> and the secondary goal
65is to be I<fast>. To reach the latter goal it was written in C.
66 59
67See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
68vice versa. 61vice versa.
69 62
70=cut 63=cut
71 64
72package CBOR::XS; 65package CBOR::XS;
73 66
74use common::sense; 67use common::sense;
75 68
76our $VERSION = 0.09; 69our $VERSION = 1.71;
77our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
78 71
79our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
80 73
81use Exporter; 74use Exporter;
119 112
120The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
121be chained: 114be chained:
122 115
123 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
117
118=item $cbor = new_safe CBOR::XS
119
120Create a new, safe/secure CBOR::XS object. This is similar to C<new>,
121but configures the coder object to be safe to use with untrusted
122data. Currently, this is equivalent to:
123
124 my $cbor = CBOR::XS
125 ->new
126 ->forbid_objects
127 ->filter (\&CBOR::XS::safe_filter)
128 ->max_size (1e8);
129
130But is more future proof (it is better to crash because of a change than
131to be exploited in other ways).
132
133=cut
134
135sub new_safe {
136 CBOR::XS
137 ->new
138 ->forbid_objects
139 ->filter (\&CBOR::XS::safe_filter)
140 ->max_size (1e8)
141}
124 142
125=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 143=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
126 144
127=item $max_depth = $cbor->get_max_depth 145=item $max_depth = $cbor->get_max_depth
128 146
144 162
145Note that nesting is implemented by recursion in C. The default value has 163Note that nesting is implemented by recursion in C. The default value has
146been chosen to be as large as typical operating systems allow without 164been chosen to be as large as typical operating systems allow without
147crashing. 165crashing.
148 166
149See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 167See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
150 168
151=item $cbor = $cbor->max_size ([$maximum_string_size]) 169=item $cbor = $cbor->max_size ([$maximum_string_size])
152 170
153=item $max_size = $cbor->get_max_size 171=item $max_size = $cbor->get_max_size
154 172
159effect on C<encode> (yet). 177effect on C<encode> (yet).
160 178
161If no argument is given, the limit check will be deactivated (same as when 179If no argument is given, the limit check will be deactivated (same as when
162C<0> is specified). 180C<0> is specified).
163 181
164See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 182See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
165 183
166=item $cbor = $cbor->allow_unknown ([$enable]) 184=item $cbor = $cbor->allow_unknown ([$enable])
167 185
168=item $enabled = $cbor->get_allow_unknown 186=item $enabled = $cbor->get_allow_unknown
169 187
186as an array, is referenced multiple times), but instead will emit a 204as an array, is referenced multiple times), but instead will emit a
187reference to the earlier value. 205reference to the earlier value.
188 206
189This means that such values will only be encoded once, and will not result 207This means that such values will only be encoded once, and will not result
190in a deep cloning of the value on decode, in decoders supporting the value 208in a deep cloning of the value on decode, in decoders supporting the value
191sharing extension. 209sharing extension. This also makes it possible to encode cyclic data
210structures (which need C<allow_cycles> to be enabled to be decoded by this
211module).
192 212
193It is recommended to leave it off unless you know your 213It is recommended to leave it off unless you know your
194communication partner supports the value sharing extensions to CBOR 214communication partner supports the value sharing extensions to CBOR
195(http://cbor.schmorp.de/value-sharing). 215(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
216resulting data structure might be unusable.
196 217
197Detecting shared values incurs a runtime overhead when values are encoded 218Detecting shared values incurs a runtime overhead when values are encoded
198that have a reference counter large than one, and might unnecessarily 219that have a reference counter large than one, and might unnecessarily
199increase the encoded size, as potentially shared values are encode as 220increase the encoded size, as potentially shared values are encoded as
200sharable whether or not they are actually shared. 221shareable whether or not they are actually shared.
201 222
202At the moment, only targets of references can be shared (e.g. scalars, 223At the moment, only targets of references can be shared (e.g. scalars,
203arrays or hashes pointed to by a reference). Weirder constructs, such as 224arrays or hashes pointed to by a reference). Weirder constructs, such as
204an array with multiple "copies" of the I<same> string, which are hard but 225an array with multiple "copies" of the I<same> string, which are hard but
205not impossible to create in Perl, are not supported (this is the same as 226not impossible to create in Perl, are not supported (this is the same as
206for L<Storable>). 227with L<Storable>).
207 228
208If C<$enable> is false (the default), then C<encode> will encode 229If C<$enable> is false (the default), then C<encode> will encode shared
209exception when it encounters anything it cannot encode as CBOR. 230data structures repeatedly, unsharing them in the process. Cyclic data
231structures cannot be encoded in this mode.
210 232
211This option does not affect C<decode> in any way - shared values and 233This option does not affect C<decode> in any way - shared values and
212references will always be decoded properly if present. 234references will always be decoded properly if present.
213 235
236=item $cbor = $cbor->allow_cycles ([$enable])
237
238=item $enabled = $cbor->get_allow_cycles
239
240If C<$enable> is true (or missing), then C<decode> will happily decode
241self-referential (cyclic) data structures. By default these will not be
242decoded, as they need manual cleanup to avoid memory leaks, so code that
243isn't prepared for this will not leak memory.
244
245If C<$enable> is false (the default), then C<decode> will throw an error
246when it encounters a self-referential/cyclic data structure.
247
248FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
249cycles - future versions of this module might chose to decode cyclic data
250structures using weak references when this option is off, instead of
251throwing an error.
252
253This option does not affect C<encode> in any way - shared values and
254references will always be encoded properly if present.
255
256=item $cbor = $cbor->forbid_objects ([$enable])
257
258=item $enabled = $cbor->get_forbid_objects
259
260Disables the use of the object serialiser protocol.
261
262If C<$enable> is true (or missing), then C<encode> will will throw an
263exception when it encounters perl objects that would be encoded using the
264perl-object tag (26). When C<decode> encounters such tags, it will fall
265back to the general filter/tagged logic as if this were an unknown tag (by
266default resulting in a C<CBOR::XC::Tagged> object).
267
268If C<$enable> is false (the default), then C<encode> will use the
269L<Types::Serialiser> object serialisation protocol to serialise objects
270into perl-object tags, and C<decode> will do the same to decode such tags.
271
272See L<SECURITY CONSIDERATIONS>, below, for more info on why forbidding this
273protocol can be useful.
274
214=item $cbor = $cbor->allow_stringref ([$enable]) 275=item $cbor = $cbor->pack_strings ([$enable])
215 276
216=item $enabled = $cbor->get_allow_stringref 277=item $enabled = $cbor->get_pack_strings
217 278
218If C<$enable> is true (or missing), then C<encode> will try not to encode 279If C<$enable> is true (or missing), then C<encode> will try not to encode
219the same string twice, but will instead encode a reference to the string 280the same string twice, but will instead encode a reference to the string
220instead. Depending on your data format. this can save a lot of space, but 281instead. Depending on your data format, this can save a lot of space, but
221also results in a very large runtime overhead (expect encoding times to be 282also results in a very large runtime overhead (expect encoding times to be
2222-4 times as high as without). 2832-4 times as high as without).
223 284
224It is recommended to leave it off unless you know your 285It is recommended to leave it off unless you know your
225communications partner supports the stringref extension to CBOR 286communications partner supports the stringref extension to CBOR
226(http://cbor.schmorp.de/stringref). 287(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
288resulting data structure might not be usable.
227 289
228If C<$enable> is false (the default), then C<encode> will encode 290If C<$enable> is false (the default), then C<encode> will encode strings
229exception when it encounters anything it cannot encode as CBOR. 291the standard CBOR way.
230 292
231This option does not affect C<decode> in any way - string references will 293This option does not affect C<decode> in any way - string references will
232always be decoded properly if present. 294always be decoded properly if present.
295
296=item $cbor = $cbor->text_keys ([$enable])
297
298=item $enabled = $cbor->get_text_keys
299
300If C<$enabled> is true (or missing), then C<encode> will encode all
301perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
302
303If C<$enable> is false (the default), then C<encode> will encode hash keys
304normally - upgraded perl strings (strings internally encoded as UTF-8) as
305CBOR text strings, and downgraded perl strings as CBOR byte strings.
306
307This option does not affect C<decode> in any way.
308
309This option is useful for interoperability with CBOR decoders that don't
310treat byte strings as a form of text. It is especially useful as Perl
311gives very little control over hash keys.
312
313Enabling this option can be slow, as all downgraded hash keys that are
314encoded need to be scanned and converted to UTF-8.
315
316=item $cbor = $cbor->text_strings ([$enable])
317
318=item $enabled = $cbor->get_text_strings
319
320This option works similar to C<text_keys>, above, but works on all strings
321(including hash keys), so C<text_keys> has no further effect after
322enabling C<text_strings>.
323
324If C<$enabled> is true (or missing), then C<encode> will encode all perl
325strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
326
327If C<$enable> is false (the default), then C<encode> will encode strings
328normally (but see C<text_keys>) - upgraded perl strings (strings
329internally encoded as UTF-8) as CBOR text strings, and downgraded perl
330strings as CBOR byte strings.
331
332This option does not affect C<decode> in any way.
333
334This option has similar advantages and disadvantages as C<text_keys>. In
335addition, this option effectively removes the ability to automatically
336encode byte strings, which might break some C<FREEZE> and C<TO_CBOR>
337methods that rely on this.
338
339A workaround is to use explicit type casts, which are unaffected by this option.
340
341=item $cbor = $cbor->validate_utf8 ([$enable])
342
343=item $enabled = $cbor->get_validate_utf8
344
345If C<$enable> is true (or missing), then C<decode> will validate that
346elements (text strings) containing UTF-8 data in fact contain valid UTF-8
347data (instead of blindly accepting it). This validation obviously takes
348extra time during decoding.
349
350The concept of "valid UTF-8" used is perl's concept, which is a superset
351of the official UTF-8.
352
353If C<$enable> is false (the default), then C<decode> will blindly accept
354UTF-8 data, marking them as valid UTF-8 in the resulting data structure
355regardless of whether that's true or not.
356
357Perl isn't too happy about corrupted UTF-8 in strings, but should
358generally not crash or do similarly evil things. Extensions might be not
359so forgiving, so it's recommended to turn on this setting if you receive
360untrusted CBOR.
361
362This option does not affect C<encode> in any way - strings that are
363supposedly valid UTF-8 will simply be dumped into the resulting CBOR
364string without checking whether that is, in fact, true or not.
233 365
234=item $cbor = $cbor->filter ([$cb->($tag, $value)]) 366=item $cbor = $cbor->filter ([$cb->($tag, $value)])
235 367
236=item $cb_or_undef = $cbor->get_filter 368=item $cb_or_undef = $cbor->get_filter
237 369
250replace the tagged value in the decoded data structure, or no values, 382replace the tagged value in the decoded data structure, or no values,
251which will result in default handling, which currently means the decoder 383which will result in default handling, which currently means the decoder
252creates a C<CBOR::XS::Tagged> object to hold the tag and the value. 384creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
253 385
254When the filter is cleared (the default state), the default filter 386When the filter is cleared (the default state), the default filter
255function, C<CBOR::XS::default_filter>, is used. This function simply looks 387function, C<CBOR::XS::default_filter>, is used. This function simply
256up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be 388looks up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists
257a code reference that is called with tag and value, and is responsible for 389it must be a code reference that is called with tag and value, and is
258decoding the value. If no entry exists, it returns no values. 390responsible for decoding the value. If no entry exists, it returns no
391values. C<CBOR::XS> provides a number of default filter functions already,
392the the C<%CBOR::XS::FILTER> hash can be freely extended with more.
259 393
394C<CBOR::XS> additionally provides an alternative filter function that is
395supposed to be safe to use with untrusted data (which the default filter
396might not), called C<CBOR::XS::safe_filter>, which works the same as
397the C<default_filter> but uses the C<%CBOR::XS::SAFE_FILTER> variable
398instead. It is prepopulated with the tag decoding functions that are
399deemed safe (basically the same as C<%CBOR::XS::FILTER> without all
400the bignum tags), and can be extended by user code as wlel, although,
401obviously, one should be very careful about adding decoding functions
402here, since the expectation is that they are safe to use on untrusted
403data, after all.
404
260Example: decode all tags not handled internally into CBOR::XS::Tagged 405Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
261objects, with no other special handling (useful when working with 406objects, with no other special handling (useful when working with
262potentially "unsafe" CBOR data). 407potentially "unsafe" CBOR data).
263 408
264 CBOR::XS->new->filter (sub { })->decode ($cbor_data); 409 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
265 410
269 $CBOR::XS::FILTER{1347375694} = sub { 414 $CBOR::XS::FILTER{1347375694} = sub {
270 my ($tag, $value); 415 my ($tag, $value);
271 416
272 "tag 1347375694 value $value" 417 "tag 1347375694 value $value"
273 }; 418 };
419
420Example: provide your own filter function that looks up tags in your own
421hash:
422
423 my %my_filter = (
424 998347484 => sub {
425 my ($tag, $value);
426
427 "tag 998347484 value $value"
428 };
429 );
430
431 my $coder = CBOR::XS->new->filter (sub {
432 &{ $my_filter{$_[0]} or return }
433 });
434
435
436Example: use the safe filter function (see L<SECURITY CONSIDERATIONS> for
437more considerations on security).
438
439 CBOR::XS->new->filter (\&CBOR::XS::safe_filter)->decode ($cbor_data);
274 440
275=item $cbor_data = $cbor->encode ($perl_scalar) 441=item $cbor_data = $cbor->encode ($perl_scalar)
276 442
277Converts the given Perl data structure (a scalar value) to its CBOR 443Converts the given Perl data structure (a scalar value) to its CBOR
278representation. 444representation.
288when there is trailing garbage after the CBOR string, it will silently 454when there is trailing garbage after the CBOR string, it will silently
289stop parsing there and return the number of characters consumed so far. 455stop parsing there and return the number of characters consumed so far.
290 456
291This is useful if your CBOR texts are not delimited by an outer protocol 457This is useful if your CBOR texts are not delimited by an outer protocol
292and you need to know where the first CBOR string ends amd the next one 458and you need to know where the first CBOR string ends amd the next one
293starts. 459starts - CBOR strings are self-delimited, so it is possible to concatenate
460CBOR strings without any delimiters or size fields and recover their data.
294 461
295 CBOR::XS->new->decode_prefix ("......") 462 CBOR::XS->new->decode_prefix ("......")
296 => ("...", 3) 463 => ("...", 3)
464
465=back
466
467=head2 INCREMENTAL PARSING
468
469In some cases, there is the need for incremental parsing of JSON
470texts. While this module always has to keep both CBOR text and resulting
471Perl data structure in memory at one time, it does allow you to parse a
472CBOR stream incrementally, using a similar to using "decode_prefix" to see
473if a full CBOR object is available, but is much more efficient.
474
475It basically works by parsing as much of a CBOR string as possible - if
476the CBOR data is not complete yet, the pasrer will remember where it was,
477to be able to restart when more data has been accumulated. Once enough
478data is available to either decode a complete CBOR value or raise an
479error, a real decode will be attempted.
480
481A typical use case would be a network protocol that consists of sending
482and receiving CBOR-encoded messages. The solution that works with CBOR and
483about anything else is by prepending a length to every CBOR value, so the
484receiver knows how many octets to read. More compact (and slightly slower)
485would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
486a CBOR value ends, and doesn't need an explicit length.
487
488The following methods help with this:
489
490=over 4
491
492=item @decoded = $cbor->incr_parse ($buffer)
493
494This method attempts to decode exactly one CBOR value from the beginning
495of the given C<$buffer>. The value is removed from the C<$buffer> on
496success. When C<$buffer> doesn't contain a complete value yet, it returns
497nothing. Finally, when the C<$buffer> doesn't start with something
498that could ever be a valid CBOR value, it raises an exception, just as
499C<decode> would. In the latter case the decoder state is undefined and
500must be reset before being able to parse further.
501
502This method modifies the C<$buffer> in place. When no CBOR value can be
503decoded, the decoder stores the current string offset. On the next call,
504continues decoding at the place where it stopped before. For this to make
505sense, the C<$buffer> must begin with the same octets as on previous
506unsuccessful calls.
507
508You can call this method in scalar context, in which case it either
509returns a decoded value or C<undef>. This makes it impossible to
510distinguish between CBOR null values (which decode to C<undef>) and an
511unsuccessful decode, which is often acceptable.
512
513=item @decoded = $cbor->incr_parse_multiple ($buffer)
514
515Same as C<incr_parse>, but attempts to decode as many CBOR values as
516possible in one go, instead of at most one. Calls to C<incr_parse> and
517C<incr_parse_multiple> can be interleaved.
518
519=item $cbor->incr_reset
520
521Resets the incremental decoder. This throws away any saved state, so that
522subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
523a new CBOR value from the beginning of the C<$buffer> again.
524
525This method can be called at any time, but it I<must> be called if you want
526to change your C<$buffer> or there was a decoding error and you want to
527reuse the C<$cbor> object for future incremental parsings.
297 528
298=back 529=back
299 530
300 531
301=head1 MAPPING 532=head1 MAPPING
319CBOR integers become (numeric) perl scalars. On perls without 64 bit 550CBOR integers become (numeric) perl scalars. On perls without 64 bit
320support, 64 bit integers will be truncated or otherwise corrupted. 551support, 64 bit integers will be truncated or otherwise corrupted.
321 552
322=item byte strings 553=item byte strings
323 554
324Byte strings will become octet strings in Perl (the byte values 0..255 555Byte strings will become octet strings in Perl (the Byte values 0..255
325will simply become characters of the same value in Perl). 556will simply become characters of the same value in Perl).
326 557
327=item UTF-8 strings 558=item UTF-8 strings
328 559
329UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 560UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
352=item tagged values 583=item tagged values
353 584
354Tagged items consists of a numeric tag and another CBOR value. 585Tagged items consists of a numeric tag and another CBOR value.
355 586
356See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >> 587See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
357for details. 588for details on which tags are handled how.
358 589
359=item anything else 590=item anything else
360 591
361Anything else (e.g. unsupported simple values) will raise a decoding 592Anything else (e.g. unsupported simple values) will raise a decoding
362error. 593error.
365 596
366 597
367=head2 PERL -> CBOR 598=head2 PERL -> CBOR
368 599
369The mapping from Perl to CBOR is slightly more difficult, as Perl is a 600The mapping from Perl to CBOR is slightly more difficult, as Perl is a
370truly typeless language, so we can only guess which CBOR type is meant by 601typeless language. That means this module can only guess which CBOR type
371a Perl value. 602is meant by a perl value.
372 603
373=over 4 604=over 4
374 605
375=item hash references 606=item hash references
376 607
377Perl hash references become CBOR maps. As there is no inherent ordering in 608Perl hash references become CBOR maps. As there is no inherent ordering in
378hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 609hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
379order. 610order. This order can be different each time a hash is encoded.
380 611
381Currently, tied hashes will use the indefinite-length format, while normal 612Currently, tied hashes will use the indefinite-length format, while normal
382hashes will use the fixed-length format. 613hashes will use the fixed-length format.
383 614
384=item array references 615=item array references
385 616
386Perl array references become fixed-length CBOR arrays. 617Perl array references become fixed-length CBOR arrays.
387 618
388=item other references 619=item other references
389 620
390Other unblessed references are generally not allowed and will cause an 621Other unblessed references will be represented using
391exception to be thrown, except for references to the integers C<0> and 622the indirection tag extension (tag value C<22098>,
392C<1>, which get turned into false and true in CBOR. 623L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
624to be able to decode these values somehow, by either "doing the right
625thing", decoding into a generic tagged object, simply ignoring the tag, or
626something else.
393 627
394=item CBOR::XS::Tagged objects 628=item CBOR::XS::Tagged objects
395 629
396Objects of this type must be arrays consisting of a single C<[tag, value]> 630Objects of this type must be arrays consisting of a single C<[tag, value]>
397pair. The (numerical) tag will be encoded as a CBOR tag, the value will 631pair. The (numerical) tag will be encoded as a CBOR tag, the value will
398be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to 632be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
399create such objects. 633create such objects.
400 634
401=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 635=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
402 636
403These special values become CBOR true, CBOR false and CBOR undefined 637These special values become CBOR true, CBOR false and CBOR undefined
420 # dump as number 654 # dump as number
421 encode_cbor [2] # yields [2] 655 encode_cbor [2] # yields [2]
422 encode_cbor [-3.0e17] # yields [-3e+17] 656 encode_cbor [-3.0e17] # yields [-3e+17]
423 my $value = 5; encode_cbor [$value] # yields [5] 657 my $value = 5; encode_cbor [$value] # yields [5]
424 658
425 # used as string, so dump as string 659 # used as string, so dump as string (either byte or text)
426 print $value; 660 print $value;
427 encode_cbor [$value] # yields ["5"] 661 encode_cbor [$value] # yields ["5"]
428 662
429 # undef becomes null 663 # undef becomes null
430 encode_cbor [undef] # yields [null] 664 encode_cbor [undef] # yields [null]
433 667
434 my $x = 3.1; # some variable containing a number 668 my $x = 3.1; # some variable containing a number
435 "$x"; # stringified 669 "$x"; # stringified
436 $x .= ""; # another, more awkward way to stringify 670 $x .= ""; # another, more awkward way to stringify
437 print $x; # perl does it for you, too, quite often 671 print $x; # perl does it for you, too, quite often
672
673You can force whether a string is encoded as byte or text string by using
674C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled).
675
676 utf8::upgrade $x; # encode $x as text string
677 utf8::downgrade $x; # encode $x as byte string
678
679More options are available, see L<TYPE CASTS>, below, and the C<text_keys>
680and C<text_strings> options.
681
682Perl doesn't define what operations up- and downgrade strings, so if the
683difference between byte and text is important, you should up- or downgrade
684your string as late as possible before encoding. You can also force the
685use of CBOR text strings by using C<text_keys> or C<text_strings>.
438 686
439You can force the type to be a CBOR number by numifying it: 687You can force the type to be a CBOR number by numifying it:
440 688
441 my $x = "3"; # some variable containing a string 689 my $x = "3"; # some variable containing a string
442 $x += 0; # numify it, ensuring it will be dumped as a number 690 $x += 0; # numify it, ensuring it will be dumped as a number
453represent numerical values are supported, but might suffer loss of 701represent numerical values are supported, but might suffer loss of
454precision. 702precision.
455 703
456=back 704=back
457 705
706=head2 TYPE CASTS
707
708B<EXPERIMENTAL>: As an experimental extension, C<CBOR::XS> allows you to
709force specific cbor types to be used when encoding. That allows you to
710encode types not normally accessible (e.g. half floats) as well as force
711string types even when C<text_strings> is in effect.
712
713Type forcing is done by calling a special "cast" function which keeps a
714copy of the value and returns a new value that can be handed over to any
715CBOR encoder function.
716
717The following casts are currently available (all of which are unary operators):
718
719=over
720
721=item CBOR::XS::as_text $value
722
723Forces the value to be encoded as (UTF-8) text values.
724
725=item CBOR::XS::as_bytes $value
726
727Forces the value to be encoded as a (binary) string value.
728
729=item CBOR::XS::as_float16 $value
730
731Forces half-float (IEEE 754 binary16) encoding of the given value.
732
733=item CBOR::XS::as_float32 $value
734
735Forces single-float (IEEE 754 binary32) encoding of the given value.
736
737=item CBOR::XS::as_float64 $value
738
739Forces double-float (IEEE 754 binary64) encoding of the given value.
740
741=item, CBOR::XS::as_cbor $cbor_text
742
743Bot a type cast per-se, this type cast forces the argument to eb encoded
744as-is. This can be used to embed pre-encoded CBOR data.
745
746Note that no checking on the validity of the C<$cbor_text> is done - it's
747the callers responsibility to correctly encode values.
748
749=back
750
751Example: encode a perl string as binary even though C<text_strings> is in
752effect.
753
754 CBOR::XS->new->text_strings->encode ([4, "text", CBOR::XS::bytes "bytevalue"]);
755
756=cut
757
758sub CBOR::XS::as_cbor ($) { bless [$_[0], 0, undef], CBOR::XS::Tagged:: }
759sub CBOR::XS::as_bytes ($) { bless [$_[0], 1, undef], CBOR::XS::Tagged:: }
760sub CBOR::XS::as_text ($) { bless [$_[0], 2, undef], CBOR::XS::Tagged:: }
761sub CBOR::XS::as_float16 ($) { bless [$_[0], 3, undef], CBOR::XS::Tagged:: }
762sub CBOR::XS::as_float32 ($) { bless [$_[0], 4, undef], CBOR::XS::Tagged:: }
763sub CBOR::XS::as_float64 ($) { bless [$_[0], 5, undef], CBOR::XS::Tagged:: }
764
458=head2 OBJECT SERIALISATION 765=head2 OBJECT SERIALISATION
766
767This module implements both a CBOR-specific and the generic
768L<Types::Serialier> object serialisation protocol. The following
769subsections explain both methods.
770
771=head3 ENCODING
459 772
460This module knows two way to serialise a Perl object: The CBOR-specific 773This module knows two way to serialise a Perl object: The CBOR-specific
461way, and the generic way. 774way, and the generic way.
462 775
463Whenever the encoder encounters a Perl object that it cnanot serialise 776Whenever the encoder encounters a Perl object that it cannot serialise
464directly (most of them), it will first look up the C<TO_CBOR> method on 777directly (most of them), it will first look up the C<TO_CBOR> method on
465it. 778it.
466 779
467If it has a C<TO_CBOR> method, it will call it with the object as only 780If it has a C<TO_CBOR> method, it will call it with the object as only
468argument, and expects exactly one return value, which it will then 781argument, and expects exactly one return value, which it will then
474 787
475The C<FREEZE> method can return any number of values (i.e. zero or 788The C<FREEZE> method can return any number of values (i.e. zero or
476more). These will be encoded as CBOR perl object, together with the 789more). These will be encoded as CBOR perl object, together with the
477classname. 790classname.
478 791
792These methods I<MUST NOT> change the data structure that is being
793serialised. Failure to comply to this can result in memory corruption -
794and worse.
795
479If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail 796If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
480with an error. 797with an error.
481 798
799=head3 DECODING
800
482Objects encoded via C<TO_CBOR> cannot be automatically decoded, but 801Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
483objects encoded via C<FREEZE> can be decoded using the following protocol: 802but objects encoded via C<FREEZE> can be decoded using the following
803protocol:
484 804
485When an encoded CBOR perl object is encountered by the decoder, it will 805When an encoded CBOR perl object is encountered by the decoder, it will
486look up the C<THAW> method, by using the stored classname, and will fail 806look up the C<THAW> method, by using the stored classname, and will fail
487if the method cannot be found. 807if the method cannot be found.
488 808
489After the lookup it will call the C<THAW> method with the stored classname 809After the lookup it will call the C<THAW> method with the stored classname
490as first argument, the constant string C<CBOR> as second argument, and all 810as first argument, the constant string C<CBOR> as second argument, and all
491values returned by C<FREEZE> as remaining arguments. 811values returned by C<FREEZE> as remaining arguments.
492 812
493=head4 EXAMPLES 813=head3 EXAMPLES
494 814
495Here is an example C<TO_CBOR> method: 815Here is an example C<TO_CBOR> method:
496 816
497 sub My::Object::TO_CBOR { 817 sub My::Object::TO_CBOR {
498 my ($obj) = @_; 818 my ($obj) = @_;
509 829
510 sub URI::TO_CBOR { 830 sub URI::TO_CBOR {
511 my ($self) = @_; 831 my ($self) = @_;
512 my $uri = "$self"; # stringify uri 832 my $uri = "$self"; # stringify uri
513 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 833 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
514 CBOR::XS::tagged 32, "$_[0]" 834 CBOR::XS::tag 32, "$_[0]"
515 } 835 }
516 836
517This will encode URIs as a UTF-8 string with tag 32, which indicates an 837This will encode URIs as a UTF-8 string with tag 32, which indicates an
518URI. 838URI.
519 839
530 "$self" # encode url string 850 "$self" # encode url string
531 } 851 }
532 852
533 sub URI::THAW { 853 sub URI::THAW {
534 my ($class, $serialiser, $uri) = @_; 854 my ($class, $serialiser, $uri) = @_;
535
536 $class->new ($uri) 855 $class->new ($uri)
537 } 856 }
538 857
539Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For 858Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
540example, a C<FREEZE> method that returns "type", "id" and "variant" values 859example, a C<FREEZE> method that returns "type", "id" and "variant" values
671additional tags (such as base64url). 990additional tags (such as base64url).
672 991
673=head2 ENFORCED TAGS 992=head2 ENFORCED TAGS
674 993
675These tags are always handled when decoding, and their handling cannot be 994These tags are always handled when decoding, and their handling cannot be
676overriden by the user. 995overridden by the user.
677 996
678=over 4 997=over 4
679 998
680=item <unassigned> (perl-object, L<http://cbor.schmorp.de/perl-object>) 999=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
681 1000
682These tags are automatically created (and decoded) for serialisable 1001These tags are automatically created (and decoded) for serialisable
683objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object 1002objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
684serialisation protocol). See L<OBJECT SERIALISATION> for details. 1003serialisation protocol). See L<OBJECT SERIALISATION> for details.
685 1004
686=item <unassigned>, <unassigned> (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>) 1005=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
687 1006
688These tags are automatically decoded when encountered, resulting in 1007These tags are automatically decoded when encountered (and they do not
1008result in a cyclic data structure, see C<allow_cycles>), resulting in
689shared values in the decoded object. They are only encoded, however, when 1009shared values in the decoded object. They are only encoded, however, when
690C<allow_sharable> is enabled. 1010C<allow_sharing> is enabled.
691 1011
1012Not all shared values can be successfully decoded: values that reference
1013themselves will I<currently> decode as C<undef> (this is not the same
1014as a reference pointing to itself, which will be represented as a value
1015that contains an indirect reference to itself - these will be decoded
1016properly).
1017
1018Note that considerably more shared value data structures can be decoded
1019than will be encoded - currently, only values pointed to by references
1020will be shared, others will not. While non-reference shared values can be
1021generated in Perl with some effort, they were considered too unimportant
1022to be supported in the encoder. The decoder, however, will decode these
1023values as shared values.
1024
692=item <unassigned>, <unassigned> (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>) 1025=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
693 1026
694These tags are automatically decoded when encountered. They are only 1027These tags are automatically decoded when encountered. They are only
695encoded, however, when C<allow_stringref> is enabled. 1028encoded, however, when C<pack_strings> is enabled.
696 1029
697=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>) 1030=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
698 1031
699This tag is automatically generated when a reference are encountered (with 1032This tag is automatically generated when a reference are encountered (with
700the exception of hash and array refernces). It is converted to a reference 1033the exception of hash and array references). It is converted to a reference
701when decoding. 1034when decoding.
702 1035
703=item 55799 (self-describe CBOR, RFC 7049) 1036=item 55799 (self-describe CBOR, RFC 7049)
704 1037
705This value is not generated on encoding (unless explicitly requested by 1038This value is not generated on encoding (unless explicitly requested by
708=back 1041=back
709 1042
710=head2 NON-ENFORCED TAGS 1043=head2 NON-ENFORCED TAGS
711 1044
712These tags have default filters provided when decoding. Their handling can 1045These tags have default filters provided when decoding. Their handling can
713be overriden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by 1046be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
714providing a custom C<filter> callback when decoding. 1047providing a custom C<filter> callback when decoding.
715 1048
716When they result in decoding into a specific Perl class, the module 1049When they result in decoding into a specific Perl class, the module
717usually provides a corresponding C<TO_CBOR> method as well. 1050usually provides a corresponding C<TO_CBOR> method as well.
718 1051
721provide these modules. The decoding usually fails with an exception if the 1054provide these modules. The decoding usually fails with an exception if the
722required module cannot be loaded. 1055required module cannot be loaded.
723 1056
724=over 4 1057=over 4
725 1058
1059=item 0, 1 (date/time string, seconds since the epoch)
1060
1061These tags are decoded into L<Time::Piece> objects. The corresponding
1062C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
1063
1064The L<Time::Piece> API is generally surprisingly bad, and fractional
1065seconds are only accidentally kept intact, so watch out. On the plus side,
1066the module comes with perl since 5.10, which has to count for something.
1067
726=item 2, 3 (positive/negative bignum) 1068=item 2, 3 (positive/negative bignum)
727 1069
728These tags are decoded into L<Math::BigInt> objects. The corresponding 1070These tags are decoded into L<Math::BigInt> objects. The corresponding
729C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR 1071C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
730integers, and others into positive/negative CBOR bignums. 1072integers, and others into positive/negative CBOR bignums.
731 1073
732=item 4, 5 (decimal fraction/bigfloat) 1074=item 4, 5, 264, 265 (decimal fraction/bigfloat)
733 1075
734Both decimal fractions and bigfloats are decoded into L<Math::BigFloat> 1076Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
735objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always> 1077objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
736encodes into a decimal fraction. 1078encodes into a decimal fraction (either tag 4 or 264).
737 1079
738CBOR cannot represent bigfloats with I<very> large exponents - conversion 1080NaN and infinities are not encoded properly, as they cannot be represented
739of such big float objects is undefined. 1081in CBOR.
740 1082
741Also, NaN and infinities are not encoded properly. 1083See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
1084
1085=item 30 (rational numbers)
1086
1087These tags are decoded into L<Math::BigRat> objects. The corresponding
1088C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
1089C<1> via their numerator only, i.e., they become normal integers or
1090C<bignums>.
1091
1092See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
742 1093
743=item 21, 22, 23 (expected later JSON conversion) 1094=item 21, 22, 23 (expected later JSON conversion)
744 1095
745CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these 1096CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
746tags. 1097tags.
751C<URI::TO_CBOR> method again results in a CBOR URI value. 1102C<URI::TO_CBOR> method again results in a CBOR URI value.
752 1103
753=back 1104=back
754 1105
755=cut 1106=cut
756
757our %FILTER = (
758 # 0 # rfc4287 datetime, utf-8
759 # 1 # unix timestamp, any
760
761 2 => sub { # pos bigint
762 require Math::BigInt;
763 Math::BigInt->new ("0x" . unpack "H*", pop)
764 },
765
766 3 => sub { # neg bigint
767 require Math::BigInt;
768 -Math::BigInt->new ("0x" . unpack "H*", pop)
769 },
770
771 4 => sub { # decimal fraction, array
772 require Math::BigFloat;
773 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
774 },
775
776 5 => sub { # bigfloat, array
777 require Math::BigFloat;
778 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2)
779 },
780
781 21 => sub { pop }, # expected conversion to base64url encoding
782 22 => sub { pop }, # expected conversion to base64 encoding
783 23 => sub { pop }, # expected conversion to base16 encoding
784
785 # 24 # embedded cbor, byte string
786
787 32 => sub {
788 require URI;
789 URI->new (pop)
790 },
791
792 # 33 # base64url rfc4648, utf-8
793 # 34 # base64 rfc46484, utf-8
794 # 35 # regex pcre/ecma262, utf-8
795 # 36 # mime message rfc2045, utf-8
796);
797
798 1107
799=head1 CBOR and JSON 1108=head1 CBOR and JSON
800 1109
801CBOR is supposed to implement a superset of the JSON data model, and is, 1110CBOR is supposed to implement a superset of the JSON data model, and is,
802with some coercion, able to represent all JSON texts (something that other 1111with some coercion, able to represent all JSON texts (something that other
811CBOR intact. 1120CBOR intact.
812 1121
813 1122
814=head1 SECURITY CONSIDERATIONS 1123=head1 SECURITY CONSIDERATIONS
815 1124
816When you are using CBOR in a protocol, talking to untrusted potentially 1125Tl;dr... if you want to decode or encode CBOR from untrusted sources, you
817hostile creatures requires relatively few measures. 1126should start with a coder object created via C<new_safe> (which implements
1127the mitigations explained below):
818 1128
1129 my $coder = CBOR::XS->new_safe;
1130
1131 my $data = $coder->decode ($cbor_text);
1132 my $cbor = $coder->encode ($data);
1133
1134Longer version: When you are using CBOR in a protocol, talking to
1135untrusted potentially hostile creatures requires some thought:
1136
1137=over 4
1138
1139=item Security of the CBOR decoder itself
1140
819First of all, your CBOR decoder should be secure, that is, should not have 1141First and foremost, your CBOR decoder should be secure, that is, should
1142not have any buffer overflows or similar bugs that could potentially be
820any buffer overflows. Obviously, this module should ensure that and I am 1143exploited. Obviously, this module should ensure that and I am trying hard
821trying hard on making that true, but you never know. 1144on making that true, but you never know.
822 1145
1146=item CBOR::XS can invoke almost arbitrary callbacks during decoding
1147
1148CBOR::XS supports object serialisation - decoding CBOR can cause calls
1149to I<any> C<THAW> method in I<any> package that exists in your process
1150(that is, CBOR::XS will not try to load modules, but any existing C<THAW>
1151method or function can be called, so they all have to be secure).
1152
1153Less obviously, it will also invoke C<TO_CBOR> and C<FREEZE> methods -
1154even if all your C<THAW> methods are secure, encoding data structures from
1155untrusted sources can invoke those and trigger bugs in those.
1156
1157So, if you are not sure about the security of all the modules you
1158have loaded (you shouldn't), you should disable this part using
1159C<forbid_objects> or using C<new_safe>.
1160
1161=item CBOR can be extended with tags that call library code
1162
1163CBOR can be extended with tags, and C<CBOR::XS> has a registry of
1164conversion functions for many existing tags that can be extended via
1165third-party modules (see the C<filter> method).
1166
1167If you don't trust these, you should configure the "safe" filter function,
1168C<CBOR::XS::safe_filter> (C<new_safe> does this), which by default only
1169includes conversion functions that are considered "safe" by the author
1170(but again, they can be extended by third party modules).
1171
1172Depending on your level of paranoia, you can use the "safe" filter:
1173
1174 $cbor->filter (\&CBOR::XS::safe_filter);
1175
1176... your own filter...
1177
1178 $cbor->filter (sub { ... do your stuffs here ... });
1179
1180... or even no filter at all, disabling all tag decoding:
1181
1182 $cbor->filter (sub { });
1183
1184This is never a problem for encoding, as the tag mechanism only exists in
1185CBOR texts.
1186
1187=item Resource-starving attacks: object memory usage
1188
823Second, you need to avoid resource-starving attacks. That means you should 1189You need to avoid resource-starving attacks. That means you should limit
824limit the size of CBOR data you accept, or make sure then when your 1190the size of CBOR data you accept, or make sure then when your resources
825resources run out, that's just fine (e.g. by using a separate process that 1191run out, that's just fine (e.g. by using a separate process that can
826can crash safely). The size of a CBOR string in octets is usually a good 1192crash safely). The size of a CBOR string in octets is usually a good
827indication of the size of the resources required to decode it into a Perl 1193indication of the size of the resources required to decode it into a Perl
828structure. While CBOR::XS can check the size of the CBOR text, it might be 1194structure. While CBOR::XS can check the size of the CBOR text (using
829too late when you already have it in memory, so you might want to check 1195C<max_size> - done by C<new_safe>), it might be too late when you already
830the size before you accept the string. 1196have it in memory, so you might want to check the size before you accept
1197the string.
831 1198
1199As for encoding, it is possible to construct data structures that are
1200relatively small but result in large CBOR texts (for example by having an
1201array full of references to the same big data structure, which will all be
1202deep-cloned during encoding by default). This is rarely an actual issue
1203(and the worst case is still just running out of memory), but you can
1204reduce this risk by using C<allow_sharing>.
1205
1206=item Resource-starving attacks: stack overflows
1207
832Third, CBOR::XS recurses using the C stack when decoding objects and 1208CBOR::XS recurses using the C stack when decoding objects and arrays. The
833arrays. The C stack is a limited resource: for instance, on my amd64 1209C stack is a limited resource: for instance, on my amd64 machine with 8MB
834machine with 8MB of stack size I can decode around 180k nested arrays but 1210of stack size I can decode around 180k nested arrays but only 14k nested
835only 14k nested CBOR objects (due to perl itself recursing deeply on croak 1211CBOR objects (due to perl itself recursing deeply on croak to free the
836to free the temporary). If that is exceeded, the program crashes. To be 1212temporary). If that is exceeded, the program crashes. To be conservative,
837conservative, the default nesting limit is set to 512. If your process 1213the default nesting limit is set to 512. If your process has a smaller
838has a smaller stack, you should adjust this setting accordingly with the 1214stack, you should adjust this setting accordingly with the C<max_depth>
839C<max_depth> method. 1215method.
1216
1217=item Resource-starving attacks: CPU en-/decoding complexity
1218
1219CBOR::XS will use the L<Math::BigInt>, L<Math::BigFloat> and
1220L<Math::BigRat> libraries to represent encode/decode bignums. These can be
1221very slow (as in, centuries of CPU time) and can even crash your program
1222(and are generally not very trustworthy). See the next section on bignum
1223security for details.
1224
1225=item Data breaches: leaking information in error messages
1226
1227CBOR::XS might leak contents of your Perl data structures in its error
1228messages, so when you serialise sensitive information you might want to
1229make sure that exceptions thrown by CBOR::XS will not end up in front of
1230untrusted eyes.
1231
1232=item Something else...
840 1233
841Something else could bomb you, too, that I forgot to think of. In that 1234Something else could bomb you, too, that I forgot to think of. In that
842case, you get to keep the pieces. I am always open for hints, though... 1235case, you get to keep the pieces. I am always open for hints, though...
843 1236
844Also keep in mind that CBOR::XS might leak contents of your Perl data 1237=back
845structures in its error messages, so when you serialise sensitive 1238
846information you might want to make sure that exceptions thrown by CBOR::XS 1239
847will not end up in front of untrusted eyes. 1240=head1 BIGNUM SECURITY CONSIDERATIONS
1241
1242CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1243L<Math::BigFloat> that tries to encode the number in the simplest possible
1244way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
12454) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1246(L<Math::BigRat>, tag 30) can also contain bignums as members.
1247
1248CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1249bigfloats (tags 5 and 265), but it will never generate these on its own.
1250
1251Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1252decimal fractions is generally fast. Decoding bigints can be slow for very
1253big numbers (tens of thousands of digits, something that could potentially
1254be caught by limiting the size of CBOR texts), and decoding bigfloats or
1255arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1256for large exponents (roughly 40 bit and longer).
1257
1258Additionally, L<Math::BigInt> can take advantage of other bignum
1259libraries, such as L<Math::GMP>, which cannot handle big floats with large
1260exponents, and might simply abort or crash your program, due to their code
1261quality.
1262
1263This can be a concern if you want to parse untrusted CBOR. If it is, you
1264might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1265types. You should also disable types 5 and 265, as these can be slow even
1266without bigints.
1267
1268Disabling bigints will also partially or fully disable types that rely on
1269them, e.g. rational numbers that use bignums.
1270
848 1271
849=head1 CBOR IMPLEMENTATION NOTES 1272=head1 CBOR IMPLEMENTATION NOTES
850 1273
851This section contains some random implementation notes. They do not 1274This section contains some random implementation notes. They do not
852describe guaranteed behaviour, but merely behaviour as-is implemented 1275describe guaranteed behaviour, but merely behaviour as-is implemented
861Only the double data type is supported for NV data types - when Perl uses 1284Only the double data type is supported for NV data types - when Perl uses
862long double to represent floating point values, they might not be encoded 1285long double to represent floating point values, they might not be encoded
863properly. Half precision types are accepted, but not encoded. 1286properly. Half precision types are accepted, but not encoded.
864 1287
865Strict mode and canonical mode are not implemented. 1288Strict mode and canonical mode are not implemented.
1289
1290
1291=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
1292
1293On perls that were built without 64 bit integer support (these are rare
1294nowadays, even on 32 bit architectures, as all major Perl distributions
1295are built with 64 bit integer support), support for any kind of 64 bit
1296value in CBOR is very limited - most likely, these 64 bit values will
1297be truncated, corrupted, or otherwise not decoded correctly. This also
1298includes string, float, array and map sizes that are stored as 64 bit
1299integers.
866 1300
867 1301
868=head1 THREADS 1302=head1 THREADS
869 1303
870This module is I<not> guaranteed to be thread safe and there are no 1304This module is I<not> guaranteed to be thread safe and there are no
884Please refrain from using rt.cpan.org or any other bug reporting 1318Please refrain from using rt.cpan.org or any other bug reporting
885service. I put the contact address into my modules for a reason. 1319service. I put the contact address into my modules for a reason.
886 1320
887=cut 1321=cut
888 1322
1323# clumsy and slow hv_store-in-hash helper function
1324sub _hv_store {
1325 $_[0]{$_[1]} = $_[2];
1326}
1327
889our %FILTER = ( 1328our %FILTER = (
890 # 0 # rfc4287 datetime, utf-8 1329 0 => sub { # rfc4287 datetime, utf-8
891 # 1 # unix timestamp, any 1330 require Time::Piece;
1331 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1332 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1333 # else either. Whats incredibe over standard strptime totally escapes me.
1334 # doesn't do fractional times, either. sigh.
1335 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1336 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1337 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1338 scalar eval {
1339 my $s = $_[1];
1340
1341 $s =~ s/Z$/+00:00/;
1342 $s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
1343 or die;
1344
1345 my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
1346 my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
1347
1348 Time::Piece::gmtime ($d->epoch + $b)
1349 } || die "corrupted CBOR date/time string ($_[0])";
1350 },
1351
1352 1 => sub { # seconds since the epoch, possibly fractional
1353 require Time::Piece;
1354 scalar Time::Piece::gmtime (pop)
1355 },
892 1356
893 2 => sub { # pos bigint 1357 2 => sub { # pos bigint
894 require Math::BigInt; 1358 require Math::BigInt;
895 Math::BigInt->new ("0x" . unpack "H*", pop) 1359 Math::BigInt->new ("0x" . unpack "H*", pop)
896 }, 1360 },
903 4 => sub { # decimal fraction, array 1367 4 => sub { # decimal fraction, array
904 require Math::BigFloat; 1368 require Math::BigFloat;
905 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0]) 1369 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
906 }, 1370 },
907 1371
1372 264 => sub { # decimal fraction with arbitrary exponent
1373 require Math::BigFloat;
1374 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1375 },
1376
908 5 => sub { # bigfloat, array 1377 5 => sub { # bigfloat, array
909 require Math::BigFloat; 1378 require Math::BigFloat;
910 scalar Math::BigFloat->new ($_[1][1])->blsft ($_[1][0], 2) 1379 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1380 },
1381
1382 265 => sub { # bigfloat with arbitrary exponent
1383 require Math::BigFloat;
1384 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1385 },
1386
1387 30 => sub { # rational number
1388 require Math::BigRat;
1389 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
911 }, 1390 },
912 1391
913 21 => sub { pop }, # expected conversion to base64url encoding 1392 21 => sub { pop }, # expected conversion to base64url encoding
914 22 => sub { pop }, # expected conversion to base64 encoding 1393 22 => sub { pop }, # expected conversion to base64 encoding
915 23 => sub { pop }, # expected conversion to base16 encoding 1394 23 => sub { pop }, # expected conversion to base16 encoding
925 # 34 # base64 rfc46484, utf-8 1404 # 34 # base64 rfc46484, utf-8
926 # 35 # regex pcre/ecma262, utf-8 1405 # 35 # regex pcre/ecma262, utf-8
927 # 36 # mime message rfc2045, utf-8 1406 # 36 # mime message rfc2045, utf-8
928); 1407);
929 1408
930sub CBOR::XS::default_filter { 1409sub default_filter {
931 &{ $FILTER{$_[0]} or return } 1410 &{ $FILTER{$_[0]} or return }
1411}
1412
1413our %SAFE_FILTER = map { $_ => $FILTER{$_} } 0, 1, 21, 22, 23, 32;
1414
1415sub safe_filter {
1416 &{ $SAFE_FILTER{$_[0]} or return }
932} 1417}
933 1418
934sub URI::TO_CBOR { 1419sub URI::TO_CBOR {
935 my $uri = $_[0]->as_string; 1420 my $uri = $_[0]->as_string;
936 utf8::upgrade $uri; 1421 utf8::upgrade $uri;
937 CBOR::XS::tag 32, $uri 1422 tag 32, $uri
938} 1423}
939 1424
940sub Math::BigInt::TO_CBOR { 1425sub Math::BigInt::TO_CBOR {
941 if ($_[0] >= -2147483648 && $_[0] <= 2147483647) { 1426 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
942 $_[0]->numify 1427 $_[0]->numify
943 } else { 1428 } else {
944 my $hex = substr $_[0]->as_hex, 2; 1429 my $hex = substr $_[0]->as_hex, 2;
945 $hex = "0$hex" if 1 & length $hex; # sigh 1430 $hex = "0$hex" if 1 & length $hex; # sigh
946 CBOR::XS::tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex 1431 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
947 } 1432 }
948} 1433}
949 1434
950sub Math::BigFloat::TO_CBOR { 1435sub Math::BigFloat::TO_CBOR {
951 my ($m, $e) = $_[0]->parts; 1436 my ($m, $e) = $_[0]->parts;
1437
1438 -9223372036854775808 <= $e && $e <= 18446744073709551615
952 CBOR::XS::tag 4, [$e->numify, $m] 1439 ? tag 4, [$e->numify, $m]
1440 : tag 264, [$e, $m]
1441}
1442
1443sub Math::BigRat::TO_CBOR {
1444 my ($n, $d) = $_[0]->parts;
1445
1446 # older versions of BigRat need *1, as they not always return numbers
1447
1448 $d*1 == 1
1449 ? $n*1
1450 : tag 30, [$n*1, $d*1]
1451}
1452
1453sub Time::Piece::TO_CBOR {
1454 tag 1, 0 + $_[0]->epoch
953} 1455}
954 1456
955XSLoader::load "CBOR::XS", $VERSION; 1457XSLoader::load "CBOR::XS", $VERSION;
956 1458
957=head1 SEE ALSO 1459=head1 SEE ALSO

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines