ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.14 by root, Tue Oct 29 20:59:16 2013 UTC vs.
Revision 1.64 by root, Fri Nov 25 23:37:27 2016 UTC

26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up to
32you to do). Furthermore, details of the implementation might change freely
33before version 1.0. And lastly, the object serialisation protocol depends
34on a pending IANA assignment, and until that assignment is official, this
35implementation is not interoperable with other implementations (even
36future versions of this module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 40data later and speed is less important you might want to compare both
41formats first).
50 42
51To give you a general idea, with texts in the megabyte range, C<CBOR::XS> 43To give you a general idea about speed, with texts in the megabyte range,
52usually encodes roughly twice as fast as L<Storable> or L<JSON::XS> and 44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
53decodes about 15%-30% faster than those. The shorter the data, the worse 45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
54L<Storable> performs in comparison. 46data, the worse L<Storable> performs in comparison.
47
48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
51
52In addition to the core CBOR data format, this module implements a
53number of extensions, to support cyclic and shared data structures
54(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55C<pack_strings>) and scalar references (always enabled).
55 56
56The primary goal of this module is to be I<correct> and the secondary goal 57The primary goal of this module is to be I<correct> and the secondary goal
57is to be I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
58 59
59See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
63 64
64package CBOR::XS; 65package CBOR::XS;
65 66
66use common::sense; 67use common::sense;
67 68
68our $VERSION = 0.06; 69our $VERSION = 1.51;
69our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
70 71
71our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
72 73
73use Exporter; 74use Exporter;
110strings. All boolean flags described below are by default I<disabled>. 111strings. All boolean flags described below are by default I<disabled>.
111 112
112The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
113be chained: 114be chained:
114 115
115#TODO
116 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
117 117
118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
119 119
120=item $max_depth = $cbor->get_max_depth 120=item $max_depth = $cbor->get_max_depth
154If no argument is given, the limit check will be deactivated (same as when 154If no argument is given, the limit check will be deactivated (same as when
155C<0> is specified). 155C<0> is specified).
156 156
157See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 157See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
158 158
159=item $cbor = $cbor->allow_unknown ([$enable])
160
161=item $enabled = $cbor->get_allow_unknown
162
163If C<$enable> is true (or missing), then C<encode> will I<not> throw an
164exception when it encounters values it cannot represent in CBOR (for
165example, filehandles) but instead will encode a CBOR C<error> value.
166
167If C<$enable> is false (the default), then C<encode> will throw an
168exception when it encounters anything it cannot encode as CBOR.
169
170This option does not affect C<decode> in any way, and it is recommended to
171leave it off unless you know your communications partner.
172
173=item $cbor = $cbor->allow_sharing ([$enable])
174
175=item $enabled = $cbor->get_allow_sharing
176
177If C<$enable> is true (or missing), then C<encode> will not double-encode
178values that have been referenced before (e.g. when the same object, such
179as an array, is referenced multiple times), but instead will emit a
180reference to the earlier value.
181
182This means that such values will only be encoded once, and will not result
183in a deep cloning of the value on decode, in decoders supporting the value
184sharing extension. This also makes it possible to encode cyclic data
185structures (which need C<allow_cycles> to be enabled to be decoded by this
186module).
187
188It is recommended to leave it off unless you know your
189communication partner supports the value sharing extensions to CBOR
190(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
191resulting data structure might be unusable.
192
193Detecting shared values incurs a runtime overhead when values are encoded
194that have a reference counter large than one, and might unnecessarily
195increase the encoded size, as potentially shared values are encode as
196shareable whether or not they are actually shared.
197
198At the moment, only targets of references can be shared (e.g. scalars,
199arrays or hashes pointed to by a reference). Weirder constructs, such as
200an array with multiple "copies" of the I<same> string, which are hard but
201not impossible to create in Perl, are not supported (this is the same as
202with L<Storable>).
203
204If C<$enable> is false (the default), then C<encode> will encode shared
205data structures repeatedly, unsharing them in the process. Cyclic data
206structures cannot be encoded in this mode.
207
208This option does not affect C<decode> in any way - shared values and
209references will always be decoded properly if present.
210
211=item $cbor = $cbor->allow_cycles ([$enable])
212
213=item $enabled = $cbor->get_allow_cycles
214
215If C<$enable> is true (or missing), then C<decode> will happily decode
216self-referential (cyclic) data structures. By default these will not be
217decoded, as they need manual cleanup to avoid memory leaks, so code that
218isn't prepared for this will not leak memory.
219
220If C<$enable> is false (the default), then C<decode> will throw an error
221when it encounters a self-referential/cyclic data structure.
222
223FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
224cycles - future versions of this module might chose to decode cyclic data
225structures using weak references when this option is off, instead of
226throwing an error.
227
228This option does not affect C<encode> in any way - shared values and
229references will always be encoded properly if present.
230
231=item $cbor = $cbor->pack_strings ([$enable])
232
233=item $enabled = $cbor->get_pack_strings
234
235If C<$enable> is true (or missing), then C<encode> will try not to encode
236the same string twice, but will instead encode a reference to the string
237instead. Depending on your data format, this can save a lot of space, but
238also results in a very large runtime overhead (expect encoding times to be
2392-4 times as high as without).
240
241It is recommended to leave it off unless you know your
242communications partner supports the stringref extension to CBOR
243(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
244resulting data structure might not be usable.
245
246If C<$enable> is false (the default), then C<encode> will encode strings
247the standard CBOR way.
248
249This option does not affect C<decode> in any way - string references will
250always be decoded properly if present.
251
252=item $cbor = $cbor->text_keys ([$enable])
253
254=item $enabled = $cbor->get_text_keys
255
256If C<$enabled> is true (or missing), then C<encode> will encode all
257perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
258
259If C<$enable> is false (the default), then C<encode> will encode hash keys
260normally - upgraded perl strings (strings internally encoded as UTF-8) as
261CBOR text strings, and downgraded perl strings as CBOR byte strings.
262
263This option does not affect C<decode> in any way.
264
265This option is useful for interoperability with CBOR decoders that don't
266treat byte strings as a form of text. It is especially useful as Perl
267gives very little control over hash keys.
268
269Enabling this option can be slow, as all downgraded hash keys that are
270encoded need to be scanned and converted to UTF-8.
271
272=item $cbor = $cbor->text_strings ([$enable])
273
274=item $enabled = $cbor->get_text_strings
275
276This option works similar to C<text_keys>, above, but works on all strings
277(including hash keys), so C<text_keys> has no further effect after
278enabling C<text_strings>.
279
280If C<$enabled> is true (or missing), then C<encode> will encode all perl
281strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
282
283If C<$enable> is false (the default), then C<encode> will encode strings
284normally (but see C<text_keys>) - upgraded perl strings (strings
285internally encoded as UTF-8) as CBOR text strings, and downgraded perl
286strings as CBOR byte strings.
287
288This option does not affect C<decode> in any way.
289
290This option has similar advantages and disadvantages as C<text_keys>. In
291addition, this option effectively removes the ability to encode byte
292strings, which might break some C<FREEZE> and C<TO_CBOR> methods that rely
293on this, such as bignum encoding, so this option is mainly useful for very
294simple data.
295
296=item $cbor = $cbor->validate_utf8 ([$enable])
297
298=item $enabled = $cbor->get_validate_utf8
299
300If C<$enable> is true (or missing), then C<decode> will validate that
301elements (text strings) containing UTF-8 data in fact contain valid UTF-8
302data (instead of blindly accepting it). This validation obviously takes
303extra time during decoding.
304
305The concept of "valid UTF-8" used is perl's concept, which is a superset
306of the official UTF-8.
307
308If C<$enable> is false (the default), then C<decode> will blindly accept
309UTF-8 data, marking them as valid UTF-8 in the resulting data structure
310regardless of whether that's true or not.
311
312Perl isn't too happy about corrupted UTF-8 in strings, but should
313generally not crash or do similarly evil things. Extensions might be not
314so forgiving, so it's recommended to turn on this setting if you receive
315untrusted CBOR.
316
317This option does not affect C<encode> in any way - strings that are
318supposedly valid UTF-8 will simply be dumped into the resulting CBOR
319string without checking whether that is, in fact, true or not.
320
321=item $cbor = $cbor->filter ([$cb->($tag, $value)])
322
323=item $cb_or_undef = $cbor->get_filter
324
325Sets or replaces the tagged value decoding filter (when C<$cb> is
326specified) or clears the filter (if no argument or C<undef> is provided).
327
328The filter callback is called only during decoding, when a non-enforced
329tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
330list of enforced tags). For specific tags, it's often better to provide a
331default converter using the C<%CBOR::XS::FILTER> hash (see below).
332
333The first argument is the numerical tag, the second is the (decoded) value
334that has been tagged.
335
336The filter function should return either exactly one value, which will
337replace the tagged value in the decoded data structure, or no values,
338which will result in default handling, which currently means the decoder
339creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
340
341When the filter is cleared (the default state), the default filter
342function, C<CBOR::XS::default_filter>, is used. This function simply looks
343up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
344a code reference that is called with tag and value, and is responsible for
345decoding the value. If no entry exists, it returns no values.
346
347Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
348objects, with no other special handling (useful when working with
349potentially "unsafe" CBOR data).
350
351 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
352
353Example: provide a global filter for tag 1347375694, converting the value
354into some string form.
355
356 $CBOR::XS::FILTER{1347375694} = sub {
357 my ($tag, $value);
358
359 "tag 1347375694 value $value"
360 };
361
159=item $cbor_data = $cbor->encode ($perl_scalar) 362=item $cbor_data = $cbor->encode ($perl_scalar)
160 363
161Converts the given Perl data structure (a scalar value) to its CBOR 364Converts the given Perl data structure (a scalar value) to its CBOR
162representation. 365representation.
163 366
176and you need to know where the first CBOR string ends amd the next one 379and you need to know where the first CBOR string ends amd the next one
177starts. 380starts.
178 381
179 CBOR::XS->new->decode_prefix ("......") 382 CBOR::XS->new->decode_prefix ("......")
180 => ("...", 3) 383 => ("...", 3)
384
385=back
386
387=head2 INCREMENTAL PARSING
388
389In some cases, there is the need for incremental parsing of JSON
390texts. While this module always has to keep both CBOR text and resulting
391Perl data structure in memory at one time, it does allow you to parse a
392CBOR stream incrementally, using a similar to using "decode_prefix" to see
393if a full CBOR object is available, but is much more efficient.
394
395It basically works by parsing as much of a CBOR string as possible - if
396the CBOR data is not complete yet, the pasrer will remember where it was,
397to be able to restart when more data has been accumulated. Once enough
398data is available to either decode a complete CBOR value or raise an
399error, a real decode will be attempted.
400
401A typical use case would be a network protocol that consists of sending
402and receiving CBOR-encoded messages. The solution that works with CBOR and
403about anything else is by prepending a length to every CBOR value, so the
404receiver knows how many octets to read. More compact (and slightly slower)
405would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
406a CBOR value ends, and doesn't need an explicit length.
407
408The following methods help with this:
409
410=over 4
411
412=item @decoded = $cbor->incr_parse ($buffer)
413
414This method attempts to decode exactly one CBOR value from the beginning
415of the given C<$buffer>. The value is removed from the C<$buffer> on
416success. When C<$buffer> doesn't contain a complete value yet, it returns
417nothing. Finally, when the C<$buffer> doesn't start with something
418that could ever be a valid CBOR value, it raises an exception, just as
419C<decode> would. In the latter case the decoder state is undefined and
420must be reset before being able to parse further.
421
422This method modifies the C<$buffer> in place. When no CBOR value can be
423decoded, the decoder stores the current string offset. On the next call,
424continues decoding at the place where it stopped before. For this to make
425sense, the C<$buffer> must begin with the same octets as on previous
426unsuccessful calls.
427
428You can call this method in scalar context, in which case it either
429returns a decoded value or C<undef>. This makes it impossible to
430distinguish between CBOR null values (which decode to C<undef>) and an
431unsuccessful decode, which is often acceptable.
432
433=item @decoded = $cbor->incr_parse_multiple ($buffer)
434
435Same as C<incr_parse>, but attempts to decode as many CBOR values as
436possible in one go, instead of at most one. Calls to C<incr_parse> and
437C<incr_parse_multiple> can be interleaved.
438
439=item $cbor->incr_reset
440
441Resets the incremental decoder. This throws away any saved state, so that
442subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
443a new CBOR value from the beginning of the C<$buffer> again.
444
445This method can be called at any time, but it I<must> be called if you want
446to change your C<$buffer> or there was a decoding error and you want to
447reuse the C<$cbor> object for future incremental parsings.
181 448
182=back 449=back
183 450
184 451
185=head1 MAPPING 452=head1 MAPPING
203CBOR integers become (numeric) perl scalars. On perls without 64 bit 470CBOR integers become (numeric) perl scalars. On perls without 64 bit
204support, 64 bit integers will be truncated or otherwise corrupted. 471support, 64 bit integers will be truncated or otherwise corrupted.
205 472
206=item byte strings 473=item byte strings
207 474
208Byte strings will become octet strings in Perl (the byte values 0..255 475Byte strings will become octet strings in Perl (the Byte values 0..255
209will simply become characters of the same value in Perl). 476will simply become characters of the same value in Perl).
210 477
211=item UTF-8 strings 478=item UTF-8 strings
212 479
213UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 480UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
231C<Types:Serialiser::false> and C<Types::Serialiser::error>, 498C<Types:Serialiser::false> and C<Types::Serialiser::error>,
232respectively. They are overloaded to act almost exactly like the numbers 499respectively. They are overloaded to act almost exactly like the numbers
233C<1> and C<0> (for true and false) or to throw an exception on access (for 500C<1> and C<0> (for true and false) or to throw an exception on access (for
234error). See the L<Types::Serialiser> manpage for details. 501error). See the L<Types::Serialiser> manpage for details.
235 502
236=item CBOR tag 256 (perl object) 503=item tagged values
237 504
238The tag value C<256> (TODO: pending iana registration) will be used
239to deserialise a Perl object serialised with C<FREEZE>. See L<OBJECT
240SERIALISATION>, below, for details.
241
242=item CBOR tag 55799 (magic header)
243
244The tag 55799 is ignored (this tag implements the magic header).
245
246=item other CBOR tags
247
248Tagged items consists of a numeric tag and another CBOR value. Tags not 505Tagged items consists of a numeric tag and another CBOR value.
249handled internally are currently converted into a L<CBOR::XS::Tagged>
250object, which is simply a blessed array reference consisting of the
251numeric tag value followed by the (decoded) CBOR value.
252 506
253In the future, support for user-supplied conversions might get added. 507See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
508for details on which tags are handled how.
254 509
255=item anything else 510=item anything else
256 511
257Anything else (e.g. unsupported simple values) will raise a decoding 512Anything else (e.g. unsupported simple values) will raise a decoding
258error. 513error.
261 516
262 517
263=head2 PERL -> CBOR 518=head2 PERL -> CBOR
264 519
265The mapping from Perl to CBOR is slightly more difficult, as Perl is a 520The mapping from Perl to CBOR is slightly more difficult, as Perl is a
266truly typeless language, so we can only guess which CBOR type is meant by 521typeless language. That means this module can only guess which CBOR type
267a Perl value. 522is meant by a perl value.
268 523
269=over 4 524=over 4
270 525
271=item hash references 526=item hash references
272 527
273Perl hash references become CBOR maps. As there is no inherent ordering in 528Perl hash references become CBOR maps. As there is no inherent ordering in
274hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 529hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
275order. 530order. This order can be different each time a hash is encoded.
276 531
277Currently, tied hashes will use the indefinite-length format, while normal 532Currently, tied hashes will use the indefinite-length format, while normal
278hashes will use the fixed-length format. 533hashes will use the fixed-length format.
279 534
280=item array references 535=item array references
281 536
282Perl array references become fixed-length CBOR arrays. 537Perl array references become fixed-length CBOR arrays.
283 538
284=item other references 539=item other references
285 540
286Other unblessed references are generally not allowed and will cause an 541Other unblessed references will be represented using
287exception to be thrown, except for references to the integers C<0> and 542the indirection tag extension (tag value C<22098>,
288C<1>, which get turned into false and true in CBOR. 543L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
544to be able to decode these values somehow, by either "doing the right
545thing", decoding into a generic tagged object, simply ignoring the tag, or
546something else.
289 547
290=item CBOR::XS::Tagged objects 548=item CBOR::XS::Tagged objects
291 549
292Objects of this type must be arrays consisting of a single C<[tag, value]> 550Objects of this type must be arrays consisting of a single C<[tag, value]>
293pair. The (numerical) tag will be encoded as a CBOR tag, the value will 551pair. The (numerical) tag will be encoded as a CBOR tag, the value will
294be encoded as appropriate for the value. You cna use C<CBOR::XS::tag> to 552be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
295create such objects. 553create such objects.
296 554
297=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 555=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
298 556
299These special values become CBOR true, CBOR false and CBOR undefined 557These special values become CBOR true, CBOR false and CBOR undefined
301if you want. 559if you want.
302 560
303=item other blessed objects 561=item other blessed objects
304 562
305Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See 563Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
306L<OBJECT SERIALISATION>, below, for details. 564L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
565module, and L<OBJECT SERIALISATION> for generic object serialisation.
307 566
308=item simple scalars 567=item simple scalars
309 568
310TODO
311Simple Perl scalars (any scalar that is not a reference) are the most 569Simple Perl scalars (any scalar that is not a reference) are the most
312difficult objects to encode: CBOR::XS will encode undefined scalars as 570difficult objects to encode: CBOR::XS will encode undefined scalars as
313CBOR null values, scalars that have last been used in a string context 571CBOR null values, scalars that have last been used in a string context
314before encoding as CBOR strings, and anything else as number value: 572before encoding as CBOR strings, and anything else as number value:
315 573
316 # dump as number 574 # dump as number
317 encode_cbor [2] # yields [2] 575 encode_cbor [2] # yields [2]
318 encode_cbor [-3.0e17] # yields [-3e+17] 576 encode_cbor [-3.0e17] # yields [-3e+17]
319 my $value = 5; encode_cbor [$value] # yields [5] 577 my $value = 5; encode_cbor [$value] # yields [5]
320 578
321 # used as string, so dump as string 579 # used as string, so dump as string (either byte or text)
322 print $value; 580 print $value;
323 encode_cbor [$value] # yields ["5"] 581 encode_cbor [$value] # yields ["5"]
324 582
325 # undef becomes null 583 # undef becomes null
326 encode_cbor [undef] # yields [null] 584 encode_cbor [undef] # yields [null]
329 587
330 my $x = 3.1; # some variable containing a number 588 my $x = 3.1; # some variable containing a number
331 "$x"; # stringified 589 "$x"; # stringified
332 $x .= ""; # another, more awkward way to stringify 590 $x .= ""; # another, more awkward way to stringify
333 print $x; # perl does it for you, too, quite often 591 print $x; # perl does it for you, too, quite often
592
593You can force whether a string is encoded as byte or text string by using
594C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled):
595
596 utf8::upgrade $x; # encode $x as text string
597 utf8::downgrade $x; # encode $x as byte string
598
599Perl doesn't define what operations up- and downgrade strings, so if the
600difference between byte and text is important, you should up- or downgrade
601your string as late as possible before encoding. You can also force the
602use of CBOR text strings by using C<text_keys> or C<text_strings>.
334 603
335You can force the type to be a CBOR number by numifying it: 604You can force the type to be a CBOR number by numifying it:
336 605
337 my $x = "3"; # some variable containing a string 606 my $x = "3"; # some variable containing a string
338 $x += 0; # numify it, ensuring it will be dumped as a number 607 $x += 0; # numify it, ensuring it will be dumped as a number
351 620
352=back 621=back
353 622
354=head2 OBJECT SERIALISATION 623=head2 OBJECT SERIALISATION
355 624
625This module implements both a CBOR-specific and the generic
626L<Types::Serialier> object serialisation protocol. The following
627subsections explain both methods.
628
629=head3 ENCODING
630
356This module knows two way to serialise a Perl object: The CBOR-specific 631This module knows two way to serialise a Perl object: The CBOR-specific
357way, and the generic way. 632way, and the generic way.
358 633
359Whenever the encoder encounters a Perl object that it cnanot serialise 634Whenever the encoder encounters a Perl object that it cannot serialise
360directly (most of them), it will first look up the C<TO_CBOR> method on 635directly (most of them), it will first look up the C<TO_CBOR> method on
361it. 636it.
362 637
363If it has a C<TO_CBOR> method, it will call it with the object as only 638If it has a C<TO_CBOR> method, it will call it with the object as only
364argument, and expects exactly one return value, which it will then 639argument, and expects exactly one return value, which it will then
370 645
371The C<FREEZE> method can return any number of values (i.e. zero or 646The C<FREEZE> method can return any number of values (i.e. zero or
372more). These will be encoded as CBOR perl object, together with the 647more). These will be encoded as CBOR perl object, together with the
373classname. 648classname.
374 649
650These methods I<MUST NOT> change the data structure that is being
651serialised. Failure to comply to this can result in memory corruption -
652and worse.
653
375If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail 654If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
376with an error. 655with an error.
377 656
657=head3 DECODING
658
378Objects encoded via C<TO_CBOR> cannot be automatically decoded, but 659Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
379objects encoded via C<FREEZE> can be decoded using the following protocol: 660but objects encoded via C<FREEZE> can be decoded using the following
661protocol:
380 662
381When an encoded CBOR perl object is encountered by the decoder, it will 663When an encoded CBOR perl object is encountered by the decoder, it will
382look up the C<THAW> method, by using the stored classname, and will fail 664look up the C<THAW> method, by using the stored classname, and will fail
383if the method cannot be found. 665if the method cannot be found.
384 666
385After the lookup it will call the C<THAW> method with the stored classname 667After the lookup it will call the C<THAW> method with the stored classname
386as first argument, the constant string C<CBOR> as second argument, and all 668as first argument, the constant string C<CBOR> as second argument, and all
387values returned by C<FREEZE> as remaining arguments. 669values returned by C<FREEZE> as remaining arguments.
388 670
389=head4 EXAMPLES 671=head3 EXAMPLES
390 672
391Here is an example C<TO_CBOR> method: 673Here is an example C<TO_CBOR> method:
392 674
393 sub My::Object::TO_CBOR { 675 sub My::Object::TO_CBOR {
394 my ($obj) = @_; 676 my ($obj) = @_;
405 687
406 sub URI::TO_CBOR { 688 sub URI::TO_CBOR {
407 my ($self) = @_; 689 my ($self) = @_;
408 my $uri = "$self"; # stringify uri 690 my $uri = "$self"; # stringify uri
409 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 691 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
410 CBOR::XS::tagged 32, "$_[0]" 692 CBOR::XS::tag 32, "$_[0]"
411 } 693 }
412 694
413This will encode URIs as a UTF-8 string with tag 32, which indicates an 695This will encode URIs as a UTF-8 string with tag 32, which indicates an
414URI. 696URI.
415 697
426 "$self" # encode url string 708 "$self" # encode url string
427 } 709 }
428 710
429 sub URI::THAW { 711 sub URI::THAW {
430 my ($class, $serialiser, $uri) = @_; 712 my ($class, $serialiser, $uri) = @_;
431
432 $class->new ($uri) 713 $class->new ($uri)
433 } 714 }
434 715
435Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For 716Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
436example, a C<FREEZE> method that returns "type", "id" and "variant" values 717example, a C<FREEZE> method that returns "type", "id" and "variant" values
452=head1 MAGIC HEADER 733=head1 MAGIC HEADER
453 734
454There is no way to distinguish CBOR from other formats 735There is no way to distinguish CBOR from other formats
455programmatically. To make it easier to distinguish CBOR from other 736programmatically. To make it easier to distinguish CBOR from other
456formats, the CBOR specification has a special "magic string" that can be 737formats, the CBOR specification has a special "magic string" that can be
457prepended to any CBOR string without changing it's meaning. 738prepended to any CBOR string without changing its meaning.
458 739
459This string is available as C<$CBOR::XS::MAGIC>. This module does not 740This string is available as C<$CBOR::XS::MAGIC>. This module does not
460prepend this string tot he CBOR data it generates, but it will ignroe it 741prepend this string to the CBOR data it generates, but it will ignore it
461if present, so users can prepend this string as a "file type" indicator as 742if present, so users can prepend this string as a "file type" indicator as
462required. 743required.
463 744
464 745
465=head1 THE CBOR::XS::Tagged CLASS 746=head1 THE CBOR::XS::Tagged CLASS
548Wrap CBOR data in CBOR: 829Wrap CBOR data in CBOR:
549 830
550 my $cbor_cbor = encode_cbor 831 my $cbor_cbor = encode_cbor
551 CBOR::XS::tag 24, 832 CBOR::XS::tag 24,
552 encode_cbor [1, 2, 3]; 833 encode_cbor [1, 2, 3];
834
835=head1 TAG HANDLING AND EXTENSIONS
836
837This section describes how this module handles specific tagged values
838and extensions. If a tag is not mentioned here and no additional filters
839are provided for it, then the default handling applies (creating a
840CBOR::XS::Tagged object on decoding, and only encoding the tag when
841explicitly requested).
842
843Tags not handled specifically are currently converted into a
844L<CBOR::XS::Tagged> object, which is simply a blessed array reference
845consisting of the numeric tag value followed by the (decoded) CBOR value.
846
847Future versions of this module reserve the right to special case
848additional tags (such as base64url).
849
850=head2 ENFORCED TAGS
851
852These tags are always handled when decoding, and their handling cannot be
853overridden by the user.
854
855=over 4
856
857=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
858
859These tags are automatically created (and decoded) for serialisable
860objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
861serialisation protocol). See L<OBJECT SERIALISATION> for details.
862
863=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
864
865These tags are automatically decoded when encountered (and they do not
866result in a cyclic data structure, see C<allow_cycles>), resulting in
867shared values in the decoded object. They are only encoded, however, when
868C<allow_sharing> is enabled.
869
870Not all shared values can be successfully decoded: values that reference
871themselves will I<currently> decode as C<undef> (this is not the same
872as a reference pointing to itself, which will be represented as a value
873that contains an indirect reference to itself - these will be decoded
874properly).
875
876Note that considerably more shared value data structures can be decoded
877than will be encoded - currently, only values pointed to by references
878will be shared, others will not. While non-reference shared values can be
879generated in Perl with some effort, they were considered too unimportant
880to be supported in the encoder. The decoder, however, will decode these
881values as shared values.
882
883=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
884
885These tags are automatically decoded when encountered. They are only
886encoded, however, when C<pack_strings> is enabled.
887
888=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
889
890This tag is automatically generated when a reference are encountered (with
891the exception of hash and array references). It is converted to a reference
892when decoding.
893
894=item 55799 (self-describe CBOR, RFC 7049)
895
896This value is not generated on encoding (unless explicitly requested by
897the user), and is simply ignored when decoding.
898
899=back
900
901=head2 NON-ENFORCED TAGS
902
903These tags have default filters provided when decoding. Their handling can
904be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
905providing a custom C<filter> callback when decoding.
906
907When they result in decoding into a specific Perl class, the module
908usually provides a corresponding C<TO_CBOR> method as well.
909
910When any of these need to load additional modules that are not part of the
911perl core distribution (e.g. L<URI>), it is (currently) up to the user to
912provide these modules. The decoding usually fails with an exception if the
913required module cannot be loaded.
914
915=over 4
916
917=item 0, 1 (date/time string, seconds since the epoch)
918
919These tags are decoded into L<Time::Piece> objects. The corresponding
920C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
921
922The L<Time::Piece> API is generally surprisingly bad, and fractional
923seconds are only accidentally kept intact, so watch out. On the plus side,
924the module comes with perl since 5.10, which has to count for something.
925
926=item 2, 3 (positive/negative bignum)
927
928These tags are decoded into L<Math::BigInt> objects. The corresponding
929C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
930integers, and others into positive/negative CBOR bignums.
931
932=item 4, 5, 264, 265 (decimal fraction/bigfloat)
933
934Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
935objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
936encodes into a decimal fraction (either tag 4 or 264).
937
938NaN and infinities are not encoded properly, as they cannot be represented
939in CBOR.
940
941See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
942
943=item 30 (rational numbers)
944
945These tags are decoded into L<Math::BigRat> objects. The corresponding
946C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
947C<1> via their numerator only, i.e., they become normal integers or
948C<bignums>.
949
950See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
951
952=item 21, 22, 23 (expected later JSON conversion)
953
954CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
955tags.
956
957=item 32 (URI)
958
959These objects decode into L<URI> objects. The corresponding
960C<URI::TO_CBOR> method again results in a CBOR URI value.
961
962=back
963
964=cut
553 965
554=head1 CBOR and JSON 966=head1 CBOR and JSON
555 967
556CBOR is supposed to implement a superset of the JSON data model, and is, 968CBOR is supposed to implement a superset of the JSON data model, and is,
557with some coercion, able to represent all JSON texts (something that other 969with some coercion, able to represent all JSON texts (something that other
573 985
574First of all, your CBOR decoder should be secure, that is, should not have 986First of all, your CBOR decoder should be secure, that is, should not have
575any buffer overflows. Obviously, this module should ensure that and I am 987any buffer overflows. Obviously, this module should ensure that and I am
576trying hard on making that true, but you never know. 988trying hard on making that true, but you never know.
577 989
990Second, CBOR::XS supports object serialisation - decoding CBOR can cause
991calls to I<any> C<THAW> method in I<any> package that exists in your
992process (that is, CBOR::XS will not try to load modules, but any existing
993C<THAW> method or function can be called, so they all have to be secure).
994
578Second, you need to avoid resource-starving attacks. That means you should 995Third, you need to avoid resource-starving attacks. That means you should
579limit the size of CBOR data you accept, or make sure then when your 996limit the size of CBOR data you accept, or make sure then when your
580resources run out, that's just fine (e.g. by using a separate process that 997resources run out, that's just fine (e.g. by using a separate process that
581can crash safely). The size of a CBOR string in octets is usually a good 998can crash safely). The size of a CBOR string in octets is usually a good
582indication of the size of the resources required to decode it into a Perl 999indication of the size of the resources required to decode it into a Perl
583structure. While CBOR::XS can check the size of the CBOR text, it might be 1000structure. While CBOR::XS can check the size of the CBOR text, it might be
584too late when you already have it in memory, so you might want to check 1001too late when you already have it in memory, so you might want to check
585the size before you accept the string. 1002the size before you accept the string.
586 1003
587Third, CBOR::XS recurses using the C stack when decoding objects and 1004Fourth, CBOR::XS recurses using the C stack when decoding objects and
588arrays. The C stack is a limited resource: for instance, on my amd64 1005arrays. The C stack is a limited resource: for instance, on my amd64
589machine with 8MB of stack size I can decode around 180k nested arrays but 1006machine with 8MB of stack size I can decode around 180k nested arrays but
590only 14k nested CBOR objects (due to perl itself recursing deeply on croak 1007only 14k nested CBOR objects (due to perl itself recursing deeply on croak
591to free the temporary). If that is exceeded, the program crashes. To be 1008to free the temporary). If that is exceeded, the program crashes. To be
592conservative, the default nesting limit is set to 512. If your process 1009conservative, the default nesting limit is set to 512. If your process
599Also keep in mind that CBOR::XS might leak contents of your Perl data 1016Also keep in mind that CBOR::XS might leak contents of your Perl data
600structures in its error messages, so when you serialise sensitive 1017structures in its error messages, so when you serialise sensitive
601information you might want to make sure that exceptions thrown by CBOR::XS 1018information you might want to make sure that exceptions thrown by CBOR::XS
602will not end up in front of untrusted eyes. 1019will not end up in front of untrusted eyes.
603 1020
1021
1022=head1 BIGNUM SECURITY CONSIDERATIONS
1023
1024CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1025L<Math::BigFloat> that tries to encode the number in the simplest possible
1026way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
10274) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1028(L<Math::BigRat>, tag 30) can also contain bignums as members.
1029
1030CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1031bigfloats (tags 5 and 265), but it will never generate these on its own.
1032
1033Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1034decimal fractions is generally fast. Decoding bigints can be slow for very
1035big numbers (tens of thousands of digits, something that could potentially
1036be caught by limiting the size of CBOR texts), and decoding bigfloats or
1037arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1038for large exponents (roughly 40 bit and longer).
1039
1040Additionally, L<Math::BigInt> can take advantage of other bignum
1041libraries, such as L<Math::GMP>, which cannot handle big floats with large
1042exponents, and might simply abort or crash your program, due to their code
1043quality.
1044
1045This can be a concern if you want to parse untrusted CBOR. If it is, you
1046might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1047types. You should also disable types 5 and 265, as these can be slow even
1048without bigints.
1049
1050Disabling bigints will also partially or fully disable types that rely on
1051them, e.g. rational numbers that use bignums.
1052
1053
604=head1 CBOR IMPLEMENTATION NOTES 1054=head1 CBOR IMPLEMENTATION NOTES
605 1055
606This section contains some random implementation notes. They do not 1056This section contains some random implementation notes. They do not
607describe guaranteed behaviour, but merely behaviour as-is implemented 1057describe guaranteed behaviour, but merely behaviour as-is implemented
608right now. 1058right now.
616Only the double data type is supported for NV data types - when Perl uses 1066Only the double data type is supported for NV data types - when Perl uses
617long double to represent floating point values, they might not be encoded 1067long double to represent floating point values, they might not be encoded
618properly. Half precision types are accepted, but not encoded. 1068properly. Half precision types are accepted, but not encoded.
619 1069
620Strict mode and canonical mode are not implemented. 1070Strict mode and canonical mode are not implemented.
1071
1072
1073=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
1074
1075On perls that were built without 64 bit integer support (these are rare
1076nowadays, even on 32 bit architectures, as all major Perl distributions
1077are built with 64 bit integer support), support for any kind of 64 bit
1078integer in CBOR is very limited - most likely, these 64 bit values will
1079be truncated, corrupted, or otherwise not decoded correctly. This also
1080includes string, array and map sizes that are stored as 64 bit integers.
621 1081
622 1082
623=head1 THREADS 1083=head1 THREADS
624 1084
625This module is I<not> guaranteed to be thread safe and there are no 1085This module is I<not> guaranteed to be thread safe and there are no
639Please refrain from using rt.cpan.org or any other bug reporting 1099Please refrain from using rt.cpan.org or any other bug reporting
640service. I put the contact address into my modules for a reason. 1100service. I put the contact address into my modules for a reason.
641 1101
642=cut 1102=cut
643 1103
1104# clumsy hv_store-in-perl
1105sub _hv_store {
1106 $_[0]{$_[1]} = $_[2];
1107}
1108
1109our %FILTER = (
1110 0 => sub { # rfc4287 datetime, utf-8
1111 require Time::Piece;
1112 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1113 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1114 # else either. Whats incredibe over standard strptime totally escapes me.
1115 # doesn't do fractional times, either. sigh.
1116 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1117 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1118 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1119 scalar eval {
1120 my $s = $_[1];
1121
1122 $s =~ s/Z$/+00:00/;
1123 $s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
1124 or die;
1125
1126 my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
1127 my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
1128
1129 Time::Piece::gmtime ($d->epoch + $b)
1130 } || die "corrupted CBOR date/time string ($_[0])";
1131 },
1132
1133 1 => sub { # seconds since the epoch, possibly fractional
1134 require Time::Piece;
1135 scalar Time::Piece::gmtime (pop)
1136 },
1137
1138 2 => sub { # pos bigint
1139 require Math::BigInt;
1140 Math::BigInt->new ("0x" . unpack "H*", pop)
1141 },
1142
1143 3 => sub { # neg bigint
1144 require Math::BigInt;
1145 -Math::BigInt->new ("0x" . unpack "H*", pop)
1146 },
1147
1148 4 => sub { # decimal fraction, array
1149 require Math::BigFloat;
1150 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1151 },
1152
1153 264 => sub { # decimal fraction with arbitrary exponent
1154 require Math::BigFloat;
1155 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1156 },
1157
1158 5 => sub { # bigfloat, array
1159 require Math::BigFloat;
1160 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1161 },
1162
1163 265 => sub { # bigfloat with arbitrary exponent
1164 require Math::BigFloat;
1165 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1166 },
1167
1168 30 => sub { # rational number
1169 require Math::BigRat;
1170 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
1171 },
1172
1173 21 => sub { pop }, # expected conversion to base64url encoding
1174 22 => sub { pop }, # expected conversion to base64 encoding
1175 23 => sub { pop }, # expected conversion to base16 encoding
1176
1177 # 24 # embedded cbor, byte string
1178
1179 32 => sub {
1180 require URI;
1181 URI->new (pop)
1182 },
1183
1184 # 33 # base64url rfc4648, utf-8
1185 # 34 # base64 rfc46484, utf-8
1186 # 35 # regex pcre/ecma262, utf-8
1187 # 36 # mime message rfc2045, utf-8
1188);
1189
1190sub CBOR::XS::default_filter {
1191 &{ $FILTER{$_[0]} or return }
1192}
1193
1194sub URI::TO_CBOR {
1195 my $uri = $_[0]->as_string;
1196 utf8::upgrade $uri;
1197 tag 32, $uri
1198}
1199
1200sub Math::BigInt::TO_CBOR {
1201 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
1202 $_[0]->numify
1203 } else {
1204 my $hex = substr $_[0]->as_hex, 2;
1205 $hex = "0$hex" if 1 & length $hex; # sigh
1206 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
1207 }
1208}
1209
1210sub Math::BigFloat::TO_CBOR {
1211 my ($m, $e) = $_[0]->parts;
1212
1213 -9223372036854775808 <= $e && $e <= 18446744073709551615
1214 ? tag 4, [$e->numify, $m]
1215 : tag 264, [$e, $m]
1216}
1217
1218sub Math::BigRat::TO_CBOR {
1219 my ($n, $d) = $_[0]->parts;
1220
1221 # older versions of BigRat need *1, as they not always return numbers
1222
1223 $d*1 == 1
1224 ? $n*1
1225 : tag 30, [$n*1, $d*1]
1226}
1227
1228sub Time::Piece::TO_CBOR {
1229 tag 1, 0 + $_[0]->epoch
1230}
1231
644XSLoader::load "CBOR::XS", $VERSION; 1232XSLoader::load "CBOR::XS", $VERSION;
645 1233
646=head1 SEE ALSO 1234=head1 SEE ALSO
647 1235
648The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 1236The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines