ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/CBOR-XS/XS.pm
(Generate patch)

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.11 by root, Tue Oct 29 00:20:26 2013 UTC vs.
Revision 1.64 by root, Fri Nov 25 23:37:27 2016 UTC

26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string 26 substr $many_cbor_strings, 0, $length, ""; # remove decoded cbor string
27 } 27 }
28 28
29=head1 DESCRIPTION 29=head1 DESCRIPTION
30 30
31WARNING! This module is very new, and not very well tested (that's up to
32you to do). Furthermore, details of the implementation might change freely
33before version 1.0. And lastly, the object serialisation protocol depends
34on a pending IANA assignment, and until that assignment is official, this
35implementation is not interoperable with other implementations (even
36future versions of this module) until the assignment is done.
37
38You are still invited to try out CBOR, and this module.
39
40This module converts Perl data structures to the Concise Binary Object 31This module converts Perl data structures to the Concise Binary Object
41Representation (CBOR) and vice versa. CBOR is a fast binary serialisation 32Representation (CBOR) and vice versa. CBOR is a fast binary serialisation
42format that aims to use a superset of the JSON data model, i.e. when you 33format that aims to use an (almost) superset of the JSON data model, i.e.
43can represent something in JSON, you should be able to represent it in 34when you can represent something useful in JSON, you should be able to
44CBOR. 35represent it in CBOR.
45 36
46In short, CBOR is a faster and very compact binary alternative to JSON, 37In short, CBOR is a faster and quite compact binary alternative to JSON,
47with the added ability of supporting serialisation of Perl objects. (JSON 38with the added ability of supporting serialisation of Perl objects. (JSON
48often compresses better than CBOR though, so if you plan to compress the 39often compresses better than CBOR though, so if you plan to compress the
49data later you might want to compare both formats first). 40data later and speed is less important you might want to compare both
41formats first).
42
43To give you a general idea about speed, with texts in the megabyte range,
44C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
45L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
46data, the worse L<Storable> performs in comparison.
47
48Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49about 20% smaller than the same data encoded as (compact) JSON or
50L<Storable>.
51
52In addition to the core CBOR data format, this module implements a
53number of extensions, to support cyclic and shared data structures
54(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55C<pack_strings>) and scalar references (always enabled).
50 56
51The primary goal of this module is to be I<correct> and the secondary goal 57The primary goal of this module is to be I<correct> and the secondary goal
52is to be I<fast>. To reach the latter goal it was written in C. 58is to be I<fast>. To reach the latter goal it was written in C.
53 59
54See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and 60See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
58 64
59package CBOR::XS; 65package CBOR::XS;
60 66
61use common::sense; 67use common::sense;
62 68
63our $VERSION = 0.05; 69our $VERSION = 1.51;
64our @ISA = qw(Exporter); 70our @ISA = qw(Exporter);
65 71
66our @EXPORT = qw(encode_cbor decode_cbor); 72our @EXPORT = qw(encode_cbor decode_cbor);
67 73
68use Exporter; 74use Exporter;
105strings. All boolean flags described below are by default I<disabled>. 111strings. All boolean flags described below are by default I<disabled>.
106 112
107The mutators for flags all return the CBOR object again and thus calls can 113The mutators for flags all return the CBOR object again and thus calls can
108be chained: 114be chained:
109 115
110#TODO
111 my $cbor = CBOR::XS->new->encode ({a => [1,2]}); 116 my $cbor = CBOR::XS->new->encode ({a => [1,2]});
112 117
113=item $cbor = $cbor->max_depth ([$maximum_nesting_depth]) 118=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
114 119
115=item $max_depth = $cbor->get_max_depth 120=item $max_depth = $cbor->get_max_depth
149If no argument is given, the limit check will be deactivated (same as when 154If no argument is given, the limit check will be deactivated (same as when
150C<0> is specified). 155C<0> is specified).
151 156
152See SECURITY CONSIDERATIONS, below, for more info on why this is useful. 157See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
153 158
159=item $cbor = $cbor->allow_unknown ([$enable])
160
161=item $enabled = $cbor->get_allow_unknown
162
163If C<$enable> is true (or missing), then C<encode> will I<not> throw an
164exception when it encounters values it cannot represent in CBOR (for
165example, filehandles) but instead will encode a CBOR C<error> value.
166
167If C<$enable> is false (the default), then C<encode> will throw an
168exception when it encounters anything it cannot encode as CBOR.
169
170This option does not affect C<decode> in any way, and it is recommended to
171leave it off unless you know your communications partner.
172
173=item $cbor = $cbor->allow_sharing ([$enable])
174
175=item $enabled = $cbor->get_allow_sharing
176
177If C<$enable> is true (or missing), then C<encode> will not double-encode
178values that have been referenced before (e.g. when the same object, such
179as an array, is referenced multiple times), but instead will emit a
180reference to the earlier value.
181
182This means that such values will only be encoded once, and will not result
183in a deep cloning of the value on decode, in decoders supporting the value
184sharing extension. This also makes it possible to encode cyclic data
185structures (which need C<allow_cycles> to be enabled to be decoded by this
186module).
187
188It is recommended to leave it off unless you know your
189communication partner supports the value sharing extensions to CBOR
190(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
191resulting data structure might be unusable.
192
193Detecting shared values incurs a runtime overhead when values are encoded
194that have a reference counter large than one, and might unnecessarily
195increase the encoded size, as potentially shared values are encode as
196shareable whether or not they are actually shared.
197
198At the moment, only targets of references can be shared (e.g. scalars,
199arrays or hashes pointed to by a reference). Weirder constructs, such as
200an array with multiple "copies" of the I<same> string, which are hard but
201not impossible to create in Perl, are not supported (this is the same as
202with L<Storable>).
203
204If C<$enable> is false (the default), then C<encode> will encode shared
205data structures repeatedly, unsharing them in the process. Cyclic data
206structures cannot be encoded in this mode.
207
208This option does not affect C<decode> in any way - shared values and
209references will always be decoded properly if present.
210
211=item $cbor = $cbor->allow_cycles ([$enable])
212
213=item $enabled = $cbor->get_allow_cycles
214
215If C<$enable> is true (or missing), then C<decode> will happily decode
216self-referential (cyclic) data structures. By default these will not be
217decoded, as they need manual cleanup to avoid memory leaks, so code that
218isn't prepared for this will not leak memory.
219
220If C<$enable> is false (the default), then C<decode> will throw an error
221when it encounters a self-referential/cyclic data structure.
222
223FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
224cycles - future versions of this module might chose to decode cyclic data
225structures using weak references when this option is off, instead of
226throwing an error.
227
228This option does not affect C<encode> in any way - shared values and
229references will always be encoded properly if present.
230
231=item $cbor = $cbor->pack_strings ([$enable])
232
233=item $enabled = $cbor->get_pack_strings
234
235If C<$enable> is true (or missing), then C<encode> will try not to encode
236the same string twice, but will instead encode a reference to the string
237instead. Depending on your data format, this can save a lot of space, but
238also results in a very large runtime overhead (expect encoding times to be
2392-4 times as high as without).
240
241It is recommended to leave it off unless you know your
242communications partner supports the stringref extension to CBOR
243(L<http://cbor.schmorp.de/stringref>), as without decoder support, the
244resulting data structure might not be usable.
245
246If C<$enable> is false (the default), then C<encode> will encode strings
247the standard CBOR way.
248
249This option does not affect C<decode> in any way - string references will
250always be decoded properly if present.
251
252=item $cbor = $cbor->text_keys ([$enable])
253
254=item $enabled = $cbor->get_text_keys
255
256If C<$enabled> is true (or missing), then C<encode> will encode all
257perl hash keys as CBOR text strings/UTF-8 string, upgrading them as needed.
258
259If C<$enable> is false (the default), then C<encode> will encode hash keys
260normally - upgraded perl strings (strings internally encoded as UTF-8) as
261CBOR text strings, and downgraded perl strings as CBOR byte strings.
262
263This option does not affect C<decode> in any way.
264
265This option is useful for interoperability with CBOR decoders that don't
266treat byte strings as a form of text. It is especially useful as Perl
267gives very little control over hash keys.
268
269Enabling this option can be slow, as all downgraded hash keys that are
270encoded need to be scanned and converted to UTF-8.
271
272=item $cbor = $cbor->text_strings ([$enable])
273
274=item $enabled = $cbor->get_text_strings
275
276This option works similar to C<text_keys>, above, but works on all strings
277(including hash keys), so C<text_keys> has no further effect after
278enabling C<text_strings>.
279
280If C<$enabled> is true (or missing), then C<encode> will encode all perl
281strings as CBOR text strings/UTF-8 strings, upgrading them as needed.
282
283If C<$enable> is false (the default), then C<encode> will encode strings
284normally (but see C<text_keys>) - upgraded perl strings (strings
285internally encoded as UTF-8) as CBOR text strings, and downgraded perl
286strings as CBOR byte strings.
287
288This option does not affect C<decode> in any way.
289
290This option has similar advantages and disadvantages as C<text_keys>. In
291addition, this option effectively removes the ability to encode byte
292strings, which might break some C<FREEZE> and C<TO_CBOR> methods that rely
293on this, such as bignum encoding, so this option is mainly useful for very
294simple data.
295
296=item $cbor = $cbor->validate_utf8 ([$enable])
297
298=item $enabled = $cbor->get_validate_utf8
299
300If C<$enable> is true (or missing), then C<decode> will validate that
301elements (text strings) containing UTF-8 data in fact contain valid UTF-8
302data (instead of blindly accepting it). This validation obviously takes
303extra time during decoding.
304
305The concept of "valid UTF-8" used is perl's concept, which is a superset
306of the official UTF-8.
307
308If C<$enable> is false (the default), then C<decode> will blindly accept
309UTF-8 data, marking them as valid UTF-8 in the resulting data structure
310regardless of whether that's true or not.
311
312Perl isn't too happy about corrupted UTF-8 in strings, but should
313generally not crash or do similarly evil things. Extensions might be not
314so forgiving, so it's recommended to turn on this setting if you receive
315untrusted CBOR.
316
317This option does not affect C<encode> in any way - strings that are
318supposedly valid UTF-8 will simply be dumped into the resulting CBOR
319string without checking whether that is, in fact, true or not.
320
321=item $cbor = $cbor->filter ([$cb->($tag, $value)])
322
323=item $cb_or_undef = $cbor->get_filter
324
325Sets or replaces the tagged value decoding filter (when C<$cb> is
326specified) or clears the filter (if no argument or C<undef> is provided).
327
328The filter callback is called only during decoding, when a non-enforced
329tagged value has been decoded (see L<TAG HANDLING AND EXTENSIONS> for a
330list of enforced tags). For specific tags, it's often better to provide a
331default converter using the C<%CBOR::XS::FILTER> hash (see below).
332
333The first argument is the numerical tag, the second is the (decoded) value
334that has been tagged.
335
336The filter function should return either exactly one value, which will
337replace the tagged value in the decoded data structure, or no values,
338which will result in default handling, which currently means the decoder
339creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
340
341When the filter is cleared (the default state), the default filter
342function, C<CBOR::XS::default_filter>, is used. This function simply looks
343up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
344a code reference that is called with tag and value, and is responsible for
345decoding the value. If no entry exists, it returns no values.
346
347Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
348objects, with no other special handling (useful when working with
349potentially "unsafe" CBOR data).
350
351 CBOR::XS->new->filter (sub { })->decode ($cbor_data);
352
353Example: provide a global filter for tag 1347375694, converting the value
354into some string form.
355
356 $CBOR::XS::FILTER{1347375694} = sub {
357 my ($tag, $value);
358
359 "tag 1347375694 value $value"
360 };
361
154=item $cbor_data = $cbor->encode ($perl_scalar) 362=item $cbor_data = $cbor->encode ($perl_scalar)
155 363
156Converts the given Perl data structure (a scalar value) to its CBOR 364Converts the given Perl data structure (a scalar value) to its CBOR
157representation. 365representation.
158 366
171and you need to know where the first CBOR string ends amd the next one 379and you need to know where the first CBOR string ends amd the next one
172starts. 380starts.
173 381
174 CBOR::XS->new->decode_prefix ("......") 382 CBOR::XS->new->decode_prefix ("......")
175 => ("...", 3) 383 => ("...", 3)
384
385=back
386
387=head2 INCREMENTAL PARSING
388
389In some cases, there is the need for incremental parsing of JSON
390texts. While this module always has to keep both CBOR text and resulting
391Perl data structure in memory at one time, it does allow you to parse a
392CBOR stream incrementally, using a similar to using "decode_prefix" to see
393if a full CBOR object is available, but is much more efficient.
394
395It basically works by parsing as much of a CBOR string as possible - if
396the CBOR data is not complete yet, the pasrer will remember where it was,
397to be able to restart when more data has been accumulated. Once enough
398data is available to either decode a complete CBOR value or raise an
399error, a real decode will be attempted.
400
401A typical use case would be a network protocol that consists of sending
402and receiving CBOR-encoded messages. The solution that works with CBOR and
403about anything else is by prepending a length to every CBOR value, so the
404receiver knows how many octets to read. More compact (and slightly slower)
405would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
406a CBOR value ends, and doesn't need an explicit length.
407
408The following methods help with this:
409
410=over 4
411
412=item @decoded = $cbor->incr_parse ($buffer)
413
414This method attempts to decode exactly one CBOR value from the beginning
415of the given C<$buffer>. The value is removed from the C<$buffer> on
416success. When C<$buffer> doesn't contain a complete value yet, it returns
417nothing. Finally, when the C<$buffer> doesn't start with something
418that could ever be a valid CBOR value, it raises an exception, just as
419C<decode> would. In the latter case the decoder state is undefined and
420must be reset before being able to parse further.
421
422This method modifies the C<$buffer> in place. When no CBOR value can be
423decoded, the decoder stores the current string offset. On the next call,
424continues decoding at the place where it stopped before. For this to make
425sense, the C<$buffer> must begin with the same octets as on previous
426unsuccessful calls.
427
428You can call this method in scalar context, in which case it either
429returns a decoded value or C<undef>. This makes it impossible to
430distinguish between CBOR null values (which decode to C<undef>) and an
431unsuccessful decode, which is often acceptable.
432
433=item @decoded = $cbor->incr_parse_multiple ($buffer)
434
435Same as C<incr_parse>, but attempts to decode as many CBOR values as
436possible in one go, instead of at most one. Calls to C<incr_parse> and
437C<incr_parse_multiple> can be interleaved.
438
439=item $cbor->incr_reset
440
441Resets the incremental decoder. This throws away any saved state, so that
442subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
443a new CBOR value from the beginning of the C<$buffer> again.
444
445This method can be called at any time, but it I<must> be called if you want
446to change your C<$buffer> or there was a decoding error and you want to
447reuse the C<$cbor> object for future incremental parsings.
176 448
177=back 449=back
178 450
179 451
180=head1 MAPPING 452=head1 MAPPING
198CBOR integers become (numeric) perl scalars. On perls without 64 bit 470CBOR integers become (numeric) perl scalars. On perls without 64 bit
199support, 64 bit integers will be truncated or otherwise corrupted. 471support, 64 bit integers will be truncated or otherwise corrupted.
200 472
201=item byte strings 473=item byte strings
202 474
203Byte strings will become octet strings in Perl (the byte values 0..255 475Byte strings will become octet strings in Perl (the Byte values 0..255
204will simply become characters of the same value in Perl). 476will simply become characters of the same value in Perl).
205 477
206=item UTF-8 strings 478=item UTF-8 strings
207 479
208UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be 480UTF-8 strings in CBOR will be decoded, i.e. the UTF-8 octets will be
226C<Types:Serialiser::false> and C<Types::Serialiser::error>, 498C<Types:Serialiser::false> and C<Types::Serialiser::error>,
227respectively. They are overloaded to act almost exactly like the numbers 499respectively. They are overloaded to act almost exactly like the numbers
228C<1> and C<0> (for true and false) or to throw an exception on access (for 500C<1> and C<0> (for true and false) or to throw an exception on access (for
229error). See the L<Types::Serialiser> manpage for details. 501error). See the L<Types::Serialiser> manpage for details.
230 502
231=item CBOR tag 256 (perl object) 503=item tagged values
232 504
233The tag value C<256> (TODO: pending iana registration) will be used
234to deserialise a Perl object serialised with C<FREEZE>. See L<OBJECT
235SERIALISATION>, below, for details.
236
237=item CBOR tag 55799 (magic header)
238
239The tag 55799 is ignored (this tag implements the magic header).
240
241=item other CBOR tags
242
243Tagged items consists of a numeric tag and another CBOR value. Tags not 505Tagged items consists of a numeric tag and another CBOR value.
244handled internally are currently converted into a L<CBOR::XS::Tagged>
245object, which is simply a blessed array reference consisting of the
246numeric tag value followed by the (decoded) CBOR value.
247 506
248In the future, support for user-supplied conversions might get added. 507See L<TAG HANDLING AND EXTENSIONS> and the description of C<< ->filter >>
508for details on which tags are handled how.
249 509
250=item anything else 510=item anything else
251 511
252Anything else (e.g. unsupported simple values) will raise a decoding 512Anything else (e.g. unsupported simple values) will raise a decoding
253error. 513error.
256 516
257 517
258=head2 PERL -> CBOR 518=head2 PERL -> CBOR
259 519
260The mapping from Perl to CBOR is slightly more difficult, as Perl is a 520The mapping from Perl to CBOR is slightly more difficult, as Perl is a
261truly typeless language, so we can only guess which CBOR type is meant by 521typeless language. That means this module can only guess which CBOR type
262a Perl value. 522is meant by a perl value.
263 523
264=over 4 524=over 4
265 525
266=item hash references 526=item hash references
267 527
268Perl hash references become CBOR maps. As there is no inherent ordering in 528Perl hash references become CBOR maps. As there is no inherent ordering in
269hash keys (or CBOR maps), they will usually be encoded in a pseudo-random 529hash keys (or CBOR maps), they will usually be encoded in a pseudo-random
270order. 530order. This order can be different each time a hash is encoded.
271 531
272Currently, tied hashes will use the indefinite-length format, while normal 532Currently, tied hashes will use the indefinite-length format, while normal
273hashes will use the fixed-length format. 533hashes will use the fixed-length format.
274 534
275=item array references 535=item array references
276 536
277Perl array references become fixed-length CBOR arrays. 537Perl array references become fixed-length CBOR arrays.
278 538
279=item other references 539=item other references
280 540
281Other unblessed references are generally not allowed and will cause an 541Other unblessed references will be represented using
282exception to be thrown, except for references to the integers C<0> and 542the indirection tag extension (tag value C<22098>,
283C<1>, which get turned into false and true in CBOR. 543L<http://cbor.schmorp.de/indirection>). CBOR decoders are guaranteed
544to be able to decode these values somehow, by either "doing the right
545thing", decoding into a generic tagged object, simply ignoring the tag, or
546something else.
284 547
285=item CBOR::XS::Tagged objects 548=item CBOR::XS::Tagged objects
286 549
287Objects of this type must be arrays consisting of a single C<[tag, value]> 550Objects of this type must be arrays consisting of a single C<[tag, value]>
288pair. The (numerical) tag will be encoded as a CBOR tag, the value will be 551pair. The (numerical) tag will be encoded as a CBOR tag, the value will
289encoded as appropriate for the value. 552be encoded as appropriate for the value. You must use C<CBOR::XS::tag> to
553create such objects.
290 554
291=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error 555=item Types::Serialiser::true, Types::Serialiser::false, Types::Serialiser::error
292 556
293These special values become CBOR true, CBOR false and CBOR undefined 557These special values become CBOR true, CBOR false and CBOR undefined
294values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly 558values, respectively. You can also use C<\1>, C<\0> and C<\undef> directly
295if you want. 559if you want.
296 560
297=item other blessed objects 561=item other blessed objects
298 562
299Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See 563Other blessed objects are serialised via C<TO_CBOR> or C<FREEZE>. See
300L<OBJECT SERIALISATION>, below, for details. 564L<TAG HANDLING AND EXTENSIONS> for specific classes handled by this
565module, and L<OBJECT SERIALISATION> for generic object serialisation.
301 566
302=item simple scalars 567=item simple scalars
303 568
304TODO
305Simple Perl scalars (any scalar that is not a reference) are the most 569Simple Perl scalars (any scalar that is not a reference) are the most
306difficult objects to encode: CBOR::XS will encode undefined scalars as 570difficult objects to encode: CBOR::XS will encode undefined scalars as
307CBOR null values, scalars that have last been used in a string context 571CBOR null values, scalars that have last been used in a string context
308before encoding as CBOR strings, and anything else as number value: 572before encoding as CBOR strings, and anything else as number value:
309 573
310 # dump as number 574 # dump as number
311 encode_cbor [2] # yields [2] 575 encode_cbor [2] # yields [2]
312 encode_cbor [-3.0e17] # yields [-3e+17] 576 encode_cbor [-3.0e17] # yields [-3e+17]
313 my $value = 5; encode_cbor [$value] # yields [5] 577 my $value = 5; encode_cbor [$value] # yields [5]
314 578
315 # used as string, so dump as string 579 # used as string, so dump as string (either byte or text)
316 print $value; 580 print $value;
317 encode_cbor [$value] # yields ["5"] 581 encode_cbor [$value] # yields ["5"]
318 582
319 # undef becomes null 583 # undef becomes null
320 encode_cbor [undef] # yields [null] 584 encode_cbor [undef] # yields [null]
323 587
324 my $x = 3.1; # some variable containing a number 588 my $x = 3.1; # some variable containing a number
325 "$x"; # stringified 589 "$x"; # stringified
326 $x .= ""; # another, more awkward way to stringify 590 $x .= ""; # another, more awkward way to stringify
327 print $x; # perl does it for you, too, quite often 591 print $x; # perl does it for you, too, quite often
592
593You can force whether a string is encoded as byte or text string by using
594C<utf8::upgrade> and C<utf8::downgrade> (if C<text_strings> is disabled):
595
596 utf8::upgrade $x; # encode $x as text string
597 utf8::downgrade $x; # encode $x as byte string
598
599Perl doesn't define what operations up- and downgrade strings, so if the
600difference between byte and text is important, you should up- or downgrade
601your string as late as possible before encoding. You can also force the
602use of CBOR text strings by using C<text_keys> or C<text_strings>.
328 603
329You can force the type to be a CBOR number by numifying it: 604You can force the type to be a CBOR number by numifying it:
330 605
331 my $x = "3"; # some variable containing a string 606 my $x = "3"; # some variable containing a string
332 $x += 0; # numify it, ensuring it will be dumped as a number 607 $x += 0; # numify it, ensuring it will be dumped as a number
345 620
346=back 621=back
347 622
348=head2 OBJECT SERIALISATION 623=head2 OBJECT SERIALISATION
349 624
625This module implements both a CBOR-specific and the generic
626L<Types::Serialier> object serialisation protocol. The following
627subsections explain both methods.
628
629=head3 ENCODING
630
350This module knows two way to serialise a Perl object: The CBOR-specific 631This module knows two way to serialise a Perl object: The CBOR-specific
351way, and the generic way. 632way, and the generic way.
352 633
353Whenever the encoder encounters a Perl object that it cnanot serialise 634Whenever the encoder encounters a Perl object that it cannot serialise
354directly (most of them), it will first look up the C<TO_CBOR> method on 635directly (most of them), it will first look up the C<TO_CBOR> method on
355it. 636it.
356 637
357If it has a C<TO_CBOR> method, it will call it with the object as only 638If it has a C<TO_CBOR> method, it will call it with the object as only
358argument, and expects exactly one return value, which it will then 639argument, and expects exactly one return value, which it will then
364 645
365The C<FREEZE> method can return any number of values (i.e. zero or 646The C<FREEZE> method can return any number of values (i.e. zero or
366more). These will be encoded as CBOR perl object, together with the 647more). These will be encoded as CBOR perl object, together with the
367classname. 648classname.
368 649
650These methods I<MUST NOT> change the data structure that is being
651serialised. Failure to comply to this can result in memory corruption -
652and worse.
653
369If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail 654If an object supports neither C<TO_CBOR> nor C<FREEZE>, encoding will fail
370with an error. 655with an error.
371 656
657=head3 DECODING
658
372Objects encoded via C<TO_CBOR> cannot be automatically decoded, but 659Objects encoded via C<TO_CBOR> cannot (normally) be automatically decoded,
373objects encoded via C<FREEZE> can be decoded using the following protocol: 660but objects encoded via C<FREEZE> can be decoded using the following
661protocol:
374 662
375When an encoded CBOR perl object is encountered by the decoder, it will 663When an encoded CBOR perl object is encountered by the decoder, it will
376look up the C<THAW> method, by using the stored classname, and will fail 664look up the C<THAW> method, by using the stored classname, and will fail
377if the method cannot be found. 665if the method cannot be found.
378 666
379After the lookup it will call the C<THAW> method with the stored classname 667After the lookup it will call the C<THAW> method with the stored classname
380as first argument, the constant string C<CBOR> as second argument, and all 668as first argument, the constant string C<CBOR> as second argument, and all
381values returned by C<FREEZE> as remaining arguments. 669values returned by C<FREEZE> as remaining arguments.
382 670
383=head4 EXAMPLES 671=head3 EXAMPLES
384 672
385Here is an example C<TO_CBOR> method: 673Here is an example C<TO_CBOR> method:
386 674
387 sub My::Object::TO_CBOR { 675 sub My::Object::TO_CBOR {
388 my ($obj) = @_; 676 my ($obj) = @_;
399 687
400 sub URI::TO_CBOR { 688 sub URI::TO_CBOR {
401 my ($self) = @_; 689 my ($self) = @_;
402 my $uri = "$self"; # stringify uri 690 my $uri = "$self"; # stringify uri
403 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string 691 utf8::upgrade $uri; # make sure it will be encoded as UTF-8 string
404 CBOR::XS::tagged 32, "$_[0]" 692 CBOR::XS::tag 32, "$_[0]"
405 } 693 }
406 694
407This will encode URIs as a UTF-8 string with tag 32, which indicates an 695This will encode URIs as a UTF-8 string with tag 32, which indicates an
408URI. 696URI.
409 697
420 "$self" # encode url string 708 "$self" # encode url string
421 } 709 }
422 710
423 sub URI::THAW { 711 sub URI::THAW {
424 my ($class, $serialiser, $uri) = @_; 712 my ($class, $serialiser, $uri) = @_;
425
426 $class->new ($uri) 713 $class->new ($uri)
427 } 714 }
428 715
429Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For 716Unlike C<TO_CBOR>, multiple values can be returned by C<FREEZE>. For
430example, a C<FREEZE> method that returns "type", "id" and "variant" values 717example, a C<FREEZE> method that returns "type", "id" and "variant" values
446=head1 MAGIC HEADER 733=head1 MAGIC HEADER
447 734
448There is no way to distinguish CBOR from other formats 735There is no way to distinguish CBOR from other formats
449programmatically. To make it easier to distinguish CBOR from other 736programmatically. To make it easier to distinguish CBOR from other
450formats, the CBOR specification has a special "magic string" that can be 737formats, the CBOR specification has a special "magic string" that can be
451prepended to any CBOR string without changing it's meaning. 738prepended to any CBOR string without changing its meaning.
452 739
453This string is available as C<$CBOR::XS::MAGIC>. This module does not 740This string is available as C<$CBOR::XS::MAGIC>. This module does not
454prepend this string tot he CBOR data it generates, but it will ignroe it 741prepend this string to the CBOR data it generates, but it will ignore it
455if present, so users can prepend this string as a "file type" indicator as 742if present, so users can prepend this string as a "file type" indicator as
456required. 743required.
457 744
745
746=head1 THE CBOR::XS::Tagged CLASS
747
748CBOR has the concept of tagged values - any CBOR value can be tagged with
749a numeric 64 bit number, which are centrally administered.
750
751C<CBOR::XS> handles a few tags internally when en- or decoding. You can
752also create tags yourself by encoding C<CBOR::XS::Tagged> objects, and the
753decoder will create C<CBOR::XS::Tagged> objects itself when it hits an
754unknown tag.
755
756These objects are simply blessed array references - the first member of
757the array being the numerical tag, the second being the value.
758
759You can interact with C<CBOR::XS::Tagged> objects in the following ways:
760
761=over 4
762
763=item $tagged = CBOR::XS::tag $tag, $value
764
765This function(!) creates a new C<CBOR::XS::Tagged> object using the given
766C<$tag> (0..2**64-1) to tag the given C<$value> (which can be any Perl
767value that can be encoded in CBOR, including serialisable Perl objects and
768C<CBOR::XS::Tagged> objects).
769
770=item $tagged->[0]
771
772=item $tagged->[0] = $new_tag
773
774=item $tag = $tagged->tag
775
776=item $new_tag = $tagged->tag ($new_tag)
777
778Access/mutate the tag.
779
780=item $tagged->[1]
781
782=item $tagged->[1] = $new_value
783
784=item $value = $tagged->value
785
786=item $new_value = $tagged->value ($new_value)
787
788Access/mutate the tagged value.
789
790=back
791
792=cut
793
794sub tag($$) {
795 bless [@_], CBOR::XS::Tagged::;
796}
797
798sub CBOR::XS::Tagged::tag {
799 $_[0][0] = $_[1] if $#_;
800 $_[0][0]
801}
802
803sub CBOR::XS::Tagged::value {
804 $_[0][1] = $_[1] if $#_;
805 $_[0][1]
806}
807
808=head2 EXAMPLES
809
810Here are some examples of C<CBOR::XS::Tagged> uses to tag objects.
811
812You can look up CBOR tag value and emanings in the IANA registry at
813L<http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml>.
814
815Prepend a magic header (C<$CBOR::XS::MAGIC>):
816
817 my $cbor = encode_cbor CBOR::XS::tag 55799, $value;
818 # same as:
819 my $cbor = $CBOR::XS::MAGIC . encode_cbor $value;
820
821Serialise some URIs and a regex in an array:
822
823 my $cbor = encode_cbor [
824 (CBOR::XS::tag 32, "http://www.nethype.de/"),
825 (CBOR::XS::tag 32, "http://software.schmorp.de/"),
826 (CBOR::XS::tag 35, "^[Pp][Ee][Rr][lL]\$"),
827 ];
828
829Wrap CBOR data in CBOR:
830
831 my $cbor_cbor = encode_cbor
832 CBOR::XS::tag 24,
833 encode_cbor [1, 2, 3];
834
835=head1 TAG HANDLING AND EXTENSIONS
836
837This section describes how this module handles specific tagged values
838and extensions. If a tag is not mentioned here and no additional filters
839are provided for it, then the default handling applies (creating a
840CBOR::XS::Tagged object on decoding, and only encoding the tag when
841explicitly requested).
842
843Tags not handled specifically are currently converted into a
844L<CBOR::XS::Tagged> object, which is simply a blessed array reference
845consisting of the numeric tag value followed by the (decoded) CBOR value.
846
847Future versions of this module reserve the right to special case
848additional tags (such as base64url).
849
850=head2 ENFORCED TAGS
851
852These tags are always handled when decoding, and their handling cannot be
853overridden by the user.
854
855=over 4
856
857=item 26 (perl-object, L<http://cbor.schmorp.de/perl-object>)
858
859These tags are automatically created (and decoded) for serialisable
860objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
861serialisation protocol). See L<OBJECT SERIALISATION> for details.
862
863=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
864
865These tags are automatically decoded when encountered (and they do not
866result in a cyclic data structure, see C<allow_cycles>), resulting in
867shared values in the decoded object. They are only encoded, however, when
868C<allow_sharing> is enabled.
869
870Not all shared values can be successfully decoded: values that reference
871themselves will I<currently> decode as C<undef> (this is not the same
872as a reference pointing to itself, which will be represented as a value
873that contains an indirect reference to itself - these will be decoded
874properly).
875
876Note that considerably more shared value data structures can be decoded
877than will be encoded - currently, only values pointed to by references
878will be shared, others will not. While non-reference shared values can be
879generated in Perl with some effort, they were considered too unimportant
880to be supported in the encoder. The decoder, however, will decode these
881values as shared values.
882
883=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
884
885These tags are automatically decoded when encountered. They are only
886encoded, however, when C<pack_strings> is enabled.
887
888=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
889
890This tag is automatically generated when a reference are encountered (with
891the exception of hash and array references). It is converted to a reference
892when decoding.
893
894=item 55799 (self-describe CBOR, RFC 7049)
895
896This value is not generated on encoding (unless explicitly requested by
897the user), and is simply ignored when decoding.
898
899=back
900
901=head2 NON-ENFORCED TAGS
902
903These tags have default filters provided when decoding. Their handling can
904be overridden by changing the C<%CBOR::XS::FILTER> entry for the tag, or by
905providing a custom C<filter> callback when decoding.
906
907When they result in decoding into a specific Perl class, the module
908usually provides a corresponding C<TO_CBOR> method as well.
909
910When any of these need to load additional modules that are not part of the
911perl core distribution (e.g. L<URI>), it is (currently) up to the user to
912provide these modules. The decoding usually fails with an exception if the
913required module cannot be loaded.
914
915=over 4
916
917=item 0, 1 (date/time string, seconds since the epoch)
918
919These tags are decoded into L<Time::Piece> objects. The corresponding
920C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
921
922The L<Time::Piece> API is generally surprisingly bad, and fractional
923seconds are only accidentally kept intact, so watch out. On the plus side,
924the module comes with perl since 5.10, which has to count for something.
925
926=item 2, 3 (positive/negative bignum)
927
928These tags are decoded into L<Math::BigInt> objects. The corresponding
929C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
930integers, and others into positive/negative CBOR bignums.
931
932=item 4, 5, 264, 265 (decimal fraction/bigfloat)
933
934Both decimal fractions and bigfloats are decoded into L<Math::BigFloat>
935objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
936encodes into a decimal fraction (either tag 4 or 264).
937
938NaN and infinities are not encoded properly, as they cannot be represented
939in CBOR.
940
941See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
942
943=item 30 (rational numbers)
944
945These tags are decoded into L<Math::BigRat> objects. The corresponding
946C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
947C<1> via their numerator only, i.e., they become normal integers or
948C<bignums>.
949
950See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
951
952=item 21, 22, 23 (expected later JSON conversion)
953
954CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore these
955tags.
956
957=item 32 (URI)
958
959These objects decode into L<URI> objects. The corresponding
960C<URI::TO_CBOR> method again results in a CBOR URI value.
961
962=back
963
964=cut
458 965
459=head1 CBOR and JSON 966=head1 CBOR and JSON
460 967
461CBOR is supposed to implement a superset of the JSON data model, and is, 968CBOR is supposed to implement a superset of the JSON data model, and is,
462with some coercion, able to represent all JSON texts (something that other 969with some coercion, able to represent all JSON texts (something that other
478 985
479First of all, your CBOR decoder should be secure, that is, should not have 986First of all, your CBOR decoder should be secure, that is, should not have
480any buffer overflows. Obviously, this module should ensure that and I am 987any buffer overflows. Obviously, this module should ensure that and I am
481trying hard on making that true, but you never know. 988trying hard on making that true, but you never know.
482 989
990Second, CBOR::XS supports object serialisation - decoding CBOR can cause
991calls to I<any> C<THAW> method in I<any> package that exists in your
992process (that is, CBOR::XS will not try to load modules, but any existing
993C<THAW> method or function can be called, so they all have to be secure).
994
483Second, you need to avoid resource-starving attacks. That means you should 995Third, you need to avoid resource-starving attacks. That means you should
484limit the size of CBOR data you accept, or make sure then when your 996limit the size of CBOR data you accept, or make sure then when your
485resources run out, that's just fine (e.g. by using a separate process that 997resources run out, that's just fine (e.g. by using a separate process that
486can crash safely). The size of a CBOR string in octets is usually a good 998can crash safely). The size of a CBOR string in octets is usually a good
487indication of the size of the resources required to decode it into a Perl 999indication of the size of the resources required to decode it into a Perl
488structure. While CBOR::XS can check the size of the CBOR text, it might be 1000structure. While CBOR::XS can check the size of the CBOR text, it might be
489too late when you already have it in memory, so you might want to check 1001too late when you already have it in memory, so you might want to check
490the size before you accept the string. 1002the size before you accept the string.
491 1003
492Third, CBOR::XS recurses using the C stack when decoding objects and 1004Fourth, CBOR::XS recurses using the C stack when decoding objects and
493arrays. The C stack is a limited resource: for instance, on my amd64 1005arrays. The C stack is a limited resource: for instance, on my amd64
494machine with 8MB of stack size I can decode around 180k nested arrays but 1006machine with 8MB of stack size I can decode around 180k nested arrays but
495only 14k nested CBOR objects (due to perl itself recursing deeply on croak 1007only 14k nested CBOR objects (due to perl itself recursing deeply on croak
496to free the temporary). If that is exceeded, the program crashes. To be 1008to free the temporary). If that is exceeded, the program crashes. To be
497conservative, the default nesting limit is set to 512. If your process 1009conservative, the default nesting limit is set to 512. If your process
504Also keep in mind that CBOR::XS might leak contents of your Perl data 1016Also keep in mind that CBOR::XS might leak contents of your Perl data
505structures in its error messages, so when you serialise sensitive 1017structures in its error messages, so when you serialise sensitive
506information you might want to make sure that exceptions thrown by CBOR::XS 1018information you might want to make sure that exceptions thrown by CBOR::XS
507will not end up in front of untrusted eyes. 1019will not end up in front of untrusted eyes.
508 1020
1021
1022=head1 BIGNUM SECURITY CONSIDERATIONS
1023
1024CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
1025L<Math::BigFloat> that tries to encode the number in the simplest possible
1026way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
10274) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
1028(L<Math::BigRat>, tag 30) can also contain bignums as members.
1029
1030CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
1031bigfloats (tags 5 and 265), but it will never generate these on its own.
1032
1033Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
1034decimal fractions is generally fast. Decoding bigints can be slow for very
1035big numbers (tens of thousands of digits, something that could potentially
1036be caught by limiting the size of CBOR texts), and decoding bigfloats or
1037arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
1038for large exponents (roughly 40 bit and longer).
1039
1040Additionally, L<Math::BigInt> can take advantage of other bignum
1041libraries, such as L<Math::GMP>, which cannot handle big floats with large
1042exponents, and might simply abort or crash your program, due to their code
1043quality.
1044
1045This can be a concern if you want to parse untrusted CBOR. If it is, you
1046might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
1047types. You should also disable types 5 and 265, as these can be slow even
1048without bigints.
1049
1050Disabling bigints will also partially or fully disable types that rely on
1051them, e.g. rational numbers that use bignums.
1052
1053
509=head1 CBOR IMPLEMENTATION NOTES 1054=head1 CBOR IMPLEMENTATION NOTES
510 1055
511This section contains some random implementation notes. They do not 1056This section contains some random implementation notes. They do not
512describe guaranteed behaviour, but merely behaviour as-is implemented 1057describe guaranteed behaviour, but merely behaviour as-is implemented
513right now. 1058right now.
521Only the double data type is supported for NV data types - when Perl uses 1066Only the double data type is supported for NV data types - when Perl uses
522long double to represent floating point values, they might not be encoded 1067long double to represent floating point values, they might not be encoded
523properly. Half precision types are accepted, but not encoded. 1068properly. Half precision types are accepted, but not encoded.
524 1069
525Strict mode and canonical mode are not implemented. 1070Strict mode and canonical mode are not implemented.
1071
1072
1073=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
1074
1075On perls that were built without 64 bit integer support (these are rare
1076nowadays, even on 32 bit architectures, as all major Perl distributions
1077are built with 64 bit integer support), support for any kind of 64 bit
1078integer in CBOR is very limited - most likely, these 64 bit values will
1079be truncated, corrupted, or otherwise not decoded correctly. This also
1080includes string, array and map sizes that are stored as 64 bit integers.
526 1081
527 1082
528=head1 THREADS 1083=head1 THREADS
529 1084
530This module is I<not> guaranteed to be thread safe and there are no 1085This module is I<not> guaranteed to be thread safe and there are no
544Please refrain from using rt.cpan.org or any other bug reporting 1099Please refrain from using rt.cpan.org or any other bug reporting
545service. I put the contact address into my modules for a reason. 1100service. I put the contact address into my modules for a reason.
546 1101
547=cut 1102=cut
548 1103
1104# clumsy hv_store-in-perl
1105sub _hv_store {
1106 $_[0]{$_[1]} = $_[2];
1107}
1108
1109our %FILTER = (
1110 0 => sub { # rfc4287 datetime, utf-8
1111 require Time::Piece;
1112 # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
1113 # from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
1114 # else either. Whats incredibe over standard strptime totally escapes me.
1115 # doesn't do fractional times, either. sigh.
1116 # In fact, it's all a lie, it uses whatever strptime it wants, and of course,
1117 # they are all incompatible. The openbsd one simply ignores %z (but according to the
1118 # docs, it would be much more incredibly flexible indeed. If it worked, that is.).
1119 scalar eval {
1120 my $s = $_[1];
1121
1122 $s =~ s/Z$/+00:00/;
1123 $s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
1124 or die;
1125
1126 my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
1127 my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
1128
1129 Time::Piece::gmtime ($d->epoch + $b)
1130 } || die "corrupted CBOR date/time string ($_[0])";
1131 },
1132
1133 1 => sub { # seconds since the epoch, possibly fractional
1134 require Time::Piece;
1135 scalar Time::Piece::gmtime (pop)
1136 },
1137
1138 2 => sub { # pos bigint
1139 require Math::BigInt;
1140 Math::BigInt->new ("0x" . unpack "H*", pop)
1141 },
1142
1143 3 => sub { # neg bigint
1144 require Math::BigInt;
1145 -Math::BigInt->new ("0x" . unpack "H*", pop)
1146 },
1147
1148 4 => sub { # decimal fraction, array
1149 require Math::BigFloat;
1150 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1151 },
1152
1153 264 => sub { # decimal fraction with arbitrary exponent
1154 require Math::BigFloat;
1155 Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
1156 },
1157
1158 5 => sub { # bigfloat, array
1159 require Math::BigFloat;
1160 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1161 },
1162
1163 265 => sub { # bigfloat with arbitrary exponent
1164 require Math::BigFloat;
1165 scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
1166 },
1167
1168 30 => sub { # rational number
1169 require Math::BigRat;
1170 Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
1171 },
1172
1173 21 => sub { pop }, # expected conversion to base64url encoding
1174 22 => sub { pop }, # expected conversion to base64 encoding
1175 23 => sub { pop }, # expected conversion to base16 encoding
1176
1177 # 24 # embedded cbor, byte string
1178
1179 32 => sub {
1180 require URI;
1181 URI->new (pop)
1182 },
1183
1184 # 33 # base64url rfc4648, utf-8
1185 # 34 # base64 rfc46484, utf-8
1186 # 35 # regex pcre/ecma262, utf-8
1187 # 36 # mime message rfc2045, utf-8
1188);
1189
1190sub CBOR::XS::default_filter {
1191 &{ $FILTER{$_[0]} or return }
1192}
1193
1194sub URI::TO_CBOR {
1195 my $uri = $_[0]->as_string;
1196 utf8::upgrade $uri;
1197 tag 32, $uri
1198}
1199
1200sub Math::BigInt::TO_CBOR {
1201 if (-2147483648 <= $_[0] && $_[0] <= 2147483647) {
1202 $_[0]->numify
1203 } else {
1204 my $hex = substr $_[0]->as_hex, 2;
1205 $hex = "0$hex" if 1 & length $hex; # sigh
1206 tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
1207 }
1208}
1209
1210sub Math::BigFloat::TO_CBOR {
1211 my ($m, $e) = $_[0]->parts;
1212
1213 -9223372036854775808 <= $e && $e <= 18446744073709551615
1214 ? tag 4, [$e->numify, $m]
1215 : tag 264, [$e, $m]
1216}
1217
1218sub Math::BigRat::TO_CBOR {
1219 my ($n, $d) = $_[0]->parts;
1220
1221 # older versions of BigRat need *1, as they not always return numbers
1222
1223 $d*1 == 1
1224 ? $n*1
1225 : tag 30, [$n*1, $d*1]
1226}
1227
1228sub Time::Piece::TO_CBOR {
1229 tag 1, 0 + $_[0]->epoch
1230}
1231
549XSLoader::load "CBOR::XS", $VERSION; 1232XSLoader::load "CBOR::XS", $VERSION;
550 1233
551=head1 SEE ALSO 1234=head1 SEE ALSO
552 1235
553The L<JSON> and L<JSON::XS> modules that do similar, but human-readable, 1236The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines