--- CBOR-XS/README 2016/02/08 04:37:12 1.16 +++ CBOR-XS/README 2020/11/29 21:35:51 1.19 @@ -35,6 +35,9 @@ compress the data later and speed is less important you might want to compare both formats first). + The primary goal of this module is to be *correct* and the secondary + goal is to be *fast*. To reach the latter goal it was written in C. + To give you a general idea about speed, with texts in the megabyte range, "CBOR::XS" usually encodes roughly twice as fast as Storable or JSON::XS and decodes about 15%-30% faster than those. The shorter the @@ -49,9 +52,6 @@ "allow_sharing" and "allow_cycles"), string deduplication (see "pack_strings") and scalar references (always enabled). - The primary goal of this module is to be *correct* and the secondary - goal is to be *fast*. To reach the latter goal it was written in C. - See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and vice versa. @@ -81,6 +81,20 @@ my $cbor = CBOR::XS->new->encode ({a => [1,2]}); + $cbor = new_safe CBOR::XS + Create a new, safe/secure CBOR::XS object. This is similar to "new", + but configures the coder object to be safe to use with untrusted + data. Currently, this is equivalent to: + + my $cbor = CBOR::XS + ->new + ->forbid_objects + ->filter (\&CBOR::XS::safe_filter) + ->max_size (1e8); + + But is more future proof (it is better to crash because of a change + than to be exploited in other ways). + $cbor = $cbor->max_depth ([$maximum_nesting_depth]) $max_depth = $cbor->get_max_depth Sets the maximum nesting level (default 512) accepted while encoding @@ -103,7 +117,7 @@ value has been chosen to be as large as typical operating systems allow without crashing. - See SECURITY CONSIDERATIONS, below, for more info on why this is + See "SECURITY CONSIDERATIONS", below, for more info on why this is useful. $cbor = $cbor->max_size ([$maximum_string_size]) @@ -117,7 +131,7 @@ If no argument is given, the limit check will be deactivated (same as when 0 is specified). - See SECURITY CONSIDERATIONS, below, for more info on why this is + See "SECURITY CONSIDERATIONS", below, for more info on why this is useful. $cbor = $cbor->allow_unknown ([$enable]) @@ -143,7 +157,7 @@ This means that such values will only be encoded once, and will not result in a deep cloning of the value on decode, in decoders supporting the value sharing extension. This also makes it possible - to encode cyclic data structures (which need "allow_cycles" to ne + to encode cyclic data structures (which need "allow_cycles" to be enabled to be decoded by this module). It is recommended to leave it off unless you know your communication @@ -154,7 +168,7 @@ Detecting shared values incurs a runtime overhead when values are encoded that have a reference counter large than one, and might unnecessarily increase the encoded size, as potentially shared - values are encode as shareable whether or not they are actually + values are encoded as shareable whether or not they are actually shared. At the moment, only targets of references can be shared (e.g. @@ -188,6 +202,25 @@ This option does not affect "encode" in any way - shared values and references will always be encoded properly if present. + $cbor = $cbor->forbid_objects ([$enable]) + $enabled = $cbor->get_forbid_objects + Disables the use of the object serialiser protocol. + + If $enable is true (or missing), then "encode" will will throw an + exception when it encounters perl objects that would be encoded + using the perl-object tag (26). When "decode" encounters such tags, + it will fall back to the general filter/tagged logic as if this were + an unknown tag (by default resulting in a "CBOR::XC::Tagged" + object). + + If $enable is false (the default), then "encode" will use the + Types::Serialiser object serialisation protocol to serialise objects + into perl-object tags, and "decode" will do the same to decode such + tags. + + See "SECURITY CONSIDERATIONS", below, for more info on why + forbidding this protocol can be useful. + $cbor = $cbor->pack_strings ([$enable]) $enabled = $cbor->get_pack_strings If $enable is true (or missing), then "encode" will try not to @@ -207,6 +240,51 @@ This option does not affect "decode" in any way - string references will always be decoded properly if present. + $cbor = $cbor->text_keys ([$enable]) + $enabled = $cbor->get_text_keys + If $enabled is true (or missing), then "encode" will encode all perl + hash keys as CBOR text strings/UTF-8 string, upgrading them as + needed. + + If $enable is false (the default), then "encode" will encode hash + keys normally - upgraded perl strings (strings internally encoded as + UTF-8) as CBOR text strings, and downgraded perl strings as CBOR + byte strings. + + This option does not affect "decode" in any way. + + This option is useful for interoperability with CBOR decoders that + don't treat byte strings as a form of text. It is especially useful + as Perl gives very little control over hash keys. + + Enabling this option can be slow, as all downgraded hash keys that + are encoded need to be scanned and converted to UTF-8. + + $cbor = $cbor->text_strings ([$enable]) + $enabled = $cbor->get_text_strings + This option works similar to "text_keys", above, but works on all + strings (including hash keys), so "text_keys" has no further effect + after enabling "text_strings". + + If $enabled is true (or missing), then "encode" will encode all perl + strings as CBOR text strings/UTF-8 strings, upgrading them as + needed. + + If $enable is false (the default), then "encode" will encode strings + normally (but see "text_keys") - upgraded perl strings (strings + internally encoded as UTF-8) as CBOR text strings, and downgraded + perl strings as CBOR byte strings. + + This option does not affect "decode" in any way. + + This option has similar advantages and disadvantages as "text_keys". + In addition, this option effectively removes the ability to + automatically encode byte strings, which might break some "FREEZE" + and "TO_CBOR" methods that rely on this. + + A workaround is to use explicit type casts, which are unaffected by + this option. + $cbor = $cbor->validate_utf8 ([$enable]) $enabled = $cbor->get_validate_utf8 If $enable is true (or missing), then "decode" will validate that @@ -219,7 +297,7 @@ If $enable is false (the default), then "decode" will blindly accept UTF-8 data, marking them as valid UTF-8 in the resulting data - structure regardless of whether thats true or not. + structure regardless of whether that's true or not. Perl isn't too happy about corrupted UTF-8 in strings, but should generally not crash or do similarly evil things. Extensions might be @@ -256,7 +334,20 @@ looks up the tag in the %CBOR::XS::FILTER hash. If an entry exists it must be a code reference that is called with tag and value, and is responsible for decoding the value. If no entry exists, it - returns no values. + returns no values. "CBOR::XS" provides a number of default filter + functions already, the the %CBOR::XS::FILTER hash can be freely + extended with more. + + "CBOR::XS" additionally provides an alternative filter function that + is supposed to be safe to use with untrusted data (which the default + filter might not), called "CBOR::XS::safe_filter", which works the + same as the "default_filter" but uses the %CBOR::XS::SAFE_FILTER + variable instead. It is prepopulated with the tag decoding functions + that are deemed safe (basically the same as %CBOR::XS::FILTER + without all the bignum tags), and can be extended by user code as + wlel, although, obviously, one should be very careful about adding + decoding functions here, since the expectation is that they are safe + to use on untrusted data, after all. Example: decode all tags not handled internally into "CBOR::XS::Tagged" objects, with no other special handling (useful @@ -273,6 +364,26 @@ "tag 1347375694 value $value" }; + Example: provide your own filter function that looks up tags in your + own hash: + + my %my_filter = ( + 998347484 => sub { + my ($tag, $value); + + "tag 998347484 value $value" + }; + ); + + my $coder = CBOR::XS->new->filter (sub { + &{ $my_filter{$_[0]} or return } + }); + + Example: use the safe filter function (see "SECURITY CONSIDERATIONS" + for more considerations on security). + + CBOR::XS->new->filter (\&CBOR::XS::safe_filter)->decode ($cbor_data); + $cbor_data = $cbor->encode ($perl_scalar) Converts the given Perl data structure (a scalar value) to its CBOR representation. @@ -289,7 +400,9 @@ This is useful if your CBOR texts are not delimited by an outer protocol and you need to know where the first CBOR string ends amd - the next one starts. + the next one starts - CBOR strings are self-delimited, so it is + possible to concatenate CBOR strings without any delimiters or size + fields and recover their data. CBOR::XS->new->decode_prefix ("......") => ("...", 3) @@ -348,9 +461,9 @@ that subsequent calls to "incr_parse" or "incr_parse_multiple" start to parse a new CBOR value from the beginning of the $buffer again. - This method can be caled at any time, but it *must* be called if you - want to change your $buffer or there was a decoding error and you - want to reuse the $cbor object for future incremental parsings. + This method can be called at any time, but it *must* be called if + you want to change your $buffer or there was a decoding error and + you want to reuse the $cbor object for future incremental parsings. MAPPING This section describes how CBOR::XS maps Perl values to CBOR values and @@ -411,7 +524,7 @@ Perl hash references become CBOR maps. As there is no inherent ordering in hash keys (or CBOR maps), they will usually be encoded in a pseudo-random order. This order can be different each time a - hahs is encoded. + hash is encoded. Currently, tied hashes will use the indefinite-length format, while normal hashes will use the fixed-length format. @@ -470,15 +583,21 @@ $x .= ""; # another, more awkward way to stringify print $x; # perl does it for you, too, quite often - You can force whether a string ie encoded as byte or text string by - using "utf8::upgrade" and "utf8::downgrade"): + You can force whether a string is encoded as byte or text string by + using "utf8::upgrade" and "utf8::downgrade" (if "text_strings" is + disabled). utf8::upgrade $x; # encode $x as text string utf8::downgrade $x; # encode $x as byte string + More options are available, see "TYPE CASTS", below, and the + "text_keys" and "text_strings" options. + Perl doesn't define what operations up- and downgrade strings, so if the difference between byte and text is important, you should up- or - downgrade your string as late as possible before encoding. + downgrade your string as late as possible before encoding. You can + also force the use of CBOR text strings by using "text_keys" or + "text_strings". You can force the type to be a CBOR number by numifying it: @@ -497,6 +616,51 @@ than IEEE double to represent numerical values are supported, but might suffer loss of precision. + TYPE CASTS + EXPERIMENTAL: As an experimental extension, "CBOR::XS" allows you to + force specific cbor types to be used when encoding. That allows you to + encode types not normally accessible (e.g. half floats) as well as force + string types even when "text_strings" is in effect. + + Type forcing is done by calling a special "cast" function which keeps a + copy of the value and returns a new value that can be handed over to any + CBOR encoder function. + + The following casts are currently available (all of which are unary + operators): + + CBOR::XS::as_int $value + Forces the value to be encoded as some form of (basic, not bignum) + integer type. + + CBOR::XS::as_text $value + Forces the value to be encoded as (UTF-8) text values. + + CBOR::XS::as_bytes $value + Forces the value to be encoded as a (binary) string value. + + CBOR::XS::as_float16 $value + Forces half-float (IEEE 754 binary16) encoding of the given value. + + CBOR::XS::as_float32 $value + Forces single-float (IEEE 754 binary32) encoding of the given value. + + CBOR::XS::as_float64 $value + Forces double-float (IEEE 754 binary64) encoding of the given value. + + =item, CBOR::XS::as_cbor $cbor_text + + Bot a type cast per-se, this type cast forces the argument to eb + encoded as-is. This can be used to embed pre-encoded CBOR data. + + Note that no checking on the validity of the $cbor_text is done - + it's the callers responsibility to correctly encode values. + + Example: encode a perl string as binary even though "text_strings" is in + effect. + + CBOR::XS->new->text_strings->encode ([4, "text", CBOR::XS::bytes "bytevalue"]); + OBJECT SERIALISATION This module implements both a CBOR-specific and the generic Types::Serialier object serialisation protocol. The following @@ -583,7 +747,6 @@ sub URI::THAW { my ($class, $serialiser, $uri) = @_; - $class->new ($uri) } @@ -689,7 +852,7 @@ ENFORCED TAGS These tags are always handled when decoding, and their handling cannot - be overriden by the user. + be overridden by the user. 26 (perl-object, ) These tags are automatically created (and decoded) for serialisable @@ -722,8 +885,8 @@ 22098 (indirection, ) This tag is automatically generated when a reference are encountered - (with the exception of hash and array refernces). It is converted to - a reference when decoding. + (with the exception of hash and array references). It is converted + to a reference when decoding. 55799 (self-describe CBOR, RFC 7049) This value is not generated on encoding (unless explicitly requested @@ -731,8 +894,8 @@ NON-ENFORCED TAGS These tags have default filters provided when decoding. Their handling - can be overriden by changing the %CBOR::XS::FILTER entry for the tag, or - by providing a custom "filter" callback when decoding. + can be overridden by changing the %CBOR::XS::FILTER entry for the tag, + or by providing a custom "filter" callback when decoding. When they result in decoding into a specific Perl class, the module usually provides a corresponding "TO_CBOR" method as well. @@ -757,15 +920,23 @@ "Math::BigInt::TO_CBOR" method encodes "small" bigints into normal CBOR integers, and others into positive/negative CBOR bignums. - 4, 5 (decimal fraction/bigfloat) + 4, 5, 264, 265 (decimal fraction/bigfloat) Both decimal fractions and bigfloats are decoded into Math::BigFloat objects. The corresponding "Math::BigFloat::TO_CBOR" method *always* - encodes into a decimal fraction. + encodes into a decimal fraction (either tag 4 or 264). + + NaN and infinities are not encoded properly, as they cannot be + represented in CBOR. + + See "BIGNUM SECURITY CONSIDERATIONS" for more info. - CBOR cannot represent bigfloats with *very* large exponents - - conversion of such big float objects is undefined. + 30 (rational numbers) + These tags are decoded into Math::BigRat objects. The corresponding + "Math::BigRat::TO_CBOR" method encodes rational numbers with + denominator 1 via their numerator only, i.e., they become normal + integers or "bignums". - Also, NaN and infinities are not encoded properly. + See "BIGNUM SECURITY CONSIDERATIONS" for more info. 21, 22, 23 (expected later JSON conversion) CBOR::XS is not a CBOR-to-JSON converter, and will simply ignore @@ -789,38 +960,140 @@ CBOR intact. SECURITY CONSIDERATIONS - When you are using CBOR in a protocol, talking to untrusted potentially - hostile creatures requires relatively few measures. + Tl;dr... if you want to decode or encode CBOR from untrusted sources, + you should start with a coder object created via "new_safe" (which + implements the mitigations explained below): + + my $coder = CBOR::XS->new_safe; + + my $data = $coder->decode ($cbor_text); + my $cbor = $coder->encode ($data); + + Longer version: When you are using CBOR in a protocol, talking to + untrusted potentially hostile creatures requires some thought: + + Security of the CBOR decoder itself + First and foremost, your CBOR decoder should be secure, that is, + should not have any buffer overflows or similar bugs that could + potentially be exploited. Obviously, this module should ensure that + and I am trying hard on making that true, but you never know. + + CBOR::XS can invoke almost arbitrary callbacks during decoding + CBOR::XS supports object serialisation - decoding CBOR can cause + calls to *any* "THAW" method in *any* package that exists in your + process (that is, CBOR::XS will not try to load modules, but any + existing "THAW" method or function can be called, so they all have + to be secure). + + Less obviously, it will also invoke "TO_CBOR" and "FREEZE" methods - + even if all your "THAW" methods are secure, encoding data structures + from untrusted sources can invoke those and trigger bugs in those. + + So, if you are not sure about the security of all the modules you + have loaded (you shouldn't), you should disable this part using + "forbid_objects" or using "new_safe". + + CBOR can be extended with tags that call library code + CBOR can be extended with tags, and "CBOR::XS" has a registry of + conversion functions for many existing tags that can be extended via + third-party modules (see the "filter" method). + + If you don't trust these, you should configure the "safe" filter + function, "CBOR::XS::safe_filter" ("new_safe" does this), which by + default only includes conversion functions that are considered + "safe" by the author (but again, they can be extended by third party + modules). + + Depending on your level of paranoia, you can use the "safe" filter: + + $cbor->filter (\&CBOR::XS::safe_filter); + + ... your own filter... + + $cbor->filter (sub { ... do your stuffs here ... }); + + ... or even no filter at all, disabling all tag decoding: + + $cbor->filter (sub { }); + + This is never a problem for encoding, as the tag mechanism only + exists in CBOR texts. + + Resource-starving attacks: object memory usage + You need to avoid resource-starving attacks. That means you should + limit the size of CBOR data you accept, or make sure then when your + resources run out, that's just fine (e.g. by using a separate + process that can crash safely). The size of a CBOR string in octets + is usually a good indication of the size of the resources required + to decode it into a Perl structure. While CBOR::XS can check the + size of the CBOR text (using "max_size" - done by "new_safe"), it + might be too late when you already have it in memory, so you might + want to check the size before you accept the string. + + As for encoding, it is possible to construct data structures that + are relatively small but result in large CBOR texts (for example by + having an array full of references to the same big data structure, + which will all be deep-cloned during encoding by default). This is + rarely an actual issue (and the worst case is still just running out + of memory), but you can reduce this risk by using "allow_sharing". + + Resource-starving attacks: stack overflows + CBOR::XS recurses using the C stack when decoding objects and + arrays. The C stack is a limited resource: for instance, on my amd64 + machine with 8MB of stack size I can decode around 180k nested + arrays but only 14k nested CBOR objects (due to perl itself + recursing deeply on croak to free the temporary). If that is + exceeded, the program crashes. To be conservative, the default + nesting limit is set to 512. If your process has a smaller stack, + you should adjust this setting accordingly with the "max_depth" + method. + + Resource-starving attacks: CPU en-/decoding complexity + CBOR::XS will use the Math::BigInt, Math::BigFloat and Math::BigRat + libraries to represent encode/decode bignums. These can be very slow + (as in, centuries of CPU time) and can even crash your program (and + are generally not very trustworthy). See the next section on bignum + security for details. + + Data breaches: leaking information in error messages + CBOR::XS might leak contents of your Perl data structures in its + error messages, so when you serialise sensitive information you + might want to make sure that exceptions thrown by CBOR::XS will not + end up in front of untrusted eyes. + + Something else... + Something else could bomb you, too, that I forgot to think of. In + that case, you get to keep the pieces. I am always open for hints, + though... + +BIGNUM SECURITY CONSIDERATIONS + CBOR::XS provides a "TO_CBOR" method for both Math::BigInt and + Math::BigFloat that tries to encode the number in the simplest possible + way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag + 4) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers + (Math::BigRat, tag 30) can also contain bignums as members. + + CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent + bigfloats (tags 5 and 265), but it will never generate these on its own. + + Using the built-in Math::BigInt::Calc support, encoding and decoding + decimal fractions is generally fast. Decoding bigints can be slow for + very big numbers (tens of thousands of digits, something that could + potentially be caught by limiting the size of CBOR texts), and decoding + bigfloats or arbitrary-exponent bigfloats can be *extremely* slow + (minutes, decades) for large exponents (roughly 40 bit and longer). + + Additionally, Math::BigInt can take advantage of other bignum libraries, + such as Math::GMP, which cannot handle big floats with large exponents, + and might simply abort or crash your program, due to their code quality. + + This can be a concern if you want to parse untrusted CBOR. If it is, you + might want to disable decoding of tag 2 (bigint) and 3 (negative bigint) + types. You should also disable types 5 and 265, as these can be slow + even without bigints. - First of all, your CBOR decoder should be secure, that is, should not - have any buffer overflows. Obviously, this module should ensure that and - I am trying hard on making that true, but you never know. - - Second, you need to avoid resource-starving attacks. That means you - should limit the size of CBOR data you accept, or make sure then when - your resources run out, that's just fine (e.g. by using a separate - process that can crash safely). The size of a CBOR string in octets is - usually a good indication of the size of the resources required to - decode it into a Perl structure. While CBOR::XS can check the size of - the CBOR text, it might be too late when you already have it in memory, - so you might want to check the size before you accept the string. - - Third, CBOR::XS recurses using the C stack when decoding objects and - arrays. The C stack is a limited resource: for instance, on my amd64 - machine with 8MB of stack size I can decode around 180k nested arrays - but only 14k nested CBOR objects (due to perl itself recursing deeply on - croak to free the temporary). If that is exceeded, the program crashes. - To be conservative, the default nesting limit is set to 512. If your - process has a smaller stack, you should adjust this setting accordingly - with the "max_depth" method. - - Something else could bomb you, too, that I forgot to think of. In that - case, you get to keep the pieces. I am always open for hints, though... - - Also keep in mind that CBOR::XS might leak contents of your Perl data - structures in its error messages, so when you serialise sensitive - information you might want to make sure that exceptions thrown by - CBOR::XS will not end up in front of untrusted eyes. + Disabling bigints will also partially or fully disable types that rely + on them, e.g. rational numbers that use bignums. CBOR IMPLEMENTATION NOTES This section contains some random implementation notes. They do not @@ -843,9 +1116,10 @@ On perls that were built without 64 bit integer support (these are rare nowadays, even on 32 bit architectures, as all major Perl distributions are built with 64 bit integer support), support for any kind of 64 bit - integer in CBOR is very limited - most likely, these 64 bit values will - be truncated, corrupted, or otherwise not decoded correctly. This also - includes string, array and map sizes that are stored as 64 bit integers. + value in CBOR is very limited - most likely, these 64 bit values will be + truncated, corrupted, or otherwise not decoded correctly. This also + includes string, float, array and map sizes that are stored as 64 bit + integers. THREADS This module is *not* guaranteed to be thread safe and there are no plans