[ViewVC] Diff of: cvs/CBOR-XS/XS.pm

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.56 by root, Mon Apr 25 21:44:13 2016 UTC vs.
Revision 1.70 by root, Sat Nov 9 07:30:36 2019 UTC

 with the added ability of supporting serialisation of Perl objects. (JSON
 often compresses better than CBOR though, so if you plan to compress the
 data later and speed is less important you might want to compare both
 formats first).
+The primary goal of this module is to be I<correct> and the secondary goal
+is to be I<fast>. To reach the latter goal it was written in C.
 To give you a general idea about speed, with texts in the megabyte range,
 C<CBOR::XS> usually encodes roughly twice as fast as L<Storable> or
 L<JSON::XS> and decodes about 15%-30% faster than those. The shorter the
 data, the worse L<Storable> performs in comparison.
 In addition to the core CBOR data format, this module implements a
 number of extensions, to support cyclic and shared data structures
 (see C<allow_sharing> and C<allow_cycles>), string deduplication (see
 C<pack_strings>) and scalar references (always enabled).
-The primary goal of this module is to be I<correct> and the secondary goal
-is to be I<fast>. To reach the latter goal it was written in C.
 See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
 vice versa.
 =cut
 package CBOR::XS;
 use common::sense;
-our $VERSION = 1.5;
+our $VERSION = 1.71;
 our @ISA = qw(Exporter);
 our @EXPORT = qw(encode_cbor decode_cbor);
 use Exporter;
 The mutators for flags all return the CBOR object again and thus calls can
 be chained:
    my $cbor = CBOR::XS->new->encode ({a => [1,2]});
+=item $cbor = new_safe CBOR::XS
+Create a new, safe/secure CBOR::XS object. This is similar to C<new>,
+but configures the coder object to be safe to use with untrusted
+data. Currently, this is equivalent to:
+   my $cbor = CBOR::XS
+      ->new
+      ->forbid_objects
+      ->filter (\&CBOR::XS::safe_filter)
+      ->max_size (1e8);
+But is more future proof (it is better to crash because of a change than
+to be exploited in other ways).
+=cut
+sub new_safe {
+   CBOR::XS
+      ->new
+      ->forbid_objects
+      ->filter (\&CBOR::XS::safe_filter)
+      ->max_size (1e8)
+}
 =item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
 =item $max_depth = $cbor->get_max_depth
 Note that nesting is implemented by recursion in C. The default value has
 been chosen to be as large as typical operating systems allow without
 crashing.
-See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
+See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
 =item $cbor = $cbor->max_size ([$maximum_string_size])
 =item $max_size = $cbor->get_max_size
 effect on C<encode> (yet).
 If no argument is given, the limit check will be deactivated (same as when
 C<0> is specified).
-See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
+See L<SECURITY CONSIDERATIONS>, below, for more info on why this is useful.
 =item $cbor = $cbor->allow_unknown ([$enable])
 =item $enabled = $cbor->get_allow_unknown
 reference to the earlier value.
 This means that such values will only be encoded once, and will not result
 in a deep cloning of the value on decode, in decoders supporting the value
 sharing extension. This also makes it possible to encode cyclic data
-structures (which need C<allow_cycles> to ne enabled to be decoded by this
+structures (which need C<allow_cycles> to be enabled to be decoded by this
 module).
 It is recommended to leave it off unless you know your
 communication partner supports the value sharing extensions to CBOR
 (L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
 resulting data structure might be unusable.
 Detecting shared values incurs a runtime overhead when values are encoded
 that have a reference counter large than one, and might unnecessarily
-increase the encoded size, as potentially shared values are encode as
+increase the encoded size, as potentially shared values are encoded as
 shareable whether or not they are actually shared.
 At the moment, only targets of references can be shared (e.g. scalars,
 arrays or hashes pointed to by a reference). Weirder constructs, such as
 an array with multiple "copies" of the I<same> string, which are hard but
 throwing an error.
 This option does not affect C<encode> in any way - shared values and
 references will always be encoded properly if present.
+=item $cbor = $cbor->forbid_objects ([$enable])
+=item $enabled = $cbor->get_forbid_objects
+Disables the use of the object serialiser protocol.
+If C<$enable> is true (or missing), then C<encode> will will throw an
+exception when it encounters perl objects that would be encoded using the
+perl-object tag (26). When C<decode> encounters such tags, it will fall
+back to the general filter/tagged logic as if this were an unknown tag (by
+default resulting in a C<CBOR::XC::Tagged> object).
+If C<$enable> is false (the default), then C<encode> will use the
+L<Types::Serialiser> object serialisation protocol to serialise objects
+into perl-object tags, and C<decode> will do the same to decode such tags.
+See L<SECURITY CONSIDERATIONS>, below, for more info on why forbidding this
+protocol can be useful.
 =item $cbor = $cbor->pack_strings ([$enable])
 =item $enabled = $cbor->get_pack_strings
 If C<$enable> is true (or missing), then C<encode> will try not to encode
 replace the tagged value in the decoded data structure, or no values,
 which will result in default handling, which currently means the decoder
 creates a C<CBOR::XS::Tagged> object to hold the tag and the value.
 When the filter is cleared (the default state), the default filter
-function, C<CBOR::XS::default_filter>, is used. This function simply looks
+function, C<CBOR::XS::default_filter>, is used. This function simply
-up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists it must be
+looks up the tag in the C<%CBOR::XS::FILTER> hash. If an entry exists
-a code reference that is called with tag and value, and is responsible for
+it must be a code reference that is called with tag and value, and is
-decoding the value. If no entry exists, it returns no values.
+responsible for decoding the value. If no entry exists, it returns no
+values. C<CBOR::XS> provides a number of default filter functions already,
+the the C<%CBOR::XS::FILTER> hash can be freely extended with more.
+C<CBOR::XS> additionally provides an alternative filter function that is
+supposed to be safe to use with untrusted data (which the default filter
+might not), called C<CBOR::XS::safe_filter>, which works the same as
+the C<default_filter> but uses the C<%CBOR::XS::SAFE_FILTER> variable
+instead. It is prepopulated with the tag decoding functions that are
+deemed safe (basically the same as C<%CBOR::XS::FILTER> without all
+the bignum tags), and can be extended by user code as wlel, although,
+obviously, one should be very careful about adding decoding functions
+here, since the expectation is that they are safe to use on untrusted
+data, after all.
 Example: decode all tags not handled internally into C<CBOR::XS::Tagged>
 objects, with no other special handling (useful when working with
 potentially "unsafe" CBOR data).
       my ($tag, $value);
       "tag 1347375694 value $value"
    };
+Example: provide your own filter function that looks up tags in your own
+hash:
+   my %my_filter = (
+      998347484 => sub {
+         my ($tag, $value);
+         "tag 998347484 value $value"
+      };
+   );
+   my $coder = CBOR::XS->new->filter (sub {
+      &{ $my_filter{$_[0]} or return }
+   });
+Example: use the safe filter function (see L<SECURITY CONSIDERATIONS> for
+more considerations on security).
+   CBOR::XS->new->filter (\&CBOR::XS::safe_filter)->decode ($cbor_data);
 =item $cbor_data = $cbor->encode ($perl_scalar)
 Converts the given Perl data structure (a scalar value) to its CBOR
 representation.
 when there is trailing garbage after the CBOR string, it will silently
 stop parsing there and return the number of characters consumed so far.
 This is useful if your CBOR texts are not delimited by an outer protocol
 and you need to know where the first CBOR string ends amd the next one
-starts.
+starts - CBOR strings are self-delimited, so it is possible to concatenate
+CBOR strings without any delimiters or size fields and recover their data.
    CBOR::XS->new->decode_prefix ("......")
    => ("...", 3)
 =back
 Resets the incremental decoder. This throws away any saved state, so that
 subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
 a new CBOR value from the beginning of the C<$buffer> again.
-This method can be caled at any time, but it I<must> be called if you want
+This method can be called at any time, but it I<must> be called if you want
 to change your C<$buffer> or there was a decoding error and you want to
 reuse the C<$cbor> object for future incremental parsings.
 =back
 objects. The corresponding C<Math::BigFloat::TO_CBOR> method I<always>
 encodes into a decimal fraction (either tag 4 or 264).
 NaN and infinities are not encoded properly, as they cannot be represented
 in CBOR.
+See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
+=item 30 (rational numbers)
+These tags are decoded into L<Math::BigRat> objects. The corresponding
+C<Math::BigRat::TO_CBOR> method encodes rational numbers with denominator
+C<1> via their numerator only, i.e., they become normal integers or
+C<bignums>.
 See L<BIGNUM SECURITY CONSIDERATIONS> for more info.
 =item 21, 22, 23 (expected later JSON conversion)
 CBOR intact.
 =head1 SECURITY CONSIDERATIONS
-When you are using CBOR in a protocol, talking to untrusted potentially
+Tl;dr... if you want to decode or encode CBOR from untrusted sources, you
-hostile creatures requires relatively few measures.
+should start with a coder object created via C<new_safe> (which implements
+the mitigations explained below):
+   my $coder = CBOR::XS->new_safe;
+   my $data = $coder->decode ($cbor_text);
+   my $cbor = $coder->encode ($data);
+Longer version: When you are using CBOR in a protocol, talking to
+untrusted potentially hostile creatures requires some thought:
+=over 4
+=item Security of the CBOR decoder itself
-First of all, your CBOR decoder should be secure, that is, should not have
+First and foremost, your CBOR decoder should be secure, that is, should
+not have any buffer overflows or similar bugs that could potentially be
-any buffer overflows. Obviously, this module should ensure that and I am
+exploited. Obviously, this module should ensure that and I am trying hard
-trying hard on making that true, but you never know.
+on making that true, but you never know.
+=item CBOR::XS can invoke almost arbitrary callbacks during decoding
+CBOR::XS supports object serialisation - decoding CBOR can cause calls
+to I<any> C<THAW> method in I<any> package that exists in your process
+(that is, CBOR::XS will not try to load modules, but any existing C<THAW>
+method or function can be called, so they all have to be secure).
+Less obviously, it will also invoke C<TO_CBOR> and C<FREEZE> methods -
+even if all your C<THAW> methods are secure, encoding data structures from
+untrusted sources can invoke those and trigger bugs in those.
+So, if you are not sure about the security of all the modules you
+have loaded (you shouldn't), you should disable this part using
+C<forbid_objects> or using C<new_safe>.
+=item CBOR can be extended with tags that call library code
+CBOR can be extended with tags, and C<CBOR::XS> has a registry of
+conversion functions for many existing tags that can be extended via
+third-party modules (see the C<filter> method).
+If you don't trust these, you should configure the "safe" filter function,
+C<CBOR::XS::safe_filter> (C<new_safe> does this), which by default only
+includes conversion functions that are considered "safe" by the author
+(but again, they can be extended by third party modules).
+Depending on your level of paranoia, you can use the "safe" filter:
+   $cbor->filter (\&CBOR::XS::safe_filter);
+... your own filter...
+   $cbor->filter (sub { ... do your stuffs here ... });
+... or even no filter at all, disabling all tag decoding:
+   $cbor->filter (sub { });
+This is never a problem for encoding, as the tag mechanism only exists in
+CBOR texts.
+=item Resource-starving attacks: object memory usage
-Second, you need to avoid resource-starving attacks. That means you should
+You need to avoid resource-starving attacks. That means you should limit
-limit the size of CBOR data you accept, or make sure then when your
+the size of CBOR data you accept, or make sure then when your resources
-resources run out, that's just fine (e.g. by using a separate process that
+run out, that's just fine (e.g. by using a separate process that can
-can crash safely). The size of a CBOR string in octets is usually a good
+crash safely). The size of a CBOR string in octets is usually a good
 indication of the size of the resources required to decode it into a Perl
-structure. While CBOR::XS can check the size of the CBOR text, it might be
+structure. While CBOR::XS can check the size of the CBOR text (using
-too late when you already have it in memory, so you might want to check
+C<max_size> - done by C<new_safe>), it might be too late when you already
-the size before you accept the string.
+have it in memory, so you might want to check the size before you accept
+the string.
+As for encoding, it is possible to construct data structures that are
+relatively small but result in large CBOR texts (for example by having an
+array full of references to the same big data structure, which will all be
+deep-cloned during encoding by default). This is rarely an actual issue
+(and the worst case is still just running out of memory), but you can
+reduce this risk by using C<allow_sharing>.
+=item Resource-starving attacks: stack overflows
-Third, CBOR::XS recurses using the C stack when decoding objects and
+CBOR::XS recurses using the C stack when decoding objects and arrays. The
-arrays. The C stack is a limited resource: for instance, on my amd64
+C stack is a limited resource: for instance, on my amd64 machine with 8MB
-machine with 8MB of stack size I can decode around 180k nested arrays but
+of stack size I can decode around 180k nested arrays but only 14k nested
-only 14k nested CBOR objects (due to perl itself recursing deeply on croak
+CBOR objects (due to perl itself recursing deeply on croak to free the
-to free the temporary). If that is exceeded, the program crashes. To be
+temporary). If that is exceeded, the program crashes. To be conservative,
-conservative, the default nesting limit is set to 512. If your process
+the default nesting limit is set to 512. If your process has a smaller
-has a smaller stack, you should adjust this setting accordingly with the
+stack, you should adjust this setting accordingly with the C<max_depth>
-C<max_depth> method.
+method.
+=item Resource-starving attacks: CPU en-/decoding complexity
+CBOR::XS will use the L<Math::BigInt>, L<Math::BigFloat> and
+L<Math::BigRat> libraries to represent encode/decode bignums. These can be
+very slow (as in, centuries of CPU time) and can even crash your program
+(and are generally not very trustworthy). See the next section on bignum
+security for details.
+=item Data breaches: leaking information in error messages
+CBOR::XS might leak contents of your Perl data structures in its error
+messages, so when you serialise sensitive information you might want to
+make sure that exceptions thrown by CBOR::XS will not end up in front of
+untrusted eyes.
+=item Something else...
 Something else could bomb you, too, that I forgot to think of. In that
 case, you get to keep the pieces. I am always open for hints, though...
-Also keep in mind that CBOR::XS might leak contents of your Perl data
+=back
-structures in its error messages, so when you serialise sensitive
-information you might want to make sure that exceptions thrown by CBOR::XS
-will not end up in front of untrusted eyes.
 =head1 BIGNUM SECURITY CONSIDERATIONS
 CBOR::XS provides a C<TO_CBOR> method for both L<Math::BigInt> and
 L<Math::BigFloat> that tries to encode the number in the simplest possible
 way, that is, either a CBOR integer, a CBOR bigint/decimal fraction (tag
-4) or an arbitrary-exponent decimal fraction (tag 264).
+4) or an arbitrary-exponent decimal fraction (tag 264). Rational numbers
+(L<Math::BigRat>, tag 30) can also contain bignums as members.
-It will also understand base-2 bigfloat or arbitrary-exponent bigfloats
+CBOR::XS will also understand base-2 bigfloat or arbitrary-exponent
-(tags 5 and 265), but it will never generate these on its own.
+bigfloats (tags 5 and 265), but it will never generate these on its own.
 Using the built-in L<Math::BigInt::Calc> support, encoding and decoding
 decimal fractions is generally fast. Decoding bigints can be slow for very
-big numbers, and decoding bigfloats or arbitrary-exponent bigfloats can be
+big numbers (tens of thousands of digits, something that could potentially
-extremely slow (minutes, decades) for large exponents.
+be caught by limiting the size of CBOR texts), and decoding bigfloats or
+arbitrary-exponent bigfloats can be I<extremely> slow (minutes, decades)
+for large exponents (roughly 40 bit and longer).
 Additionally, L<Math::BigInt> can take advantage of other bignum
-libraries, such as L<Math::GMP>, which cannot handle big
+libraries, such as L<Math::GMP>, which cannot handle big floats with large
-floats with large exponents, and might simply abort or crash your program,
+exponents, and might simply abort or crash your program, due to their code
-due to their code quality.
+quality.
 This can be a concern if you want to parse untrusted CBOR. If it is, you
-need to disable decoding of tag 2 (bigint) and 3 (negative bigint) types,
+might want to disable decoding of tag 2 (bigint) and 3 (negative bigint)
-which will also disable bigfloat support (to be sure, you can also disable
+types. You should also disable types 5 and 265, as these can be slow even
-types 4, 5, 264 and 265).
+without bigints.
+Disabling bigints will also partially or fully disable types that rely on
+them, e.g. rational numbers that use bignums.
 =head1 CBOR IMPLEMENTATION NOTES
 This section contains some random implementation notes. They do not
 Please refrain from using rt.cpan.org or any other bug reporting
 service. I put the contact address into my modules for a reason.
 =cut
+# clumsy and slow hv_store-in-hash helper function
+sub _hv_store {
+   $_[0]{$_[1]} = $_[2];
+}
 our %FILTER = (
    0 => sub { # rfc4287 datetime, utf-8
       require Time::Piece;
       # Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
       scalar Time::Piece::gmtime (pop)
    },
    2 => sub { # pos bigint
       require Math::BigInt;
-      Math::BigInt->from_hex ("0x" . unpack "H*", pop)
+      Math::BigInt->new ("0x" . unpack "H*", pop)
    },
    3 => sub { # neg bigint
       require Math::BigInt;
-      -Math::BigInt->from_hex ("0x" . unpack "H*", pop)
+      -Math::BigInt->new ("0x" . unpack "H*", pop)
    },
    4 => sub { # decimal fraction, array
       require Math::BigFloat;
       Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
    },
+   264 => sub { # decimal fraction with arbitrary exponent
+      require Math::BigFloat;
+      Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
+   },
    5 => sub { # bigfloat, array
       require Math::BigFloat;
       scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
    },
+   265 => sub { # bigfloat with arbitrary exponent
+      require Math::BigFloat;
+      scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
+   },
+   30 => sub { # rational number
+      require Math::BigRat;
+      Math::BigRat->new ("$_[1][0]/$_[1][1]") # separate parameters only work in recent versons
+   },
    21 => sub { pop }, # expected conversion to base64url encoding
    22 => sub { pop }, # expected conversion to base64 encoding
    23 => sub { pop }, # expected conversion to base16 encoding
    # 24 # embedded cbor, byte string
    # 33 # base64url rfc4648, utf-8
    # 34 # base64 rfc46484, utf-8
    # 35 # regex pcre/ecma262, utf-8
    # 36 # mime message rfc2045, utf-8
-   264 => sub { # decimal fraction with arbitrary exponent
-      require Math::BigFloat;
-      Math::BigFloat->new ($_[1][1] . "E" . $_[1][0])
-   },
-   265 => sub { # bigfloat with arbitrary exponent
-      require Math::BigFloat;
-      scalar Math::BigFloat->new ($_[1][1]) * Math::BigFloat->new (2)->bpow ($_[1][0])
-   },
 );
-sub CBOR::XS::default_filter {
+sub default_filter {
    &{ $FILTER{$_[0]} or return }
+}
+our %SAFE_FILTER = map { $_ => $FILTER{$_} } 0, 1, 21, 22, 23, 32;
+sub safe_filter {
+   &{ $SAFE_FILTER{$_[0]} or return }
 }
 sub URI::TO_CBOR {
    my $uri = $_[0]->as_string;
    utf8::upgrade $uri;
    -9223372036854775808 <= $e && $e <= 18446744073709551615
       ? tag 4, [$e->numify, $m]
       : tag 264, [$e, $m]
 }
+sub Math::BigRat::TO_CBOR {
+   my ($n, $d) = $_[0]->parts;
+   # older versions of BigRat need *1, as they not always return numbers
+   $d*1 == 1
+      ? $n*1
+      : tag 30, [$n*1, $d*1]
+}
 sub Time::Piece::TO_CBOR {
    tag 1, 0 + $_[0]->epoch
 }
 XSLoader::load "CBOR::XS", $VERSION;

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing CBOR-XS/XS.pm (file contents): Revision 1.56 by root, Mon Apr 25 21:44:13 2016 UTC vs. Revision 1.70 by root, Sat Nov 9 07:30:36 2019 UTC

Diff Legend

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.56 by root, Mon Apr 25 21:44:13 2016 UTC vs.
Revision 1.70 by root, Sat Nov 9 07:30:36 2019 UTC