CBOR-XS/XS.pm

=head1 NAME

CBOR::XS - Concise Binary Object Representation (CBOR, RFC7049)

=encoding utf-8

=head1 SYNOPSIS

 use CBOR::XS;

 $binary_cbor_data = encode_cbor $perl_value;
 $perl_value       = decode_cbor $binary_cbor_data;

 # OO-interface

 $coder = CBOR::XS->new;
 #TODO

=head1 DESCRIPTION

WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA AND
EAT YOUR CHILDREN!

This module converts Perl data structures to CBOR and vice versa. Its
primary goal is to be I<correct> and its secondary goal is to be
I<fast>. To reach the latter goal it was written in C.

See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
vice versa.

=cut

package CBOR::XS;

use common::sense;

our $VERSION = 0.01;
our @ISA = qw(Exporter);

our @EXPORT = qw(encode_cbor decode_cbor);

use Exporter;
use XSLoader;

=head1 FUNCTIONAL INTERFACE

The following convenience methods are provided by this module. They are
exported by default:

=over 4

=item $cbor_data = encode_cbor $perl_scalar

Converts the given Perl data structure to CBOR representation. Croaks on
error.

=item $perl_scalar = decode_cbor $cbor_data

The opposite of C<encode_cbor>: expects a valid CBOR string to parse,
returning the resulting perl scalar. Croaks on error.

=back


=head1 OBJECT-ORIENTED INTERFACE

The object oriented interface lets you configure your own encoding or
decoding style, within the limits of supported formats.

=over 4

=item $cbor = new CBOR::XS

Creates a new CBOR::XS object that can be used to de/encode CBOR
strings. All boolean flags described below are by default I<disabled>.

The mutators for flags all return the CBOR object again and thus calls can
be chained:

#TODO
   my $cbor = CBOR::XS->new->encode ({a => [1,2]});

=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])

=item $max_depth = $cbor->get_max_depth

Sets the maximum nesting level (default C<512>) accepted while encoding
or decoding. If a higher nesting level is detected in CBOR data or a Perl
data structure, then the encoder and decoder will stop and croak at that
point.

Nesting level is defined by number of hash- or arrayrefs that the encoder
needs to traverse to reach a given point or the number of C<{> or C<[>
characters without their matching closing parenthesis crossed to reach a
given character in a string.

Setting the maximum depth to one disallows any nesting, so that ensures
that the object is only a single hash/object or array.

If no argument is given, the highest possible setting will be used, which
is rarely useful.

Note that nesting is implemented by recursion in C. The default value has
been chosen to be as large as typical operating systems allow without
crashing.

See SECURITY CONSIDERATIONS, below, for more info on why this is useful.

=item $cbor = $cbor->max_size ([$maximum_string_size])

=item $max_size = $cbor->get_max_size

Set the maximum length a CBOR string may have (in bytes) where decoding
is being attempted. The default is C<0>, meaning no limit. When C<decode>
is called on a string that is longer then this many bytes, it will not
attempt to decode the string but throw an exception. This setting has no
effect on C<encode> (yet).

If no argument is given, the limit check will be deactivated (same as when
C<0> is specified).

See SECURITY CONSIDERATIONS, below, for more info on why this is useful.

=item $cbor_data = $cbor->encode ($perl_scalar)

Converts the given Perl data structure (a scalar value) to its CBOR
representation.

=item $perl_scalar = $cbor->decode ($cbor_data)

The opposite of C<encode>: expects CBOR data and tries to parse it,
returning the resulting simple scalar or reference. Croaks on error.

=item ($perl_scalar, $octets) = $cbor->decode_prefix ($cbor_data)

This works like the C<decode> method, but instead of raising an exception
when there is trailing garbage after the CBOR string, it will silently
stop parsing there and return the number of characters consumed so far.

This is useful if your CBOR texts are not delimited by an outer protocol
and you need to know where the first CBOR string ends amd the next one
starts.

   CBOR::XS->new->decode_prefix ("......")
   => ("...", 3)

=back


=head1 MAPPING

This section describes how CBOR::XS maps Perl values to CBOR values and
vice versa. These mappings are designed to "do the right thing" in most
circumstances automatically, preserving round-tripping characteristics
(what you put in comes out as something equivalent).

For the more enlightened: note that in the following descriptions,
lowercase I<perl> refers to the Perl interpreter, while uppercase I<Perl>
refers to the abstract Perl language itself.


=head2 CBOR -> PERL

=over 4

=item True, False

These CBOR values become C<CBOR::XS::true> and C<CBOR::XS::false>,
respectively. They are overloaded to act almost exactly like the numbers
C<1> and C<0>. You can check whether a scalar is a CBOR boolean by using
the C<CBOR::XS::is_bool> function.

=item Null, Undefined

CBOR Null and Undefined values becomes C<undef> in Perl (in the future,
Undefined may raise an exception).

=back


=head2 PERL -> CBOR

The mapping from Perl to CBOR is slightly more difficult, as Perl is a
truly typeless language, so we can only guess which CBOR type is meant by
a Perl value.

=over 4

=item hash references

Perl hash references become CBOR maps. As there is no inherent ordering
in hash keys (or CBOR maps), they will usually be encoded in a
pseudo-random order.

=item array references

Perl array references become CBOR arrays.

=item other references

Other unblessed references are generally not allowed and will cause an
exception to be thrown, except for references to the integers C<0> and
C<1>, which get turned into C<False> and C<True> in CBOR.

=item CBOR::XS::true, CBOR::XS::false

These special values become CBOR True and CBOR False values,
respectively. You can also use C<\1> and C<\0> directly if you want.

=item blessed objects

Blessed objects are not directly representable in CBOR. TODO
See the
C<allow_blessed> and C<convert_blessed> methods on various options on
how to deal with this: basically, you can choose between throwing an
exception, encoding the reference as if it weren't blessed, or provide
your own serialiser method.

=item simple scalars

TODO
Simple Perl scalars (any scalar that is not a reference) are the most
difficult objects to encode: CBOR::XS will encode undefined scalars as
CBOR C<Null> values, scalars that have last been used in a string context
before encoding as CBOR strings, and anything else as number value:

   # dump as number
   encode_cbor [2]                      # yields [2]
   encode_cbor [-3.0e17]                # yields [-3e+17]
   my $value = 5; encode_cbor [$value]  # yields [5]

   # used as string, so dump as string
   print $value;
   encode_cbor [$value]                 # yields ["5"]

   # undef becomes null
   encode_cbor [undef]                  # yields [null]

You can force the type to be a CBOR string by stringifying it:

   my $x = 3.1; # some variable containing a number
   "$x";        # stringified
   $x .= "";    # another, more awkward way to stringify
   print $x;    # perl does it for you, too, quite often

You can force the type to be a CBOR number by numifying it:

   my $x = "3"; # some variable containing a string
   $x += 0;     # numify it, ensuring it will be dumped as a number
   $x *= 1;     # same thing, the choice is yours.

You can not currently force the type in other, less obscure, ways. Tell me
if you need this capability (but don't forget to explain why it's needed
:).

Note that numerical precision has the same meaning as under Perl (so
binary to decimal conversion follows the same rules as in Perl, which
can differ to other languages). Also, your perl interpreter might expose
extensions to the floating point numbers of your platform, such as
infinities or NaN's - these cannot be represented in CBOR, and it is an
error to pass those in.

=back


=head2 CBOR and JSON

TODO


=head1 SECURITY CONSIDERATIONS

When you are using CBOR in a protocol, talking to untrusted potentially
hostile creatures requires relatively few measures.

First of all, your CBOR decoder should be secure, that is, should not have
any buffer overflows. Obviously, this module should ensure that and I am
trying hard on making that true, but you never know.

Second, you need to avoid resource-starving attacks. That means you should
limit the size of CBOR data you accept, or make sure then when your
resources run out, that's just fine (e.g. by using a separate process that
can crash safely). The size of a CBOR string in octets is usually a good
indication of the size of the resources required to decode it into a Perl
structure. While CBOR::XS can check the size of the CBOR text, it might be
too late when you already have it in memory, so you might want to check
the size before you accept the string.

Third, CBOR::XS recurses using the C stack when decoding objects and
arrays. The C stack is a limited resource: for instance, on my amd64
machine with 8MB of stack size I can decode around 180k nested arrays but
only 14k nested CBOR objects (due to perl itself recursing deeply on croak
to free the temporary). If that is exceeded, the program crashes. To be
conservative, the default nesting limit is set to 512. If your process
has a smaller stack, you should adjust this setting accordingly with the
C<max_depth> method.

Something else could bomb you, too, that I forgot to think of. In that
case, you get to keep the pieces. I am always open for hints, though...

Also keep in mind that CBOR::XS might leak contents of your Perl data
structures in its error messages, so when you serialise sensitive
information you might want to make sure that exceptions thrown by CBOR::XS
will not end up in front of untrusted eyes.

=head1 CBOR IMPLEMENTATION NOTES

This section contains some random implementation notes. They do not
describe guaranteed behaviour, but merely behaviour as-is implemented
right now.

64 bit integers are only properly decoded when Perl was built with 64 bit
support.

Strings and arrays are encoded with a definite length. Hashes as well,
unless they are tied (or otherwise magical).

Only the double data type is supported for NV data types - when Perl uses
long double to represent floating point values, they might not be encoded
properly. Half precision types are accepted, but not encoded.

Strict mode and canonical mode are not implemented.


=head1 THREADS

This module is I<not> guaranteed to be thread safe and there are no
plans to change this until Perl gets thread support (as opposed to the
horribly slow so-called "threads" which are simply slow and bloated
process simulations - use fork, it's I<much> faster, cheaper, better).

(It might actually work, but you have been warned).


=head1 BUGS

While the goal of this module is to be correct, that unfortunately does
not mean it's bug-free, only that I think its design is bug-free. If you
keep reporting bugs they will be fixed swiftly, though.

Please refrain from using rt.cpan.org or any other bug reporting
service. I put the contact address into my modules for a reason.

=cut

our $true  = do { bless \(my $dummy = 1), "CBOR::XS::Boolean" };
our $false = do { bless \(my $dummy = 0), "CBOR::XS::Boolean" };

sub true()  { $true  }
sub false() { $false }

sub is_bool($) {
   UNIVERSAL::isa $_[0], "CBOR::XS::Boolean"
#      or UNIVERSAL::isa $_[0], "CBOR::Literal"
}

XSLoader::load "CBOR::XS", $VERSION;

package CBOR::XS::Boolean;

use overload
   "0+"     => sub { ${$_[0]} },
   "++"     => sub { $_[0] = ${$_[0]} + 1 },
   "--"     => sub { $_[0] = ${$_[0]} - 1 },
   fallback => 1;

1;

=head1 SEE ALSO

The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,
serialisation.

=head1 AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://home.schmorp.de/

=cut

Revision:	1.2
Committed:	Sat Oct 26 10:41:12 2013 UTC (10 years, 6 months ago) by root
Branch:	MAIN
Changes since 1.1:	+3 -2 lines
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3			CBOR::XS - Concise Binary Object Representation (CBOR, RFC7049)
4
5			=encoding utf-8
6
7			=head1 SYNOPSIS
8
9			use CBOR::XS;
10
11			$binary_cbor_data = encode_cbor $perl_value;
12			$perl_value = decode_cbor $binary_cbor_data;
13
14			# OO-interface
15
16			$coder = CBOR::XS->new;
17			#TODO
18
19			=head1 DESCRIPTION
20
21			WARNING! THIS IS A PRE-ALPHA RELEASE! IT WILL CRASH, CORRUPT YOUR DATA AND
22			EAT YOUR CHILDREN!
23
24			This module converts Perl data structures to CBOR and vice versa. Its
25			primary goal is to be I<correct> and its secondary goal is to be
26			I<fast>. To reach the latter goal it was written in C.
27
28			See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
29			vice versa.
30
31			=cut
32
33			package CBOR::XS;
34
35			use common::sense;
36
37			our $VERSION = 0.01;
38			our @ISA = qw(Exporter);
39
40			our @EXPORT = qw(encode_cbor decode_cbor);
41
42			use Exporter;
43			use XSLoader;
44
45			=head1 FUNCTIONAL INTERFACE
46
47			The following convenience methods are provided by this module. They are
48			exported by default:
49
50			=over 4
51
52			=item $cbor_data = encode_cbor $perl_scalar
53
54			Converts the given Perl data structure to CBOR representation. Croaks on
55			error.
56
57			=item $perl_scalar = decode_cbor $cbor_data
58
59			The opposite of C<encode_cbor>: expects a valid CBOR string to parse,
60			returning the resulting perl scalar. Croaks on error.
61
62			=back
63
64
65			=head1 OBJECT-ORIENTED INTERFACE
66
67			The object oriented interface lets you configure your own encoding or
68			decoding style, within the limits of supported formats.
69
70			=over 4
71
72			=item $cbor = new CBOR::XS
73
74			Creates a new CBOR::XS object that can be used to de/encode CBOR
75			strings. All boolean flags described below are by default I<disabled>.
76
77			The mutators for flags all return the CBOR object again and thus calls can
78			be chained:
79
80			#TODO
81			my $cbor = CBOR::XS->new->encode ({a => [1,2]});
82
83			=item $cbor = $cbor->max_depth ([$maximum_nesting_depth])
84
85			=item $max_depth = $cbor->get_max_depth
86
87			Sets the maximum nesting level (default C<512>) accepted while encoding
88			or decoding. If a higher nesting level is detected in CBOR data or a Perl
89			data structure, then the encoder and decoder will stop and croak at that
90			point.
91
92			Nesting level is defined by number of hash- or arrayrefs that the encoder
93			needs to traverse to reach a given point or the number of C<{> or C<[>
94			characters without their matching closing parenthesis crossed to reach a
95			given character in a string.
96
97			Setting the maximum depth to one disallows any nesting, so that ensures
98			that the object is only a single hash/object or array.
99
100			If no argument is given, the highest possible setting will be used, which
101			is rarely useful.
102
103			Note that nesting is implemented by recursion in C. The default value has
104			been chosen to be as large as typical operating systems allow without
105			crashing.
106
107			See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
108
109			=item $cbor = $cbor->max_size ([$maximum_string_size])
110
111			=item $max_size = $cbor->get_max_size
112
113			Set the maximum length a CBOR string may have (in bytes) where decoding
114			is being attempted. The default is C<0>, meaning no limit. When C<decode>
115			is called on a string that is longer then this many bytes, it will not
116			attempt to decode the string but throw an exception. This setting has no
117			effect on C<encode> (yet).
118
119			If no argument is given, the limit check will be deactivated (same as when
120			C<0> is specified).
121
122			See SECURITY CONSIDERATIONS, below, for more info on why this is useful.
123
124			=item $cbor_data = $cbor->encode ($perl_scalar)
125
126			Converts the given Perl data structure (a scalar value) to its CBOR
127			representation.
128
129			=item $perl_scalar = $cbor->decode ($cbor_data)
130
131			The opposite of C<encode>: expects CBOR data and tries to parse it,
132			returning the resulting simple scalar or reference. Croaks on error.
133
134			=item ($perl_scalar, $octets) = $cbor->decode_prefix ($cbor_data)
135
136			This works like the C<decode> method, but instead of raising an exception
137			when there is trailing garbage after the CBOR string, it will silently
138			stop parsing there and return the number of characters consumed so far.
139
140			This is useful if your CBOR texts are not delimited by an outer protocol
141			and you need to know where the first CBOR string ends amd the next one
142			starts.
143
144			CBOR::XS->new->decode_prefix ("......")
145			=> ("...", 3)
146
147			=back
148
149
150			=head1 MAPPING
151
152			This section describes how CBOR::XS maps Perl values to CBOR values and
153			vice versa. These mappings are designed to "do the right thing" in most
154			circumstances automatically, preserving round-tripping characteristics
155			(what you put in comes out as something equivalent).
156
157			For the more enlightened: note that in the following descriptions,
158			lowercase I<perl> refers to the Perl interpreter, while uppercase I<Perl>
159			refers to the abstract Perl language itself.
160
161
162			=head2 CBOR -> PERL
163
164			=over 4
165
166			=item True, False
167
168			These CBOR values become C<CBOR::XS::true> and C<CBOR::XS::false>,
169			respectively. They are overloaded to act almost exactly like the numbers
170			C<1> and C<0>. You can check whether a scalar is a CBOR boolean by using
171			the C<CBOR::XS::is_bool> function.
172
173	root	1.2	=item Null, Undefined
174	root	1.1
175	root	1.2	CBOR Null and Undefined values becomes C<undef> in Perl (in the future,
176			Undefined may raise an exception).
177	root	1.1
178			=back
179
180
181			=head2 PERL -> CBOR
182
183			The mapping from Perl to CBOR is slightly more difficult, as Perl is a
184			truly typeless language, so we can only guess which CBOR type is meant by
185			a Perl value.
186
187			=over 4
188
189			=item hash references
190
191			Perl hash references become CBOR maps. As there is no inherent ordering
192			in hash keys (or CBOR maps), they will usually be encoded in a
193			pseudo-random order.
194
195			=item array references
196
197			Perl array references become CBOR arrays.
198
199			=item other references
200
201			Other unblessed references are generally not allowed and will cause an
202			exception to be thrown, except for references to the integers C<0> and
203			C<1>, which get turned into C<False> and C<True> in CBOR.
204
205			=item CBOR::XS::true, CBOR::XS::false
206
207			These special values become CBOR True and CBOR False values,
208			respectively. You can also use C<\1> and C<\0> directly if you want.
209
210			=item blessed objects
211
212			Blessed objects are not directly representable in CBOR. TODO
213			See the
214			C<allow_blessed> and C<convert_blessed> methods on various options on
215			how to deal with this: basically, you can choose between throwing an
216			exception, encoding the reference as if it weren't blessed, or provide
217			your own serialiser method.
218
219			=item simple scalars
220
221			TODO
222			Simple Perl scalars (any scalar that is not a reference) are the most
223			difficult objects to encode: CBOR::XS will encode undefined scalars as
224			CBOR C<Null> values, scalars that have last been used in a string context
225			before encoding as CBOR strings, and anything else as number value:
226
227			# dump as number
228			encode_cbor [2] # yields [2]
229			encode_cbor [-3.0e17] # yields [-3e+17]
230			my $value = 5; encode_cbor [$value] # yields [5]
231
232			# used as string, so dump as string
233			print $value;
234			encode_cbor [$value] # yields ["5"]
235
236			# undef becomes null
237			encode_cbor [undef] # yields [null]
238
239			You can force the type to be a CBOR string by stringifying it:
240
241			my $x = 3.1; # some variable containing a number
242			"$x"; # stringified
243			$x .= ""; # another, more awkward way to stringify
244			print $x; # perl does it for you, too, quite often
245
246			You can force the type to be a CBOR number by numifying it:
247
248			my $x = "3"; # some variable containing a string
249			$x += 0; # numify it, ensuring it will be dumped as a number
250			$x *= 1; # same thing, the choice is yours.
251
252			You can not currently force the type in other, less obscure, ways. Tell me
253			if you need this capability (but don't forget to explain why it's needed
254			:).
255
256			Note that numerical precision has the same meaning as under Perl (so
257			binary to decimal conversion follows the same rules as in Perl, which
258			can differ to other languages). Also, your perl interpreter might expose
259			extensions to the floating point numbers of your platform, such as
260			infinities or NaN's - these cannot be represented in CBOR, and it is an
261			error to pass those in.
262
263			=back
264
265
266			=head2 CBOR and JSON
267
268			TODO
269
270
271			=head1 SECURITY CONSIDERATIONS
272
273			When you are using CBOR in a protocol, talking to untrusted potentially
274			hostile creatures requires relatively few measures.
275
276			First of all, your CBOR decoder should be secure, that is, should not have
277			any buffer overflows. Obviously, this module should ensure that and I am
278			trying hard on making that true, but you never know.
279
280			Second, you need to avoid resource-starving attacks. That means you should
281			limit the size of CBOR data you accept, or make sure then when your
282			resources run out, that's just fine (e.g. by using a separate process that
283			can crash safely). The size of a CBOR string in octets is usually a good
284			indication of the size of the resources required to decode it into a Perl
285			structure. While CBOR::XS can check the size of the CBOR text, it might be
286			too late when you already have it in memory, so you might want to check
287			the size before you accept the string.
288
289			Third, CBOR::XS recurses using the C stack when decoding objects and
290			arrays. The C stack is a limited resource: for instance, on my amd64
291			machine with 8MB of stack size I can decode around 180k nested arrays but
292			only 14k nested CBOR objects (due to perl itself recursing deeply on croak
293			to free the temporary). If that is exceeded, the program crashes. To be
294			conservative, the default nesting limit is set to 512. If your process
295			has a smaller stack, you should adjust this setting accordingly with the
296			C<max_depth> method.
297
298			Something else could bomb you, too, that I forgot to think of. In that
299			case, you get to keep the pieces. I am always open for hints, though...
300
301			Also keep in mind that CBOR::XS might leak contents of your Perl data
302			structures in its error messages, so when you serialise sensitive
303			information you might want to make sure that exceptions thrown by CBOR::XS
304			will not end up in front of untrusted eyes.
305
306			=head1 CBOR IMPLEMENTATION NOTES
307
308			This section contains some random implementation notes. They do not
309			describe guaranteed behaviour, but merely behaviour as-is implemented
310			right now.
311
312			64 bit integers are only properly decoded when Perl was built with 64 bit
313			support.
314
315			Strings and arrays are encoded with a definite length. Hashes as well,
316			unless they are tied (or otherwise magical).
317
318			Only the double data type is supported for NV data types - when Perl uses
319			long double to represent floating point values, they might not be encoded
320			properly. Half precision types are accepted, but not encoded.
321
322			Strict mode and canonical mode are not implemented.
323
324
325			=head1 THREADS
326
327			This module is I<not> guaranteed to be thread safe and there are no
328			plans to change this until Perl gets thread support (as opposed to the
329			horribly slow so-called "threads" which are simply slow and bloated
330			process simulations - use fork, it's I<much> faster, cheaper, better).
331
332			(It might actually work, but you have been warned).
333
334
335			=head1 BUGS
336
337			While the goal of this module is to be correct, that unfortunately does
338			not mean it's bug-free, only that I think its design is bug-free. If you
339			keep reporting bugs they will be fixed swiftly, though.
340
341			Please refrain from using rt.cpan.org or any other bug reporting
342			service. I put the contact address into my modules for a reason.
343
344			=cut
345
346			our $true = do { bless \(my $dummy = 1), "CBOR::XS::Boolean" };
347			our $false = do { bless \(my $dummy = 0), "CBOR::XS::Boolean" };
348
349			sub true() { $true }
350			sub false() { $false }
351
352			sub is_bool($) {
353			UNIVERSAL::isa $_[0], "CBOR::XS::Boolean"
354			# or UNIVERSAL::isa $_[0], "CBOR::Literal"
355			}
356
357			XSLoader::load "CBOR::XS", $VERSION;
358
359			package CBOR::XS::Boolean;
360
361			use overload
362			"0+" => sub { ${$_[0]} },
363			"++" => sub { $_[0] = ${$_[0]} + 1 },
364			"--" => sub { $_[0] = ${$_[0]} - 1 },
365			fallback => 1;
366
367			1;
368
369			=head1 SEE ALSO
370
371			The L<JSON> and L<JSON::XS> modules that do similar, but human-readable,
372			serialisation.
373
374			=head1 AUTHOR
375
376			Marc Lehmann <schmorp@schmorp.de>
377			http://home.schmorp.de/
378
379			=cut
380