[ViewVC] Diff of: cvs/CBOR-XS/XS.pm

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.29 by root, Sat Nov 30 15:23:59 2013 UTC vs.
Revision 1.40 by root, Sun Jan 5 14:24:54 2014 UTC

…		…
48	Regarding compactness, C<CBOR::XS>-encoded data structures are usually	48	Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49	about 20% smaller than the same data encoded as (compact) JSON or	49	about 20% smaller than the same data encoded as (compact) JSON or
50	L<Storable>.	50	L<Storable>.
51		51
52	In addition to the core CBOR data format, this module implements a	52	In addition to the core CBOR data format, this module implements a
53	number of extensions, to support cyclic and shared data structures (see	53	number of extensions, to support cyclic and shared data structures
54	C<allow_sharing>), string deduplication (see C<pack_strings>) and scalar	54	(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55	references (always enabled).	55	C<pack_strings>) and scalar references (always enabled).
56		56
57	The primary goal of this module is to be I<correct> and the secondary goal	57	The primary goal of this module is to be I<correct> and the secondary goal
58	is to be I<fast>. To reach the latter goal it was written in C.	58	is to be I<fast>. To reach the latter goal it was written in C.
59		59
60	See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and	60	See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
…		…
64		64
65	package CBOR::XS;	65	package CBOR::XS;
66		66
67	use common::sense;	67	use common::sense;
68		68
69	our $VERSION = '1.0';	69	our $VERSION = 1.25;
70	our @ISA = qw(Exporter);	70	our @ISA = qw(Exporter);
71		71
72	our @EXPORT = qw(encode_cbor decode_cbor);	72	our @EXPORT = qw(encode_cbor decode_cbor);
73		73
74	use Exporter;	74	use Exporter;
…		…
180	reference to the earlier value.	180	reference to the earlier value.
181		181
182	This means that such values will only be encoded once, and will not result	182	This means that such values will only be encoded once, and will not result
183	in a deep cloning of the value on decode, in decoders supporting the value	183	in a deep cloning of the value on decode, in decoders supporting the value
184	sharing extension. This also makes it possible to encode cyclic data	184	sharing extension. This also makes it possible to encode cyclic data
185	structures.	185	structures (which need C<allow_cycles> to ne enabled to be decoded by this
		186	module).
186		187
187	It is recommended to leave it off unless you know your	188	It is recommended to leave it off unless you know your
188	communication partner supports the value sharing extensions to CBOR	189	communication partner supports the value sharing extensions to CBOR
189	(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the	190	(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
190	resulting data structure might be unusable.	191	resulting data structure might be unusable.
191		192
192	Detecting shared values incurs a runtime overhead when values are encoded	193	Detecting shared values incurs a runtime overhead when values are encoded
193	that have a reference counter large than one, and might unnecessarily	194	that have a reference counter large than one, and might unnecessarily
194	increase the encoded size, as potentially shared values are encode as	195	increase the encoded size, as potentially shared values are encode as
195	sharable whether or not they are actually shared.	196	shareable whether or not they are actually shared.
196		197
197	At the moment, only targets of references can be shared (e.g. scalars,	198	At the moment, only targets of references can be shared (e.g. scalars,
198	arrays or hashes pointed to by a reference). Weirder constructs, such as	199	arrays or hashes pointed to by a reference). Weirder constructs, such as
199	an array with multiple "copies" of the I<same> string, which are hard but	200	an array with multiple "copies" of the I<same> string, which are hard but
200	not impossible to create in Perl, are not supported (this is the same as	201	not impossible to create in Perl, are not supported (this is the same as
…		…
205	structures cannot be encoded in this mode.	206	structures cannot be encoded in this mode.
206		207
207	This option does not affect C<decode> in any way - shared values and	208	This option does not affect C<decode> in any way - shared values and
208	references will always be decoded properly if present.	209	references will always be decoded properly if present.
209		210
		211	=item $cbor = $cbor->allow_cycles ([$enable])
		212
		213	=item $enabled = $cbor->get_allow_cycles
		214
		215	If C<$enable> is true (or missing), then C<decode> will happily decode
		216	self-referential (cyclic) data structures. By default these will not be
		217	decoded, as they need manual cleanup to avoid memory leaks, so code that
		218	isn't prepared for this will not leak memory.
		219
		220	If C<$enable> is false (the default), then C<decode> will throw an error
		221	when it encounters a self-referential/cyclic data structure.
		222
		223	This option does not affect C<encode> in any way - shared values and
		224	references will always be decoded properly if present.
		225
210	=item $cbor = $cbor->pack_strings ([$enable])	226	=item $cbor = $cbor->pack_strings ([$enable])
211		227
212	=item $enabled = $cbor->get_pack_strings	228	=item $enabled = $cbor->get_pack_strings
213		229
214	If C<$enable> is true (or missing), then C<encode> will try not to encode	230	If C<$enable> is true (or missing), then C<encode> will try not to encode
…		…
226	the standard CBOR way.	242	the standard CBOR way.
227		243
228	This option does not affect C<decode> in any way - string references will	244	This option does not affect C<decode> in any way - string references will
229	always be decoded properly if present.	245	always be decoded properly if present.
230		246
		247	=item $cbor = $cbor->validate_utf8 ([$enable])
		248
		249	=item $enabled = $cbor->get_validate_utf8
		250
		251	If C<$enable> is true (or missing), then C<decode> will validate that
		252	elements (text strings) containing UTF-8 data in fact contain valid UTF-8
		253	data (instead of blindly accepting it). This validation obviously takes
		254	extra time during decoding.
		255
		256	The concept of "valid UTF-8" used is perl's concept, which is a superset
		257	of the official UTF-8.
		258
		259	If C<$enable> is false (the default), then C<decode> will blindly accept
		260	UTF-8 data, marking them as valid UTF-8 in the resulting data structure
		261	regardless of whether thats true or not.
		262
		263	Perl isn't too happy about corrupted UTF-8 in strings, but should
		264	generally not crash or do similarly evil things. Extensions might be not
		265	so forgiving, so it's recommended to turn on this setting if you receive
		266	untrusted CBOR.
		267
		268	This option does not affect C<encode> in any way - strings that are
		269	supposedly valid UTF-8 will simply be dumped into the resulting CBOR
		270	string without checking whether that is, in fact, true or not.
		271
231	=item $cbor = $cbor->filter ([$cb->($tag, $value)])	272	=item $cbor = $cbor->filter ([$cb->($tag, $value)])
232		273
233	=item $cb_or_undef = $cbor->get_filter	274	=item $cb_or_undef = $cbor->get_filter
234		275
235	Sets or replaces the tagged value decoding filter (when C<$cb> is	276	Sets or replaces the tagged value decoding filter (when C<$cb> is
…		…
289	and you need to know where the first CBOR string ends amd the next one	330	and you need to know where the first CBOR string ends amd the next one
290	starts.	331	starts.
291		332
292	CBOR::XS->new->decode_prefix ("......")	333	CBOR::XS->new->decode_prefix ("......")
293	=> ("...", 3)	334	=> ("...", 3)
		335
		336	=back
		337
		338	=head2 INCREMENTAL PARSING
		339
		340	In some cases, there is the need for incremental parsing of JSON
		341	texts. While this module always has to keep both CBOR text and resulting
		342	Perl data structure in memory at one time, it does allow you to parse a
		343	CBOR stream incrementally, using a similar to using "decode_prefix" to see
		344	if a full CBOR object is available, but is much more efficient.
		345
		346	It basically works by parsing as much of a CBOR string as possible - if
		347	the CBOR data is not complete yet, the pasrer will remember where it was,
		348	to be able to restart when more data has been accumulated. Once enough
		349	data is available to either decode a complete CBOR value or raise an
		350	error, a real decode will be attempted.
		351
		352	A typical use case would be a network protocol that consists of sending
		353	and receiving CBOR-encoded messages. The solution that works with CBOR and
		354	about anything else is by prepending a length to every CBOR value, so the
		355	receiver knows how many octets to read. More compact (and slightly slower)
		356	would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
		357	a CBOR value ends, and doesn't need an explicit length.
		358
		359	The following methods help with this:
		360
		361	=over 4
		362
		363	=item @decoded = $cbor->incr_parse ($buffer)
		364
		365	This method attempts to decode exactly one CBOR value from the beginning
		366	of the given C<$buffer>. The value is removed from the C<$buffer> on
		367	success. When C<$buffer> doesn't contain a complete value yet, it returns
		368	nothing. Finally, when the C<$buffer> doesn't start with something
		369	that could ever be a valid CBOR value, it raises an exception, just as
		370	C<decode> would. In the latter case the decoder state is undefined and
		371	must be reset before being able to parse further.
		372
		373	This method modifies the C<$buffer> in place. When no CBOR value can be
		374	decoded, the decoder stores the current string offset. On the next call,
		375	continues decoding at the place where it stopped before. For this to make
		376	sense, the C<$buffer> must begin with the same octets as on previous
		377	unsuccessful calls.
		378
		379	You can call this method in scalar context, in which case it either
		380	returns a decoded value or C<undef>. This makes it impossible to
		381	distinguish between CBOR null values (which decode to C<undef>) and an
		382	unsuccessful decode, which is often acceptable.
		383
		384	=item @decoded = $cbor->incr_parse_multiple ($buffer)
		385
		386	Same as C<incr_parse>, but attempts to decode as many CBOR values as
		387	possible in one go, instead of at most one. Calls to C<incr_parse> and
		388	C<incr_parse_multiple> can be interleaved.
		389
		390	=item $cbor->incr_reset
		391
		392	Resets the incremental decoder. This throws away any saved state, so that
		393	subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
		394	a new CBOR value from the beginning of the C<$buffer> again.
		395
		396	This method can be caled at any time, but it I<must> be called if you want
		397	to change your C<$buffer> or there was a decoding error and you want to
		398	reuse the C<$cbor> object for future incremental parsings.
294		399
295	=back	400	=back
296		401
297		402
298	=head1 MAPPING	403	=head1 MAPPING
…		…
704		809
705	These tags are automatically created (and decoded) for serialisable	810	These tags are automatically created (and decoded) for serialisable
706	objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object	811	objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
707	serialisation protocol). See L<OBJECT SERIALISATION> for details.	812	serialisation protocol). See L<OBJECT SERIALISATION> for details.
708		813
709	=item 28, 29 (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)	814	=item 28, 29 (shareable, sharedref, L <http://cbor.schmorp.de/value-sharing>)
710		815
711	These tags are automatically decoded when encountered, resulting in	816	These tags are automatically decoded when encountered (and they do not
		817	result in a cyclic data structure, see C<allow_cycles>), resulting in
712	shared values in the decoded object. They are only encoded, however, when	818	shared values in the decoded object. They are only encoded, however, when
713	C<allow_sharable> is enabled.	819	C<allow_sharing> is enabled.
		820
		821	Not all shared values can be successfully decoded: values that reference
		822	themselves will I<currently> decode as C<undef> (this is not the same
		823	as a reference pointing to itself, which will be represented as a value
		824	that contains an indirect reference to itself - these will be decoded
		825	properly).
		826
		827	Note that considerably more shared value data structures can be decoded
		828	than will be encoded - currently, only values pointed to by references
		829	will be shared, others will not. While non-reference shared values can be
		830	generated in Perl with some effort, they were considered too unimportant
		831	to be supported in the encoder. The decoder, however, will decode these
		832	values as shared values.
714		833
715	=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)	834	=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)
716		835
717	These tags are automatically decoded when encountered. They are only	836	These tags are automatically decoded when encountered. They are only
718	encoded, however, when C<pack_strings> is enabled.	837	encoded, however, when C<pack_strings> is enabled.
…		…
743	perl core distribution (e.g. L<URI>), it is (currently) up to the user to	862	perl core distribution (e.g. L<URI>), it is (currently) up to the user to
744	provide these modules. The decoding usually fails with an exception if the	863	provide these modules. The decoding usually fails with an exception if the
745	required module cannot be loaded.	864	required module cannot be loaded.
746		865
747	=over 4	866	=over 4
		867
		868	=item 0, 1 (date/time string, seconds since the epoch)
		869
		870	These tags are decoded into L<Time::Piece> objects. The corresponding
		871	C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
		872
		873	The L<Time::Piece> API is generally surprisingly bad, and fractional
		874	seconds are only accidentally kept intact, so watch out. On the plus side,
		875	the module comes with perl since 5.10, which has to count for something.
748		876
749	=item 2, 3 (positive/negative bignum)	877	=item 2, 3 (positive/negative bignum)
750		878
751	These tags are decoded into L<Math::BigInt> objects. The corresponding	879	These tags are decoded into L<Math::BigInt> objects. The corresponding
752	C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR	880	C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
…		…
886	properly. Half precision types are accepted, but not encoded.	1014	properly. Half precision types are accepted, but not encoded.
887		1015
888	Strict mode and canonical mode are not implemented.	1016	Strict mode and canonical mode are not implemented.
889		1017
890		1018
		1019	=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
		1020
		1021	On perls that were built without 64 bit integer support (these are rare
		1022	nowadays, even on 32 bit architectures), support for any kind of 64 bit
		1023	integer in CBOR is very limited - most likely, these 64 bit values will
		1024	be truncated, corrupted, or otherwise not decoded correctly. This also
		1025	includes string, array and map sizes that are stored as 64 bit integers.
		1026
		1027
891	=head1 THREADS	1028	=head1 THREADS
892		1029
893	This module is I<not> guaranteed to be thread safe and there are no	1030	This module is I<not> guaranteed to be thread safe and there are no
894	plans to change this until Perl gets thread support (as opposed to the	1031	plans to change this until Perl gets thread support (as opposed to the
895	horribly slow so-called "threads" which are simply slow and bloated	1032	horribly slow so-called "threads" which are simply slow and bloated
…		…
908	service. I put the contact address into my modules for a reason.	1045	service. I put the contact address into my modules for a reason.
909		1046
910	=cut	1047	=cut
911		1048
912	our %FILTER = (	1049	our %FILTER = (
913	# 0 # rfc4287 datetime, utf-8	1050	0 => sub { # rfc4287 datetime, utf-8
914	# 1 # unix timestamp, any	1051	require Time::Piece;
		1052	# Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
		1053	# from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
		1054	# else either. Whats incredibe over standard strptime totally escapes me.
		1055	# doesn't do fractional times, either. sigh.
		1056	# In fact, it's all a lie, it uses whatever strptime it wants, and of course,
		1057	# they are all incomptible. The openbsd one simply ignores %z (but according to the
		1058	# docs, it would be much more incredibly flexible indeed. If it worked, that is.).
		1059	scalar eval {
		1060	my $s = $_[1];
		1061
		1062	$s =~ s/Z$/+00:00/;
		1063	$s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
		1064	or die;
		1065
		1066	my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
		1067	my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
		1068
		1069	Time::Piece::gmtime ($d->epoch + $b)
		1070	} \|\| die "corrupted CBOR date/time string ($_[0])";
		1071	},
		1072
		1073	1 => sub { # seconds since the epoch, possibly fractional
		1074	require Time::Piece;
		1075	scalar Time::Piece::gmtime (pop)
		1076	},
915		1077
916	2 => sub { # pos bigint	1078	2 => sub { # pos bigint
917	require Math::BigInt;	1079	require Math::BigInt;
918	Math::BigInt->new ("0x" . unpack "H*", pop)	1080	Math::BigInt->new ("0x" . unpack "H*", pop)
919	},	1081	},
…		…
955	}	1117	}
956		1118
957	sub URI::TO_CBOR {	1119	sub URI::TO_CBOR {
958	my $uri = $_[0]->as_string;	1120	my $uri = $_[0]->as_string;
959	utf8::upgrade $uri;	1121	utf8::upgrade $uri;
960	CBOR::XS::tag 32, $uri	1122	tag 32, $uri
961	}	1123	}
962		1124
963	sub Math::BigInt::TO_CBOR {	1125	sub Math::BigInt::TO_CBOR {
964	if ($_[0] >= -2147483648 && $_[0] <= 2147483647) {	1126	if ($_[0] >= -2147483648 && $_[0] <= 2147483647) {
965	$_[0]->numify	1127	$_[0]->numify
966	} else {	1128	} else {
967	my $hex = substr $_[0]->as_hex, 2;	1129	my $hex = substr $_[0]->as_hex, 2;
968	$hex = "0$hex" if 1 & length $hex; # sigh	1130	$hex = "0$hex" if 1 & length $hex; # sigh
969	CBOR::XS::tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex	1131	tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
970	}	1132	}
971	}	1133	}
972		1134
973	sub Math::BigFloat::TO_CBOR {	1135	sub Math::BigFloat::TO_CBOR {
974	my ($m, $e) = $_[0]->parts;	1136	my ($m, $e) = $_[0]->parts;
975	CBOR::XS::tag 4, [$e->numify, $m]	1137	tag 4, [$e->numify, $m]
		1138	}
		1139
		1140	sub Time::Piece::TO_CBOR {
		1141	tag 1, 0 + $_[0]->epoch
976	}	1142	}
977		1143
978	XSLoader::load "CBOR::XS", $VERSION;	1144	XSLoader::load "CBOR::XS", $VERSION;
979		1145
980	=head1 SEE ALSO	1146	=head1 SEE ALSO

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing CBOR-XS/XS.pm (file contents): Revision 1.29 by root, Sat Nov 30 15:23:59 2013 UTC vs. Revision 1.40 by root, Sun Jan 5 14:24:54 2014 UTC

Diff Legend

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.29 by root, Sat Nov 30 15:23:59 2013 UTC vs.
Revision 1.40 by root, Sun Jan 5 14:24:54 2014 UTC