[ViewVC] Diff of: cvs/CBOR-XS/XS.pm

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.30 by root, Sat Nov 30 16:19:59 2013 UTC vs.
Revision 1.47 by root, Mon Feb 8 04:26:01 2016 UTC

…		…
48	Regarding compactness, C<CBOR::XS>-encoded data structures are usually	48	Regarding compactness, C<CBOR::XS>-encoded data structures are usually
49	about 20% smaller than the same data encoded as (compact) JSON or	49	about 20% smaller than the same data encoded as (compact) JSON or
50	L<Storable>.	50	L<Storable>.
51		51
52	In addition to the core CBOR data format, this module implements a	52	In addition to the core CBOR data format, this module implements a
53	number of extensions, to support cyclic and shared data structures (see	53	number of extensions, to support cyclic and shared data structures
54	C<allow_sharing>), string deduplication (see C<pack_strings>) and scalar	54	(see C<allow_sharing> and C<allow_cycles>), string deduplication (see
55	references (always enabled).	55	C<pack_strings>) and scalar references (always enabled).
56		56
57	The primary goal of this module is to be I<correct> and the secondary goal	57	The primary goal of this module is to be I<correct> and the secondary goal
58	is to be I<fast>. To reach the latter goal it was written in C.	58	is to be I<fast>. To reach the latter goal it was written in C.
59		59
60	See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and	60	See MAPPING, below, on how CBOR::XS maps perl values to CBOR values and
…		…
64		64
65	package CBOR::XS;	65	package CBOR::XS;
66		66
67	use common::sense;	67	use common::sense;
68		68
69	our $VERSION = '1.0';	69	our $VERSION = 1.4;
70	our @ISA = qw(Exporter);	70	our @ISA = qw(Exporter);
71		71
72	our @EXPORT = qw(encode_cbor decode_cbor);	72	our @EXPORT = qw(encode_cbor decode_cbor);
73		73
74	use Exporter;	74	use Exporter;
…		…
180	reference to the earlier value.	180	reference to the earlier value.
181		181
182	This means that such values will only be encoded once, and will not result	182	This means that such values will only be encoded once, and will not result
183	in a deep cloning of the value on decode, in decoders supporting the value	183	in a deep cloning of the value on decode, in decoders supporting the value
184	sharing extension. This also makes it possible to encode cyclic data	184	sharing extension. This also makes it possible to encode cyclic data
185	structures.	185	structures (which need C<allow_cycles> to ne enabled to be decoded by this
		186	module).
186		187
187	It is recommended to leave it off unless you know your	188	It is recommended to leave it off unless you know your
188	communication partner supports the value sharing extensions to CBOR	189	communication partner supports the value sharing extensions to CBOR
189	(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the	190	(L<http://cbor.schmorp.de/value-sharing>), as without decoder support, the
190	resulting data structure might be unusable.	191	resulting data structure might be unusable.
191		192
192	Detecting shared values incurs a runtime overhead when values are encoded	193	Detecting shared values incurs a runtime overhead when values are encoded
193	that have a reference counter large than one, and might unnecessarily	194	that have a reference counter large than one, and might unnecessarily
194	increase the encoded size, as potentially shared values are encode as	195	increase the encoded size, as potentially shared values are encode as
195	sharable whether or not they are actually shared.	196	shareable whether or not they are actually shared.
196		197
197	At the moment, only targets of references can be shared (e.g. scalars,	198	At the moment, only targets of references can be shared (e.g. scalars,
198	arrays or hashes pointed to by a reference). Weirder constructs, such as	199	arrays or hashes pointed to by a reference). Weirder constructs, such as
199	an array with multiple "copies" of the I<same> string, which are hard but	200	an array with multiple "copies" of the I<same> string, which are hard but
200	not impossible to create in Perl, are not supported (this is the same as	201	not impossible to create in Perl, are not supported (this is the same as
…		…
205	structures cannot be encoded in this mode.	206	structures cannot be encoded in this mode.
206		207
207	This option does not affect C<decode> in any way - shared values and	208	This option does not affect C<decode> in any way - shared values and
208	references will always be decoded properly if present.	209	references will always be decoded properly if present.
209		210
		211	=item $cbor = $cbor->allow_cycles ([$enable])
		212
		213	=item $enabled = $cbor->get_allow_cycles
		214
		215	If C<$enable> is true (or missing), then C<decode> will happily decode
		216	self-referential (cyclic) data structures. By default these will not be
		217	decoded, as they need manual cleanup to avoid memory leaks, so code that
		218	isn't prepared for this will not leak memory.
		219
		220	If C<$enable> is false (the default), then C<decode> will throw an error
		221	when it encounters a self-referential/cyclic data structure.
		222
		223	FUTURE DIRECTION: the motivation behind this option is to avoid I<real>
		224	cycles - future versions of this module might chose to decode cyclic data
		225	structures using weak references when this option is off, instead of
		226	throwing an error.
		227
		228	This option does not affect C<encode> in any way - shared values and
		229	references will always be encoded properly if present.
		230
210	=item $cbor = $cbor->pack_strings ([$enable])	231	=item $cbor = $cbor->pack_strings ([$enable])
211		232
212	=item $enabled = $cbor->get_pack_strings	233	=item $enabled = $cbor->get_pack_strings
213		234
214	If C<$enable> is true (or missing), then C<encode> will try not to encode	235	If C<$enable> is true (or missing), then C<encode> will try not to encode
…		…
226	the standard CBOR way.	247	the standard CBOR way.
227		248
228	This option does not affect C<decode> in any way - string references will	249	This option does not affect C<decode> in any way - string references will
229	always be decoded properly if present.	250	always be decoded properly if present.
230		251
		252	=item $cbor = $cbor->validate_utf8 ([$enable])
		253
		254	=item $enabled = $cbor->get_validate_utf8
		255
		256	If C<$enable> is true (or missing), then C<decode> will validate that
		257	elements (text strings) containing UTF-8 data in fact contain valid UTF-8
		258	data (instead of blindly accepting it). This validation obviously takes
		259	extra time during decoding.
		260
		261	The concept of "valid UTF-8" used is perl's concept, which is a superset
		262	of the official UTF-8.
		263
		264	If C<$enable> is false (the default), then C<decode> will blindly accept
		265	UTF-8 data, marking them as valid UTF-8 in the resulting data structure
		266	regardless of whether thats true or not.
		267
		268	Perl isn't too happy about corrupted UTF-8 in strings, but should
		269	generally not crash or do similarly evil things. Extensions might be not
		270	so forgiving, so it's recommended to turn on this setting if you receive
		271	untrusted CBOR.
		272
		273	This option does not affect C<encode> in any way - strings that are
		274	supposedly valid UTF-8 will simply be dumped into the resulting CBOR
		275	string without checking whether that is, in fact, true or not.
		276
231	=item $cbor = $cbor->filter ([$cb->($tag, $value)])	277	=item $cbor = $cbor->filter ([$cb->($tag, $value)])
232		278
233	=item $cb_or_undef = $cbor->get_filter	279	=item $cb_or_undef = $cbor->get_filter
234		280
235	Sets or replaces the tagged value decoding filter (when C<$cb> is	281	Sets or replaces the tagged value decoding filter (when C<$cb> is
…		…
289	and you need to know where the first CBOR string ends amd the next one	335	and you need to know where the first CBOR string ends amd the next one
290	starts.	336	starts.
291		337
292	CBOR::XS->new->decode_prefix ("......")	338	CBOR::XS->new->decode_prefix ("......")
293	=> ("...", 3)	339	=> ("...", 3)
		340
		341	=back
		342
		343	=head2 INCREMENTAL PARSING
		344
		345	In some cases, there is the need for incremental parsing of JSON
		346	texts. While this module always has to keep both CBOR text and resulting
		347	Perl data structure in memory at one time, it does allow you to parse a
		348	CBOR stream incrementally, using a similar to using "decode_prefix" to see
		349	if a full CBOR object is available, but is much more efficient.
		350
		351	It basically works by parsing as much of a CBOR string as possible - if
		352	the CBOR data is not complete yet, the pasrer will remember where it was,
		353	to be able to restart when more data has been accumulated. Once enough
		354	data is available to either decode a complete CBOR value or raise an
		355	error, a real decode will be attempted.
		356
		357	A typical use case would be a network protocol that consists of sending
		358	and receiving CBOR-encoded messages. The solution that works with CBOR and
		359	about anything else is by prepending a length to every CBOR value, so the
		360	receiver knows how many octets to read. More compact (and slightly slower)
		361	would be to just send CBOR values back-to-back, as C<CBOR::XS> knows where
		362	a CBOR value ends, and doesn't need an explicit length.
		363
		364	The following methods help with this:
		365
		366	=over 4
		367
		368	=item @decoded = $cbor->incr_parse ($buffer)
		369
		370	This method attempts to decode exactly one CBOR value from the beginning
		371	of the given C<$buffer>. The value is removed from the C<$buffer> on
		372	success. When C<$buffer> doesn't contain a complete value yet, it returns
		373	nothing. Finally, when the C<$buffer> doesn't start with something
		374	that could ever be a valid CBOR value, it raises an exception, just as
		375	C<decode> would. In the latter case the decoder state is undefined and
		376	must be reset before being able to parse further.
		377
		378	This method modifies the C<$buffer> in place. When no CBOR value can be
		379	decoded, the decoder stores the current string offset. On the next call,
		380	continues decoding at the place where it stopped before. For this to make
		381	sense, the C<$buffer> must begin with the same octets as on previous
		382	unsuccessful calls.
		383
		384	You can call this method in scalar context, in which case it either
		385	returns a decoded value or C<undef>. This makes it impossible to
		386	distinguish between CBOR null values (which decode to C<undef>) and an
		387	unsuccessful decode, which is often acceptable.
		388
		389	=item @decoded = $cbor->incr_parse_multiple ($buffer)
		390
		391	Same as C<incr_parse>, but attempts to decode as many CBOR values as
		392	possible in one go, instead of at most one. Calls to C<incr_parse> and
		393	C<incr_parse_multiple> can be interleaved.
		394
		395	=item $cbor->incr_reset
		396
		397	Resets the incremental decoder. This throws away any saved state, so that
		398	subsequent calls to C<incr_parse> or C<incr_parse_multiple> start to parse
		399	a new CBOR value from the beginning of the C<$buffer> again.
		400
		401	This method can be caled at any time, but it I<must> be called if you want
		402	to change your C<$buffer> or there was a decoding error and you want to
		403	reuse the C<$cbor> object for future incremental parsings.
294		404
295	=back	405	=back
296		406
297		407
298	=head1 MAPPING	408	=head1 MAPPING
…		…
704		814
705	These tags are automatically created (and decoded) for serialisable	815	These tags are automatically created (and decoded) for serialisable
706	objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object	816	objects using the C<FREEZE/THAW> methods (the L<Types::Serialier> object
707	serialisation protocol). See L<OBJECT SERIALISATION> for details.	817	serialisation protocol). See L<OBJECT SERIALISATION> for details.
708		818
709	=item 28, 29 (sharable, sharedref, L <http://cbor.schmorp.de/value-sharing>)	819	=item 28, 29 (shareable, sharedref, L<http://cbor.schmorp.de/value-sharing>)
710		820
711	These tags are automatically decoded when encountered, resulting in	821	These tags are automatically decoded when encountered (and they do not
		822	result in a cyclic data structure, see C<allow_cycles>), resulting in
712	shared values in the decoded object. They are only encoded, however, when	823	shared values in the decoded object. They are only encoded, however, when
713	C<allow_sharable> is enabled.	824	C<allow_sharing> is enabled.
714		825
		826	Not all shared values can be successfully decoded: values that reference
		827	themselves will I<currently> decode as C<undef> (this is not the same
		828	as a reference pointing to itself, which will be represented as a value
		829	that contains an indirect reference to itself - these will be decoded
		830	properly).
		831
		832	Note that considerably more shared value data structures can be decoded
		833	than will be encoded - currently, only values pointed to by references
		834	will be shared, others will not. While non-reference shared values can be
		835	generated in Perl with some effort, they were considered too unimportant
		836	to be supported in the encoder. The decoder, however, will decode these
		837	values as shared values.
		838
715	=item 256, 25 (stringref-namespace, stringref, L <http://cbor.schmorp.de/stringref>)	839	=item 256, 25 (stringref-namespace, stringref, L<http://cbor.schmorp.de/stringref>)
716		840
717	These tags are automatically decoded when encountered. They are only	841	These tags are automatically decoded when encountered. They are only
718	encoded, however, when C<pack_strings> is enabled.	842	encoded, however, when C<pack_strings> is enabled.
719		843
720	=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)	844	=item 22098 (indirection, L<http://cbor.schmorp.de/indirection>)
…		…
743	perl core distribution (e.g. L<URI>), it is (currently) up to the user to	867	perl core distribution (e.g. L<URI>), it is (currently) up to the user to
744	provide these modules. The decoding usually fails with an exception if the	868	provide these modules. The decoding usually fails with an exception if the
745	required module cannot be loaded.	869	required module cannot be loaded.
746		870
747	=over 4	871	=over 4
		872
		873	=item 0, 1 (date/time string, seconds since the epoch)
		874
		875	These tags are decoded into L<Time::Piece> objects. The corresponding
		876	C<Time::Piece::TO_CBOR> method always encodes into tag 1 values currently.
		877
		878	The L<Time::Piece> API is generally surprisingly bad, and fractional
		879	seconds are only accidentally kept intact, so watch out. On the plus side,
		880	the module comes with perl since 5.10, which has to count for something.
748		881
749	=item 2, 3 (positive/negative bignum)	882	=item 2, 3 (positive/negative bignum)
750		883
751	These tags are decoded into L<Math::BigInt> objects. The corresponding	884	These tags are decoded into L<Math::BigInt> objects. The corresponding
752	C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR	885	C<Math::BigInt::TO_CBOR> method encodes "small" bigints into normal CBOR
…		…
889		1022
890		1023
891	=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT	1024	=head1 LIMITATIONS ON PERLS WITHOUT 64-BIT INTEGER SUPPORT
892		1025
893	On perls that were built without 64 bit integer support (these are rare	1026	On perls that were built without 64 bit integer support (these are rare
894	nowadays, even on 32 bit architectures), support for any kind of 64 bit	1027	nowadays, even on 32 bit architectures, as all major Perl distributions
		1028	are built with 64 bit integer support), support for any kind of 64 bit
895	integer in CBOR is very limited - most likely, these 64 bit values will	1029	integer in CBOR is very limited - most likely, these 64 bit values will
896	be truncated, corrupted, or otherwise not decoded correctly. This also	1030	be truncated, corrupted, or otherwise not decoded correctly. This also
897	includes string, array and map sizes that are stored as 64 bit integers.	1031	includes string, array and map sizes that are stored as 64 bit integers.
898		1032
899		1033
…		…
917	service. I put the contact address into my modules for a reason.	1051	service. I put the contact address into my modules for a reason.
918		1052
919	=cut	1053	=cut
920		1054
921	our %FILTER = (	1055	our %FILTER = (
922	# 0 # rfc4287 datetime, utf-8	1056	0 => sub { # rfc4287 datetime, utf-8
923	# 1 # unix timestamp, any	1057	require Time::Piece;
		1058	# Time::Piece::Strptime uses the "incredibly flexible date parsing routine"
		1059	# from FreeBSD, which can't parse ISO 8601, RFC3339, RFC4287 or much of anything
		1060	# else either. Whats incredibe over standard strptime totally escapes me.
		1061	# doesn't do fractional times, either. sigh.
		1062	# In fact, it's all a lie, it uses whatever strptime it wants, and of course,
		1063	# they are all incompatible. The openbsd one simply ignores %z (but according to the
		1064	# docs, it would be much more incredibly flexible indeed. If it worked, that is.).
		1065	scalar eval {
		1066	my $s = $_[1];
		1067
		1068	$s =~ s/Z$/+00:00/;
		1069	$s =~ s/(\.[0-9]+)?([+-][0-9][0-9]):([0-9][0-9])$//
		1070	or die;
		1071
		1072	my $b = $1 - ($2 * 60 + $3) * 60; # fractional part + offset. hopefully
		1073	my $d = Time::Piece->strptime ($s, "%Y-%m-%dT%H:%M:%S");
		1074
		1075	Time::Piece::gmtime ($d->epoch + $b)
		1076	} \|\| die "corrupted CBOR date/time string ($_[0])";
		1077	},
		1078
		1079	1 => sub { # seconds since the epoch, possibly fractional
		1080	require Time::Piece;
		1081	scalar Time::Piece::gmtime (pop)
		1082	},
924		1083
925	2 => sub { # pos bigint	1084	2 => sub { # pos bigint
926	require Math::BigInt;	1085	require Math::BigInt;
927	Math::BigInt->new ("0x" . unpack "H*", pop)	1086	Math::BigInt->new ("0x" . unpack "H*", pop)
928	},	1087	},
…		…
964	}	1123	}
965		1124
966	sub URI::TO_CBOR {	1125	sub URI::TO_CBOR {
967	my $uri = $_[0]->as_string;	1126	my $uri = $_[0]->as_string;
968	utf8::upgrade $uri;	1127	utf8::upgrade $uri;
969	CBOR::XS::tag 32, $uri	1128	tag 32, $uri
970	}	1129	}
971		1130
972	sub Math::BigInt::TO_CBOR {	1131	sub Math::BigInt::TO_CBOR {
973	if ($_[0] >= -2147483648 && $_[0] <= 2147483647) {	1132	if ($_[0] >= -2147483648 && $_[0] <= 2147483647) {
974	$_[0]->numify	1133	$_[0]->numify
975	} else {	1134	} else {
976	my $hex = substr $_[0]->as_hex, 2;	1135	my $hex = substr $_[0]->as_hex, 2;
977	$hex = "0$hex" if 1 & length $hex; # sigh	1136	$hex = "0$hex" if 1 & length $hex; # sigh
978	CBOR::XS::tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex	1137	tag $_[0] >= 0 ? 2 : 3, pack "H*", $hex
979	}	1138	}
980	}	1139	}
981		1140
982	sub Math::BigFloat::TO_CBOR {	1141	sub Math::BigFloat::TO_CBOR {
983	my ($m, $e) = $_[0]->parts;	1142	my ($m, $e) = $_[0]->parts;
984	CBOR::XS::tag 4, [$e->numify, $m]	1143	tag 4, [$e->numify, $m]
		1144	}
		1145
		1146	sub Time::Piece::TO_CBOR {
		1147	tag 1, 0 + $_[0]->epoch
985	}	1148	}
986		1149
987	XSLoader::load "CBOR::XS", $VERSION;	1150	XSLoader::load "CBOR::XS", $VERSION;
988		1151
989	=head1 SEE ALSO	1152	=head1 SEE ALSO

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing CBOR-XS/XS.pm (file contents): Revision 1.30 by root, Sat Nov 30 16:19:59 2013 UTC vs. Revision 1.47 by root, Mon Feb 8 04:26:01 2016 UTC

Diff Legend

Comparing CBOR-XS/XS.pm (file contents):
Revision 1.30 by root, Sat Nov 30 16:19:59 2013 UTC vs.
Revision 1.47 by root, Mon Feb 8 04:26:01 2016 UTC