Convert-BER-XS/XS.pm

=head1 NAME

Convert::BER::XS - I<very> low level BER en-/decoding

=head1 SYNOPSIS

 use Convert::BER::XS ':all';

 my $ber = ber_decode $buf
    or die "unable to decode SNMP v1/v2c Message";

 # the above results in a data structure consisting of (class, tag,
 # constructed, data) tuples. here is such a message, SNMPv1 trap
 # with a cisoc mac change notification

 [ ASN_UNIVERSAL, ASN_SEQUENCE, 1,
   [
      [ ASN_UNIVERSAL, ASN_INTEGER32, 0, 0 ], # snmp version 1
      [ ASN_UNIVERSAL, 4, 0, "public" ], # community
      [ ASN_CONTEXT, 4, 1, # CHOICE, constructed
         [
            [ ASN_UNIVERSAL, ASN_OBJECT_IDENTIFIER, 0, "1.3.6.1.4.1.9.9.215.2" ], # enterprise oid
            [ ASN_APPLICATION, 0, 0, "\x0a\x00\x00\x01" ], # SNMP IpAddress, 10.0.0.1
            [ ASN_UNIVERSAL, ASN_INTEGER32, 0, 6 ], # generic trap
            [ ASN_UNIVERSAL, ASN_INTEGER32, 0, 1 ], # specific trap
            [ ASN_APPLICATION, ASN_TIMETICKS, 0, 1817903850 ], # SNMP TimeTicks
            [ ASN_UNIVERSAL, ASN_SEQUENCE, 1, # the varbindlist
               [
                  [ ASN_UNIVERSAL, ASN_SEQUENCE, 1, # a single varbind, "key value" pair
                     [
                        [ ASN_UNIVERSAL, ASN_OBJECT_IDENTIFIER, 0, "1.3.6.1.4.1.9.9.215.1.1.8.1.2.1" ], # the oid
                        [ ASN_UNIVERSAL, ASN_OCTET_STRING, 0, "...data..." # the value
                        ]
                     ]
                  ],
                  ...

 # let's decode it a bit with some helper functions

 my $msg = ber_is_seq $ber
    or die "SNMP message does not start with a sequence";

 ber_is $msg->[0], ASN_UNIVERSAL, ASN_INTEGER32, 0
    or die "SNMP message does not start with snmp version\n";

 # message is SNMP v1 or v2c?
 if ($msg->[0][BER_DATA] == 0 || $msg->[0][BER_DATA] == 1) {

    # message is v1 trap?
    if (ber_is $msg->[2], ASN_CONTEXT, 4, 1) {
       my $trap = $msg->[2][BER_DATA];

       # check whether trap is a cisco mac notification mac changed message
       if (
          (ber_is_oid $trap->[0], "1.3.6.1.4.1.9.9.215.2") # cmnInterfaceObjects
          and (ber_is_i32 $trap->[2], 6)
          and (ber_is_i32 $trap->[3], 1) # mac changed msg
       ) {
          ... and so on

 # finally, let's encode it again and hope it results in the same bit pattern

 my $buf = ber_encode $ber;

=head1 DESCRIPTION

This module implements a I<very> low level BER/DER en-/decoder.

If is tuned for low memory and high speed, while still maintaining some
level of user-friendlyness.

Currently, not much is documented, as this is an initial release to
reserve CPAN namespace, stay tuned for a few days.

=head2 ASN.1/BER/DER/... BASICS

ASN.1 is a strange language that can be sed to describe protocols and
data structures. It supports various mappings to JSON, XML, but most
importantly, to a various binary encodings such as BER, that is the topic
of this module, and is used in SNMP or LDAP for example.

While ASN.1 defines a schema that is useful to interpret encoded data,
the BER encoding is actually somehat self-describing: you might not know
whether something is a string or a number or a sequence or something else,
but you can nevertheless decode the overall structure, even if you end up
with just a binary blob for the actual value.

This works because BER values are tagged with a type and a namespace,
and also have a flag that says whther a value consists of subvalues (is
"constructed") or not (is "primitive").

Tags are simple integers, and ASN.1 defines a somewhat weird assortment of
those - for example, you have 32 bit signed integers and 16(!) different
string types, but there is no unsigned32 type for example. Different
applications work around this in different ways, for example, SNMP defines
application-specific Gauge32, Counter32 and Unsigned32, which are mapped
to two different tags: you can distinguish between Counter32 and the
others, but not between Gause32 and Unsigned32, without the ASN.1 schema.

Ugh.

=head2 DECODED BER REPRESENTATION

This module represents every BER value as a 4-element tuple (actually an
array-reference):

   [CLASS, TAG, CONSTRUCTED, DATA]

I<CLASS> is something like a namespace for I<TAG>s - there is the
C<ASN_UNIVERSAL> namespace which defines tags common to all ASN.1
implementations, the C<ASN_APPLICATION> namespace which defines tags for
specific applications (for example, the SNMP C<Unsigned32> type is in this
namespace), a special-purpose context namespace (C<ASN_CONTEXT>, used e.g.
for C<CHOICE>) and a private namespace (C<ASN_PRIVATE>).

The meaning of the I<TAG> depends on the namespace, and defines a
(partial) interpretation of the data value. For example, right now, SNMP
application namespace knowledge ix hardcoded into this module, so it
knows that SNMP C<Unsigned32> values need to be decoded into actual perl
integers.

The most common tags in the C<ASN_UNIVERSAL> namespace are
C<ASN_INTEGER32>, C<ASN_BIT_STRING>, C<ASN_NULL>, C<ASN_OCTET_STRING>,
C<ASN_OBJECT_IDENTIFIER>, C<ASN_SEQUENCE>, C<ASN_SET> and
C<ASN_IA5_STRING>.

The most common tags in SNMP's C<ASN_APPLICATION> namespace
are C<SNMP_IPADDRESS>, C<SNMP_COUNTER32>, C<SNMP_UNSIGNED32>,
C<SNMP_TIMETICKS>, C<SNMP_OPAQUE> and C<SNMP_COUNTER64>.

The I<CONSTRUCTED> flag is really just a boolean - if it is false, the
the value is "primitive" and contains no subvalues, kind of like a
non-reference perl scalar. IF it is true, then the value is "constructed"
which just means it contains a list of subvalues which this module will
en-/decode as BER tuples themselves.

The I<DATA> value is either a reference to an array of further tuples (if
the value is I<CONSTRUCTED>), some decoded representation of the value,
if this module knows how to decode it (e.g. for the integer types above)
or a binary string with the raw octets if this module doesn't know how to
interpret the namespace/tag.

Thus, you can always decode a BER data structure and at worst you get a
string in place of some nice decoded value.

See the SYNOPSIS for an example of such an encoded tuple representation.

=head2 RELATIONSHIP TO L<Convert::BER> and L<Convert::ASN1>

This module is I<not> the XS version of L<Convert::BER>, but a different
take at doing the same thing. I imagine this module would be a good base
for speeding up either of these, or write a similar module, or write your
own LDAP or SNMP module for example.

=cut

package Convert::BER::XS;

use common::sense;

use XSLoader ();
use Exporter qw(import);

our $VERSION = '0.0';

XSLoader::load __PACKAGE__, $VERSION;

our %EXPORT_TAGS = (
   const => [qw(
      BER_CLASS BER_TAG BER_CONSTRUCTED BER_DATA

      ASN_BOOLEAN ASN_INTEGER32 ASN_BIT_STRING ASN_OCTET_STRING ASN_NULL ASN_OBJECT_IDENTIFIER ASN_TAG_BER ASN_TAG_MASK
      ASN_CONSTRUCTED ASN_UNIVERSAL ASN_APPLICATION ASN_CONTEXT ASN_PRIVATE ASN_CLASS_MASK ASN_CLASS_SHIFT
      ASN_SEQUENCE
      
      SNMP_IPADDRESS SNMP_COUNTER32 SNMP_UNSIGNED32 SNMP_TIMETICKS SNMP_OPAQUE SNMP_COUNTER64
   )],
   encode => [qw(
      ber_decode
      ber_is ber_is_seq ber_is_i32 ber_is_oid
   )],
   decode => [qw(
      ber_encode
   )],
);

our @EXPORT_OK = map @$_, values %EXPORT_TAGS;

$EXPORT_TAGS{all} = \@EXPORT_OK;

1;

=head2 BUGS / SHORTCOMINGs

This module does have a number of SNMPisms hardcoded, such as the SNMP
tags for Unsigned32 and so on. More configurability is needed, and, if
ever implemented, will come in a form similar to how L<JSON::XS> and
L<CBOR::XS> respresent things, namely with an object-oriented interface.

=head1 AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://software.schmorp.de/pkg/Convert-BER-XS

=cut

Revision:	1.4
Committed:	Fri Apr 19 19:46:29 2019 UTC (5 years, 1 month ago) by root
Branch:	MAIN
Changes since 1.3:	+101 -9 lines
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3	root	1.4	Convert::BER::XS - I<very> low level BER en-/decoding
4	root	1.1
5			=head1 SYNOPSIS
6
7			use Convert::BER::XS ':all';
8
9			my $ber = ber_decode $buf
10			or die "unable to decode SNMP v1/v2c Message";
11
12	root	1.3	# the above results in a data structure consisting of (class, tag,
13			# constructed, data) tuples. here is such a message, SNMPv1 trap
14			# with a cisoc mac change notification
15
16			[ ASN_UNIVERSAL, ASN_SEQUENCE, 1,
17			[
18			[ ASN_UNIVERSAL, ASN_INTEGER32, 0, 0 ], # snmp version 1
19			[ ASN_UNIVERSAL, 4, 0, "public" ], # community
20			[ ASN_CONTEXT, 4, 1, # CHOICE, constructed
21			[
22			[ ASN_UNIVERSAL, ASN_OBJECT_IDENTIFIER, 0, "1.3.6.1.4.1.9.9.215.2" ], # enterprise oid
23			[ ASN_APPLICATION, 0, 0, "\x0a\x00\x00\x01" ], # SNMP IpAddress, 10.0.0.1
24			[ ASN_UNIVERSAL, ASN_INTEGER32, 0, 6 ], # generic trap
25			[ ASN_UNIVERSAL, ASN_INTEGER32, 0, 1 ], # specific trap
26			[ ASN_APPLICATION, ASN_TIMETICKS, 0, 1817903850 ], # SNMP TimeTicks
27			[ ASN_UNIVERSAL, ASN_SEQUENCE, 1, # the varbindlist
28			[
29			[ ASN_UNIVERSAL, ASN_SEQUENCE, 1, # a single varbind, "key value" pair
30			[
31			[ ASN_UNIVERSAL, ASN_OBJECT_IDENTIFIER, 0, "1.3.6.1.4.1.9.9.215.1.1.8.1.2.1" ], # the oid
32			[ ASN_UNIVERSAL, ASN_OCTET_STRING, 0, "...data..." # the value
33			]
34			]
35			],
36			...
37
38			# let's decode it a bit with some helper functions
39
40	root	1.1	my $msg = ber_is_seq $ber
41			or die "SNMP message does not start with a sequence";
42
43			ber_is $msg->[0], ASN_UNIVERSAL, ASN_INTEGER32, 0
44			or die "SNMP message does not start with snmp version\n";
45
46	root	1.3	# message is SNMP v1 or v2c?
47	root	1.1	if ($msg->[0][BER_DATA] == 0 \|\| $msg->[0][BER_DATA] == 1) {
48
49	root	1.3	# message is v1 trap?
50	root	1.1	if (ber_is $msg->[2], ASN_CONTEXT, 4, 1) {
51			my $trap = $msg->[2][BER_DATA];
52
53			# check whether trap is a cisco mac notification mac changed message
54			if (
55			(ber_is_oid $trap->[0], "1.3.6.1.4.1.9.9.215.2") # cmnInterfaceObjects
56			and (ber_is_i32 $trap->[2], 6)
57			and (ber_is_i32 $trap->[3], 1) # mac changed msg
58			) {
59			... and so on
60
61	root	1.4	# finally, let's encode it again and hope it results in the same bit pattern
62
63			my $buf = ber_encode $ber;
64
65	root	1.1	=head1 DESCRIPTION
66
67	root	1.4	This module implements a I<very> low level BER/DER en-/decoder.
68	root	1.1
69			If is tuned for low memory and high speed, while still maintaining some
70			level of user-friendlyness.
71
72			Currently, not much is documented, as this is an initial release to
73			reserve CPAN namespace, stay tuned for a few days.
74
75	root	1.4	=head2 ASN.1/BER/DER/... BASICS
76
77			ASN.1 is a strange language that can be sed to describe protocols and
78			data structures. It supports various mappings to JSON, XML, but most
79			importantly, to a various binary encodings such as BER, that is the topic
80			of this module, and is used in SNMP or LDAP for example.
81
82			While ASN.1 defines a schema that is useful to interpret encoded data,
83			the BER encoding is actually somehat self-describing: you might not know
84			whether something is a string or a number or a sequence or something else,
85			but you can nevertheless decode the overall structure, even if you end up
86			with just a binary blob for the actual value.
87
88			This works because BER values are tagged with a type and a namespace,
89			and also have a flag that says whther a value consists of subvalues (is
90			"constructed") or not (is "primitive").
91
92			Tags are simple integers, and ASN.1 defines a somewhat weird assortment of
93			those - for example, you have 32 bit signed integers and 16(!) different
94			string types, but there is no unsigned32 type for example. Different
95			applications work around this in different ways, for example, SNMP defines
96			application-specific Gauge32, Counter32 and Unsigned32, which are mapped
97			to two different tags: you can distinguish between Counter32 and the
98			others, but not between Gause32 and Unsigned32, without the ASN.1 schema.
99
100			Ugh.
101
102			=head2 DECODED BER REPRESENTATION
103
104			This module represents every BER value as a 4-element tuple (actually an
105			array-reference):
106
107			[CLASS, TAG, CONSTRUCTED, DATA]
108
109			I<CLASS> is something like a namespace for I<TAG>s - there is the
110			C<ASN_UNIVERSAL> namespace which defines tags common to all ASN.1
111			implementations, the C<ASN_APPLICATION> namespace which defines tags for
112			specific applications (for example, the SNMP C<Unsigned32> type is in this
113			namespace), a special-purpose context namespace (C<ASN_CONTEXT>, used e.g.
114			for C<CHOICE>) and a private namespace (C<ASN_PRIVATE>).
115
116			The meaning of the I<TAG> depends on the namespace, and defines a
117			(partial) interpretation of the data value. For example, right now, SNMP
118			application namespace knowledge ix hardcoded into this module, so it
119			knows that SNMP C<Unsigned32> values need to be decoded into actual perl
120			integers.
121
122			The most common tags in the C<ASN_UNIVERSAL> namespace are
123			C<ASN_INTEGER32>, C<ASN_BIT_STRING>, C<ASN_NULL>, C<ASN_OCTET_STRING>,
124			C<ASN_OBJECT_IDENTIFIER>, C<ASN_SEQUENCE>, C<ASN_SET> and
125			C<ASN_IA5_STRING>.
126
127			The most common tags in SNMP's C<ASN_APPLICATION> namespace
128			are C<SNMP_IPADDRESS>, C<SNMP_COUNTER32>, C<SNMP_UNSIGNED32>,
129			C<SNMP_TIMETICKS>, C<SNMP_OPAQUE> and C<SNMP_COUNTER64>.
130
131			The I<CONSTRUCTED> flag is really just a boolean - if it is false, the
132			the value is "primitive" and contains no subvalues, kind of like a
133			non-reference perl scalar. IF it is true, then the value is "constructed"
134			which just means it contains a list of subvalues which this module will
135			en-/decode as BER tuples themselves.
136
137			The I<DATA> value is either a reference to an array of further tuples (if
138			the value is I<CONSTRUCTED>), some decoded representation of the value,
139			if this module knows how to decode it (e.g. for the integer types above)
140			or a binary string with the raw octets if this module doesn't know how to
141			interpret the namespace/tag.
142
143			Thus, you can always decode a BER data structure and at worst you get a
144			string in place of some nice decoded value.
145
146			See the SYNOPSIS for an example of such an encoded tuple representation.
147
148	root	1.2	=head2 RELATIONSHIP TO L<Convert::BER> and L<Convert::ASN1>
149
150			This module is I<not> the XS version of L<Convert::BER>, but a different
151			take at doing the same thing. I imagine this module would be a good base
152	root	1.4	for speeding up either of these, or write a similar module, or write your
153	root	1.2	own LDAP or SNMP module for example.
154
155	root	1.1	=cut
156
157			package Convert::BER::XS;
158
159			use common::sense;
160
161			use XSLoader ();
162			use Exporter qw(import);
163
164			our $VERSION = '0.0';
165
166			XSLoader::load __PACKAGE__, $VERSION;
167
168			our %EXPORT_TAGS = (
169	root	1.4	const => [qw(
170	root	1.1	BER_CLASS BER_TAG BER_CONSTRUCTED BER_DATA
171	root	1.4
172	root	1.1	ASN_BOOLEAN ASN_INTEGER32 ASN_BIT_STRING ASN_OCTET_STRING ASN_NULL ASN_OBJECT_IDENTIFIER ASN_TAG_BER ASN_TAG_MASK
173			ASN_CONSTRUCTED ASN_UNIVERSAL ASN_APPLICATION ASN_CONTEXT ASN_PRIVATE ASN_CLASS_MASK ASN_CLASS_SHIFT
174	root	1.4	ASN_SEQUENCE
175
176			SNMP_IPADDRESS SNMP_COUNTER32 SNMP_UNSIGNED32 SNMP_TIMETICKS SNMP_OPAQUE SNMP_COUNTER64
177			)],
178			encode => [qw(
179			ber_decode
180			ber_is ber_is_seq ber_is_i32 ber_is_oid
181			)],
182			decode => [qw(
183			ber_encode
184	root	1.1	)],
185			);
186
187			our @EXPORT_OK = map @$_, values %EXPORT_TAGS;
188
189	root	1.4	$EXPORT_TAGS{all} = \@EXPORT_OK;
190
191	root	1.1	1;
192
193	root	1.4	=head2 BUGS / SHORTCOMINGs
194
195			This module does have a number of SNMPisms hardcoded, such as the SNMP
196			tags for Unsigned32 and so on. More configurability is needed, and, if
197			ever implemented, will come in a form similar to how L<JSON::XS> and
198			L<CBOR::XS> respresent things, namely with an object-oriented interface.
199
200	root	1.1	=head1 AUTHOR
201
202			Marc Lehmann <schmorp@schmorp.de>
203			http://software.schmorp.de/pkg/Convert-BER-XS
204
205			=cut
206