Coro-Mysql/Mysql.pm

=head1 NAME

Coro::Mysql - let other threads run while doing mysql/mariadb requests

=head1 SYNOPSIS

 use Coro::Mysql;

 my $DBH = Coro::Mysql::unblock DBI->connect (...);

=head1 DESCRIPTION

(Note that in this manual, "thread" refers to real threads as implemented
by the Coro module, not to the built-in windows process emulation which
unfortunately is also called "threads")

This module replaces the I/O handlers for a database connection, with the
effect that "patched" database handles no longer block the all threads of
a process, but only the thread that does the request.

This can be used to make parallel sql requests using Coro, or to do other
stuff while mariadb is rumbling in the background.

=head2 CAVEAT

Note that this module must be linked against exactly the same (shared,
possibly not working with all OSes) F<libmariadb>/F<libmysqlclient>
library as L<DBD::mysql>, otherwise it will not work.

Also, while this module makes database handles non-blocking, you still
cannot run multiple requests in parallel on the same database handle. If
you want to run multiple queries in parallel, you have to create multiple
database connections, one for each thread that runs queries. Not doing
so can corrupt your data - use a Coro::Semaphore to protetc access to a
shared database handle when in doubt.

If you make sure that you never run two or more requests in parallel, you
can freely share the database handles between threads, of course.

=head2 SPEED

This module is implemented in XS, and as long as mysqld replies quickly
enough, it adds no overhead to the standard libmysql communication
routines (which are very badly written, btw.). In fact, since it has a
more efficient buffering and allows requests to run in parallel, it often
decreases the actual time to run many queries considerably.

For very fast queries ("select 0"), this module can add noticable overhead
(around 15%, 7% when EV can be used) as it tries to switch to other
coroutines when mysqld doesn't deliver the data immediately, although,
again, when running queries in parallel, they will usually execute faster.

For most types of queries, there will be no extra latency, especially on
multicore systems where your perl process can do other things while mysqld
does its stuff.

=head2 LIMITATIONS

This module only supports "standard" mysql connection handles - this
means unix domain or TCP sockets, and excludes SSL/TLS connections, named
pipes (windows) and shared memory (also windows). No support for these
connection types is planned, either.

=head1 CANCELLATION

Cancelling a thread that is within a mysql query will likely make the
handle unusable. As far as Coro::Mysql is concerned, the handle can be
safely destroyed, but it's not clear how mysql itself will react to a
cancellation.

=head1 FUNCTIONS

Coro::Mysql offers a single user-accessible function:

=over 4

=cut

package Coro::Mysql;

use strict qw(vars subs);
no warnings;

use Scalar::Util ();
use Carp qw(croak);

use Guard;
use AnyEvent ();
use Coro ();
use Coro::AnyEvent (); # not necessary with newer Coro versions

# we need this extra indirection, as Coro doesn't support
# calling SLF-like functions via call_sv.

sub readable { &Coro::Handle::FH::readable }
sub writable { &Coro::Handle::FH::writable }

BEGIN {
   our $VERSION = '2.0';

   require XSLoader;
   XSLoader::load Coro::Mysql::, $VERSION;
}

=item $DBH = Coro::Mysql::unblock $DBH

This function takes a DBI database handles and "patches" it
so it becomes compatible to Coro threads.

After that, it returns the patched handle - you should always use the
newly returned database handle.

It is safe to call this function on any database handle (or just about any
value), but it will only do anything to L<DBD::mysql> handles, others are
returned unchanged. That means it is harmless when applied to database
handles of other databases.

It is also safe to pass C<undef>, so code like this is works as expected:

   my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
      or die $DBI::errstr;

=cut

sub unblock {
   my ($DBH) = @_;

   if ($DBH && $DBH->{Driver}{Name} eq "mysql") {
      my $sock = $DBH->{sock};

      open my $fh, "+>&" . $DBH->{sockfd}
         or croak "Coro::Mysql unable to clone mysql fd";

      if (AnyEvent::detect ne "AnyEvent::Impl::EV" || !_use_ev) {
         require Coro::Handle;
         $fh = Coro::Handle::unblock ($fh);
      }

      _patch $sock, $DBH->{sockfd}, $DBH->{mysql_clientversion}, $fh, tied *$$fh;
   }

   $DBH
}

1;

=back

=head1 USAGE EXAMPLE

This example uses L<PApp::SQL> and L<Coro::on_enter> to implement a
function C<with_db>, that connects to a database, uses C<unblock> on the
resulting handle and then makes sure that C<$PApp::SQL::DBH> is set to the
(per-thread) database handle when the given thread is running (it does not
restore any previous value of $PApp::SQL::DBH, however):

   use Coro;
   use Coro::Mysql;
   use PApp::SQL;

   sub with_db($$$&) {
      my ($database, $user, $pass, $cb) = @_;

      my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
         or die $DBI::errstr;

      Coro::on_enter { $PApp::SQL::DBH = $dbh };

      $cb->();
   }

This function makes it possible to easily use L<PApp::SQL> with
L<Coro::Mysql>, without worrying about database handles.

   # now start 10 threads doing stuff
   async {

      with_db "DBI:mysql:test", "", "", sub {
         sql_exec "update table set col = 5 where id = 7";

         my $st = sql_exec \my ($id, $name),
                           "select id, name from table where name like ?",
                           "a%";

         while ($st->fetch) {
            ...
         }

         my $id = sql_insertid sql_exec "insert into table values (1,2,3)";
         # etc.
      };

   } for 1..10;

=head1 SEE ALSO

L<Coro>, L<PApp::SQL> (a user friendly but efficient wrapper around DBI).

=head1 HISTORY

This module was initially hacked together within a few hours on a long
flight to Malaysia, and seems to have worked ever since, with minor
adjustments for newer libmysqlclient libraries.

Well, at least until mariadb introduced the new Pluggable Virtual IO API
in mariadb 10.3, which changed and broke everything. On the positive
side, the old system was horrible to use, as many GNU/Linux distributions
forgot to include the required heaqder files and there were frequent small
changes, while the new PVIO system seems to be "official" and hopefully
better supported.

=head1 AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://home.schmorp.de/

=cut

Revision:	1.23
Committed:	Mon Mar 4 05:44:43 2019 UTC (5 years, 2 months ago) by root
Branch:	MAIN
Changes since 1.22:	+12 -16 lines
Log Message:	* empty log message *
#	Content
1	=head1 NAME
2
3	Coro::Mysql - let other threads run while doing mysql/mariadb requests
4
5	=head1 SYNOPSIS
6
7	use Coro::Mysql;
8
9	my $DBH = Coro::Mysql::unblock DBI->connect (...);
10
11	=head1 DESCRIPTION
12
13	(Note that in this manual, "thread" refers to real threads as implemented
14	by the Coro module, not to the built-in windows process emulation which
15	unfortunately is also called "threads")
16
17	This module replaces the I/O handlers for a database connection, with the
18	effect that "patched" database handles no longer block the all threads of
19	a process, but only the thread that does the request.
20
21	This can be used to make parallel sql requests using Coro, or to do other
22	stuff while mariadb is rumbling in the background.
23
24	=head2 CAVEAT
25
26	Note that this module must be linked against exactly the same (shared,
27	possibly not working with all OSes) F<libmariadb>/F<libmysqlclient>
28	library as L<DBD::mysql>, otherwise it will not work.
29
30	Also, while this module makes database handles non-blocking, you still
31	cannot run multiple requests in parallel on the same database handle. If
32	you want to run multiple queries in parallel, you have to create multiple
33	database connections, one for each thread that runs queries. Not doing
34	so can corrupt your data - use a Coro::Semaphore to protetc access to a
35	shared database handle when in doubt.
36
37	If you make sure that you never run two or more requests in parallel, you
38	can freely share the database handles between threads, of course.
39
40	=head2 SPEED
41
42	This module is implemented in XS, and as long as mysqld replies quickly
43	enough, it adds no overhead to the standard libmysql communication
44	routines (which are very badly written, btw.). In fact, since it has a
45	more efficient buffering and allows requests to run in parallel, it often
46	decreases the actual time to run many queries considerably.
47
48	For very fast queries ("select 0"), this module can add noticable overhead
49	(around 15%, 7% when EV can be used) as it tries to switch to other
50	coroutines when mysqld doesn't deliver the data immediately, although,
51	again, when running queries in parallel, they will usually execute faster.
52
53	For most types of queries, there will be no extra latency, especially on
54	multicore systems where your perl process can do other things while mysqld
55	does its stuff.
56
57	=head2 LIMITATIONS
58
59	This module only supports "standard" mysql connection handles - this
60	means unix domain or TCP sockets, and excludes SSL/TLS connections, named
61	pipes (windows) and shared memory (also windows). No support for these
62	connection types is planned, either.
63
64	=head1 CANCELLATION
65
66	Cancelling a thread that is within a mysql query will likely make the
67	handle unusable. As far as Coro::Mysql is concerned, the handle can be
68	safely destroyed, but it's not clear how mysql itself will react to a
69	cancellation.
70
71	=head1 FUNCTIONS
72
73	Coro::Mysql offers a single user-accessible function:
74
75	=over 4
76
77	=cut
78
79	package Coro::Mysql;
80
81	use strict qw(vars subs);
82	no warnings;
83
84	use Scalar::Util ();
85	use Carp qw(croak);
86
87	use Guard;
88	use AnyEvent ();
89	use Coro ();
90	use Coro::AnyEvent (); # not necessary with newer Coro versions
91
92	# we need this extra indirection, as Coro doesn't support
93	# calling SLF-like functions via call_sv.
94
95	sub readable { &Coro::Handle::FH::readable }
96	sub writable { &Coro::Handle::FH::writable }
97
98	BEGIN {
99	our $VERSION = '2.0';
100
101	require XSLoader;
102	XSLoader::load Coro::Mysql::, $VERSION;
103	}
104
105	=item $DBH = Coro::Mysql::unblock $DBH
106
107	This function takes a DBI database handles and "patches" it
108	so it becomes compatible to Coro threads.
109
110	After that, it returns the patched handle - you should always use the
111	newly returned database handle.
112
113	It is safe to call this function on any database handle (or just about any
114	value), but it will only do anything to L<DBD::mysql> handles, others are
115	returned unchanged. That means it is harmless when applied to database
116	handles of other databases.
117
118	It is also safe to pass C<undef>, so code like this is works as expected:
119
120	my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
121	or die $DBI::errstr;
122
123	=cut
124
125	sub unblock {
126	my ($DBH) = @_;
127
128	if ($DBH && $DBH->{Driver}{Name} eq "mysql") {
129	my $sock = $DBH->{sock};
130
131	open my $fh, "+>&" . $DBH->{sockfd}
132	or croak "Coro::Mysql unable to clone mysql fd";
133
134	if (AnyEvent::detect ne "AnyEvent::Impl::EV" \|\| !_use_ev) {
135	require Coro::Handle;
136	$fh = Coro::Handle::unblock ($fh);
137	}
138
139	_patch $sock, $DBH->{sockfd}, $DBH->{mysql_clientversion}, $fh, tied *$$fh;
140	}
141
142	$DBH
143	}
144
145	1;
146
147	=back
148
149	=head1 USAGE EXAMPLE
150
151	This example uses L<PApp::SQL> and L<Coro::on_enter> to implement a
152	function C<with_db>, that connects to a database, uses C<unblock> on the
153	resulting handle and then makes sure that C<$PApp::SQL::DBH> is set to the
154	(per-thread) database handle when the given thread is running (it does not
155	restore any previous value of $PApp::SQL::DBH, however):
156
157	use Coro;
158	use Coro::Mysql;
159	use PApp::SQL;
160
161	sub with_db($$$&) {
162	my ($database, $user, $pass, $cb) = @_;
163
164	my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
165	or die $DBI::errstr;
166
167	Coro::on_enter { $PApp::SQL::DBH = $dbh };
168
169	$cb->();
170	}
171
172	This function makes it possible to easily use L<PApp::SQL> with
173	L<Coro::Mysql>, without worrying about database handles.
174
175	# now start 10 threads doing stuff
176	async {
177
178	with_db "DBI:mysql:test", "", "", sub {
179	sql_exec "update table set col = 5 where id = 7";
180
181	my $st = sql_exec \my ($id, $name),
182	"select id, name from table where name like ?",
183	"a%";
184
185	while ($st->fetch) {
186	...
187	}
188
189	my $id = sql_insertid sql_exec "insert into table values (1,2,3)";
190	# etc.
191	};
192
193	} for 1..10;
194
195	=head1 SEE ALSO
196
197	L<Coro>, L<PApp::SQL> (a user friendly but efficient wrapper around DBI).
198
199	=head1 HISTORY
200
201	This module was initially hacked together within a few hours on a long
202	flight to Malaysia, and seems to have worked ever since, with minor
203	adjustments for newer libmysqlclient libraries.
204
205	Well, at least until mariadb introduced the new Pluggable Virtual IO API
206	in mariadb 10.3, which changed and broke everything. On the positive
207	side, the old system was horrible to use, as many GNU/Linux distributions
208	forgot to include the required heaqder files and there were frequent small
209	changes, while the new PVIO system seems to be "official" and hopefully
210	better supported.
211
212	=head1 AUTHOR
213
214	Marc Lehmann <schmorp@schmorp.de>
215	http://home.schmorp.de/
216
217	=cut
218