Coro-Mysql/Mysql.pm

=head1 NAME

Coro::Mysql - let other threads run while doing mysql requests

=head1 SYNOPSIS

 use Coro::Mysql;

 my $DBH = Coro::Mysql::unblock DBI->connect (...);

=head1 DESCRIPTION

(Note that in this manual, "thread" refers to real threads as implemented
by the Coro module, not to the built-in windows process emulation which
unfortunately is also called "threads")

This module replaces the I/O handlers for a database connection, with the
effect that "patched" database handles no longer block the all threads of
a process, but only the thread that does the request.

This can be used to make parallel sql requests using Coro, or to do other
stuff while mysql is rumbling in the background.

=head2 CAVEAT

Note that this module must be linked against exactly the same (shared,
possibly not working with all OSes) F<libmysqlclient> library as
DBD::mysql, otherwise it will not work.

Also, this module requires a header file that apparently isn't installed
everywhere (F<violite.h>), and therefore comes with it's own copy, which
might or might not be compatible to the F<violite.h> of your library -
when in doubt, make sure all the libmysqlclient header files are installed
and delete the F<violite.h> header that comes with this module.

On the good side, this module does a multitude of checks to ensure that
the libray versions match on the binary level, so on incompatibilities you
should expect an exception when trying to unblock a handle, rather than
data corruption.

Also, while this module makes database handles non-blocking, you still
cannot run multiple requests in parallel on the same database handle. If
you want to run multiple queries in parallel, you have to create multiple
database connections, one for each thread that runs queries. Not doing
so can corrupt your data - use a Coro::Semaphore to protetc access to a
shared database handle when in doubt.

If you make sure that you never run two or more requests in parallel, you
can freely share the database handles between threads, of course.

=head2 SPEED

This module is implemented in XS, and as long as mysqld replies quickly
enough, it adds no overhead to the standard libmysql communication
routines (which are very badly written, btw.). In fact, since it has a
more efficient buffering and allows requests to run in parallel, it often
decreases the actual time to run many queries considerably.

For very fast queries ("select 0"), this module can add noticable overhead
(around 15%, 7% when EV can be used) as it tries to switch to other
coroutines when mysqld doesn't deliver the data immediately, although,
again, when running queries in parallel, they will usually execute faster.

For most types of queries, there will be no extra latency, especially on
multicore systems where your perl process can do other things while mysqld
does its stuff.

=head2 LIMITATIONS

This module only supports "standard" mysql connection handles - this
means unix domain or TCP sockets, and excludes SSL/TLS connections, named
pipes (windows) and shared memory (also windows). No support for these
connection types is planned, either.

=head1 CANCELLATION

Cancelling a thread that is within a mysql query will likely make the
handle unusable. As far as Coro::Mysql is concerned, the handle can be
safely destroyed, but it's not clear how mysql itself will react to a
cancellation.

=head1 FUNCTIONS

Coro::Mysql offers a single user-accessible function:

=over 4

=cut

package Coro::Mysql;

use strict qw(vars subs);
no warnings;

use Scalar::Util ();
use Carp qw(croak);

use Guard;
use AnyEvent ();
use Coro ();
use Coro::AnyEvent (); # not necessary with newer Coro versions

# we need this extra indirection, as Coro doesn't support
# calling SLF-like functions via call_sv.

sub readable { &Coro::Handle::FH::readable }
sub writable { &Coro::Handle::FH::writable }

BEGIN {
   our $VERSION = '1.24';

   require XSLoader;
   XSLoader::load Coro::Mysql::, $VERSION;
}

=item $DBH = Coro::Mysql::unblock $DBH

This function takes a DBI database handles and "patches" it
so it becomes compatible to Coro threads.

After that, it returns the patched handle - you should always use the
newly returned database handle.

It is safe to call this function on any database handle (or just about any
value), but it will only do anything to L<DBD::mysql> handles, others are
returned unchanged. That means it is harmless when applied to database
handles of other databases.

It is also safe to pass C<undef>, so code like this is works as expected:

   my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
      or die $DBI::errstr;

=cut

sub unblock {
   my ($DBH) = @_;

   if ($DBH && $DBH->{Driver}{Name} eq "mysql") {
      my $sock = $DBH->{sock};

      open my $fh, "+>&" . $DBH->{sockfd}
         or croak "Coro::Mysql unable to clone mysql fd";

      if (AnyEvent::detect ne "AnyEvent::Impl::EV" || !_use_ev) {
         require Coro::Handle;
         $fh = Coro::Handle::unblock ($fh);
      }

      warn "$sock,$DBH->{sockfd},$DBH->{mysql_clientversion},$fh\n";#d#
      _patch $sock, $DBH->{sockfd}, $DBH->{mysql_clientversion}, $fh, tied *$$fh;
   }

   $DBH
}

1;

=back

=head1 USAGE EXAMPLE

This example uses L<PApp::SQL> and L<Coro::on_enter> to implement a
function C<with_db>, that connects to a database, uses C<unblock> on the
resulting handle and then makes sure that C<$PApp::SQL::DBH> is set to the
(per-thread) database handle when the given thread is running (it does not
restore any previous value of $PApp::SQL::DBH, however):

   use Coro;
   use Coro::Mysql;
   use PApp::SQL;

   sub with_db($$$&) {
      my ($database, $user, $pass, $cb) = @_;

      my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
         or die $DBI::errstr;

      Coro::on_enter { $PApp::SQL::DBH = $dbh };

      $cb->();
   }  

This function makes it possible to easily use L<PApp::SQL> with
L<Coro::Mysql>, without worrying about database handles.

   # now start 10 threads doing stuff
   async {

      with_db "DBI:mysql:test", "", "", sub {
         sql_exec "update table set col = 5 where id = 7";

         my $st = sql_exec \my ($id, $name),
                           "select id, name from table where name like ?",
                           "a%";

         while ($st->fetch) {
            ...
         }

         my $id = sql_insertid sql_exec "insert into table values (1,2,3)";
         # etc.
      };

   } for 1..10;

=head1 SEE ALSO

L<Coro>, L<PApp::SQL> (a user friendly but efficient wrapper around DBI).

=head1 HISTORY

This module was initially hacked together within a few hours on a long
flight to Malaysia, and seems to have worked ever since, with minor
adjustments for newer libmysqlclient libraries.

=head1 AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://home.schmorp.de/

=cut

Revision:	1.18
Committed:	Tue Jun 3 03:08:34 2014 UTC (9 years, 11 months ago) by root
Branch:	MAIN
CVS Tags:	rel-1_24
Changes since 1.17:	+2 -1 lines
Log Message:	1.24
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3			Coro::Mysql - let other threads run while doing mysql requests
4
5			=head1 SYNOPSIS
6
7			use Coro::Mysql;
8
9			my $DBH = Coro::Mysql::unblock DBI->connect (...);
10
11			=head1 DESCRIPTION
12
13			(Note that in this manual, "thread" refers to real threads as implemented
14			by the Coro module, not to the built-in windows process emulation which
15			unfortunately is also called "threads")
16
17	root	1.13	This module replaces the I/O handlers for a database connection, with the
18			effect that "patched" database handles no longer block the all threads of
19			a process, but only the thread that does the request.
20	root	1.1
21			This can be used to make parallel sql requests using Coro, or to do other
22			stuff while mysql is rumbling in the background.
23
24			=head2 CAVEAT
25
26	root	1.7	Note that this module must be linked against exactly the same (shared,
27			possibly not working with all OSes) F<libmysqlclient> library as
28			DBD::mysql, otherwise it will not work.
29	root	1.1
30	root	1.13	Also, this module requires a header file that apparently isn't installed
31			everywhere (F<violite.h>), and therefore comes with it's own copy, which
32			might or might not be compatible to the F<violite.h> of your library -
33			when in doubt, make sure all the libmysqlclient header files are installed
34			and delete the F<violite.h> header that comes with this module.
35
36			On the good side, this module does a multitude of checks to ensure that
37			the libray versions match on the binary level, so on incompatibilities you
38			should expect an exception when trying to unblock a handle, rather than
39			data corruption.
40
41	root	1.1	Also, while this module makes database handles non-blocking, you still
42			cannot run multiple requests in parallel on the same database handle. If
43			you want to run multiple queries in parallel, you have to create multiple
44	root	1.13	database connections, one for each thread that runs queries. Not doing
45			so can corrupt your data - use a Coro::Semaphore to protetc access to a
46			shared database handle when in doubt.
47	root	1.1
48			If you make sure that you never run two or more requests in parallel, you
49	root	1.4	can freely share the database handles between threads, of course.
50	root	1.1
51			=head2 SPEED
52
53			This module is implemented in XS, and as long as mysqld replies quickly
54			enough, it adds no overhead to the standard libmysql communication
55	root	1.9	routines (which are very badly written, btw.). In fact, since it has a
56			more efficient buffering and allows requests to run in parallel, it often
57			decreases the actual time to run many queries considerably.
58	root	1.1
59			For very fast queries ("select 0"), this module can add noticable overhead
60	root	1.9	(around 15%, 7% when EV can be used) as it tries to switch to other
61			coroutines when mysqld doesn't deliver the data immediately, although,
62			again, when running queries in parallel, they will usually execute faster.
63	root	1.1
64	root	1.8	For most types of queries, there will be no extra latency, especially on
65	root	1.1	multicore systems where your perl process can do other things while mysqld
66			does its stuff.
67
68	root	1.4	=head2 LIMITATIONS
69
70			This module only supports "standard" mysql connection handles - this
71			means unix domain or TCP sockets, and excludes SSL/TLS connections, named
72			pipes (windows) and shared memory (also windows). No support for these
73			connection types is planned, either.
74
75	root	1.9	=head1 CANCELLATION
76
77			Cancelling a thread that is within a mysql query will likely make the
78			handle unusable. As far as Coro::Mysql is concerned, the handle can be
79			safely destroyed, but it's not clear how mysql itself will react to a
80			cancellation.
81
82	root	1.4	=head1 FUNCTIONS
83
84			Coro::Mysql offers a single user-accessible function:
85
86	root	1.1	=over 4
87
88			=cut
89
90			package Coro::Mysql;
91
92			use strict qw(vars subs);
93			no warnings;
94
95			use Scalar::Util ();
96			use Carp qw(croak);
97
98			use Guard;
99	root	1.9	use AnyEvent ();
100			use Coro ();
101			use Coro::AnyEvent (); # not necessary with newer Coro versions
102	root	1.1
103			# we need this extra indirection, as Coro doesn't support
104			# calling SLF-like functions via call_sv.
105
106			sub readable { &Coro::Handle::FH::readable }
107			sub writable { &Coro::Handle::FH::writable }
108
109			BEGIN {
110	root	1.18	our $VERSION = '1.24';
111	root	1.1
112			require XSLoader;
113			XSLoader::load Coro::Mysql::, $VERSION;
114			}
115
116			=item $DBH = Coro::Mysql::unblock $DBH
117
118			This function takes a DBI database handles and "patches" it
119			so it becomes compatible to Coro threads.
120
121			After that, it returns the patched handle - you should always use the
122			newly returned database handle.
123
124	root	1.4	It is safe to call this function on any database handle (or just about any
125			value), but it will only do anything to L<DBD::mysql> handles, others are
126			returned unchanged. That means it is harmless when applied to database
127			handles of other databases.
128	root	1.3
129	root	1.10	It is also safe to pass C<undef>, so code like this is works as expected:
130
131			my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
132			or die $DBI::errstr;
133
134	root	1.1	=cut
135
136			sub unblock {
137			my ($DBH) = @_;
138
139	root	1.10	if ($DBH && $DBH->{Driver}{Name} eq "mysql") {
140	root	1.3	my $sock = $DBH->{sock};
141
142			open my $fh, "+>&" . $DBH->{sockfd}
143			or croak "Coro::Mysql unable to clone mysql fd";
144	root	1.1
145	root	1.9	if (AnyEvent::detect ne "AnyEvent::Impl::EV" \|\| !_use_ev) {
146			require Coro::Handle;
147			$fh = Coro::Handle::unblock ($fh);
148			}
149	root	1.1
150	root	1.18	warn "$sock,$DBH->{sockfd},$DBH->{mysql_clientversion},$fh\n";#d#
151	root	1.15	_patch $sock, $DBH->{sockfd}, $DBH->{mysql_clientversion}, $fh, tied *$$fh;
152	root	1.3	}
153	root	1.1
154			$DBH
155			}
156
157			1;
158
159			=back
160
161	root	1.4	=head1 USAGE EXAMPLE
162
163			This example uses L<PApp::SQL> and L<Coro::on_enter> to implement a
164			function C<with_db>, that connects to a database, uses C<unblock> on the
165			resulting handle and then makes sure that C<$PApp::SQL::DBH> is set to the
166			(per-thread) database handle when the given thread is running (it does not
167			restore any previous value of $PApp::SQL::DBH, however):
168
169			use Coro;
170			use Coro::Mysql;
171			use PApp::SQL;
172
173			sub with_db($$$&) {
174			my ($database, $user, $pass, $cb) = @_;
175
176	root	1.9	my $dbh = DBI->connect ($database, $user, $pass)->Coro::Mysql::unblock
177	root	1.4	or die $DBI::errstr;
178
179			Coro::on_enter { $PApp::SQL::DBH = $dbh };
180
181			$cb->();
182			}
183
184			This function makes it possible to easily use L<PApp::SQL> with
185			L<Coro::Mysql>, without worrying about database handles.
186
187			# now start 10 threads doing stuff
188			async {
189
190			with_db "DBI:mysql:test", "", "", sub {
191			sql_exec "update table set col = 5 where id = 7";
192
193			my $st = sql_exec \my ($id, $name),
194			"select id, name from table where name like ?",
195			"a%";
196
197			while ($st->fetch) {
198			...
199			}
200
201			my $id = sql_insertid sql_exec "insert into table values (1,2,3)";
202			# etc.
203			};
204
205			} for 1..10;
206
207			=head1 SEE ALSO
208
209			L<Coro>, L<PApp::SQL> (a user friendly but efficient wrapper around DBI).
210
211	root	1.13	=head1 HISTORY
212
213			This module was initially hacked together within a few hours on a long
214			flight to Malaysia, and seems to have worked ever since, with minor
215			adjustments for newer libmysqlclient libraries.
216
217	root	1.1	=head1 AUTHOR
218
219			Marc Lehmann <schmorp@schmorp.de>
220			http://home.schmorp.de/
221
222			=cut
223