[ViewVC] Diff of: cvs/Urlader/Urlader.pm

Comparing Urlader/Urlader.pm (file contents):
Revision 1.2 by root, Tue Jan 3 10:37:45 2012 UTC vs.
Revision 1.8 by root, Mon Feb 6 22:09:40 2012 UTC

…		…
6		6
7	use Urlader;	7	use Urlader;
8		8
9	=head1 DESCRIPTION	9	=head1 DESCRIPTION
10		10
		11	Urlader (that's german for "bootloader" btw.) was created out of
		12	frustration over PAR always being horribly slow, again not working, again
		13	not being flexible enough for simple things such as software upgrades, and
		14	again causing mysterious missing file issues on various platforms.
		15
		16	That doesn't mean this module replaces PAR, in fact, you should stay with
		17	PAR for many reasons right now, user-friendlyness is one of them.
		18
		19	However, if you want to make single-file distributions out of your perl
		20	programs (or python, or C or whatever), and you are prepared to fiddle
		21	a LOT, this module might provide a faster and more versatile deployment
		22	technique then PAR. Well, if it ever gets finished.
		23
		24	Also, I<nothing in this module is considered very stable yet>, and it's
		25	far from feature-complete.
		26
		27	Having said all that, Urlader basically provides three services:
		28
11	=over 4	29	=over 4
12		30
		31	=item A simple archiver that packs a directory tree into a single file.
		32
		33	=item A small C program that works on windows and unix, which unpacks an attached
		34	archive and runs a program (perl, python, whatever...).
		35
		36	=item A perl module support module (I<this one>), that can be used to query
		37	the runtime environment, find out where to install updates and so on.
		38
		39	=back
		40
		41	=head1 EXAMPLE
		42
		43	How can it be used to provide single-file executables?
		44
		45	So simple, create a directory with everything that's needed, e.g.:
		46
		47	# find bintree
		48	bintree/perl
		49	bintree/libperl.so.5.10
		50	bintree/run
		51	bintree/pm/Guard.pm
		52	bintree/pm/auto/Guard/Guard.so
		53	bintree/pm/XSLoader.pm
		54	bintree/pm/DynaLoader.pm
		55	bintree/pm/Config.pm
		56	bintree/pm/strict.pm
		57	bintree/pm/vars.pm
		58	bintree/pm/warnings.pm
		59
		60	# cat bintree/run
		61	@INC = ("pm", "."); # "." works around buggy AutoLoader
		62	use Guard;
		63	guard { warn "hello, world!\n" }; # just to show off
		64	exit 0; # tell the urlader that everything was fine
		65
		66	Then pack it:
		67
		68	# wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86
		69	# urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \
		70	LD_LIBRARY_PATH=. ./perl run \
		71	>myprog
		72	# chmod 755 myprog
		73
		74	The packing step takes a binary loader and appends all the files in the
		75	directory tree, plus some meta information.
		76
		77	The resulting file is an executable that, when run, will unpack all the
		78	files and run the embedded program.
		79
		80	=head1 CONCEPTS
		81
		82	=over 4
		83
		84	=item urlader
		85
		86	A small (hopefully) and relatively portable (hopefully) binary that is
		87	prepended to a pack file to make it executable.
		88
		89	You can build it yourself from sources (see F<prebuilt/Makefile> in the
		90	distribution) or use one of the precompiled ones at:
		91
		92	http://urlader.schmorp.de/prebuilt/1.0/
		93
		94	The F<README> there has further information on the binaries provided.
		95
		96	=item exe_id
		97
		98	A string that uniquely identifies your program - all branches of it. It
		99	must consist of the characters C<A-Za-z0-9_-> only and should be a valid
		100	directory name on all systems you want to deploy on.
		101
		102	=item exe_ver
		103
		104	A string the uniquely identifies the contents of the archive, i.e. the
		105	version. It has the same restrictions as the C<exe_id>, and should be
		106	fixed-length, as Urlader assumes lexicographically higher versions are
		107	newer, and thus preferable.
		108
		109	=item pack file (archive)
		110
		111	This contains the C<exe_id>, the C<exe_ver>, a number of environment
		112	variable assignments, the program name to execute, the initial arguments
		113	it receives, and finally, a list of files (with contents :) and
		114	directories.
		115
		116	=item override
		117
		118	When the urlader starts, it first finds out what C<exe_id> is
		119	embedded in it. It then looks for an override file for this id
		120	(F<$URLADER_EXE_DIR/override>) and verifies that it is for the same
		121	C<exe_id>, and the version is newer. If this is the case, then it will
		122	unpack and run the override file instead of unpacking the files attched to
		123	itself.
		124
		125	This way one can implement software upgrades - download a new executable,
		126	write it safely to disk and move it to the override path.
		127
		128	=back
		129
		130	=head1 ENVIRONMENT VARIABLES
		131
		132	The urlader sets and maintains the following environment variables, in
		133	addition to any variables specified on the commandline. The values in
		134	parentheses are typical (but not gauranteed) values for unix - on windows,
		135	F<~/.urlader> is replaced by F<%AppData%/urlader>.
		136
		137	=over 4
		138
		139	=item URLADER_VERSION (C<1.0>)
		140
		141	Set to the version of the urlader binary itself. All versions with the
		142	same major number should be compatible to older versions with the same
		143	major number.
		144
		145	=item URLADER_DATADIR (F<~/.urlader>)
		146
		147	The data directory used to store whatever urlader needs to store.
		148
		149	=item URLADER_CURRDIR
		150
		151	This is set to the full path of the current working directory where
		152	the urlader was started. Atfer unpacking, the urlader changes to the
		153	C<URLADER_EXECDIR>, so any relative paths should be resolved via this
		154	path.
		155
		156	=item URLADER_EXEPATH
		157
		158	This is set to the path of the urlader executable itself, usually relative
		159	to F<$URLADER_CURRDIR>.
		160
		161	=item URLADER_EXE_ID
		162
		163	This stores the executable id of the pack file attached to the urlader.
		164
		165	=item URLADER_EXE_VER
		166
		167	This is the executable version of the pack file attached to the urlader,
		168	or the override, whichever was newer. Or in other words, this is the
		169	version of the application running at the moment.
		170
		171	=item URLADER_EXE_DIR (F<~/.urlader/$URLADER_EXE_ID>>
		172
		173	The directory where urlader stores files related to the executable with
		174	the given id.
		175
		176	=item URLADER_EXECDIR (F<~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER>)
		177
		178	The directory where the files from the pack file are unpacked and the
		179	program is being run. Also the working directory of the program when it is
		180	run.
		181
		182	=item URLADER_OVERRIDE (empty or F<override>)
		183
		184	The override file used, if any, relative to F<$URLADER_EXECDIR>. This is
		185	either missing, when no override was used, or the string F<override>, as
		186	thta is currently the only override file urlader is looking for.
		187
		188	=back
		189
		190	=head1 FUNCTIONS AND VARIABLES IN THIS MODULE
		191
		192	=over 4
		193
13	=cut	194	=cut
14		195
15	package Urlader;	196	package Urlader;
16		197
17	use common::sense;	198	use common::sense;
18		199
19	BEGIN {	200	BEGIN {
20	our $VERSION = '0.2';	201	our $VERSION = '1.0';
21		202
22	use XSLoader;	203	use XSLoader;
23	XSLoader::load __PACKAGE__, $VERSION;	204	XSLoader::load __PACKAGE__, $VERSION;
24	}	205	}
25		206
		207	=item $Urlader::URLADER_VERSION
		208
		209	Set to the urlader version (C<URLADER_VERSION>) when the program is
		210	running form within urlader, undef otherwise.
		211
		212	=item $Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER, $Urlader::EXE_DIR, $Urlader::EXECDIR
		213
		214	Contain the same value as the environment variable of the (almost) same
		215	name. You should prefer these, though, as these might even be set to
		216	correct values when not running form within an urlader environment.
		217
		218	=cut
		219
		220	our $URLADER_VERSION; # only set when running under urlader
26	our $DATADIR;	221	our $DATADIR;
27	our $EXE_ID;	222	our $EXE_ID;
28	our $EXE_VER;	223	our $EXE_VER;
29	our $EXE_DIR; # %AppData%/urlader/EXE_ID	224	our $EXE_DIR; # %AppData%/urlader/EXE_ID
30	our $EXECDIR; # %AppData%/urlader/EXE_ID/i-EXE_VER	225	our $EXECDIR; # %AppData%/urlader/EXE_ID/i-EXE_VER
31		226
32	sub _get_env {	227	sub _get_env {
		228	$URLADER_VERSION = getenv "URLADER_VERSION";
33	$DATADIR = getenv "URLADER_DATADIR";	229	$DATADIR = getenv "URLADER_DATADIR";
34	$EXE_ID = getenv "URLADER_EXE_ID";	230	$EXE_ID = getenv "URLADER_EXE_ID";
35	$EXE_VER = getenv "URLADER_EXE_VER";	231	$EXE_VER = getenv "URLADER_EXE_VER";
36	$EXE_DIR = getenv "URLADER_EXE_DIR"; # %AppData%/urlader/EXE_ID	232	$EXE_DIR = getenv "URLADER_EXE_DIR"; # %AppData%/urlader/EXE_ID
37	$EXECDIR = getenv "URLADER_EXECDIR"; # %AppData%/urlader/EXE_ID/i-EXE_VER	233	$EXECDIR = getenv "URLADER_EXECDIR"; # %AppData%/urlader/EXE_ID/i-EXE_VER
38	}	234	}
39		235
40	_get_env;	236	_get_env;
		237
		238	=item Urlader::set_exe_info $exe_id, $exe_ver
		239
		240	Sets up the paths and variables as if running the given executable and
		241	version from within urlader.
		242
		243	=cut
41		244
42	sub set_exe_info($$) {	245	sub set_exe_info($$) {
43	_set_datadir unless defined getenv "URLADER_DATADIR";	246	_set_datadir unless defined getenv "URLADER_DATADIR";
44	&_set_exe_info;	247	&_set_exe_info;
45	_get_env;	248	_get_env;
46	}	249	}
47		250
		251	=item $lock = Urlader::lock $path, $exclusive, $wait
		252
		253	Tries to acquire a lock on the given path (which must specify a file which
		254	will be created if necessary). If C<$exclusive> is true, then it tries to
		255	acquire an exclusive lock, otherwise the lock will be shared. If C<$wait>
		256	is true, then it will wait until the lock can be acquired, otherwise it
		257	only attempts to acquire it and returns immediately if it can't.
		258
		259	If successful it returns a lock object - the lock will be given up
		260	when the lock object is destroyed.
		261
		262	If the lock could not be acquired, C<undef> is returned.
		263
		264	This function will probably go awway in the future, but is provided to
		265	assist applications that want to clean up old versions, see "TIPS AND
		266	TRICKS", below.
		267
		268	=cut
48		269
49	1;	270	1;
50		271
51	=back	272	=back
		273
		274	=head1 TIPS AND TRICKS
		275
		276	=over 4
		277
		278	=item Gathering files
		279
		280	Gathering all the files needed for distribution cna be a big
		281	problem. Right now, Urlader does not assist you in this task in any way,
		282	however, just like perl source stripping, it is planned to unbundle the
		283	relevant technology from B<staticperl> (L<http://staticperl.plan9.de>) for
		284	use with this module.
		285
		286	You could always use par to find all library files, unpack the bundle and
		287	add F<perl>, F<libperl> and other support libraries (e.g. F<libgcc_s>).
		288
		289	=item Software update
		290
		291	Updating the software can be done by downloading a new packfile (with the
		292	same C<exe_id> but a higher C<exe_ver> - this can simply be the executable
		293	you create when making a release) and replacing the F<override> file in
		294	the F<$URLADER_EXE_DIR>.
		295
		296	When looking for updates, you should include C<$URLADER_VERSION>,
		297	C<$URLADER_EXE_ID> and C<$URLADER_EXE_VER> - the first two must be
		298	identical for update and currently running program, while the last one
		299	should be lexicographically higher.
		300
		301	Replacing the override file can be done like this:
		302
		303	rename "new-override.tmp", "$Urlader::EXE_DIR/override"
		304	or die "could not replace override";
		305
		306	This can fail on windows when another urlader currently reads it, but
		307	should work on all platforms even when other urlader programs execute
		308	concurrently.
		309
		310	=item Cleaning up old directories
		311
		312	Urlader only packs executables once and then caches them in the
		313	F<$URLADER_EXECDIR>. After upgrades there will be old versions in there
		314	that are not being used anymore. Or are they?
		315
		316	Each instance directory (F<i-*>) in the F<$URLADER_EXE_DIR>) has an
		317	associated lock file (F<i-*.lck>) - while urlader executes an app it keeps
		318	a shared lock on this file.
		319
		320	To detect whether a version is in use or not, you must try to acquire an
		321	exclusive lock, i.e.:
		322
		323	my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0;
		324	if (!$lock) {
		325	# instance dir is not in use and can be safely deleted
		326	}
		327
		328	If an older urlader wants to use an instance that was deleted or is
		329	currently being deleted it will wait until it's gone and simply recreate
		330	it, so while less efficient, deleting instance directories should always
		331	be safe.
		332
		333	The lockfile itself can be deleted as long as you have an exclusive lock
		334	on it (if your platform allows this).
		335
		336	=item A real world project
		337
		338	The only real world project using this that I know of at the moment is the
		339	deliantra client (http://www.deliantra.net for more info).
		340
		341	It uses some scary scripts to build the client and some dependnet modules
		342	(F<build.*>), to gather perl source files into a distribution tree, shared
		343	objects and system shared libraries (some of which have to be patched or,
		344	due to the horrible dll hell on OS X, even renamed), called C<gatherer>,
		345	and a script called C<gendist> to build executable distributions.
		346
		347	These can be found at
		348	L<http://cvs.schmorp.de/deliantra/Deliantra-Client/util/>, but looking at
		349	them can lead to premature blindless.
		350
		351	=item Shared Libraries
		352
		353	It is often desirable to package shared libraries - for example the
		354	Deliantra client packages SD>, Berkely DB, Pango and amny other libraries
		355	that are unlikely to be available on the target system.
		356
		357	This usually requires some fiddling (see below), and additionally some
		358	environment variables to be set.
		359
		360	For example, on ELF systems you usually want F<LD_LIBRARY_PATH=.> and on
		361	OS X, you want F<DYLD_LIBRARY_PATH=.> (these are effectively the default
		362	on windows).
		363
		364	These can most easily be specified when building the packfile:
		365
		366	urlader-util ... LD_LIBRARY_PATH=. ./perl run
		367
		368	=item Portability: RPATH
		369
		370	Often F<perl> is linked against a shared F<libperl.so> - and might be so
		371	using an rpath. Perl extensikns likewise might use an rpath, which means
		372	the binary will mostly ignore LD_LIBRARY_PATH, which leads to trouble.
		373
		374	There is an utility called F<chrpath>, whose F<-d> option can remove the
		375	rpath from binaries, shared library and shared objects.
		376
		377	=item Portability: OS X DLL HELL
		378
		379	OS X has the most severe form of DLL hell I have seen - if you link
		380	against system libraries, which is practically unavoidable, you get
		381	libraries of well-known names (e.g. libjpeg) that have nothing to do with
		382	what you normally expect libjpeg to be, and there is no way to get your
		383	version of libjpeg into your program.
		384
		385	Moreover, even if apple ships well-known libraries (e.g. libiconv), they
		386	often ship patched versions which have a different ABI or even API then
		387	the real releases.
		388
		389	The only way aorund this I found was to change all library names
		390	in my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and
		391	so on), by patching the paths in the share dlibraries and shared
		392	objects. F<install-name-tool> (with F<-id> and F<-change>) works in many
		393	cases, but often paths are embedded indirectly, so you might have to use a
		394	I<dirty> string replacement.
		395
		396	=back
		397
		398	=head1 SECURITY CONSIDERATIONS
		399
		400	The urlader executable itself does not support setuig/setgid operation, or
		401	running with elevated privileges - it does no input sanitisation, and is
		402	trivially exploitable.
52		403
53	=head1 AUTHOR	404	=head1 AUTHOR
54		405
55	Marc Lehmann <schmorp@schmorp.de>	406	Marc Lehmann <schmorp@schmorp.de>
56	http://home.schmorp.de/	407	http://home.schmorp.de/

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing Urlader/Urlader.pm (file contents): Revision 1.2 by root, Tue Jan 3 10:37:45 2012 UTC vs. Revision 1.8 by root, Mon Feb 6 22:09:40 2012 UTC

Diff Legend

Comparing Urlader/Urlader.pm (file contents):
Revision 1.2 by root, Tue Jan 3 10:37:45 2012 UTC vs.
Revision 1.8 by root, Mon Feb 6 22:09:40 2012 UTC