ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/Urlader/Urlader.pm
Revision: 1.11
Committed: Sun Jun 17 13:11:25 2012 UTC (11 years, 10 months ago) by root
Branch: MAIN
CVS Tags: rel-1_01
Changes since 1.10: +1 -1 lines
Log Message:
1.01

File Contents

# User Rev Content
1 root 1.1 =head1 NAME
2    
3     Urlader - installer-less single-file independent executables
4    
5     =head1 SYNOPSIS
6    
7     use Urlader;
8    
9     =head1 DESCRIPTION
10    
11 root 1.4 Urlader (that's german for "bootloader" btw.) was created out of
12 root 1.5 frustration over PAR always being horribly slow, again not working, again
13     not being flexible enough for simple things such as software upgrades, and
14     again causing mysterious missing file issues on various platforms.
15 root 1.4
16     That doesn't mean this module replaces PAR, in fact, you should stay with
17 root 1.5 PAR for many reasons right now, user-friendlyness is one of them.
18 root 1.4
19     However, if you want to make single-file distributions out of your perl
20 root 1.5 programs (or python, or C or whatever), and you are prepared to fiddle
21     a LOT, this module might provide a faster and more versatile deployment
22     technique then PAR. Well, if it ever gets finished.
23 root 1.4
24     Also, I<nothing in this module is considered very stable yet>, and it's
25     far from feature-complete.
26    
27     Having said all that, Urlader basically provides three services:
28    
29     =over 4
30    
31     =item A simple archiver that packs a directory tree into a single file.
32    
33     =item A small C program that works on windows and unix, which unpacks an attached
34 root 1.5 archive and runs a program (perl, python, whatever...).
35 root 1.4
36     =item A perl module support module (I<this one>), that can be used to query
37     the runtime environment, find out where to install updates and so on.
38    
39     =back
40    
41     =head1 EXAMPLE
42    
43     How can it be used to provide single-file executables?
44    
45     So simple, create a directory with everything that's needed, e.g.:
46    
47     # find bintree
48     bintree/perl
49     bintree/libperl.so.5.10
50     bintree/run
51     bintree/pm/Guard.pm
52     bintree/pm/auto/Guard/Guard.so
53 root 1.5 bintree/pm/XSLoader.pm
54     bintree/pm/DynaLoader.pm
55     bintree/pm/Config.pm
56     bintree/pm/strict.pm
57     bintree/pm/vars.pm
58     bintree/pm/warnings.pm
59 root 1.4
60     # cat bintree/run
61     @INC = ("pm", "."); # "." works around buggy AutoLoader
62     use Guard;
63     guard { warn "hello, world!\n" }; # just to show off
64     exit 0; # tell the urlader that everything was fine
65    
66     Then pack it:
67    
68     # wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86
69     # urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \
70     LD_LIBRARY_PATH=. ./perl run \
71     >myprog
72     # chmod 755 myprog
73    
74 root 1.5 The packing step takes a binary loader and appends all the files in the
75     directory tree, plus some meta information.
76    
77     The resulting file is an executable that, when run, will unpack all the
78     files and run the embedded program.
79    
80 root 1.4 =head1 CONCEPTS
81    
82     =over 4
83    
84     =item urlader
85    
86     A small (hopefully) and relatively portable (hopefully) binary that is
87     prepended to a pack file to make it executable.
88    
89     You can build it yourself from sources (see F<prebuilt/Makefile> in the
90     distribution) or use one of the precompiled ones at:
91    
92     http://urlader.schmorp.de/prebuilt/1.0/
93    
94     The F<README> there has further information on the binaries provided.
95    
96     =item exe_id
97    
98     A string that uniquely identifies your program - all branches of it. It
99     must consist of the characters C<A-Za-z0-9_-> only and should be a valid
100     directory name on all systems you want to deploy on.
101    
102     =item exe_ver
103    
104     A string the uniquely identifies the contents of the archive, i.e. the
105     version. It has the same restrictions as the C<exe_id>, and should be
106     fixed-length, as Urlader assumes lexicographically higher versions are
107     newer, and thus preferable.
108    
109     =item pack file (archive)
110    
111     This contains the C<exe_id>, the C<exe_ver>, a number of environment
112     variable assignments, the program name to execute, the initial arguments
113     it receives, and finally, a list of files (with contents :) and
114     directories.
115    
116     =item override
117    
118 root 1.5 When the urlader starts, it first finds out what C<exe_id> is
119     embedded in it. It then looks for an override file for this id
120     (F<$URLADER_EXE_DIR/override>) and verifies that it is for the same
121     C<exe_id>, and the version is newer. If this is the case, then it will
122     unpack and run the override file instead of unpacking the files attched to
123     itself.
124    
125     This way one can implement software upgrades - download a new executable,
126     write it safely to disk and move it to the override path.
127    
128     =back
129    
130     =head1 ENVIRONMENT VARIABLES
131    
132     The urlader sets and maintains the following environment variables, in
133     addition to any variables specified on the commandline. The values in
134     parentheses are typical (but not gauranteed) values for unix - on windows,
135     F<~/.urlader> is replaced by F<%AppData%/urlader>.
136    
137     =over 4
138    
139     =item URLADER_VERSION (C<1.0>)
140    
141     Set to the version of the urlader binary itself. All versions with the
142     same major number should be compatible to older versions with the same
143     major number.
144    
145     =item URLADER_DATADIR (F<~/.urlader>)
146    
147     The data directory used to store whatever urlader needs to store.
148    
149     =item URLADER_CURRDIR
150    
151     This is set to the full path of the current working directory where
152     the urlader was started. Atfer unpacking, the urlader changes to the
153     C<URLADER_EXECDIR>, so any relative paths should be resolved via this
154     path.
155    
156     =item URLADER_EXEPATH
157    
158     This is set to the path of the urlader executable itself, usually relative
159     to F<$URLADER_CURRDIR>.
160    
161     =item URLADER_EXE_ID
162    
163     This stores the executable id of the pack file attached to the urlader.
164    
165     =item URLADER_EXE_VER
166    
167     This is the executable version of the pack file attached to the urlader,
168     or the override, whichever was newer. Or in other words, this is the
169     version of the application running at the moment.
170    
171     =item URLADER_EXE_DIR (F<~/.urlader/$URLADER_EXE_ID>>
172    
173     The directory where urlader stores files related to the executable with
174     the given id.
175    
176     =item URLADER_EXECDIR (F<~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER>)
177    
178     The directory where the files from the pack file are unpacked and the
179     program is being run. Also the working directory of the program when it is
180     run.
181    
182     =item URLADER_OVERRIDE (empty or F<override>)
183    
184     The override file used, if any, relative to F<$URLADER_EXECDIR>. This is
185     either missing, when no override was used, or the string F<override>, as
186     thta is currently the only override file urlader is looking for.
187    
188 root 1.4 =back
189    
190     =head1 FUNCTIONS AND VARIABLES IN THIS MODULE
191    
192 root 1.1 =over 4
193    
194     =cut
195    
196     package Urlader;
197    
198     use common::sense;
199    
200     BEGIN {
201 root 1.11 our $VERSION = '1.01';
202 root 1.1
203     use XSLoader;
204     XSLoader::load __PACKAGE__, $VERSION;
205     }
206    
207 root 1.5 =item $Urlader::URLADER_VERSION
208    
209     Set to the urlader version (C<URLADER_VERSION>) when the program is
210     running form within urlader, undef otherwise.
211    
212     =item $Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER, $Urlader::EXE_DIR, $Urlader::EXECDIR
213    
214     Contain the same value as the environment variable of the (almost) same
215     name. You should prefer these, though, as these might even be set to
216     correct values when not running form within an urlader environment.
217    
218     =cut
219    
220     our $URLADER_VERSION; # only set when running under urlader
221 root 1.1 our $DATADIR;
222     our $EXE_ID;
223     our $EXE_VER;
224     our $EXE_DIR; # %AppData%/urlader/EXE_ID
225     our $EXECDIR; # %AppData%/urlader/EXE_ID/i-EXE_VER
226    
227     sub _get_env {
228 root 1.3 $URLADER_VERSION = getenv "URLADER_VERSION";
229     $DATADIR = getenv "URLADER_DATADIR";
230     $EXE_ID = getenv "URLADER_EXE_ID";
231     $EXE_VER = getenv "URLADER_EXE_VER";
232     $EXE_DIR = getenv "URLADER_EXE_DIR"; # %AppData%/urlader/EXE_ID
233     $EXECDIR = getenv "URLADER_EXECDIR"; # %AppData%/urlader/EXE_ID/i-EXE_VER
234 root 1.1 }
235    
236     _get_env;
237    
238 root 1.5 =item Urlader::set_exe_info $exe_id, $exe_ver
239    
240     Sets up the paths and variables as if running the given executable and
241     version from within urlader.
242    
243     =cut
244    
245 root 1.1 sub set_exe_info($$) {
246 root 1.2 _set_datadir unless defined getenv "URLADER_DATADIR";
247 root 1.1 &_set_exe_info;
248     _get_env;
249     }
250    
251 root 1.5 =item $lock = Urlader::lock $path, $exclusive, $wait
252    
253     Tries to acquire a lock on the given path (which must specify a file which
254     will be created if necessary). If C<$exclusive> is true, then it tries to
255     acquire an exclusive lock, otherwise the lock will be shared. If C<$wait>
256     is true, then it will wait until the lock can be acquired, otherwise it
257     only attempts to acquire it and returns immediately if it can't.
258    
259 root 1.9 If successful it returns a lock object - the lock will be given up when
260     the lock object is destroyed or when the process exits (even on a crash)
261     and has a good chance of working on network drives as well.
262 root 1.5
263     If the lock could not be acquired, C<undef> is returned.
264    
265 root 1.9 This function is provided to assist applications that want to clean up old
266     versions, see "TIPS AND TRICKS", below.
267 root 1.5
268     =cut
269 root 1.1
270     1;
271    
272     =back
273    
274 root 1.5 =head1 TIPS AND TRICKS
275    
276     =over 4
277    
278     =item Gathering files
279    
280 root 1.10 Gathering all the files needed for distribution can be a big
281 root 1.5 problem. Right now, Urlader does not assist you in this task in any way,
282     however, just like perl source stripping, it is planned to unbundle the
283     relevant technology from B<staticperl> (L<http://staticperl.plan9.de>) for
284     use with this module.
285    
286     You could always use par to find all library files, unpack the bundle and
287     add F<perl>, F<libperl> and other support libraries (e.g. F<libgcc_s>).
288    
289     =item Software update
290    
291     Updating the software can be done by downloading a new packfile (with the
292     same C<exe_id> but a higher C<exe_ver> - this can simply be the executable
293     you create when making a release) and replacing the F<override> file in
294     the F<$URLADER_EXE_DIR>.
295    
296     When looking for updates, you should include C<$URLADER_VERSION>,
297     C<$URLADER_EXE_ID> and C<$URLADER_EXE_VER> - the first two must be
298     identical for update and currently running program, while the last one
299     should be lexicographically higher.
300    
301     Replacing the override file can be done like this:
302    
303     rename "new-override.tmp", "$Urlader::EXE_DIR/override"
304     or die "could not replace override";
305    
306     This can fail on windows when another urlader currently reads it, but
307     should work on all platforms even when other urlader programs execute
308     concurrently.
309    
310     =item Cleaning up old directories
311    
312     Urlader only packs executables once and then caches them in the
313     F<$URLADER_EXECDIR>. After upgrades there will be old versions in there
314     that are not being used anymore. Or are they?
315    
316     Each instance directory (F<i-*>) in the F<$URLADER_EXE_DIR>) has an
317     associated lock file (F<i-*.lck>) - while urlader executes an app it keeps
318     a shared lock on this file.
319    
320     To detect whether a version is in use or not, you must try to acquire an
321     exclusive lock, i.e.:
322    
323     my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0;
324     if (!$lock) {
325     # instance dir is not in use and can be safely deleted
326     }
327    
328     If an older urlader wants to use an instance that was deleted or is
329     currently being deleted it will wait until it's gone and simply recreate
330     it, so while less efficient, deleting instance directories should always
331     be safe.
332    
333     The lockfile itself can be deleted as long as you have an exclusive lock
334     on it (if your platform allows this).
335    
336     =item A real world project
337    
338     The only real world project using this that I know of at the moment is the
339     deliantra client (http://www.deliantra.net for more info).
340    
341     It uses some scary scripts to build the client and some dependnet modules
342     (F<build.*>), to gather perl source files into a distribution tree, shared
343     objects and system shared libraries (some of which have to be patched or,
344     due to the horrible dll hell on OS X, even renamed), called C<gatherer>,
345     and a script called C<gendist> to build executable distributions.
346    
347     These can be found at
348     L<http://cvs.schmorp.de/deliantra/Deliantra-Client/util/>, but looking at
349     them can lead to premature blindless.
350    
351 root 1.6 =item Shared Libraries
352    
353     It is often desirable to package shared libraries - for example the
354     Deliantra client packages SD>, Berkely DB, Pango and amny other libraries
355     that are unlikely to be available on the target system.
356    
357     This usually requires some fiddling (see below), and additionally some
358     environment variables to be set.
359    
360     For example, on ELF systems you usually want F<LD_LIBRARY_PATH=.> and on
361     OS X, you want F<DYLD_LIBRARY_PATH=.> (these are effectively the default
362     on windows).
363    
364     These can most easily be specified when building the packfile:
365    
366     urlader-util ... LD_LIBRARY_PATH=. ./perl run
367    
368     =item Portability: RPATH
369    
370     Often F<perl> is linked against a shared F<libperl.so> - and might be so
371     using an rpath. Perl extensikns likewise might use an rpath, which means
372     the binary will mostly ignore LD_LIBRARY_PATH, which leads to trouble.
373    
374     There is an utility called F<chrpath>, whose F<-d> option can remove the
375     rpath from binaries, shared library and shared objects.
376    
377     =item Portability: OS X DLL HELL
378    
379     OS X has the most severe form of DLL hell I have seen - if you link
380     against system libraries, which is practically unavoidable, you get
381     libraries of well-known names (e.g. libjpeg) that have nothing to do with
382     what you normally expect libjpeg to be, and there is no way to get your
383     version of libjpeg into your program.
384    
385     Moreover, even if apple ships well-known libraries (e.g. libiconv), they
386     often ship patched versions which have a different ABI or even API then
387     the real releases.
388    
389     The only way aorund this I found was to change all library names
390     in my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and
391     so on), by patching the paths in the share dlibraries and shared
392     objects. F<install-name-tool> (with F<-id> and F<-change>) works in many
393     cases, but often paths are embedded indirectly, so you might have to use a
394     I<dirty> string replacement.
395    
396 root 1.5 =back
397    
398 root 1.7 =head1 SECURITY CONSIDERATIONS
399    
400     The urlader executable itself does not support setuig/setgid operation, or
401     running with elevated privileges - it does no input sanitisation, and is
402     trivially exploitable.
403    
404 root 1.1 =head1 AUTHOR
405    
406     Marc Lehmann <schmorp@schmorp.de>
407     http://home.schmorp.de/
408    
409     =cut
410