[ViewVC] Annotation of: cvs/Urlader/README

NAME
    Urlader - installer-less single-file independent executables

SYNOPSIS
     use Urlader;

DESCRIPTION
    Urlader (that's german for "bootloader" btw.) was created out of
    frustration over PAR always being horribly slow, again not working,
    again not being flexible enough for simple things such as software
    upgrades, and again causing mysterious missing file issues on various
    platforms.

    That doesn't mean this module replaces PAR, in fact, you should stay
    with PAR for many reasons right now, user-friendlyness is one of them.

    However, if you want to make single-file distributions out of your perl
    programs (or python, or C or whatever), and you are prepared to fiddle a
    LOT, this module might provide a faster and more versatile deployment
    technique then PAR. Well, if it ever gets finished.

    Also, *nothing in this module is considered very stable yet*, and it's
    far from feature-complete.

    Having said all that, Urlader basically provides three services:

    A simple archiver that packs a directory tree into a single file.
    A small C program that works on windows and unix, which unpacks an
    attached archive and runs a program (perl, python, whatever...).
    A perl module support module (*this one*), that can be used to query the
    runtime environment, find out where to install updates and so on.

EXAMPLE
    How can it be used to provide single-file executables?

    So simple, create a directory with everything that's needed, e.g.:

       # find bintree
       bintree/perl
       bintree/libperl.so.5.10
       bintree/run
       bintree/pm/Guard.pm
       bintree/pm/auto/Guard/Guard.so
       bintree/pm/XSLoader.pm
       bintree/pm/DynaLoader.pm
       bintree/pm/Config.pm
       bintree/pm/strict.pm
       bintree/pm/vars.pm
       bintree/pm/warnings.pm

       # cat bintree/run
       @INC = ("pm", "."); # "." works around buggy AutoLoader
       use Guard;
       guard { warn "hello, world!\n" }; # just to show off
       exit 0; # tell the urlader that everything was fine

    Then pack it:

       # wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86
       # urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \
                      LD_LIBRARY_PATH=. ./perl run \
                      >myprog
       # chmod 755 myprog

    The packing step takes a binary loader and appends all the files in the
    directory tree, plus some meta information.

    The resulting file is an executable that, when run, will unpack all the
    files and run the embedded program.

CONCEPTS
    urlader
        A small (hopefully) and relatively portable (hopefully) binary that
        is prepended to a pack file to make it executable.

        You can build it yourself from sources (see prebuilt/Makefile in the
        distribution) or use one of the precompiled ones at:

           http://urlader.schmorp.de/prebuilt/1.0/

        The README there has further information on the binaries provided.

    exe_id
        A string that uniquely identifies your program - all branches of it.
        It must consist of the characters "A-Za-z0-9_-" only and should be a
        valid directory name on all systems you want to deploy on.

    exe_ver
        A string the uniquely identifies the contents of the archive, i.e.
        the version. It has the same restrictions as the "exe_id", and
        should be fixed-length, as Urlader assumes lexicographically higher
        versions are newer, and thus preferable.

    pack file (archive)
        This contains the "exe_id", the "exe_ver", a number of environment
        variable assignments, the program name to execute, the initial
        arguments it receives, and finally, a list of files (with contents
        :) and directories.

    override
        When the urlader starts, it first finds out what "exe_id" is
        embedded in it. It then looks for an override file for this id
        ($URLADER_EXE_DIR/override) and verifies that it is for the same
        "exe_id", and the version is newer. If this is the case, then it
        will unpack and run the override file instead of unpacking the files
        attched to itself.

        This way one can implement software upgrades - download a new
        executable, write it safely to disk and move it to the override
        path.

ENVIRONMENT VARIABLES
    The urlader sets and maintains the following environment variables, in
    addition to any variables specified on the commandline. The values in
    parentheses are typical (but not gauranteed) values for unix - on
    windows, ~/.urlader is replaced by %AppData%/urlader.

    URLADER_VERSION (1.0)
        Set to the version of the urlader binary itself. All versions with
        the same major number should be compatible to older versions with
        the same major number.

    URLADER_DATADIR (~/.urlader)
        The data directory used to store whatever urlader needs to store.

    URLADER_CURRDIR
        This is set to the full path of the current working directory where
        the urlader was started. Atfer unpacking, the urlader changes to the
        "URLADER_EXECDIR", so any relative paths should be resolved via this
        path.

    URLADER_EXEPATH
        This is set to the path of the urlader executable itself, usually
        relative to $URLADER_CURRDIR.

    URLADER_EXE_ID
        This stores the executable id of the pack file attached to the
        urlader.

    URLADER_EXE_VER
        This is the executable version of the pack file attached to the
        urlader, or the override, whichever was newer. Or in other words,
        this is the version of the application running at the moment.

    URLADER_EXE_DIR (~/.urlader/$URLADER_EXE_ID>
        The directory where urlader stores files related to the executable
        with the given id.

    URLADER_EXECDIR (~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER)
        The directory where the files from the pack file are unpacked and
        the program is being run. Also the working directory of the program
        when it is run.

    URLADER_OVERRIDE (empty or override)
        The override file used, if any, relative to $URLADER_EXECDIR. This
        is either missing, when no override was used, or the string
        override, as thta is currently the only override file urlader is
        looking for.

FUNCTIONS AND VARIABLES IN THIS MODULE
    $Urlader::URLADER_VERSION
        Set to the urlader version ("URLADER_VERSION") when the program is
        running form within urlader, undef otherwise.

    $Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER,
    $Urlader::EXE_DIR, $Urlader::EXECDIR
        Contain the same value as the environment variable of the (almost)
        same name. You should prefer these, though, as these might even be
        set to correct values when not running form within an urlader
        environment.

    Urlader::set_exe_info $exe_id, $exe_ver
        Sets up the paths and variables as if running the given executable
        and version from within urlader.

    $lock = Urlader::lock $path, $exclusive, $wait
        Tries to acquire a lock on the given path (which must specify a file
        which will be created if necessary). If $exclusive is true, then it
        tries to acquire an exclusive lock, otherwise the lock will be
        shared. If $wait is true, then it will wait until the lock can be
        acquired, otherwise it only attempts to acquire it and returns
        immediately if it can't.

        If successful it returns a lock object - the lock will be given up
        when the lock object is destroyed.

        If the lock could not be acquired, "undef" is returned.

        This function will probably go awway in the future, but is provided
        to assist applications that want to clean up old versions, see "TIPS
        AND TRICKS", below.

TIPS AND TRICKS
    Gathering files
        Gathering all the files needed for distribution cna be a big
        problem. Right now, Urlader does not assist you in this task in any
        way, however, just like perl source stripping, it is planned to
        unbundle the relevant technology from staticperl
        (<http://staticperl.plan9.de>) for use with this module.

        You could always use par to find all library files, unpack the
        bundle and add perl, libperl and other support libraries (e.g.
        libgcc_s).

    Software update
        Updating the software can be done by downloading a new packfile
        (with the same "exe_id" but a higher "exe_ver" - this can simply be
        the executable you create when making a release) and replacing the
        override file in the $URLADER_EXE_DIR.

        When looking for updates, you should include $URLADER_VERSION,
        $URLADER_EXE_ID and $URLADER_EXE_VER - the first two must be
        identical for update and currently running program, while the last
        one should be lexicographically higher.

        Replacing the override file can be done like this:

           rename "new-override.tmp", "$Urlader::EXE_DIR/override"
              or die "could not replace override";

        This can fail on windows when another urlader currently reads it,
        but should work on all platforms even when other urlader programs
        execute concurrently.

    Cleaning up old directories
        Urlader only packs executables once and then caches them in the
        $URLADER_EXECDIR. After upgrades there will be old versions in there
        that are not being used anymore. Or are they?

        Each instance directory (i-*) in the $URLADER_EXE_DIR) has an
        associated lock file (i-*.lck) - while urlader executes an app it
        keeps a shared lock on this file.

        To detect whether a version is in use or not, you must try to
        acquire an exclusive lock, i.e.:

          my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0;
          if (!$lock) {
             # instance dir is not in use and can be safely deleted
          }

        If an older urlader wants to use an instance that was deleted or is
        currently being deleted it will wait until it's gone and simply
        recreate it, so while less efficient, deleting instance directories
        should always be safe.

        The lockfile itself can be deleted as long as you have an exclusive
        lock on it (if your platform allows this).

    A real world project
        The only real world project using this that I know of at the moment
        is the deliantra client (http://www.deliantra.net for more info).

        It uses some scary scripts to build the client and some dependnet
        modules (build.*), to gather perl source files into a distribution
        tree, shared objects and system shared libraries (some of which have
        to be patched or, due to the horrible dll hell on OS X, even
        renamed), called "gatherer", and a script called "gendist" to build
        executable distributions.

        These can be found at
        <http://cvs.schmorp.de/deliantra/Deliantra-Client/util/>, but
        looking at them can lead to premature blindless.

    Shared Libraries
        It is often desirable to package shared libraries - for example the
        Deliantra client packages SD>, Berkely DB, Pango and amny other
        libraries that are unlikely to be available on the target system.

        This usually requires some fiddling (see below), and additionally
        some environment variables to be set.

        For example, on ELF systems you usually want LD_LIBRARY_PATH=. and
        on OS X, you want DYLD_LIBRARY_PATH=. (these are effectively the
        default on windows).

        These can most easily be specified when building the packfile:

           urlader-util ... LD_LIBRARY_PATH=. ./perl run

    Portability: RPATH
        Often perl is linked against a shared libperl.so - and might be so
        using an rpath. Perl extensikns likewise might use an rpath, which
        means the binary will mostly ignore LD_LIBRARY_PATH, which leads to
        trouble.

        There is an utility called chrpath, whose -d option can remove the
        rpath from binaries, shared library and shared objects.

    Portability: OS X DLL HELL
        OS X has the most severe form of DLL hell I have seen - if you link
        against system libraries, which is practically unavoidable, you get
        libraries of well-known names (e.g. libjpeg) that have nothing to do
        with what you normally expect libjpeg to be, and there is no way to
        get your version of libjpeg into your program.

        Moreover, even if apple ships well-known libraries (e.g. libiconv),
        they often ship patched versions which have a different ABI or even
        API then the real releases.

        The only way aorund this I found was to change all library names in
        my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and so
        on), by patching the paths in the share dlibraries and shared
        objects. install-name-tool (with -id and -change) works in many
        cases, but often paths are embedded indirectly, so you might have to
        use a *dirty* string replacement.

SECURITY CONSIDERATIONS
    The urlader executable itself does not support setuig/setgid operation,
    or running with elevated privileges - it does no input sanitisation, and
    is trivially exploitable.

AUTHOR
     Marc Lehmann <schmorp@schmorp.de>
     http://home.schmorp.de/

Revision:	1.4
Committed:	Mon Feb 6 22:09:40 2012 UTC (14 years, 7 months ago) by root
Branch:	MAIN
CVS Tags:	rel-1_0
Changes since 1.3:	+48 -0 lines
Log Message:	1.0
#	User	Rev	Content
1	root	1.2	NAME
2			Urlader - installer-less single-file independent executables
3
4			SYNOPSIS
5			use Urlader;
6
7			DESCRIPTION
8			Urlader (that's german for "bootloader" btw.) was created out of
9	root	1.3	frustration over PAR always being horribly slow, again not working,
10			again not being flexible enough for simple things such as software
11			upgrades, and again causing mysterious missing file issues on various
12			platforms.
13	root	1.2
14			That doesn't mean this module replaces PAR, in fact, you should stay
15	root	1.3	with PAR for many reasons right now, user-friendlyness is one of them.
16	root	1.2
17			However, if you want to make single-file distributions out of your perl
18			programs (or python, or C or whatever), and you are prepared to fiddle a
19	root	1.3	LOT, this module might provide a faster and more versatile deployment
20			technique then PAR. Well, if it ever gets finished.
21	root	1.2
22			Also, nothing in this module is considered very stable yet, and it's
23			far from feature-complete.
24
25			Having said all that, Urlader basically provides three services:
26
27			A simple archiver that packs a directory tree into a single file.
28			A small C program that works on windows and unix, which unpacks an
29	root	1.3	attached archive and runs a program (perl, python, whatever...).
30	root	1.2	A perl module support module (this one), that can be used to query the
31			runtime environment, find out where to install updates and so on.
32
33			EXAMPLE
34			How can it be used to provide single-file executables?
35
36			So simple, create a directory with everything that's needed, e.g.:
37
38			# find bintree
39			bintree/perl
40			bintree/libperl.so.5.10
41			bintree/run
42			bintree/pm/Guard.pm
43			bintree/pm/auto/Guard/Guard.so
44	root	1.3	bintree/pm/XSLoader.pm
45			bintree/pm/DynaLoader.pm
46			bintree/pm/Config.pm
47			bintree/pm/strict.pm
48			bintree/pm/vars.pm
49			bintree/pm/warnings.pm
50	root	1.2
51			# cat bintree/run
52			@INC = ("pm", "."); # "." works around buggy AutoLoader
53			use Guard;
54			guard { warn "hello, world!\n" }; # just to show off
55			exit 0; # tell the urlader that everything was fine
56
57			Then pack it:
58
59			# wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86
60			# urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \
61			LD_LIBRARY_PATH=. ./perl run \
62			>myprog
63			# chmod 755 myprog
64
65	root	1.3	The packing step takes a binary loader and appends all the files in the
66			directory tree, plus some meta information.
67
68			The resulting file is an executable that, when run, will unpack all the
69			files and run the embedded program.
70
71	root	1.2	CONCEPTS
72			urlader
73			A small (hopefully) and relatively portable (hopefully) binary that
74			is prepended to a pack file to make it executable.
75
76			You can build it yourself from sources (see prebuilt/Makefile in the
77			distribution) or use one of the precompiled ones at:
78
79			http://urlader.schmorp.de/prebuilt/1.0/
80
81			The README there has further information on the binaries provided.
82
83			exe_id
84			A string that uniquely identifies your program - all branches of it.
85			It must consist of the characters "A-Za-z0-9_-" only and should be a
86			valid directory name on all systems you want to deploy on.
87
88			exe_ver
89			A string the uniquely identifies the contents of the archive, i.e.
90			the version. It has the same restrictions as the "exe_id", and
91			should be fixed-length, as Urlader assumes lexicographically higher
92			versions are newer, and thus preferable.
93
94			pack file (archive)
95			This contains the "exe_id", the "exe_ver", a number of environment
96			variable assignments, the program name to execute, the initial
97			arguments it receives, and finally, a list of files (with contents
98			:) and directories.
99
100			override
101	root	1.3	When the urlader starts, it first finds out what "exe_id" is
102			embedded in it. It then looks for an override file for this id
103			($URLADER_EXE_DIR/override) and verifies that it is for the same
104			"exe_id", and the version is newer. If this is the case, then it
105			will unpack and run the override file instead of unpacking the files
106			attched to itself.
107
108			This way one can implement software upgrades - download a new
109			executable, write it safely to disk and move it to the override
110			path.
111
112			ENVIRONMENT VARIABLES
113			The urlader sets and maintains the following environment variables, in
114			addition to any variables specified on the commandline. The values in
115			parentheses are typical (but not gauranteed) values for unix - on
116			windows, ~/.urlader is replaced by %AppData%/urlader.
117
118			URLADER_VERSION (1.0)
119			Set to the version of the urlader binary itself. All versions with
120			the same major number should be compatible to older versions with
121			the same major number.
122
123			URLADER_DATADIR (~/.urlader)
124			The data directory used to store whatever urlader needs to store.
125
126			URLADER_CURRDIR
127			This is set to the full path of the current working directory where
128			the urlader was started. Atfer unpacking, the urlader changes to the
129			"URLADER_EXECDIR", so any relative paths should be resolved via this
130			path.
131
132			URLADER_EXEPATH
133			This is set to the path of the urlader executable itself, usually
134			relative to $URLADER_CURRDIR.
135
136			URLADER_EXE_ID
137			This stores the executable id of the pack file attached to the
138			urlader.
139
140			URLADER_EXE_VER
141			This is the executable version of the pack file attached to the
142			urlader, or the override, whichever was newer. Or in other words,
143			this is the version of the application running at the moment.
144
145			URLADER_EXE_DIR (~/.urlader/$URLADER_EXE_ID>
146			The directory where urlader stores files related to the executable
147			with the given id.
148
149			URLADER_EXECDIR (~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER)
150			The directory where the files from the pack file are unpacked and
151			the program is being run. Also the working directory of the program
152			when it is run.
153
154			URLADER_OVERRIDE (empty or override)
155			The override file used, if any, relative to $URLADER_EXECDIR. This
156			is either missing, when no override was used, or the string
157			override, as thta is currently the only override file urlader is
158			looking for.
159	root	1.2
160			FUNCTIONS AND VARIABLES IN THIS MODULE
161	root	1.3	$Urlader::URLADER_VERSION
162			Set to the urlader version ("URLADER_VERSION") when the program is
163			running form within urlader, undef otherwise.
164
165			$Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER,
166			$Urlader::EXE_DIR, $Urlader::EXECDIR
167			Contain the same value as the environment variable of the (almost)
168			same name. You should prefer these, though, as these might even be
169			set to correct values when not running form within an urlader
170			environment.
171
172			Urlader::set_exe_info $exe_id, $exe_ver
173			Sets up the paths and variables as if running the given executable
174			and version from within urlader.
175
176			$lock = Urlader::lock $path, $exclusive, $wait
177			Tries to acquire a lock on the given path (which must specify a file
178			which will be created if necessary). If $exclusive is true, then it
179			tries to acquire an exclusive lock, otherwise the lock will be
180			shared. If $wait is true, then it will wait until the lock can be
181			acquired, otherwise it only attempts to acquire it and returns
182			immediately if it can't.
183
184			If successful it returns a lock object - the lock will be given up
185			when the lock object is destroyed.
186
187			If the lock could not be acquired, "undef" is returned.
188
189			This function will probably go awway in the future, but is provided
190			to assist applications that want to clean up old versions, see "TIPS
191			AND TRICKS", below.
192
193			TIPS AND TRICKS
194			Gathering files
195			Gathering all the files needed for distribution cna be a big
196			problem. Right now, Urlader does not assist you in this task in any
197			way, however, just like perl source stripping, it is planned to
198			unbundle the relevant technology from staticperl
199			(<http://staticperl.plan9.de>) for use with this module.
200
201			You could always use par to find all library files, unpack the
202			bundle and add perl, libperl and other support libraries (e.g.
203			libgcc_s).
204
205			Software update
206			Updating the software can be done by downloading a new packfile
207			(with the same "exe_id" but a higher "exe_ver" - this can simply be
208			the executable you create when making a release) and replacing the
209			override file in the $URLADER_EXE_DIR.
210
211			When looking for updates, you should include $URLADER_VERSION,
212			$URLADER_EXE_ID and $URLADER_EXE_VER - the first two must be
213			identical for update and currently running program, while the last
214			one should be lexicographically higher.
215
216			Replacing the override file can be done like this:
217
218			rename "new-override.tmp", "$Urlader::EXE_DIR/override"
219			or die "could not replace override";
220
221			This can fail on windows when another urlader currently reads it,
222			but should work on all platforms even when other urlader programs
223			execute concurrently.
224
225			Cleaning up old directories
226			Urlader only packs executables once and then caches them in the
227			$URLADER_EXECDIR. After upgrades there will be old versions in there
228			that are not being used anymore. Or are they?
229
230			Each instance directory (i-*) in the $URLADER_EXE_DIR) has an
231			associated lock file (i-*.lck) - while urlader executes an app it
232			keeps a shared lock on this file.
233
234			To detect whether a version is in use or not, you must try to
235			acquire an exclusive lock, i.e.:
236
237			my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0;
238			if (!$lock) {
239			# instance dir is not in use and can be safely deleted
240			}
241
242			If an older urlader wants to use an instance that was deleted or is
243			currently being deleted it will wait until it's gone and simply
244			recreate it, so while less efficient, deleting instance directories
245			should always be safe.
246
247			The lockfile itself can be deleted as long as you have an exclusive
248			lock on it (if your platform allows this).
249
250			A real world project
251			The only real world project using this that I know of at the moment
252			is the deliantra client (http://www.deliantra.net for more info).
253
254			It uses some scary scripts to build the client and some dependnet
255			modules (build.*), to gather perl source files into a distribution
256			tree, shared objects and system shared libraries (some of which have
257			to be patched or, due to the horrible dll hell on OS X, even
258			renamed), called "gatherer", and a script called "gendist" to build
259			executable distributions.
260
261			These can be found at
262			<http://cvs.schmorp.de/deliantra/Deliantra-Client/util/>, but
263			looking at them can lead to premature blindless.
264
265	root	1.4	Shared Libraries
266			It is often desirable to package shared libraries - for example the
267			Deliantra client packages SD>, Berkely DB, Pango and amny other
268			libraries that are unlikely to be available on the target system.
269
270			This usually requires some fiddling (see below), and additionally
271			some environment variables to be set.
272
273			For example, on ELF systems you usually want LD_LIBRARY_PATH=. and
274			on OS X, you want DYLD_LIBRARY_PATH=. (these are effectively the
275			default on windows).
276
277			These can most easily be specified when building the packfile:
278
279			urlader-util ... LD_LIBRARY_PATH=. ./perl run
280
281			Portability: RPATH
282			Often perl is linked against a shared libperl.so - and might be so
283			using an rpath. Perl extensikns likewise might use an rpath, which
284			means the binary will mostly ignore LD_LIBRARY_PATH, which leads to
285			trouble.
286
287			There is an utility called chrpath, whose -d option can remove the
288			rpath from binaries, shared library and shared objects.
289
290			Portability: OS X DLL HELL
291			OS X has the most severe form of DLL hell I have seen - if you link
292			against system libraries, which is practically unavoidable, you get
293			libraries of well-known names (e.g. libjpeg) that have nothing to do
294			with what you normally expect libjpeg to be, and there is no way to
295			get your version of libjpeg into your program.
296
297			Moreover, even if apple ships well-known libraries (e.g. libiconv),
298			they often ship patched versions which have a different ABI or even
299			API then the real releases.
300
301			The only way aorund this I found was to change all library names in
302			my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and so
303			on), by patching the paths in the share dlibraries and shared
304			objects. install-name-tool (with -id and -change) works in many
305			cases, but often paths are embedded indirectly, so you might have to
306			use a dirty string replacement.
307
308			SECURITY CONSIDERATIONS
309			The urlader executable itself does not support setuig/setgid operation,
310			or running with elevated privileges - it does no input sanitisation, and
311			is trivially exploitable.
312
313	root	1.2	AUTHOR
314			Marc Lehmann <schmorp@schmorp.de>
315			http://home.schmorp.de/
316