--- Urlader/Urlader.pm 2012/01/03 10:37:45 1.2 +++ Urlader/Urlader.pm 2012/06/17 13:06:10 1.10 @@ -8,6 +8,187 @@ =head1 DESCRIPTION +Urlader (that's german for "bootloader" btw.) was created out of +frustration over PAR always being horribly slow, again not working, again +not being flexible enough for simple things such as software upgrades, and +again causing mysterious missing file issues on various platforms. + +That doesn't mean this module replaces PAR, in fact, you should stay with +PAR for many reasons right now, user-friendlyness is one of them. + +However, if you want to make single-file distributions out of your perl +programs (or python, or C or whatever), and you are prepared to fiddle +a LOT, this module might provide a faster and more versatile deployment +technique then PAR. Well, if it ever gets finished. + +Also, I, and it's +far from feature-complete. + +Having said all that, Urlader basically provides three services: + +=over 4 + +=item A simple archiver that packs a directory tree into a single file. + +=item A small C program that works on windows and unix, which unpacks an attached +archive and runs a program (perl, python, whatever...). + +=item A perl module support module (I), that can be used to query +the runtime environment, find out where to install updates and so on. + +=back + +=head1 EXAMPLE + +How can it be used to provide single-file executables? + +So simple, create a directory with everything that's needed, e.g.: + + # find bintree + bintree/perl + bintree/libperl.so.5.10 + bintree/run + bintree/pm/Guard.pm + bintree/pm/auto/Guard/Guard.so + bintree/pm/XSLoader.pm + bintree/pm/DynaLoader.pm + bintree/pm/Config.pm + bintree/pm/strict.pm + bintree/pm/vars.pm + bintree/pm/warnings.pm + + # cat bintree/run + @INC = ("pm", "."); # "." works around buggy AutoLoader + use Guard; + guard { warn "hello, world!\n" }; # just to show off + exit 0; # tell the urlader that everything was fine + +Then pack it: + + # wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86 + # urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \ + LD_LIBRARY_PATH=. ./perl run \ + >myprog + # chmod 755 myprog + +The packing step takes a binary loader and appends all the files in the +directory tree, plus some meta information. + +The resulting file is an executable that, when run, will unpack all the +files and run the embedded program. + +=head1 CONCEPTS + +=over 4 + +=item urlader + +A small (hopefully) and relatively portable (hopefully) binary that is +prepended to a pack file to make it executable. + +You can build it yourself from sources (see F in the +distribution) or use one of the precompiled ones at: + + http://urlader.schmorp.de/prebuilt/1.0/ + +The F there has further information on the binaries provided. + +=item exe_id + +A string that uniquely identifies your program - all branches of it. It +must consist of the characters C only and should be a valid +directory name on all systems you want to deploy on. + +=item exe_ver + +A string the uniquely identifies the contents of the archive, i.e. the +version. It has the same restrictions as the C, and should be +fixed-length, as Urlader assumes lexicographically higher versions are +newer, and thus preferable. + +=item pack file (archive) + +This contains the C, the C, a number of environment +variable assignments, the program name to execute, the initial arguments +it receives, and finally, a list of files (with contents :) and +directories. + +=item override + +When the urlader starts, it first finds out what C is +embedded in it. It then looks for an override file for this id +(F<$URLADER_EXE_DIR/override>) and verifies that it is for the same +C, and the version is newer. If this is the case, then it will +unpack and run the override file instead of unpacking the files attched to +itself. + +This way one can implement software upgrades - download a new executable, +write it safely to disk and move it to the override path. + +=back + +=head1 ENVIRONMENT VARIABLES + +The urlader sets and maintains the following environment variables, in +addition to any variables specified on the commandline. The values in +parentheses are typical (but not gauranteed) values for unix - on windows, +F<~/.urlader> is replaced by F<%AppData%/urlader>. + +=over 4 + +=item URLADER_VERSION (C<1.0>) + +Set to the version of the urlader binary itself. All versions with the +same major number should be compatible to older versions with the same +major number. + +=item URLADER_DATADIR (F<~/.urlader>) + +The data directory used to store whatever urlader needs to store. + +=item URLADER_CURRDIR + +This is set to the full path of the current working directory where +the urlader was started. Atfer unpacking, the urlader changes to the +C, so any relative paths should be resolved via this +path. + +=item URLADER_EXEPATH + +This is set to the path of the urlader executable itself, usually relative +to F<$URLADER_CURRDIR>. + +=item URLADER_EXE_ID + +This stores the executable id of the pack file attached to the urlader. + +=item URLADER_EXE_VER + +This is the executable version of the pack file attached to the urlader, +or the override, whichever was newer. Or in other words, this is the +version of the application running at the moment. + +=item URLADER_EXE_DIR (F<~/.urlader/$URLADER_EXE_ID>> + +The directory where urlader stores files related to the executable with +the given id. + +=item URLADER_EXECDIR (F<~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER>) + +The directory where the files from the pack file are unpacked and the +program is being run. Also the working directory of the program when it is +run. + +=item URLADER_OVERRIDE (empty or F) + +The override file used, if any, relative to F<$URLADER_EXECDIR>. This is +either missing, when no override was used, or the string F, as +thta is currently the only override file urlader is looking for. + +=back + +=head1 FUNCTIONS AND VARIABLES IN THIS MODULE + =over 4 =cut @@ -17,12 +198,26 @@ use common::sense; BEGIN { - our $VERSION = '0.2'; + our $VERSION = '1.0'; use XSLoader; XSLoader::load __PACKAGE__, $VERSION; } +=item $Urlader::URLADER_VERSION + +Set to the urlader version (C) when the program is +running form within urlader, undef otherwise. + +=item $Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER, $Urlader::EXE_DIR, $Urlader::EXECDIR + +Contain the same value as the environment variable of the (almost) same +name. You should prefer these, though, as these might even be set to +correct values when not running form within an urlader environment. + +=cut + +our $URLADER_VERSION; # only set when running under urlader our $DATADIR; our $EXE_ID; our $EXE_VER; @@ -30,26 +225,182 @@ our $EXECDIR; # %AppData%/urlader/EXE_ID/i-EXE_VER sub _get_env { - $DATADIR = getenv "URLADER_DATADIR"; - $EXE_ID = getenv "URLADER_EXE_ID"; - $EXE_VER = getenv "URLADER_EXE_VER"; - $EXE_DIR = getenv "URLADER_EXE_DIR"; # %AppData%/urlader/EXE_ID - $EXECDIR = getenv "URLADER_EXECDIR"; # %AppData%/urlader/EXE_ID/i-EXE_VER + $URLADER_VERSION = getenv "URLADER_VERSION"; + $DATADIR = getenv "URLADER_DATADIR"; + $EXE_ID = getenv "URLADER_EXE_ID"; + $EXE_VER = getenv "URLADER_EXE_VER"; + $EXE_DIR = getenv "URLADER_EXE_DIR"; # %AppData%/urlader/EXE_ID + $EXECDIR = getenv "URLADER_EXECDIR"; # %AppData%/urlader/EXE_ID/i-EXE_VER } _get_env; +=item Urlader::set_exe_info $exe_id, $exe_ver + +Sets up the paths and variables as if running the given executable and +version from within urlader. + +=cut + sub set_exe_info($$) { _set_datadir unless defined getenv "URLADER_DATADIR"; &_set_exe_info; _get_env; } +=item $lock = Urlader::lock $path, $exclusive, $wait + +Tries to acquire a lock on the given path (which must specify a file which +will be created if necessary). If C<$exclusive> is true, then it tries to +acquire an exclusive lock, otherwise the lock will be shared. If C<$wait> +is true, then it will wait until the lock can be acquired, otherwise it +only attempts to acquire it and returns immediately if it can't. + +If successful it returns a lock object - the lock will be given up when +the lock object is destroyed or when the process exits (even on a crash) +and has a good chance of working on network drives as well. + +If the lock could not be acquired, C is returned. + +This function is provided to assist applications that want to clean up old +versions, see "TIPS AND TRICKS", below. + +=cut 1; =back +=head1 TIPS AND TRICKS + +=over 4 + +=item Gathering files + +Gathering all the files needed for distribution can be a big +problem. Right now, Urlader does not assist you in this task in any way, +however, just like perl source stripping, it is planned to unbundle the +relevant technology from B (L) for +use with this module. + +You could always use par to find all library files, unpack the bundle and +add F, F and other support libraries (e.g. F). + +=item Software update + +Updating the software can be done by downloading a new packfile (with the +same C but a higher C - this can simply be the executable +you create when making a release) and replacing the F file in +the F<$URLADER_EXE_DIR>. + +When looking for updates, you should include C<$URLADER_VERSION>, +C<$URLADER_EXE_ID> and C<$URLADER_EXE_VER> - the first two must be +identical for update and currently running program, while the last one +should be lexicographically higher. + +Replacing the override file can be done like this: + + rename "new-override.tmp", "$Urlader::EXE_DIR/override" + or die "could not replace override"; + +This can fail on windows when another urlader currently reads it, but +should work on all platforms even when other urlader programs execute +concurrently. + +=item Cleaning up old directories + +Urlader only packs executables once and then caches them in the +F<$URLADER_EXECDIR>. After upgrades there will be old versions in there +that are not being used anymore. Or are they? + +Each instance directory (F) in the F<$URLADER_EXE_DIR>) has an +associated lock file (F) - while urlader executes an app it keeps +a shared lock on this file. + +To detect whether a version is in use or not, you must try to acquire an +exclusive lock, i.e.: + + my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0; + if (!$lock) { + # instance dir is not in use and can be safely deleted + } + +If an older urlader wants to use an instance that was deleted or is +currently being deleted it will wait until it's gone and simply recreate +it, so while less efficient, deleting instance directories should always +be safe. + +The lockfile itself can be deleted as long as you have an exclusive lock +on it (if your platform allows this). + +=item A real world project + +The only real world project using this that I know of at the moment is the +deliantra client (http://www.deliantra.net for more info). + +It uses some scary scripts to build the client and some dependnet modules +(F), to gather perl source files into a distribution tree, shared +objects and system shared libraries (some of which have to be patched or, +due to the horrible dll hell on OS X, even renamed), called C, +and a script called C to build executable distributions. + +These can be found at +L, but looking at +them can lead to premature blindless. + +=item Shared Libraries + +It is often desirable to package shared libraries - for example the +Deliantra client packages SD>, Berkely DB, Pango and amny other libraries +that are unlikely to be available on the target system. + +This usually requires some fiddling (see below), and additionally some +environment variables to be set. + +For example, on ELF systems you usually want F and on +OS X, you want F (these are effectively the default +on windows). + +These can most easily be specified when building the packfile: + + urlader-util ... LD_LIBRARY_PATH=. ./perl run + +=item Portability: RPATH + +Often F is linked against a shared F - and might be so +using an rpath. Perl extensikns likewise might use an rpath, which means +the binary will mostly ignore LD_LIBRARY_PATH, which leads to trouble. + +There is an utility called F, whose F<-d> option can remove the +rpath from binaries, shared library and shared objects. + +=item Portability: OS X DLL HELL + +OS X has the most severe form of DLL hell I have seen - if you link +against system libraries, which is practically unavoidable, you get +libraries of well-known names (e.g. libjpeg) that have nothing to do with +what you normally expect libjpeg to be, and there is no way to get your +version of libjpeg into your program. + +Moreover, even if apple ships well-known libraries (e.g. libiconv), they +often ship patched versions which have a different ABI or even API then +the real releases. + +The only way aorund this I found was to change all library names +in my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and +so on), by patching the paths in the share dlibraries and shared +objects. F (with F<-id> and F<-change>) works in many +cases, but often paths are embedded indirectly, so you might have to use a +I string replacement. + +=back + +=head1 SECURITY CONSIDERATIONS + +The urlader executable itself does not support setuig/setgid operation, or +running with elevated privileges - it does no input sanitisation, and is +trivially exploitable. + =head1 AUTHOR Marc Lehmann