--- Urlader/Urlader.pm 2012/01/04 14:19:05 1.4 +++ Urlader/Urlader.pm 2012/02/06 22:09:40 1.8 @@ -9,17 +9,17 @@ =head1 DESCRIPTION Urlader (that's german for "bootloader" btw.) was created out of -frustration over PAR again not working, again not being flexible enough -for simple things, and again causing mysterious missing files issues on -various platforms. +frustration over PAR always being horribly slow, again not working, again +not being flexible enough for simple things such as software upgrades, and +again causing mysterious missing file issues on various platforms. That doesn't mean this module replaces PAR, in fact, you should stay with -it for many reasons, user-friendlyness is one of them. +PAR for many reasons right now, user-friendlyness is one of them. However, if you want to make single-file distributions out of your perl -programs (or python, or C or whatever), and you are prepared to fiddle a -LOT, this module might provide a tiny step towards your goal. Well, if it -ever gets finished. +programs (or python, or C or whatever), and you are prepared to fiddle +a LOT, this module might provide a faster and more versatile deployment +technique then PAR. Well, if it ever gets finished. Also, I, and it's far from feature-complete. @@ -31,7 +31,7 @@ =item A simple archiver that packs a directory tree into a single file. =item A small C program that works on windows and unix, which unpacks an attached -archive and runs a program. +archive and runs a program (perl, python, whatever...). =item A perl module support module (I), that can be used to query the runtime environment, find out where to install updates and so on. @@ -50,6 +50,12 @@ bintree/run bintree/pm/Guard.pm bintree/pm/auto/Guard/Guard.so + bintree/pm/XSLoader.pm + bintree/pm/DynaLoader.pm + bintree/pm/Config.pm + bintree/pm/strict.pm + bintree/pm/vars.pm + bintree/pm/warnings.pm # cat bintree/run @INC = ("pm", "."); # "." works around buggy AutoLoader @@ -65,6 +71,12 @@ >myprog # chmod 755 myprog +The packing step takes a binary loader and appends all the files in the +directory tree, plus some meta information. + +The resulting file is an executable that, when run, will unpack all the +files and run the embedded program. + =head1 CONCEPTS =over 4 @@ -103,6 +115,76 @@ =item override +When the urlader starts, it first finds out what C is +embedded in it. It then looks for an override file for this id +(F<$URLADER_EXE_DIR/override>) and verifies that it is for the same +C, and the version is newer. If this is the case, then it will +unpack and run the override file instead of unpacking the files attched to +itself. + +This way one can implement software upgrades - download a new executable, +write it safely to disk and move it to the override path. + +=back + +=head1 ENVIRONMENT VARIABLES + +The urlader sets and maintains the following environment variables, in +addition to any variables specified on the commandline. The values in +parentheses are typical (but not gauranteed) values for unix - on windows, +F<~/.urlader> is replaced by F<%AppData%/urlader>. + +=over 4 + +=item URLADER_VERSION (C<1.0>) + +Set to the version of the urlader binary itself. All versions with the +same major number should be compatible to older versions with the same +major number. + +=item URLADER_DATADIR (F<~/.urlader>) + +The data directory used to store whatever urlader needs to store. + +=item URLADER_CURRDIR + +This is set to the full path of the current working directory where +the urlader was started. Atfer unpacking, the urlader changes to the +C, so any relative paths should be resolved via this +path. + +=item URLADER_EXEPATH + +This is set to the path of the urlader executable itself, usually relative +to F<$URLADER_CURRDIR>. + +=item URLADER_EXE_ID + +This stores the executable id of the pack file attached to the urlader. + +=item URLADER_EXE_VER + +This is the executable version of the pack file attached to the urlader, +or the override, whichever was newer. Or in other words, this is the +version of the application running at the moment. + +=item URLADER_EXE_DIR (F<~/.urlader/$URLADER_EXE_ID>> + +The directory where urlader stores files related to the executable with +the given id. + +=item URLADER_EXECDIR (F<~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER>) + +The directory where the files from the pack file are unpacked and the +program is being run. Also the working directory of the program when it is +run. + +=item URLADER_OVERRIDE (empty or F) + +The override file used, if any, relative to F<$URLADER_EXECDIR>. This is +either missing, when no override was used, or the string F, as +thta is currently the only override file urlader is looking for. + =back =head1 FUNCTIONS AND VARIABLES IN THIS MODULE @@ -116,13 +198,26 @@ use common::sense; BEGIN { - our $VERSION = '0.2'; + our $VERSION = '1.0'; use XSLoader; XSLoader::load __PACKAGE__, $VERSION; } -our $URLADER_VERSION; # only ste when running under urlader +=item $Urlader::URLADER_VERSION + +Set to the urlader version (C) when the program is +running form within urlader, undef otherwise. + +=item $Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER, $Urlader::EXE_DIR, $Urlader::EXECDIR + +Contain the same value as the environment variable of the (almost) same +name. You should prefer these, though, as these might even be set to +correct values when not running form within an urlader environment. + +=cut + +our $URLADER_VERSION; # only set when running under urlader our $DATADIR; our $EXE_ID; our $EXE_VER; @@ -140,17 +235,172 @@ _get_env; +=item Urlader::set_exe_info $exe_id, $exe_ver + +Sets up the paths and variables as if running the given executable and +version from within urlader. + +=cut + sub set_exe_info($$) { _set_datadir unless defined getenv "URLADER_DATADIR"; &_set_exe_info; _get_env; } +=item $lock = Urlader::lock $path, $exclusive, $wait + +Tries to acquire a lock on the given path (which must specify a file which +will be created if necessary). If C<$exclusive> is true, then it tries to +acquire an exclusive lock, otherwise the lock will be shared. If C<$wait> +is true, then it will wait until the lock can be acquired, otherwise it +only attempts to acquire it and returns immediately if it can't. + +If successful it returns a lock object - the lock will be given up +when the lock object is destroyed. + +If the lock could not be acquired, C is returned. + +This function will probably go awway in the future, but is provided to +assist applications that want to clean up old versions, see "TIPS AND +TRICKS", below. + +=cut 1; =back +=head1 TIPS AND TRICKS + +=over 4 + +=item Gathering files + +Gathering all the files needed for distribution cna be a big +problem. Right now, Urlader does not assist you in this task in any way, +however, just like perl source stripping, it is planned to unbundle the +relevant technology from B (L) for +use with this module. + +You could always use par to find all library files, unpack the bundle and +add F, F and other support libraries (e.g. F). + +=item Software update + +Updating the software can be done by downloading a new packfile (with the +same C but a higher C - this can simply be the executable +you create when making a release) and replacing the F file in +the F<$URLADER_EXE_DIR>. + +When looking for updates, you should include C<$URLADER_VERSION>, +C<$URLADER_EXE_ID> and C<$URLADER_EXE_VER> - the first two must be +identical for update and currently running program, while the last one +should be lexicographically higher. + +Replacing the override file can be done like this: + + rename "new-override.tmp", "$Urlader::EXE_DIR/override" + or die "could not replace override"; + +This can fail on windows when another urlader currently reads it, but +should work on all platforms even when other urlader programs execute +concurrently. + +=item Cleaning up old directories + +Urlader only packs executables once and then caches them in the +F<$URLADER_EXECDIR>. After upgrades there will be old versions in there +that are not being used anymore. Or are they? + +Each instance directory (F) in the F<$URLADER_EXE_DIR>) has an +associated lock file (F) - while urlader executes an app it keeps +a shared lock on this file. + +To detect whether a version is in use or not, you must try to acquire an +exclusive lock, i.e.: + + my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0; + if (!$lock) { + # instance dir is not in use and can be safely deleted + } + +If an older urlader wants to use an instance that was deleted or is +currently being deleted it will wait until it's gone and simply recreate +it, so while less efficient, deleting instance directories should always +be safe. + +The lockfile itself can be deleted as long as you have an exclusive lock +on it (if your platform allows this). + +=item A real world project + +The only real world project using this that I know of at the moment is the +deliantra client (http://www.deliantra.net for more info). + +It uses some scary scripts to build the client and some dependnet modules +(F), to gather perl source files into a distribution tree, shared +objects and system shared libraries (some of which have to be patched or, +due to the horrible dll hell on OS X, even renamed), called C, +and a script called C to build executable distributions. + +These can be found at +L, but looking at +them can lead to premature blindless. + +=item Shared Libraries + +It is often desirable to package shared libraries - for example the +Deliantra client packages SD>, Berkely DB, Pango and amny other libraries +that are unlikely to be available on the target system. + +This usually requires some fiddling (see below), and additionally some +environment variables to be set. + +For example, on ELF systems you usually want F and on +OS X, you want F (these are effectively the default +on windows). + +These can most easily be specified when building the packfile: + + urlader-util ... LD_LIBRARY_PATH=. ./perl run + +=item Portability: RPATH + +Often F is linked against a shared F - and might be so +using an rpath. Perl extensikns likewise might use an rpath, which means +the binary will mostly ignore LD_LIBRARY_PATH, which leads to trouble. + +There is an utility called F, whose F<-d> option can remove the +rpath from binaries, shared library and shared objects. + +=item Portability: OS X DLL HELL + +OS X has the most severe form of DLL hell I have seen - if you link +against system libraries, which is practically unavoidable, you get +libraries of well-known names (e.g. libjpeg) that have nothing to do with +what you normally expect libjpeg to be, and there is no way to get your +version of libjpeg into your program. + +Moreover, even if apple ships well-known libraries (e.g. libiconv), they +often ship patched versions which have a different ABI or even API then +the real releases. + +The only way aorund this I found was to change all library names +in my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and +so on), by patching the paths in the share dlibraries and shared +objects. F (with F<-id> and F<-change>) works in many +cases, but often paths are embedded indirectly, so you might have to use a +I string replacement. + +=back + +=head1 SECURITY CONSIDERATIONS + +The urlader executable itself does not support setuig/setgid operation, or +running with elevated privileges - it does no input sanitisation, and is +trivially exploitable. + =head1 AUTHOR Marc Lehmann