--- App-Staticperl/staticperl.pod 2010/12/06 19:33:57 1.1 +++ App-Staticperl/staticperl.pod 2010/12/08 23:03:21 1.16 @@ -1,6 +1,6 @@ =head1 NAME -staticperl - perl, libc, 50 modules all in one 500kb file +staticperl - perl, libc, 100 modules, all in one 500kb file =head1 SYNOPSIS @@ -16,6 +16,7 @@ staticperl instcpan modulename... # install modules from CPAN staticperl mkbundle # see documentation staticperl mkperl # see documentation + staticperl mkapp appname # see documentation Typical Examples: @@ -24,21 +25,24 @@ staticperl mkperl -M '"Config_heavy.pl"' # build a perl that supports -V staticperl mkperl -MAnyEvent::Impl::Perl -MAnyEvent::HTTPD -MURI -MURI::http # build a perl with the above modules linked in + staticperl mkapp myapp --boot mainprog mymodules + # build a binary "myapp" from mainprog and mymodules =head1 DESCRIPTION -This script helps you creating single-file perl interpreters, or embedding -a pelr interpreter in your apps. Single-file means that it is fully -self-contained - no separate shared objects, no autoload fragments, no .pm -or .pl files are needed. And when linking statically, you can create (or -embed) a single file that contains perl interpreter, libc, all the modules -you need and all the libraries you need. - -With uclibc and upx on x86, you can create a single 500kb binary that -contains perl and 50 modules such as AnyEvent, EV, IO::AIO, Coro and so -on. Or any other choice of modules. +This script helps you to create single-file perl interpreters +or applications, or embedding a perl interpreter in your +applications. Single-file means that it is fully self-contained - no +separate shared objects, no autoload fragments, no .pm or .pl files are +needed. And when linking statically, you can create (or embed) a single +file that contains perl interpreter, libc, all the modules you need, all +the libraries you need and of course your actual program. + +With F and F on x86, you can create a single 500kb binary +that contains perl and 100 modules such as POSIX, AnyEvent, EV, IO::AIO, +Coro and so on. Or any other choice of modules. -The created files do not need write access to the filesystem (like PAR +The created files do not need write access to the file system (like PAR does). In fact, since this script is in many ways similar to PAR::Packer, here are the differences: @@ -67,9 +71,10 @@ =item * More control over included files. -PAR tries to be maintainance and hassle-free - it tries to include more files -than necessary to make sure everything works out of the box. The extra files -(such as the unicode database) can take substantial amounts of memory and filesize. +PAR tries to be maintenance and hassle-free - it tries to include more +files than necessary to make sure everything works out of the box. The +extra files (such as the unicode database) can take substantial amounts of +memory and file size. With F, the burden is mostly with the developer - only direct compile-time dependencies and L are handled automatically. @@ -82,6 +87,11 @@ build and possibly fiddling with some modules. PAR is likely to produce results faster. +Ok, PAR never has worked for me out of the box, and for some people, +F does work out of the box, as they don't count "fiddling with +module use lists" against it, but nevertheless, F is certainly +a bit more difficult to use. + =back =head1 HOW DOES IT WORK? @@ -90,12 +100,12 @@ your choice in F<~/.staticperl>. You can add extra modules either by letting F install them for you automatically, or by using CPAN and doing it interactively. This usually takes 5-10 minutes, depending on -the speed of your computer and your internet conenction. +the speed of your computer and your internet connection. It is possible to do program development at this stage, too. Afterwards, you create a list of files and modules you want to include, -and then either build a new perl binary (that acts just like a normla perl +and then either build a new perl binary (that acts just like a normal perl except everything is compiled in), or you create bundle files (basically C sources you can use to embed all files into your project). @@ -136,7 +146,7 @@ Most of the following commands simply run one or more steps of this sequence. -To force recompilation or reinstalaltion, you need to run F first. =over 4 @@ -156,13 +166,13 @@ =item F -Wipes the perl installation directory (usually F<~/.staticperl/perl>) and installs -the perl distribution, potentially aftering building it first. +Wipes the perl installation directory (usually F<~/.staticperl/perl>) and +installs the perl distribution, potentially after building it first. =item F [args...] -Starts an interactive CPAN shell that you cna use to install further -modules. Installs the perl first if neccessary, but apart from that, +Starts an interactive CPAN shell that you can use to install further +modules. Installs the perl first if necessary, but apart from that, no magic is involved: you could just as well run it manually via F<~/.staticperl/perl/bin/cpan>. @@ -179,15 +189,18 @@ =item F directory... In the unlikely case that you have unpacked perl modules around and want -to install from these instead of from CPAN, you cna do this using this +to install from these instead of from CPAN, you can do this using this command by specifying all the directories with modules in them that you want to have built. =item F -Runs F in the perl source directory (and potentially -cleans up other intermediate files). This can be used to clean up -intermediate files without removing the installed perl interpreter. +Deletes the perl source directory (and potentially cleans up other +intermediate files). This can be used to clean up files only needed for +building perl, without removing the installed perl interpreter, or to +force a re-build from scratch. + +At the moment, it doesn't delete downloaded tarballs. =item F @@ -208,11 +221,11 @@ staticperl mkbundle mkbundle-args... In the oh so unlikely case of something not working here, you -cna run the script manually as well (by default it is written to +can run the script manually as well (by default it is written to F<~/.staticperl/mkbundle>). F is a more conventional command and expect the argument -syntax commonly used on unix clones. For example, this command builds +syntax commonly used on UNIX clones. For example, this command builds a new F binary and includes F (for F), F, F and a custom F script (from F in this distribution): @@ -231,16 +244,27 @@ As you can see, things are not quite as trivial: the L module has a hidden dependency which is not even a perl module (F), L needs at least one event loop backend that we have to -specifymanually (here L), and the F module +specify manually (here L), and the F module (required by L) implements various URI schemes as extra modules - since L only needs C URIs, we only need -to include that module. +to include that module. I found out about these dependencies by carefully +watching any error messages about missing modules... + +Instead of building a new perl binary, you can also build a standalone +application: + + # build the app + staticperl mkapp app --boot eg/httpd \ + -MAnyEvent::Impl::Perl -MAnyEvent::HTTPD -MURI::http + + # run it + ./app =head3 OPTION PROCESSING -All options can be given as arguments on the commandline (typically using -long (e.g. C<--verbose>) or short option (e.g. C<-v>) style). Since -specifying a lot of modules can make the commandlien very cumbersome, +All options can be given as arguments on the command line (typically +using long (e.g. C<--verbose>) or short option (e.g. C<-v>) style). Since +specifying a lot of modules can make the command line very cumbersome, you can put all long options into a "bundle specification file" (with or without C<--> prefix) and specify this bundle file instead. @@ -256,31 +280,512 @@ use URI::http add eg/httpd httpd.pm +All options that specify modules or files to be added are processed in the +order given on the command line (that affects the C<--use> and C<--eval> +options at the moment). + =head3 MKBUNDLE OPTIONS =over 4 - "strip=s" => \$STRIP, - "verbose|v" => sub { ++$VERBOSE }, - "quiet|q" => sub { --$VERBOSE }, - "perl" => \$PERL, - "eval=s" => sub { trace_eval $_[1] }, - "use|M=s" => sub { trace_module $_[1] }, - "boot=s" => sub { cmd_boot $_[1] }, - "add=s" => sub { cmd_add $_[1] }, - "static" => sub { $STATIC = 1 }, - "<>" => sub { cmd_file $_[1] }, +=item --verbose | -v + +Increases the verbosity level by one (the default is C<1>). + +=item --quiet | -q + +Decreases the verbosity level by one. + +=item --strip none|pod|ppi + +Specify the stripping method applied to reduce the file of the perl +sources included. + +The default is C, which uses the L module to remove all +pod documentation, which is very fast and reduces file size a lot. + +The C method uses L to parse and condense the perl sources. This +saves a lot more than just L, and is generally safer, but +is also a lot slower, so is best used for production builds. Note that +this method doesn't optimise for raw file size, but for best compression +(that means that the uncompressed file size is a bit larger, but the files +compress better, e.g. with F). + +Last not least, if you need accurate line numbers in error messages, +or in the unlikely case where C is too slow, or some module gets +mistreated, you can specify C to not mangle included perl sources in +any way. + +=item --perl + +After writing out the bundle files, try to link a new perl interpreter. It +will be called F and will be left in the current working +directory. The bundle files will be removed. + +This switch is automatically used when F is invoked with the +C command (instead of C): + + # build a new ./perl with only common::sense in it - very small :) + staticperl mkperl -Mcommon::sense + +=item --app name + +After writing out the bundle files, try to link a new standalone +program. It will be called C, and the bundle files get removed after +linking it. + +The difference to the (mutually exclusive) C<--perl> option is that the +binary created by this option will not try to act as a perl interpreter - +instead it will simply initialise the perl interpreter, clean it up and +exit. + +This switch is automatically used when F is invoked with the +C command (instead of C): + +To let it do something useful you I add some boot code, e.g. with +the C<--boot> option. + +Example: create a standalone perl binary that will execute F when +it is started. + + staticperl mkbundle --app myexe --boot appfile + +=item --use module | -Mmodule + +Include the named module and all direct dependencies. This is done by +C'ing the module in a subprocess and tracing which other modules +and files it actually loads. If the module uses L, then all +splitfiles will be included as well. + +Example: include AnyEvent and AnyEvent::Impl::Perl. + + staticperl mkbundle --use AnyEvent --use AnyEvent::Impl::Perl + +Sometimes you want to load old-style "perl libraries" (F<.pl> files), or +maybe other weirdly named files. To do that, you need to quote the name in +single or double quotes. When given on the command line, you probably need +to quote once more to avoid your shell interpreting it. Common cases that +need this are F and F. + +Example: include the required files for F to work in all its +glory (F is included automatically by this). + + # bourne shell + staticperl mkbundle --use '"Config_heavy.pl"' + + # bundle specification file + use "Config_heavy.pl" + +The C<-Mmodule> syntax is included as an alias that might be easier to +remember than C. Or maybe it confuses people. Time will tell. Or +maybe not. Argh. + +=item --eval "perl code" | -e "perl code" + +Sometimes it is easier (or necessary) to specify dependencies using perl +code, or maybe one of the modules you use need a special use statement. In +that case, you can use C to execute some perl snippet or set some +variables or whatever you need. All files C'd or C'd in the +script are included in the final bundle. + +Keep in mind that F will only C the modules named +by the C<--use> option, so do not expect the symbols from modules you +C<--use>'d earlier on the command line to be available. + +Example: force L to detect a backend and therefore include it +in the final bundle. + + staticperl mkbundle --eval 'use AnyEvent; AnyEvent::detect' + + # or like this + staticperl mkbundle -MAnyEvent --eval 'use AnyEvent; AnyEvent::detect' + +Example: use a separate "bootstrap" script that C's lots of modules +and include this in the final bundle, to be executed automatically. + + staticperl mkbundle --eval 'do "bootstrap"' --boot bootstrap + +=item --boot filename + +Include the given file in the bundle and arrange for it to be executed +(using a C) before anything else when the new perl is +initialised. This can be used to modify C<@INC> or anything else before +the perl interpreter executes scripts given on the command line (or via +C<-e>). This works even in an embedded interpreter. + +=item --add "file" | --add "file alias" + +Adds the given (perl) file into the bundle (and optionally call it +"alias"). This is useful to include any custom files into the bundle. + +Example: embed the file F as F when creating the bundle. + + staticperl mkperl --add "httpd httpd.pm" + +It is also a great way to add any custom modules: + + # specification file + add file1 myfiles/file1 + add file2 myfiles/file2 + add file3 myfiles/file3 + +=item --binadd "file" | --add "file alias" + +Just like C<--add>, except that it treats the file as binary and adds it +without any processing. + +You should probably add a C prefix to avoid clashing with embedded +perl files (whose paths do not start with C), and/or use a special +directory, such as C. + +You can later get a copy of these files by calling C. + +=item --static + +When C<--perl> is also given, link statically instead of dynamically. The +default is to link the new perl interpreter fully dynamic (that means all +perl modules are linked statically, but all external libraries are still +referenced dynamically). + +Keep in mind that Solaris doesn't support static linking at all, and +systems based on GNU libc don't really support it in a usable fashion +either. Try uClibc if you want to create fully statically linked +executables, or try the C<--staticlibs> option to link only some libraries +statically. + +=item any other argument + +Any other argument is interpreted as a bundle specification file, which +supports most long options (without extra quoting), one option per line. =back -=head2 F CONFIGURATION AND HOOKS +=head2 F CONFIGURATION AND HOOKS -#TODO +During (each) startup, F tries to source the following shell +files in order: -=head1 AUTHOR + /etc/staticperlrc + ~/.staticperlrc + $STATICPERL/rc - Marc Lehmann - http://software.schmorp.de/pkg/staticperl.html +They can be used to override shell variables, or define functions to be +called at specific phases. + +Note that the last file is erased during F, so +generally should not be used. + +=head3 CONFIGURATION VARIABLES + +=head4 Variables you I override + +=over 4 + +=item C + +The e-mail address of the person who built this binary. Has no good +default, so should be specified by you. + +=item C + +The URL of the CPAN mirror to use (e.g. L). + +=item C + +Additional modules installed during F. Here you can +set which modules you want have to installed from CPAN. + +Example: I really really need EV, AnyEvent, Coro and AnyEvent::AIO. + + EXTRA_MODULES="EV AnyEvent Coro AnyEvent::AIO" + +Note that you can also use a C hook to achieve this, and +more. + +=back + +=head4 Variables you might I to override + +=over 4 + +=item C + +The directory where staticperl stores all its files +(default: F<~/.staticperl>). + +=item C, C, ... + +Usually set to C<1> to make modules "less inquisitive" during their +installation, you can set any environment variable you want - some modules +(such as L or L) use environment variables for further tweaking. + +=item C +The perl version to install - default is currently C<5.12.2>, but C<5.8.9> +is also a good choice (5.8.9 is much smaller than 5.12.2, while 5.10.1 is +about as big as 5.12.2). +=item C +The prefix where perl gets installed (default: F<$STATICPERL/perl>), +i.e. where the F and F subdirectories will end up. + +=item C + +Additional Configure options - these are simply passed to the perl +Configure script. For example, if you wanted to enable dynamic loading, +you could pass C<-Dusedl>. To enable ithreads (Why would you want that +insanity? Don't! Use L instead!) you would pass C<-Duseithreads> +and so on. + +More commonly, you would either activate 64 bit integer support +(C<-Duse64bitint>), or disable large files support (-Uuselargefiles), to +reduce filesize further. + +=item C, C, C, C + +These flags are passed to perl's F script, and are generally +optimised for small size (at the cost of performance). Since they also +contain subtle workarounds around various build issues, changing these +usually requires understanding their default values - best look at the top +of the F script for more info on these. + +=back + +=head4 Variables you probably I to override + +=over 4 + +=item C + +Where F writes the C command to +(default: F<$STATICPERL/mkbundle>). + +=item C + +Additional modules needed by C - should therefore not be changed +unless you know what you are doing. + +=back + +=head3 OVERRIDABLE HOOKS + +In addition to environment variables, it is possible to provide some +shell functions that are called at specific times. To provide your own +commands, just define the corresponding function. + +Example: install extra modules from CPAN and from some directories +at F time. + + postinstall() { + rm -rf lib/threads* # weg mit Schaden + instcpan IO::AIO EV + instsrc ~/src/AnyEvent + instsrc ~/src/XML-Sablotron-1.0100001 + instcpan Anyevent::AIO AnyEvent::HTTPD + } + +=over 4 + +=item preconfigure + +Called just before running F<./Configur> in the perl source +directory. Current working directory is the perl source directory. + +This can be used to set any C variables, which might be costly +to compute. + +=item postconfigure + +Called after configuring, but before building perl. Current working +directory is the perl source directory. + +Could be used to tailor/patch config.sh (followed by F) +or do any other modifications. + +=item postbuild + +Called after building, but before installing perl. Current working +directory is the perl source directory. + +I have no clue what this could be used for - tell me. + +=item postinstall + +Called after perl and any extra modules have been installed in C<$PREFIX>, +but before setting the "installation O.K." flag. + +The current working directory is C<$PREFIX>, but maybe you should not rely +on that. + +This hook is most useful to customise the installation, by deleting files, +or installing extra modules using the C or C functions. + +The script must return with a zero exit status, or the installation will +fail. + +=back + +=head1 ANATOMY OF A BUNDLE + +When not building a new perl binary, C will leave a number of +files in the current working directory, which can be used to embed a perl +interpreter in your program. + +Intimate knowledge of L and preferably some experience with +embedding perl is highly recommended. + +C (or the C<--perl> option) basically does this to link the new +interpreter (it also adds a main program to F): + + $Config{cc} $(cat bundle.ccopts) -o perl bundle.c $(cat bundle.ldopts) + +=over 4 + +=item bundle.h + +A header file that contains the prototypes of the few symbols "exported" +by bundle.c, and also exposes the perl headers to the application. + +=over 4 + +=item staticperl_init () + +Initialises the perl interpreter. You can use the normal perl functions +after calling this function, for example, to define extra functions or +to load a .pm file that contains some initialisation code, or the main +program function: + + XS (xsfunction) + { + dXSARGS; + + // now we have items, ST(i) etc. + } + + static void + run_myapp(void) + { + staticperl_init (); + newXSproto ("myapp::xsfunction", xsfunction, __FILE__, "$$;$"); + eval_pv ("require myapp::main", 1); // executes "myapp/main.pm" + } + +=item staticperl_xs_init (pTHX) + +Sometimes you need direct control over C and C, in +which case you do not want to use C but call them on your +own. + +Then you need this function - either pass it directly as the C +function to C, or call it from your own C function. + +=item staticperl_cleanup () + +In the unlikely case that you want to destroy the perl interpreter, here +is the corresponding function. + +=item PerlInterpreter *staticperl + +The perl interpreter pointer used by staticperl. Not normally so useful, +but there it is. + +=back + +=item bundle.ccopts + +Contains the compiler options required to compile at least F and +any file that includes F - you should probably use it in your +C. + +=item bundle.ldopts + +The linker options needed to link the final program. + +=back + +=head1 RUNTIME FUNCTIONALITY + +Binaries created with C/C contain extra functions, which +are required to access the bundled perl sources, but might be useful for +other purposes. + +In addition, for the embedded loading of perl files to work, F +overrides the C<@INC> array. + +=over 4 + +=item $file = staticperl::find $path + +Returns the data associated with the given C<$path> +(e.g. C, C), which is basically +the UNIX path relative to the perl library directory. + +Returns C if the file isn't embedded. + +=item @paths = staticperl::list + +Returns the list of all paths embedded in this binary. + +=back + +=head1 FULLY STATIC BINARIES - BUILDROOT + +To make truly static (Linux-) libraries, you might want to have a look at +buildroot (L). + +Buildroot is primarily meant to set up a cross-compile environment (which +is not so useful as perl doesn't quite like cross compiles), but it can also compile +a chroot environment where you can use F. + +To do so, download buildroot, and enable "Build options => development +files in target filesystem" and optionally "Build options => gcc +optimization level (optimize for size)". At the time of writing, I had +good experiences with GCC 4.4.x but not GCC 4.5. + +To minimise code size, I used C<-pipe -ffunction-sections -fdata-sections +-finline-limit=8 -fno-builtin-strlen -mtune=i386>. The C<-mtune=i386> +doesn't decrease codesize much, but it makes the file much more +compressible. + +If you don't need Coro or threads, you can go with "linuxthreads.old" (or +no thread support). For Coro, it is highly recommended to switch to a +uClibc newer than 0.9.31 (at the time of this writing, I used the 20101201 +snapshot) and enable NPTL, otherwise Coro needs to be configured with the +ultra-slow pthreads backend to work around linuxthreads bugs (it also uses +twice the address space needed for stacks). + +If you use C, then you should also be aware that +uClibc shares C between all threads when statically linking. See +L for a +workaround (And L for discussion). + +C support is also recommended, especially if you want +to play around with buildroot options. Enabling the C +package will probably enable all options required for a successful +perl build. F itself additionally needs either C +(recommended, for CPAN) or C. + +As for shells, busybox should provide all that is needed, but the default +busybox configuration doesn't include F which is needed by perl - +either make a custom busybox config, or compile coreutils. + +For the latter route, you might find that bash has some bugs that keep +it from working properly in a chroot - either use dash (and link it to +F inside the chroot) or link busybox to F, using it's +built-in ash shell. + +Finally, you need F inside the chroot for many scripts to work +- F or bind-mounting your F will +both provide this. + +After you have compiled and set up your buildroot target, you can copy +F from the C distribution or from your +perl f directory (if you installed it) into the F +filesystem, chroot inside and run it. + +=head1 AUTHOR + + Marc Lehmann + http://software.schmorp.de/pkg/staticperl.html