[ViewVC] Annotation of: cvs/App-Staticperl/README

NAME
    staticperl - perl, libc, 100 modules, all in one standalone 500kb file

SYNOPSIS
       staticperl help      # print the embedded documentation
       staticperl fetch     # fetch and unpack perl sources
       staticperl configure # fetch and then configure perl
       staticperl build     # configure and then build perl
       staticperl install   # build and then install perl
       staticperl clean     # clean most intermediate files (restart at configure)
       staticperl distclean # delete everything installed by this script
       staticperl perl ...  # invoke the perlinterpreter
       staticperl cpan      # invoke CPAN shell
       staticperl instsrc path...        # install unpacked modules
       staticperl instcpan modulename... # install modules from CPAN
       staticperl mkbundle <bundle-args...> # see documentation
       staticperl mkperl <bundle-args...>   # see documentation
       staticperl mkapp appname <bundle-args...> # see documentation

    Typical Examples:

       staticperl install   # fetch, configure, build and install perl
       staticperl cpan      # run interactive cpan shell
       staticperl mkperl -MConfig_heavy.pl # build a perl that supports -V
       staticperl mkperl -MAnyEvent::Impl::Perl -MAnyEvent::HTTPD -MURI -MURI::http
                            # build a perl with the above modules linked in
       staticperl mkapp myapp --boot mainprog mymodules
                            # build a binary "myapp" from mainprog and mymodules

DESCRIPTION
    This script helps you to create single-file perl interpreters or
    applications, or embedding a perl interpreter in your applications.
    Single-file means that it is fully self-contained - no separate shared
    objects, no autoload fragments, no .pm or .pl files are needed. And when
    linking statically, you can create (or embed) a single file that
    contains perl interpreter, libc, all the modules you need, all the
    libraries you need and of course your actual program.

    With uClibc and upx on x86, you can create a single 500kb binary that
    contains perl and 100 modules such as POSIX, AnyEvent, EV, IO::AIO, Coro
    and so on. Or any other choice of modules (and some other size :).

    To see how this turns out, you can try out smallperl and bigperl, two
    pre-built static and compressed perl binaries with many and even more
    modules: just follow the links at <http://staticperl.schmorp.de/>.

    The created files do not need write access to the file system (like PAR
    does). In fact, since this script is in many ways similar to
    PAR::Packer, here are the differences:

    *   The generated executables are much smaller than PAR created ones.

        Shared objects and the perl binary contain a lot of extra info,
        while the static nature of staticperl allows the linker to remove
        all functionality and meta-info not required by the final
        executable. Even extensions statically compiled into perl at build
        time will only be present in the final executable when needed.

        In addition, staticperl can strip perl sources much more effectively
        than PAR.

    *   The generated executables start much faster.

        There is no need to unpack files, or even to parse Zip archives
        (which is slow and memory-consuming business).

    *   The generated executables don't need a writable filesystem.

        staticperl loads all required files directly from memory. There is
        no need to unpack files into a temporary directory.

    *   More control over included files, more burden.

        PAR tries to be maintenance and hassle-free - it tries to include
        more files than necessary to make sure everything works out of the
        box. It mostly succeeds at this, but he extra files (such as the
        unicode database) can take substantial amounts of memory and file
        size.

        With staticperl, the burden is mostly with the developer - only
        direct compile-time dependencies and AutoLoader are handled
        automatically. This means the modules to include often need to be
        tweaked manually.

        All this does not preclude more permissive modes to be implemented
        in the future, but right now, you have to resolve hidden
        dependencies manually.

    *   PAR works out of the box, staticperl does not.

        Maintaining your own custom perl build can be a pain in the ass, and
        while staticperl tries to make this easy, it still requires a custom
        perl build and possibly fiddling with some modules. PAR is likely to
        produce results faster.

        Ok, PAR never has worked for me out of the box, and for some people,
        staticperl does work out of the box, as they don't count "fiddling
        with module use lists" against it, but nevertheless, staticperl is
        certainly a bit more difficult to use.

HOW DOES IT WORK?
    Simple: staticperl downloads, compile and installs a perl version of
    your choice in ~/.staticperl. You can add extra modules either by
    letting staticperl install them for you automatically, or by using CPAN
    and doing it interactively. This usually takes 5-10 minutes, depending
    on the speed of your computer and your internet connection.

    It is possible to do program development at this stage, too.

    Afterwards, you create a list of files and modules you want to include,
    and then either build a new perl binary (that acts just like a normal
    perl except everything is compiled in), or you create bundle files
    (basically C sources you can use to embed all files into your project).

    This step is very fast (a few seconds if PPI is not used for stripping,
    or the stripped files are in the cache), and can be tweaked and repeated
    as often as necessary.

THE STATICPERL SCRIPT
    This module installs a script called staticperl into your perl binary
    directory. The script is fully self-contained, and can be used without
    perl (for example, in an uClibc chroot environment). In fact, it can be
    extracted from the "App::Staticperl" distribution tarball as
    bin/staticperl, without any installation. The newest (possibly alpha)
    version can also be downloaded from
    <http://staticperl.schmorp.de/staticperl>.

    staticperl interprets the first argument as a command to execute,
    optionally followed by any parameters.

    There are two command categories: the "phase 1" commands which deal with
    installing perl and perl modules, and the "phase 2" commands, which deal
    with creating binaries and bundle files.

  PHASE 1 COMMANDS: INSTALLING PERL
    The most important command is install, which does basically everything.
    The default is to download and install perl 5.12.3 and a few modules
    required by staticperl itself, but all this can (and should) be changed
    - see CONFIGURATION, below.

    The command

       staticperl install

    is normally all you need: It installs the perl interpreter in
    ~/.staticperl/perl. It downloads, configures, builds and installs the
    perl interpreter if required.

    Most of the following staticperl subcommands simply run one or more
    steps of this sequence.

    If it fails, then most commonly because the compiler options I selected
    are not supported by your compiler - either edit the staticperl script
    yourself or create ~/.staticperl shell script where your set working
    "PERL_CCFLAGS" etc. variables.

    To force recompilation or reinstallation, you need to run staticperl
    distclean first.

    staticperl version
        Prints some info about the version of the staticperl script you are
        using.

    staticperl fetch
        Runs only the download and unpack phase, unless this has already
        happened.

    staticperl configure
        Configures the unpacked perl sources, potentially after downloading
        them first.

    staticperl build
        Builds the configured perl sources, potentially after automatically
        configuring them.

    staticperl install
        Wipes the perl installation directory (usually ~/.staticperl/perl)
        and installs the perl distribution, potentially after building it
        first.

    staticperl perl [args...]
        Invokes the compiled perl interpreter with the given args. Basically
        the same as starting perl directly (usually via
        ~/.staticperl/bin/perl), but beats typing the path sometimes.

        Example: check that the Gtk2 module is installed and loadable.

           staticperl perl -MGtk2 -e0

    staticperl cpan [args...]
        Starts an interactive CPAN shell that you can use to install further
        modules. Installs the perl first if necessary, but apart from that,
        no magic is involved: you could just as well run it manually via
        ~/.staticperl/perl/bin/cpan, except that staticperl additionally
        sets the environment variable $PERL to the path of the perl
        interpreter, which is handy in subshells.

        Any additional arguments are simply passed to the cpan command.

    staticperl instcpan module...
        Tries to install all the modules given and their dependencies, using
        CPAN.

        Example:

           staticperl instcpan EV AnyEvent::HTTPD Coro

    staticperl instsrc directory...
        In the unlikely case that you have unpacked perl modules around and
        want to install from these instead of from CPAN, you can do this
        using this command by specifying all the directories with modules in
        them that you want to have built.

    staticperl clean
        Deletes the perl source directory (and potentially cleans up other
        intermediate files). This can be used to clean up files only needed
        for building perl, without removing the installed perl interpreter.

        At the moment, it doesn't delete downloaded tarballs.

        The exact semantics of this command will probably change.

    staticperl distclean
        This wipes your complete ~/.staticperl directory. Be careful with
        this, it nukes your perl download, perl sources, perl distribution
        and any installed modules. It is useful if you wish to start over
        "from scratch" or when you want to uninstall staticperl.

  PHASE 2 COMMANDS: BUILDING PERL BUNDLES
    Building (linking) a new perl binary is handled by a separate script. To
    make it easy to use staticperl from a chroot, the script is embedded
    into staticperl, which will write it out and call for you with any
    arguments you pass:

       staticperl mkbundle mkbundle-args...

    In the oh so unlikely case of something not working here, you can run
    the script manually as well (by default it is written to
    ~/.staticperl/mkbundle).

    mkbundle is a more conventional command and expect the argument syntax
    commonly used on UNIX clones. For example, this command builds a new
    perl binary and includes Config.pm (for perl -V), AnyEvent::HTTPD, URI
    and a custom httpd script (from eg/httpd in this distribution):

       # first make sure we have perl and the required modules
       staticperl instcpan AnyEvent::HTTPD

       # now build the perl
       staticperl mkperl -MConfig_heavy.pl -MAnyEvent::Impl::Perl \
                         -MAnyEvent::HTTPD -MURI::http \
                         --add 'eg/httpd httpd.pm'

       # finally, invoke it
       ./perl -Mhttpd

    As you can see, things are not quite as trivial: the Config module has a
    hidden dependency which is not even a perl module (Config_heavy.pl),
    AnyEvent needs at least one event loop backend that we have to specify
    manually (here AnyEvent::Impl::Perl), and the URI module (required by
    AnyEvent::HTTPD) implements various URI schemes as extra modules - since
    AnyEvent::HTTPD only needs "http" URIs, we only need to include that
    module. I found out about these dependencies by carefully watching any
    error messages about missing modules...

    Instead of building a new perl binary, you can also build a standalone
    application:

       # build the app
       staticperl mkapp app --boot eg/httpd \
                        -MAnyEvent::Impl::Perl -MAnyEvent::HTTPD -MURI::http

       # run it
       ./app

    Here are the three phase 2 commands:

    staticperl mkbundle args...
        The "default" bundle command - it interprets the given bundle
        options and writes out bundle.h, bundle.c, bundle.ccopts and
        bundle.ldopts files, useful for embedding.

    staticperl mkperl args...
        Creates a bundle just like staticperl mkbundle (in fact, it's the
        same as invoking staticperl mkbundle --perl args...), but then
        compiles and links a new perl interpreter that embeds the created
        bundle, then deletes all intermediate files.

    staticperl mkapp filename args...
        Does the same as staticperl mkbundle (in fact, it's the same as
        invoking staticperl mkbundle --app filename args...), but then
        compiles and links a new standalone application that simply
        initialises the perl interpreter.

        The difference to staticperl mkperl is that the standalone
        application does not act like a perl interpreter would - in fact, by
        default it would just do nothing and exit immediately, so you should
        specify some code to be executed via the --boot option.

   OPTION PROCESSING
    All options can be given as arguments on the command line (typically
    using long (e.g. "--verbose") or short option (e.g. "-v") style). Since
    specifying a lot of options can make the command line very long and
    unwieldy, you can put all long options into a "bundle specification
    file" (one option per line, with or without "--" prefix) and specify
    this bundle file instead.

    For example, the command given earlier to link a new perl could also
    look like this:

       staticperl mkperl httpd.bundle

    With all options stored in the httpd.bundle file (one option per line,
    everything after the option is an argument):

       use "Config_heavy.pl"
       use AnyEvent::Impl::Perl
       use AnyEvent::HTTPD
       use URI::http
       add eg/httpd httpd.pm

    All options that specify modules or files to be added are processed in
    the order given on the command line.

   BUNDLE CREATION WORKFLOW / STATICPERL MKBUNDLE OPTIONS
    staticperl mkbundle works by first assembling a list of candidate files
    and modules to include, then filtering them by include/exclude patterns.
    The remaining modules (together with their direct dependencies, such as
    link libraries and AutoLoader files) are then converted into bundle
    files suitable for embedding. staticperl mkbundle can then optionally
    build a new perl interpreter or a standalone application.

    Step 0: Generic argument processing.
        The following options influence staticperl mkbundle itself.

        "--verbose" | "-v"
            Increases the verbosity level by one (the default is 1).

        "--quiet" | "-q"
            Decreases the verbosity level by one.

        any other argument
            Any other argument is interpreted as a bundle specification
            file, which supports all options (without extra quoting), one
            option per line, in the format "option" or "option argument".
            They will effectively be expanded and processed as if they were
            directly written on the command line, in place of the file name.

    Step 1: gather candidate files and modules
        In this step, modules, perl libraries (.pl files) and other files
        are selected for inclusion in the bundle. The relevant options are
        executed in order (this makes a difference mostly for "--eval",
        which can rely on earlier "--use" options to have been executed).

        "--use" module | "-M"module
            Include the named module or perl library and trace direct
            dependencies. This is done by loading the module in a subprocess
            and tracing which other modules and files it actually loads.

            Example: include AnyEvent and AnyEvent::Impl::Perl.

               staticperl mkbundle --use AnyEvent --use AnyEvent::Impl::Perl

            Sometimes you want to load old-style "perl libraries" (.pl
            files), or maybe other weirdly named files. To support this, the
            "--use" option actually tries to do what you mean, depending on
            the string you specify:

            a possibly valid module name, e.g. common::sense, Carp,
            Coro::Mysql.
                If the string contains no quotes, no / and no ., then
                "--use" assumes that it is a normal module name. It will
                create a new package and evaluate a "use module" in it, i.e.
                it will load the package and do a default import.

                The import step is done because many modules trigger more
                dependencies when something is imported than without.

            anything that contains / or . characters, e.g. utf8_heavy.pl,
            Module/private/data.pl.
                The string will be quoted and passed to require, as if you
                used "require $module". Nothing will be imported.

            "path" or 'path', e.g. "utf8_heavy.pl".
                If you enclose the name into single or double quotes, then
                the quotes will be removed and the resulting string will be
                passed to require. This syntax is form compatibility with
                older versions of staticperl and should not be used anymore.

            Example: "use" AnyEvent::Socket, once using "use" (importing the
            symbols), and once via "require", not importing any symbols. The
            first form is preferred as many modules load some extra
            dependencies when asked to export symbols.

               staticperl mkbundle -MAnyEvent::Socket     # use + import
               staticperl mkbundle -MAnyEvent/Socket.pm   # require only

            Example: include the required files for perl -V to work in all
            its glory (Config.pm is included automatically by the dependency
            tracker).

               # shell command
               staticperl mkbundle -MConfig_heavy.pl

               # bundle specification file
               use Config_heavy.pl

            The "-M"module syntax is included as a convenience that might be
            easier to remember than "--use" - it's the same switch as perl
            itself uses to load modules. Or maybe it confuses people. Time
            will tell. Or maybe not. Sigh.

        "--eval" "perl code" | "-e" "perl code"
            Sometimes it is easier (or necessary) to specify dependencies
            using perl code, or maybe one of the modules you use need a
            special use statement. In that case, you can use "--eval" to
            execute some perl snippet or set some variables or whatever you
            need. All files "require"'d or "use"'d while executing the
            snippet are included in the final bundle.

            Keep in mind that mkbundle will not import any symbols from the
            modules named by the "--use" option, so do not expect the
            symbols from modules you "--use"'d earlier on the command line
            to be available.

            Example: force AnyEvent to detect a backend and therefore
            include it in the final bundle.

               staticperl mkbundle --eval 'use AnyEvent; AnyEvent::detect'

               # or like this
               staticperl mkbundle -MAnyEvent --eval 'AnyEvent::detect'

            Example: use a separate "bootstrap" script that "use"'s lots of
            modules and also include this in the final bundle, to be
            executed automatically when the interpreter is initialised.

               staticperl mkbundle --eval 'do "bootstrap"' --boot bootstrap

        "--boot" filename
            Include the given file in the bundle and arrange for it to be
            executed (using "require") before the main program when the new
            perl is initialised. This can be used to modify @INC or do
            similar modifications before the perl interpreter executes
            scripts given on the command line (or via "-e"). This works even
            in an embedded interpreter - the file will be executed during
            interpreter initialisation in that case.

        "--incglob" pattern
            This goes through all standard library directories and tries to
            match any .pm and .pl files against the extended glob pattern
            (see below). If a file matches, it is added. The pattern is
            matched against the full path of the file (sans the library
            directory prefix), e.g. Sys/Syslog.pm.

            This is very useful to include "everything":

               --incglob '*'

            It is also useful for including perl libraries, or trees of
            those, such as the unicode database files needed by some perl
            built-ins, the regex engine and other modules.

               --incglob '/unicore/**.pl'

        "--add" file | "--add" "file alias"
            Adds the given (perl) file into the bundle (and optionally call
            it "alias"). The file is either an absolute path or a path
            relative to the current directory. If an alias is specified,
            then this is the name it will use for @INC searches, otherwise
            the path file will be used as the internal name.

            This switch is used to include extra files into the bundle.

            Example: embed the file httpd in the current directory as
            httpd.pm when creating the bundle.

               staticperl mkperl --add "httpd httpd.pm"

               # can be accessed via "use httpd"

            Example: add a file initcode from the current directory.

               staticperl mkperl --add 'initcode &initcode'

               # can be accessed via "do '&initcode'"

            Example: add local files as extra modules in the bundle.

               # specification file
               add file1 myfiles/file1.pm
               add file2 myfiles/file2.pm
               add file3 myfiles/file3.pl

               # then later, in perl, use
               use myfiles::file1;
               require myfiles::file2;
               my $res = do "myfiles/file3.pl";

        "--binadd" file | "--add" "file alias"
            Just like "--add", except that it treats the file as binary and
            adds it without any postprocessing (perl files might get
            stripped to reduce their size).

            If you specify an alias you should probably add a "/" prefix to
            avoid clashing with embedded perl files (whose paths never start
            with "/"), and/or use a special directory prefix, such as
            "/res/name".

            You can later get a copy of these files by calling "static::find
            "alias"".

            An alternative way to embed binary files is to convert them to
            perl and use "do" to get the contents - this method is a bit
            cumbersome, but works both inside and outside of a staticperl
            bundle, without extra ado:

               # a "binary" file, call it "bindata.pl"
               <<'SOME_MARKER'
               binary data NOT containing SOME_MARKER
               SOME_MARKER

               # load the binary
               chomp (my $data = do "bindata.pl");

        "--allow-dynamic"
            By default, when mkbundle hits a dynamic perl extension (e.g. a
            .so or .dll file), it will stop with a fatal error.

            When this option is enabled, mkbundle packages the shared object
            into the bundle instead, with a prefix of ! (e.g.
            !auto/List/Util/Util.so). What you do with that is currently up
            to you, staticperl has no special support for this at the
            moment, apart from working around the lack of availability of
            PerlIO::scalar while bootstrapping, at a speed cost.

            One way to deal with this is to write all files starting with !
            into some directory and then "unshift" that path onto @INC.

            #TODO: example

    Step 2: filter all files using "--include" and "--exclude" options.
        After all candidate files and modules are added, they are *filtered*
        by a combination of "--include" and "--exclude" patterns (there is
        an implicit "--include *" at the end, so if no filters are
        specified, all files are included).

        All that this step does is potentially reduce the number of files
        that are to be included - no new files are added during this step.

        "--include" pattern | "-i" pattern | "--exclude" pattern | "-x"
        pattern
            These specify an include or exclude pattern to be applied to the
            candidate file list. An include makes sure that the given files
            will be part of the resulting file set, an exclude will exclude
            remaining files. The patterns are "extended glob patterns" (see
            below).

            The patterns are applied "in order" - files included via earlier
            "--include" specifications cannot be removed by any following
            "--exclude", and likewise, and file excluded by an earlier
            "--exclude" cannot be added by any following "--include".

            For example, to include everything except "Devel" modules, but
            still include Devel::PPPort, you could use this:

               --incglob '*' -i '/Devel/PPPort.pm' -x '/Devel/**'

    Step 3: add any extra or "hidden" dependencies.
        staticperl currently knows about three extra types of depdendencies
        that are added automatically. Only one (.packlist files) is
        currently optional and can be influenced, the others are always
        included:

        "--usepacklists"
            Read .packlist files for each distribution that happens to match
            a module name you specified. Sounds weird, and it is, so expect
            semantics to change somehow in the future.

            The idea is that most CPAN distributions have a .pm file that
            matches the name of the distribution (which is rather reasonable
            after all).

            If this switch is enabled, then if any of the .pm files that
            have been selected match an install distribution, then all .pm,
            .pl, .al and .ix files installed by this distribution are also
            included.

            For example, using this switch, when the URI module is
            specified, then all URI submodules that have been installed via
            the CPAN distribution are included as well, so you don't have to
            manually specify them.

        AutoLoader splitfiles
            Some modules use AutoLoader - less commonly (hopefully) used
            functions are split into separate .al files, and an index (.ix)
            file contains the prototypes.

            Both .ix and .al files will be detected automatically and added
            to the bundle.

        link libraries (.a files)
            Modules using XS (or any other non-perl language extension
            compiled at installation time) will have a static archive
            (typically .a). These will automatically be added to the linker
            options in bundle.ldopts.

            Should staticperl find a dynamic link library (typically .so) it
            will warn about it - obviously this shouldn't happen unless you
            use staticperl on the wrong perl, or one (probably wrongly)
            configured to use dynamic loading.

        extra libraries (extralibs.ld)
            Some modules need linking against external libraries - these are
            found in extralibs.ld and added to bundle.ldopts.

    Step 4: write bundle files and optionally link a program
        At this point, the select files will be read, processed (stripped)
        and finally the bundle files get written to disk, and staticperl
        mkbundle is normally finished. Optionally, it can go a step further
        and either link a new perl binary with all selected modules and
        files inside, or build a standalone application.

        Both the contents of the bundle files and any extra linking is
        controlled by these options:

        "--strip" "none"|"pod"|"ppi"
            Specify the stripping method applied to reduce the file of the
            perl sources included.

            The default is "pod", which uses the Pod::Strip module to remove
            all pod documentation, which is very fast and reduces file size
            a lot.

            The "ppi" method uses PPI to parse and condense the perl
            sources. This saves a lot more than just Pod::Strip, and is
            generally safer, but is also a lot slower (some files take
            almost a minute to strip - staticperl maintains a cache of
            stripped files to speed up subsequent runs for this reason).
            Note that this method doesn't optimise for raw file size, but
            for best compression (that means that the uncompressed file size
            is a bit larger, but the files compress better, e.g. with upx).

            Last not least, if you need accurate line numbers in error
            messages, or in the unlikely case where "pod" is too slow, or
            some module gets mistreated, you can specify "none" to not
            mangle included perl sources in any way.

        "--perl"
            After writing out the bundle files, try to link a new perl
            interpreter. It will be called perl and will be left in the
            current working directory. The bundle files will be removed.

            This switch is automatically used when staticperl is invoked
            with the "mkperl" command instead of "mkbundle".

            Example: build a new ./perl binary with only common::sense
            inside - it will be even smaller than the standard perl
            interpreter as none of the modules of the base distribution
            (such as Fcntl) will be included.

               staticperl mkperl -Mcommon::sense

        "--app" name
            After writing out the bundle files, try to link a new standalone
            program. It will be called "name", and the bundle files get
            removed after linking it.

            This switch is automatically used when staticperl is invoked
            with the "mkapp" command instead of "mkbundle".

            The difference to the (mutually exclusive) "--perl" option is
            that the binary created by this option will not try to act as a
            perl interpreter - instead it will simply initialise the perl
            interpreter, clean it up and exit.

            This means that, by default, it will do nothing but burn a few
            CPU cycles - for it to do something useful you *must* add some
            boot code, e.g. with the "--boot" option.

            Example: create a standalone perl binary called ./myexe that
            will execute appfile when it is started.

               staticperl mkbundle --app myexe --boot appfile

        "--ignore-env"
            Generates extra code to unset some environment variables before
            initialising/running perl. Perl supports a lot of environment
            variables that might alter execution in ways that might be
            undesirablre for standalone applications, and this option
            removes those known to cause trouble.

            Specifically, these are removed:

            "PERL_HASH_SEED_DEBUG" and "PERL_DEBUG_MSTATS" can cause
            undesirable output, "PERL5OPT", "PERL_DESTRUCT_LEVEL",
            "PERL_HASH_SEED" and "PERL_SIGNALS" can alter execution
            significantly, and "PERL_UNICODE", "PERLIO_DEBUG" and "PERLIO"
            can affect input and output.

            The variables "PERL_LIB" and "PERL5_LIB" are always ignored
            because the startup code used by staticperl overrides @INC in
            all cases.

            This option will not make your program more secure (unless you
            are running with elevated privileges), but it will reduce the
            surprise effect when a user has these environment variables set
            and doesn't expect your standalone program to act like a perl
            interpreter.

        "--static"
            Add "-static" to bundle.ldopts, which means a fully static (if
            supported by the OS) executable will be created. This is not
            immensely useful when just creating the bundle files, but is
            most useful when linking a binary with the "--perl" or "--app"
            options.

            The default is to link the new binary dynamically (that means
            all perl modules are linked statically, but all external
            libraries are still referenced dynamically).

            Keep in mind that Solaris doesn't support static linking at all,
            and systems based on GNU libc don't really support it in a very
            usable fashion either. Try uClibc if you want to create fully
            statically linked executables, or try the "--staticlib" option
            to link only some libraries statically.

        "--staticlib" libname
            When not linking fully statically, this option allows you to
            link specific libraries statically. What it does is simply
            replace all occurrences of "-llibname" with the GCC-specific
            "-Wl,-Bstatic -llibname -Wl,-Bdynamic" option.

            This will have no effect unless the library is actually linked
            against, specifically, "--staticlib" will not link against the
            named library unless it would be linked against anyway.

            Example: link libcrypt statically into the final binary.

               staticperl mkperl -MIO::AIO --staticlib crypt

               # ldopts might now contain:
               # -lm -Wl,-Bstatic -lcrypt -Wl,-Bdynamic -lpthread

   EXTENDED GLOB PATTERNS
    Some options of staticperl mkbundle expect an *extended glob pattern*.
    This is neither a normal shell glob nor a regex, but something in
    between. The idea has been copied from rsync, and there are the current
    matching rules:

    Patterns starting with / will be a anchored at the root of the library
    tree.
        That is, /unicore will match the unicore directory in @INC, but
        nothing inside, and neither any other file or directory called
        unicore anywhere else in the hierarchy.

    Patterns not starting with / will be anchored at the end of the path.
        That is, idna.pl will match any file called idna.pl anywhere in the
        hierarchy, but not any directories of the same name.

    A * matches anything within a single path component.
        That is, /unicore/*.pl would match all .pl files directly inside
        "/unicore", not any deeper level .pl files. Or in other words, *
        will not match slashes.

    A ** matches anything.
        That is, /unicore/**.pl would match all .pl files under /unicore, no
        matter how deeply nested they are inside subdirectories.

    A ? matches a single character within a component.
        That is, /Encode/??.pm matches /Encode/JP.pm, but not the
        hypothetical /Encode/J/.pm, as ? does not match /.

  STATICPERL CONFIGURATION AND HOOKS
    During (each) startup, staticperl tries to source some shell files to
    allow you to fine-tune/override configuration settings.

    In them you can override shell variables, or define shell functions
    ("hooks") to be called at specific phases during installation. For
    example, you could define a "postinstall" hook to install additional
    modules from CPAN each time you start from scratch.

    If the env variable $STATICPERLRC is set, then staticperl will try to
    source the file named with it only. Otherwise, it tries the following
    shell files in order:

       /etc/staticperlrc
       ~/.staticperlrc
       $STATICPERL/rc

    Note that the last file is erased during staticperl distclean, so
    generally should not be used.

   CONFIGURATION VARIABLES
   Variables you *should* override
    "EMAIL"
        The e-mail address of the person who built this binary. Has no good
        default, so should be specified by you.

    "CPAN"
        The URL of the CPAN mirror to use (e.g.
        <http://mirror.netcologne.de/cpan/>).

    "EXTRA_MODULES"
        Additional modules installed during staticperl install. Here you can
        set which modules you want have to installed from CPAN.

        Example: I really really need EV, AnyEvent, Coro and AnyEvent::AIO.

           EXTRA_MODULES="EV AnyEvent Coro AnyEvent::AIO"

        Note that you can also use a "postinstall" hook to achieve this, and
        more.

   Variables you might *want* to override
    "STATICPERL"
        The directory where staticperl stores all its files (default:
        ~/.staticperl).

    "DLCACHE"
        The path to a directory (will be created if it doesn't exist) where
        downloaded perl sources are being cached, to avoid downloading them
        again. The default is empty, which means there is no cache.

    "PERL_VERSION"
        The perl version to install - default is currently 5.12.3, but 5.8.9
        is also a good choice (5.8.9 is much smaller than 5.12.3, while
        5.10.1 is about as big as 5.12.3).

    "PERL_MM_USE_DEFAULT", "EV_EXTRA_DEFS", ...
        Usually set to 1 to make modules "less inquisitive" during their
        installation. You can set (and export!) any environment variable you
        want - some modules (such as Coro or EV) use environment variables
        for further tweaking.

    "PERL_PREFIX"
        The prefix where perl gets installed (default: $STATICPERL/perl),
        i.e. where the bin and lib subdirectories will end up.

    "PERL_CONFIGURE"
        Additional Configure options - these are simply passed to the perl
        Configure script. For example, if you wanted to enable dynamic
        loading, you could pass "-Dusedl". To enable ithreads (Why would you
        want that insanity? Don't! Use forks instead!) you would pass
        "-Duseithreads" and so on.

        More commonly, you would either activate 64 bit integer support
        ("-Duse64bitint"), or disable large files support (-Uuselargefiles),
        to reduce filesize further.

    "PERL_CC", "PERL_CCFLAGS", "PERL_OPTIMIZE", "PERL_LDFLAGS", "PERL_LIBS"
        These flags are passed to perl's Configure script, and are generally
        optimised for small size (at the cost of performance). Since they
        also contain subtle workarounds around various build issues,
        changing these usually requires understanding their default values -
        best look at the top of the staticperl script for more info on
        these, and use a ~/.staticperlrc to override them.

        Most of the variables override (or modify) the corresponding
        Configure variable, except "PERL_CCFLAGS", which gets appended.

        The default for "PERL_OPTIMIZE" is "-Os" (assuming gcc), and for
        "PERL_LIBS" is "-lm -lcrypt", which should be good for most (but not
        all) systems.

        For other compilers or more customised optimisation settings, you
        need to adjust these, e.g. in your ~/.staticperlrc.

        With gcc on x86 and amd64, you can get more space-savings by using:

           -Os -ffunction-sections -fdata-sections -finline-limit=8 -mpush-args
           -mno-inline-stringops-dynamically -mno-align-stringops

        And on x86 and pentium3 and newer (basically everything you might
        ever want to run on), adding these is even better for space-savings
        (use -mtune=core2 or something newer for much faster code, too):

           -fomit-frame-pointer -march=pentium3 -mtune=i386

   Variables you probably *do not want* to override
    "MAKE"
        The make command to use - default is "make".

    "MKBUNDLE"
        Where staticperl writes the "mkbundle" command to (default:
        $STATICPERL/mkbundle).

    "STATICPERL_MODULES"
        Additional modules needed by "mkbundle" - should therefore not be
        changed unless you know what you are doing.

   OVERRIDABLE HOOKS
    In addition to environment variables, it is possible to provide some
    shell functions that are called at specific times. To provide your own
    commands, just define the corresponding function.

    The actual order in which hooks are invoked during a full install from
    scratch is "preconfigure", "patchconfig", "postconfigure", "postbuild",
    "postinstall".

    Example: install extra modules from CPAN and from some directories at
    staticperl install time.

       postinstall() {
          rm -rf lib/threads* # weg mit Schaden
          instcpan IO::AIO EV
          instsrc ~/src/AnyEvent
          instsrc ~/src/XML-Sablotron-1.0100001
          instcpan Anyevent::AIO AnyEvent::HTTPD
       }

    preconfigure
        Called just before running ./Configure in the perl source directory.
        Current working directory is the perl source directory.

        This can be used to set any "PERL_xxx" variables, which might be
        costly to compute.

    patchconfig
        Called after running ./Configure in the perl source directory to
        create ./config.sh, but before running ./Configure -S to actually
        apply the config. Current working directory is the perl source
        directory.

        Can be used to tailor/patch config.sh or do any other modifications.

    postconfigure
        Called after configuring, but before building perl. Current working
        directory is the perl source directory.

    postbuild
        Called after building, but before installing perl. Current working
        directory is the perl source directory.

        I have no clue what this could be used for - tell me.

    postinstall
        Called after perl and any extra modules have been installed in
        $PREFIX, but before setting the "installation O.K." flag.

        The current working directory is $PREFIX, but maybe you should not
        rely on that.

        This hook is most useful to customise the installation, by deleting
        files, or installing extra modules using the "instcpan" or "instsrc"
        functions.

        The script must return with a zero exit status, or the installation
        will fail.

ANATOMY OF A BUNDLE
    When not building a new perl binary, "mkbundle" will leave a number of
    files in the current working directory, which can be used to embed a
    perl interpreter in your program.

    Intimate knowledge of perlembed and preferably some experience with
    embedding perl is highly recommended.

    "mkperl" (or the "--perl" option) basically does this to link the new
    interpreter (it also adds a main program to bundle.):

       $Config{cc} $(cat bundle.ccopts) -o perl bundle.c $(cat bundle.ldopts)

    bundle.h
        A header file that contains the prototypes of the few symbols
        "exported" by bundle.c, and also exposes the perl headers to the
        application.

        staticperl_init (xs_init = 0)
            Initialises the perl interpreter. You can use the normal perl
            functions after calling this function, for example, to define
            extra functions or to load a .pm file that contains some
            initialisation code, or the main program function:

               XS (xsfunction)
               {
                 dXSARGS;

                 // now we have items, ST(i) etc.
               }

               static void
               run_myapp(void)
               {
                  staticperl_init (0);
                  newXSproto ("myapp::xsfunction", xsfunction, __FILE__, "$$;$");
                  eval_pv ("require myapp::main", 1); // executes "myapp/main.pm"
               }

            When your bootcode already wants to access some XS functions at
            compiletime, then you need to supply an "xs_init" function
            pointer that is called as soon as perl is initialised enough to
            define XS functions, but before the preamble code is executed:

               static void
               xs_init (pTHX)
               {
                 newXSproto ("myapp::xsfunction", xsfunction, __FILE__, "$$;$");
               }

               static void
               run_myapp(void)
               {
                  staticperl_init (xs_init);
               }

        staticperl_cleanup ()
            In the unlikely case that you want to destroy the perl
            interpreter, here is the corresponding function.

        staticperl_xs_init (pTHX)
            Sometimes you need direct control over "perl_parse" and
            "perl_run", in which case you do not want to use
            "staticperl_init" but call them on your own.

            Then you need this function - either pass it directly as the
            "xs_init" function to "perl_parse", or call it as one of the
            first things from your own "xs_init" function.

        PerlInterpreter *staticperl
            The perl interpreter pointer used by staticperl. Not normally so
            useful, but there it is.

    bundle.ccopts
        Contains the compiler options required to compile at least bundle.c
        and any file that includes bundle.h - you should probably use it in
        your "CFLAGS".

    bundle.ldopts
        The linker options needed to link the final program.

RUNTIME FUNCTIONALITY
    Binaries created with "mkbundle"/"mkperl" contain extra functionality,
    mostly related to the extra files bundled in the binary (the virtual
    filesystem). All of this data is statically compiled into the binary,
    and accessing means copying it from a read-only section of your binary.
    Data pages in this way is usually freed by the operating system, as it
    isn't use more the onace.

  VIRTUAL FILESYSTEM
    Every bundle has a virtual filesystem. The only information stored in it
    is the path and contents of each file that was bundled.

   LAYOUT
    Any path starting with an ampersand (&) or exclamation mark (!) are
    reserved by staticperl. They must only be used as described in this
    section.

    !   All files that typically cannot be loaded from memory (such as
        dynamic objects or shared libraries), but have to reside in the
        filesystem, are prefixed with !. Typically these files get written
        out to some (semi-)temporary directory shortly after program
        startup, or before being used.

    !boot
        The bootstrap file, if specified during bundling.

    !auto/
        Shared objects or dlls corresponding to dynamically-linked perl
        extensions are stored with an !auto/ prefix.

    !lib/
        External shared libraries are stored in this directory.

    any letter
        Any path starting with a letter is a perl library file. For example,
        Coro/AIO.pm corresponds to the file loaded by "use Coro::AIO", and
        Coro/jit.pl corresponds to "require "Coro/jit.pl"".

        Obviously, module names shouldn't start with any other characters
        than letters :)

   FUNCTIONS
    $file = static::find $path
        Returns the data associated with the given $path (e.g.
        "Digest/MD5.pm", "auto/POSIX/autosplit.ix").

        Returns "undef" if the file isn't embedded.

    @paths = static::list
        Returns the list of all paths embedded in this binary.

  EXTRA FEATURES
    In addition, for the embedded loading of perl files to work, staticperl
    overrides the @INC array.

FULLY STATIC BINARIES - UCLIBC AND BUILDROOT
    To make truly static (Linux-) libraries, you might want to have a look
    at buildroot (<http://buildroot.uclibc.org/>).

    Buildroot is primarily meant to set up a cross-compile environment
    (which is not so useful as perl doesn't quite like cross compiles), but
    it can also compile a chroot environment where you can use staticperl.

    To do so, download buildroot, and enable "Build options => development
    files in target filesystem" and optionally "Build options => gcc
    optimization level (optimize for size)". At the time of writing, I had
    good experiences with GCC 4.4.x but not GCC 4.5.

    To minimise code size, I used "-pipe -ffunction-sections -fdata-sections
    -finline-limit=8 -fno-builtin-strlen -mtune=i386". The "-mtune=i386"
    doesn't decrease codesize much, but it makes the file much more
    compressible (and the execution a lot slower...).

    If you don't need Coro or threads, you can go with "linuxthreads.old"
    (or no thread support). For Coro, it is highly recommended to switch to
    a uClibc newer than 0.9.31 (at the time of this writing, I used the
    20101201 snapshot) and enable NPTL, otherwise Coro needs to be
    configured with the ultra-slow pthreads backend to work around
    linuxthreads bugs (it also uses twice the address space needed for
    stacks).

    If you use "linuxthreads.old", then you should also be aware that uClibc
    shares "errno" between all threads when statically linking. See
    <http://lists.uclibc.org/pipermail/uclibc/2010-June/044157.html> for a
    workaround (and <https://bugs.uclibc.org/2089> for discussion).

    "ccache" support is also recommended, especially if you want to play
    around with buildroot options. Enabling the "miniperl" package will
    probably enable all options required for a successful perl build.
    staticperl itself additionally needs either "wget" (recommended, for
    CPAN) or "curl".

    As for shells, busybox should provide all that is needed, but the
    default busybox configuration doesn't include comm which is needed by
    perl - either make a custom busybox config, or compile coreutils.

    For the latter route, you might find that bash has some bugs that keep
    it from working properly in a chroot - either use dash (and link it to
    /bin/sh inside the chroot) or link busybox to /bin/sh, using it's
    built-in ash shell.

    Finally, you need /dev/null inside the chroot for many scripts to work -
    either cp /dev/null output/target/dev or bind-mounting your /dev will
    provide this.

    After you have compiled and set up your buildroot target, you can copy
    staticperl from the "App::Staticperl" distribution or from your perl bin
    directory (if you installed it) into the output/target filesystem,
    chroot inside and run it.

RECIPES / SPECIFIC MODULES
    This section contains some common(?) recipes and information about
    problems with some common modules or perl constructs that require extra
    files to be included.

  MODULES
    utf8
        Some functionality in the utf8 module, such as swash handling (used
        for unicode character ranges in regexes) is implemented in the
        "utf8_heavy.pl" library:

           -Mutf8_heavy.pl

        Many Unicode properties in turn are defined in separate modules,
        such as "unicore/Heavy.pl" and more specific data tables such as
        "unicore/To/Digit.pl" or "unicore/lib/Perl/Word.pl". These tables
        are big (7MB uncompressed, although staticperl contains special
        handling for those files), so including them on demand by your
        application only might pay off.

        To simply include the whole unicode database, use:

           --incglob '/unicore/**.pl'

    AnyEvent
        AnyEvent needs a backend implementation that it will load in a
        delayed fashion. The AnyEvent::Impl::Perl backend is the default
        choice for AnyEvent if it can't find anything else, and is usually a
        safe fallback. If you plan to use e.g. EV (POE...), then you need to
        include the AnyEvent::Impl::EV (AnyEvent::Impl::POE...) backend as
        well.

        If you want to handle IRIs or IDNs (AnyEvent::Util punycode and idn
        functions), you also need to include "AnyEvent/Util/idna.pl" and
        "AnyEvent/Util/uts46data.pl".

        Or you can use "--usepacklists" and specify "-MAnyEvent" to include
        everything.

    Cairo
        See Glib, same problem, same solution.

    Carp
        Carp had (in older versions of perl) a dependency on Carp::Heavy. As
        of perl 5.12.2 (maybe earlier), this dependency no longer exists.

    Config
        The perl -V switch (as well as many modules) needs Config, which in
        turn might need "Config_heavy.pl". Including the latter gives you
        both.

    Glib
        Glib literally requires Glib to be installed already to build - it
        tries to fake this by running Glib out of the build directory before
        being built. staticperl tries to work around this by forcing
        "MAN1PODS" and "MAN3PODS" to be empty via the "PERL_MM_OPT"
        environment variable.

    Gtk2
        See Pango, same problems, same solution.

    Pango
        In addition to the "MAN3PODS" problem in Glib, Pango also routes
        around ExtUtils::MakeMaker by compiling its files on its own.
        staticperl tries to patch ExtUtils::MM_Unix to route around Pango.

    Term::ReadLine::Perl
        Also needs Term::ReadLine::readline, or "--usepacklists".

    URI URI implements schemes as separate modules - the generic URL scheme
        is implemented in URI::_generic, HTTP is implemented in URI::http.
        If you need to use any of these schemes, you should include these
        manually, or use "--usepacklists".

  RECIPES
    Just link everything in
        To link just about everything installed in the perl library into a
        new perl, try this (the first time this runs it will take a long
        time, as a lot of files need to be parsed):

           staticperl mkperl -v --strip ppi --incglob '*'

        If you don't mind the extra megabytes, this can be a very effective
        way of creating bundles without having to worry about forgetting any
        modules.

        You get even more useful variants of this method by first selecting
        everything, and then excluding stuff you are reasonable sure not to
        need - bigperl <http://staticperl.schmorp.de/bigperl.html> uses this
        approach.

    Getting rid of netdb functions
        The perl core has lots of netdb functions ("getnetbyname",
        "getgrent" and so on) that few applications use. You can avoid
        compiling them in by putting the following fragment into a
        "preconfigure" hook:

           preconfigure() {
              for sym in \
                 d_getgrnam_r d_endgrent d_endgrent_r d_endhent \
                 d_endhostent_r d_endnent d_endnetent_r d_endpent \
                 d_endprotoent_r d_endpwent d_endpwent_r d_endsent \
                 d_endservent_r d_getgrent d_getgrent_r d_getgrgid_r \
                 d_getgrnam_r d_gethbyaddr d_gethent d_getsbyport \
                 d_gethostbyaddr_r d_gethostbyname_r d_gethostent_r \
                 d_getlogin_r d_getnbyaddr d_getnbyname d_getnent \
                 d_getnetbyaddr_r d_getnetbyname_r d_getnetent_r \
                 d_getpent d_getpbyname d_getpbynumber d_getprotobyname_r \
                 d_getprotobynumber_r d_getprotoent_r d_getpwent \
                 d_getpwent_r d_getpwnam_r d_getpwuid_r d_getsent \
                 d_getservbyname_r d_getservbyport_r d_getservent_r \
                 d_getspnam_r d_getsbyname
                 # d_gethbyname
              do
                 PERL_CONFIGURE="$PERL_CONFIGURE -U$sym"
              done
           }

        This mostly gains space when linking statically, as the functions
        will likely not be linked in. The gain for dynamically-linked
        binaries is smaller.

        Also, this leaves "gethostbyname" in - not only is it actually used
        often, the Socket module also exposes it, so leaving it out usually
        gains little. Why Socket exposes a C function that is in the core
        already is anybody's guess.

AUTHOR
     Marc Lehmann <schmorp@schmorp.de>
     http://software.schmorp.de/pkg/staticperl.html

Revision:	1.17
Committed:	Mon Mar 5 07:14:32 2012 UTC (14 years, 6 months ago) by root
Branch:	MAIN
CVS Tags:	rel-1_42, rel-1_41
Changes since 1.16:	+3 -3 lines
Log Message:	1.41