[DRAFT] PDD 30: Installation

Abstract

This PDD outlines Parrot's installation system and support. Parrot's core installation system will provide support for binary packages, a working make install target, compiled installables, and FHS compliant search paths for the installables. This document also aims to solve the current problem of accessing installed source-only files, and to allow the optimization of config bootstrapping if a frozen config_hash is already linked.

Version

$Revision$

Synopsis

Parrot installation process (from the parrot source directory):

  perl Configure.pl --prefix=/usr/lib
  make
  make test
  make install

Language installation process (from the language source directory):

  make
  make test
  make install

Description

Parrot uses Filesystem Hierarchy Standard (FHS) compliant install directories by default. Each install location is configurable with options passed to the configure script.

/usr/bin/parrot
/usr/lib/parrot/<version/library/>
/usr/lib/parrot/<version/include/>
/usr/lib/parrot/<version/dynext/>
/usr/lib/parrot/<version/languages/>
/usr/lib/parrot/<version/languages/*>
/usr/lib/parrot/<version/tools/>
/usr/share/doc/parrot/<version/

Bootstrapping the configuration hash should not read a config file when the hash is already contained in the pmc or executable. {{ See RT #57418 [TODO] optimize _config to omit .include "library/config.pir" on installables. }} The same problem is for every .include, loadlib and load_bytecode statement in installed files where the target is not installed. If in doubt install the missing library by patching the installation code.

Test executables are binary different to installable executables because of this embedded configuration hash. Test executables contain configuration hash with the prefix to the build directory, installables to the given prefix from Configure.pl. The executables that are tested should always be the same as the ones that are installed. Otherwise, subtle bugs can leak into the installed executables which can never be caught by the tests.

There are's also a long-standing conflict in building Parrot with an already installed shared libparrot.so. See RT #39742. For win32 this was solved to put the dll into build_dir, for compilers withour rpath support this is still a problem.

Dependencies

Building core Parrot depends on Perl (including perldoc, which may be a separate package), libgdm and libreadline.

Building a language depends on a series of Parrot build tools, installed in /usr/lib/parrot/<version/tools. These tools will generally not be included in the default parrot package on most systems, but will require a parrot-dev package to be installed before they can be built.

Definitions

The build_dir is the full path where Parrot was built. It is defined in the configuration hash. When building from source the build_dir is also the PARROT_RUNTIME prefix.

An installable is a bytecode or executable file which must not access the build_dir paths. The build_dir path is not available in a binary package. This is solved by generating and linking a special install_config.fpmc. Custom Python modules have a similar packaging problem, which they haven't solved yet.

The destination directory is the path of the installed Parrot tree after the prefix (/usr, /usr/local, or some other platform-specific or custom location). Creating a virtual installation path like this simplifies packaging by installing into a separate install tree and creating a tarball from that tree.

The configuration hash is the return value of the global function _config(), generated in config_lib.pasm, and either defined in library/config.pir, or as frozen PMC embedded in the test executable (config.fpmc), the installable executable (install_config.fpmc) or empty for miniparrot (null_config.fpmc).

make install

The Parrot build system is currently optimized for building and testing in the build directory, but not for building with an already installed Parrot. This is complicated by some simple build system bugs. It is also not optimized to build and test installables, which should not access libraries in the build directory, but in the destination directory.

The short-term goal of this document is to make installation work for Parrot, its libraries and tools, and its languages so that packagers can provide binary packages for parrot and its languages.

The longer-term goal is a framework so that external libraries and languages not within the current parrot source tree can also be properly built, tested and installed.

We do not only support GNU make or Win32 nmake but also other platform make variants, so we generate the Makefile at first from a generic template.

Currently for Parrot and its libraries the install actions are defined in the main Makefile. For the languages the install actions are defined in the language's Makefile. {{ See an implementation of this in #56554-make-install-lang.patch.}}

One general comment: We need to be careful not to install bytecode files in the same directories as executables. The C includes and the PIR includes also need to be separate.

make install currently does not work with an already installed shared libparrot.so on some platforms. {{See a patch for this in RT #39742.}}

bin_dir: Copy installables to the destination directory's bin directory as parrot-mylanguage.
script_dir: Optionally copy the main language file mylanguage.pbc to the destination directory's script directory. (/usr/lib/parrot/bin/ ?)
dynext: Copy shared libraries (groups and ops) to the destination directory's lib directory under /parrot/dynext/.

The subdirs are currently needed for forth and WMLScript, the other language pbc's are php_*.pbc, pipplib.pbc and tcllib.pbc.

lib_dir: Optionally copy include PASM and PIR files to the destination directory's lib directory under parrot/include/ (not yet).
doc_dir: Copy documentation files to a $doc_dir/languages/mylanguage/ subdirectory tree.
man_dir: Generate man(1) pages and copy to destination directory's $man_dir/man1/ directory. {{ what about man(2) and man(3) pages? info_dir ditto }}
html_dir: Optionally generate HTML and copy to destination directory's html directory, possibly under a language specific subdirectory. This should be selectable by a Configure or make install option.

Configuration bootstrapping

Bootstrapping the configuration hash should not read a config file when the hash is already contained in the PMC or executable. .include "library/config.pir" and load_bytecode "config.pbc" should be omitted on installables if possible.

{{NOTE: We need to step back and take a broader view here. Why is library/config.pir currently included in the installables? It sounds like a hack to work around earlier limitations in the build system. This is an ideal opportunity to eliminate the hack. -allison}}

Accessing not-installed files

Makefile and MANIFEST cleanup

Problem: Various PIR files load source-only PIR, PASM or compiler bytecode files, which are not installed in binary packages. This shows up when trying to run an installable with the build directory removed or renamed.

 $ parrot-forth.exe xx
 "load_bytecode" couldn't find file 'languages/forth/tokenstream.pbc'
 current instr.: ' init' pc 942 (forth.pir:9)
 called from Sub 'main' pc 994 (forth.pir:40)

 $ parrot-pheme.exe
 "load_bytecode" couldn't find file 'compilers/tge/TGE/Rule.pbc'
 current instr.: 'parrot;TGE;__onload' pc 19 (TGE.pir:94)
 called from Sub 'parrot;Pheme::AST::Grammar;__onload' pc 7175
    (languages/pheme/lib/ASTGrammar.pir:5)
 called from Sub 'parrot;Pheme::Compiler;main' pc -1 ((unknown file):-1)

 $ parrot-pipp
 Parrot VM: Can't stat
    /usr/src/perl/parrot/parrot-0.7.0-1/
         build/languages/pipp/src/common/pipplib.pbc, code 2.
 Unable to append PBC to the current directory
 current instr.: 'parrot;Pipp;__onload' pc 47 (src/common/pipp.pir:92)
 called from Sub 'parrot;Pipp;pipp' pc -1 ((unknown file):-1)

Fix 1: Install all deps and make sure that every HLL is installed at lib_dir/parrot/version/languages. See also pdd31_hll_interop.pod

{{NOTE: This may be a sign that we need to rethink our language build strategy. Trying to glom everything into a single C executable is less than ideal. Especially since it causes problems for language interoperability if every language is running off its own independent executable. -allison}}

The simple Forth and Pipp problem could be solved by merging the missing bytecode files to a single file forth.pbc and generate from this the installable.

The simple Pheme problem could be solved by installing also all TGE and other compiler bytecode files at the parrot/library/compilers path. Since TGE is not used elsewhere anymore, Pheme should be fixed to get rid of this. {{NOTE: commonly used libraries should be installed somewhere. -allison}}

The same problem is for every .include, loadlib and load_bytecode statement in installed files where the target is not installed.

Fix 2: Module system.

Avoid already loaded pbc files.

Source loading PIR statements like loadlib and load_bytecode should cache the file name and skip the file if it has already been loaded (as in perl5)

Fix 3: pbc_merge fixups

pbc_merge could patch up the bytecode (if possible) to omit loading load_bytecode pbc-files which are being merged, but hacking bytecode during pbc_merge is not desirable.

Implementation

A new language is generated by tools/dev/mk_language_shell.pl

The makefiles are generated from a config/makefiles/root.in template, which can make use of conditional platform and config logic, the forward slashes are automatically converted to backslashes for MSWin32 and \n is converted to \r\n for MSWin32 nmake. See Parrot::Configure::Compiler.

A new lib/Parrot/Install.pm or Parrot/Install.pir library should provide the same support as described above and simplify the Makefile and installation maintenance. The entry point could be a Makefile.pl or Makefile.pir then.