Architecture

Terminology

  • Package identifier: a package name and version, e.g. text-1.2.1.0
  • GhcPkgId: a package identifier plus the unique hash for the generated binary, e.g. text-1.2.1.0-bb83023b42179dd898ebe815ada112c2
  • Package index: a collection of packages available for download. This is a combination of an index containing all of the .cabal files (either a tarball downloaded via HTTP(S) or a Git repository) and some way to download package tarballs.
    • By default, stack uses a single package index (the Github/S3 mirrors of Hackage), but supports customization and adding more than one index
  • Package database: a collection of metadata about built libraries
  • Install root: a destination for installing packages into. Contains a bin path (for generated executables), lib (for the compiled libraries), pkgdb (for the package database), and a few other things
  • Snapshot: an LTS Haskell or Stackage Nightly, which gives information on a complete set of packages. This contains a lot of metadata, but importantly it can be converted into a mini build plan...
  • Mini build plan: a collection of package identifiers and their build flags that are known to build together
  • Resolver: the means by which stack resolves dependencies for your packages. The two currently supported options are snapshot (using LTS or Nightly), and GHC (which installs no extra dependencies). Others may be added in the future (such as a SAT-based dependency solver). These packages are always taken from a package index
  • extra-deps: additional packages to be taken from the package index for dependencies. This list will shadow packages provided by the resolver
  • Local packages: source code actually present on your file system, and referred to by the packages field in your stack.yaml file. Each local package has exactly one .cabal file
  • Project: a stack.yaml config file and all of the local packages it refers to.

Databases

Every build uses three distinct install roots, which means three separate package databases and bin paths. These are:

  • Global: the packages that ship with GHC. We never install anything into this database
  • Snapshot: a database shared by all projects using the same snapshot. Packages installed in this database must use the exact same dependencies and build flags as specified in the snapshot, and cannot be affected by user flags, ensuring that one project cannot corrupt another. There are two caveats to this:
    • If different projects use different package indices, then their definitions of what package foo-1.2.3 are may be different, in which case they can corrupt each other's shared databases. This is warned about in the FAQ
    • Turning on profiling may cause a package to be recompiled, which will result in a different GhcPkgId
  • Local: extra-deps, local packages, and snapshot packages which depend on them (more on that in shadowing)

Building

Shadowing

Every project must have precisely one version of a package. If one of your local packages or extra dependencies conflicts with a package in the snapshot, the local/extradep shadows the snapshot version. The way this works is:

  • The package is removed from the list of packages in the snapshot
  • Any package that depends on that package (directly or indirectly) is moved from the snapshot to extra-deps, so that it is available to your packages as dependencies.
    • Note that there is no longer any guarantee that this package will build, since you're using an untested dependency

After shadowing, you end up with what is called internally a SourceMap, which is Map PackageName PackageSource, where a PackageSource can be either a local package, or a package taken from a package index (specified as a version number and the build flags).

Installed packages

Once you have a SourceMap, you can inspect your three available databases and decide which of the installed packages you wish to use from them. We move from the global, to snapshot, and finally local, with the following rules:

  • If we require profiling, and the library does not provide profiling, do not use it
  • If the package is in the SourceMap, but belongs to a difference database, or has a different version, do not use it
  • If after the above two steps, any of the dependencies are unavailable, do not use it
  • Otherwise: include the package in the list of installed packages

We do something similar for executables, but maintain our own database of installed executables, since GHC does not track them for us.

Plan construction

When running a build, we know which packages we want installed (inventively called "wanteds"), which packages are available to install, and which are already installed. In plan construction, we put this information together to decide which packages must be built. The code in Stack.Build.ConstructPlan is authoritative on this and should be consulted. The basic idea though is:

  • If any of the dependencies have changed, reconfigure and rebuild
  • If a local package has any files changed, rebuild (but don't bother reconfiguring)
  • If a local package is wanted and we're running tests or benchmarks, run the test or benchmark even if the code and dependencies haven't changed

Plan execution

Once we have the plan, execution is a relatively simple process of calling runghc Setup.hs in the correct order with the correct parameters. See Stack.Build.Execute for the implementation.

The Setup.hs file uses the Cabal library to build packages.

Parsing Cabal files

Stack itself builds against a version of the Cabal library, which it uses for parsing Cabal files.

This version is not necessarily present on stack users’ machines.

This version determines which version of the Cabal file format stack is able to parse. Where possible, releases of stack will be compiled using the most recent version of Cabal-the-library, in order to support the most recent versions of the Cabal file format.

Building packages

The version of the Cabal library used to build packages Stack builds a Setup.hs file against a version of the Cabal library, in order to build packages.

The boot version of Cabal used by GHC, which is globally available to stack, is used to compile the build-type: Simple setup executable. All packages using the same compiler and Cabal version are built with the same executable. These executables are cached in the setup-exe-cache configuration directory.

Build artefacts are placed in the corresponding .stack-work/dist/Cabal-xxxxx directory.

For packages with build-type: Custom, Stack compiles Setup.hs against the version of the Cabal library present in the snapshot (which may be overridden using extra-deps), and uses that setup executable to perform builds. This treats Cabal as any other dependency package. The process is as follows:

  1. Stack uses the boot version of Cabal to build the required version of Cabal, which is treated as though built with build-type: Simple. This will take a short while — typically a few minutes.

  2. Stack uses this version of Cabal to build the setup executable from the Setup.hs file. The resulting executable is placed in the .stack-work/dist/Cabal-xxxxx directory corresponding to the boot version of Cabal — even though it was built with the snapshot version.

  3. This setup executable builds the package. Again, build artefacts are placed in the .stack-work/dist/Cabal-xxxxx directory — even though it was built with the snapshot version. A resulting executable is copied to the .stack-work/install/$compiler-variant/$snapshot/$compiler-version/bin directory.

Importing Cabal as a library

Packages may themselves depend on the Cabal library.

As any other dependency, they will use the snapshot version (which may be overridden using extra-deps).

What this means

There are a number of consequences of this design.

  1. Snapshot packages only depend on the GHC compiler version, not the Cabal library version (with the exception of build-type: Custom packages).

  2. The most recent stack can usually read the most recent Cabal files.

  3. However, stack may not be able to build packages defined using those files.

This occurs when

i. The package uses build-type: Simple and the Cabal file format requires a more recent version of Cabal than the global (boot) version for the compiler.

  e.g. Stack lts-11.22 uses GHC 8.2.2, corresponding to Cabal 2.0.1.0.
  A library using `build-type: Simple` and SPDX license identifiers (introduced in Cabal 2.2) will not build, though stack will process the Cabal file correctly.

      Cabal-simple_mPHDZzAJ_2.0.1.0_ghc-8.2.2: ./file.cabal:7: Parse of field 'license' failed.
  1. Stack only occasionally needs to build Cabal-the-library. This is a resource intensive build, so avoiding it improves performance.

  2. Stack uses ghc-pkg to identify the Cabal version it should use for the setup executable for a build. The build output is stored in a directory .stack-work/dist/Cabal-xxxxx named for the boot version of Cabal corresponding to the GHC version.. Together, this means it is only possible to know for sure which Cabal version ought to be used if the corresponding compiler is installed.

This behaviour will change with stack 2.0.

Configuration

stack has two layers of configuration: project and non-project. All of these are stored in stack.yaml files, but the former has extra fields (resolver, packages, extra-deps, and flags). The latter can be monoidally combined so that a system config file provides defaults, which a user can override with ~/.stack/config.yaml, and a project can further customize. In addition, environment variables STACK_ROOT and STACK_YAML can be used to tweak where stack gets its configuration from.

stack follows a simple algorithm for finding your project configuration file: start in the current directory, and keep going to the parent until it finds a stack.yaml. When using stack ghc or stack exec as mentioned above, you'll sometimes want to override that behavior and point to a specific project in order to use its databases and bin directories. To do so, simply set the STACK_YAML environment variable to point to the relevant stack.yaml file.

Snapshot auto-detection

When you run stack build with no stack.yaml, it will create a basic configuration with a single package (the current directory) and an auto-detected snapshot. The algorithm it uses for selecting this snapshot is:

  • Try the latest two LTS major versions at their most recent minor version release, and the most recent Stackage Nightly. For example, at the time of writing, this would be lts-2.10, lts-1.15, and nightly-2015-05-26
  • For each of these, test the version bounds in the package's .cabal file to see if they are compatible with the snapshot, choosing the first one that matches
  • If no snapshot matches, uses the most recent LTS snapshot, even though it will not compile

If you end up in the no compatible snapshot case, you typically have three options to fix things:

  • Manually specify a different snapshot that you know to be compatible. If you can do that, great, but typically if the auto-detection fails, it means that there's no compatible snapshot
  • Modify version bounds in your .cabal file to be compatible with the selected snapshot
  • Add extra-deps to your stack.yaml file to fix compatibility problems

Remember that running stack build will give you information on why your build cannot occur, which should help guide you through the steps necessary for the second and third option above. Also, note that those options can be mixed-and-matched, e.g. you may decide to relax some version bounds in your .cabal file, while also adding some extra-deps.

Explicit breakage

As mentioned above, updating your package indices will not cause stack to invalidate any existing package databases. That's because stack is always explicit about build plans, via:

  1. the selected snapshot
  2. the extra-deps
  3. local packages

The only way to change a plan for packages to be installed is by modifying one of the above. This means that breakage of a set of installed packages is an explicit and contained activity. Specifically, you get the following guarantees:

  • Since snapshots are immutable, the snapshot package database will not be invalidated by any action. If you change the snapshot you're using, however, you may need to build those packages from scratch.
  • If you modify your extra-deps, stack may need to unregister and reinstall them.
  • Any changes to your local packages trigger a rebuild of that package and its dependencies.