Edeliver

Downloads
14732
This Week
1365
Releases
13
Latest Release
1.2.5
Stars
415
Watchers
20
Open Issues
15
Forks
31
Last Commit
13 May 16
From Github Readme

edeliver Hex.pm Hex.pm

edeliver logo

Deployment for Elixir and Erlang

edeliver is based on deliver and provides a bash script to build and deploy Elixir and Erlang applications and perform hot-code upgrades.

The erlang releases are built on a remote host that have a similar configuration to the deployment target systems and can then be deployed to several production systems.

This is necessary because the release contains the full erts (erlang runtime system), all dependencies (erlang or elixir applications), elixir runtime, native port drivers and your own erlang/elixir application(s) in a standalone embedded node.

Examples:

Build an erlang/elixir release and deploy it on your production hosts:

mix edeliver build release --branch=feature
mix edeliver deploy release to production
mix edeliver start production

Build a live upgrade from v1.0 to v2.0 for an erlang/elixir release and deploy it to your production hosts:

# build upgrade from tag v1.0 to v2.0

mix edeliver build upgrade --from=v1.0 --to=v2.0
mix edeliver deploy upgrade to production

# or if you have the old release in your release store,
# you can build the upgrade with that old release instead of the old git revision/tag

mix edeliver build upgrade --with=v1.0 --to=v2.0
mix edeliver deploy upgrade to production

# run ecto migrations manually:
mix edeliver migrate production
# or automatically during upgrade when upgrade is built with --run-migrations

The deployed upgrade will be available immediately, without restarting your application. If the generated upgrade instructions (relup) for the hot code upgrade are not sufficient, you can modify these files before installing the upgrade by using the edeliver edit relup command.

To execute that steps by a single command and upgrade e.g. all production nodes automatically from their running version to the current version using hot code upgrade without restarting, you can use the upgrade command:

mix edeliver upgrade production

This performs the following steps automatically:

  • Detect current version on all running nodes
  • Validate that all nodes run the same version
  • Build new upgrade from that version to the current version
  • Auto-patch the relup file
  • Deploy (hot code) upgrade while nodes are running
  • Validate that all nodes run the upgraded version
  • Deploy the release to not running nodes

Installation

Because it is based on deliver, is uses only shell scripts and has no further dependencies except the erlang / elixir build system. It can be used with one of these build systems:

  • rebar for pure erlang releases
  • mix in conjunction with exrm for elixir/erlang releases
  • mix in conjunction with relx for elixir/erlang releases

Edeliver tries to autodetect which system to use to compile the sources and build the release:

  • If a ./mix.exs file exists, mix is used fetch the dependencies, compile the sources and exrm is used to generate the releases / upgrades.
  • If a ./relx.config file exists in addition to a ./mix.exs file, mix is used fetch the dependencies, compile the sources and relx is used to generate the releases / upgrades.
  • Otherwise rebar is used to fetch the dependencies, compile the sources and generate the releases / upgrades.

This can be overridden by the config variables BUILD_CMD=rebar|mix and RELEASE_CMD=rebar|mix|relx.

If using mix, add it as hex package to you mix.exs config:

defp deps do
    [{:edeliver, ">= 1.2.5"}]
end

And run mix do deps.get, deps.compile. Edeliver is then available as mix task: mix edeliver.

For additional edeliver commands version, migrate and show migrations add edeliver as application for your relase in the mix.exs file:

def application, do: [
    applications: [
      # ...
      :edeliver,
    ],
  ]

When using rebar, edeliver can be added as rebar depencency. Just add it to your rebar.config (and ensure that a ./rebar binary/link is in your project directory:

{deps, [
  % ...
  {edeliver, "1.2.5",
    {git, "git://github.com/boldpoker/edeliver.git", {branch, master}}}
]}.

And link the edeliver binary to the root of your project directory:

./rebar get-deps # when using rebar, or ...
ln -s ./deps/edeliver/bin/edeliver .

Then use the linked binary ./edeliver instead of the mix edeliver tasks from the examples.

Configuration

Create a .deliver directory in your project folder and add the config file:

#!/usr/bin/env bash

APP="your-erlang-app" # name of your release

BUILD_HOST="build-system.acme.org" # host where to build the release
BUILD_USER="build" # local user at build host
BUILD_AT="/tmp/erlang/my-app/builds" # build directory on build host

STAGING_HOSTS="test1.acme.org test2.acme.org" # staging / test hosts separated by space
STAGING_USER="test" # local user at staging hosts
TEST_AT="/test/my-erlang-app" # deploy directory on staging hosts. default is DELIVER_TO

PRODUCTION_HOSTS="deploy1.acme.org deploy2.acme.org" # deploy / production hosts separated by space
PRODUCTION_USER="production" # local user at deploy hosts
DELIVER_TO="/opt/my-erlang-app" # deploy directory on production hosts

It uses ssh and scp to build and deploy the releases. Is is recommended that you use ssh and scp with keys + passphrase only. You can use ssh-add if your don’t want to enter your passphrase every time.

Maybe it is required to configure git on your build host (git user name / email) or to clone the repository initially at the BUILD_AT path. And of course you need to install erlang and elixir on the BUILD_HOST. If you use mix to build the releases, you should install hex on the build hosts before the first build (otherwise mix asks interactively to install it). Run the build command with --verbose if it fails the first time.

To use different configurations on different hosts for your erlang/elixir release, you can configure edeliver to link the vm.args and / or the sys.config files in the release package by setting the LINK_VM_ARGS=/path/to/vm.args and/or LINK_SYS_CONFIG=/path/to/sys.config variables in the edeliver config if you use mix and exrm to build the releases.

There are four kinds of commands: build commands which compile the sources and build the erlang release on the remote build system, deploy commands which deliver the built releases to the remote production systems, node commands that control the nodes (e.g. starting/stopping) and local commands.

Build Commands

The releases must be built on a system that is similar to the target system. E.g. if you want to deploy to a production system based on linux, the release must also be built on a linux system. Furthermore the erlang runtime / OTP version (e.g. OTP 17.5) of the remote build system and the elixir runtime is included into the release built and delivered to all production system. It is not required to install the otp runtime nor elixir on the production systems. For build commands the following configuration variables must be set:

  • APP: the name of your release which should be built
  • BUILD_HOST: the host where to build the release
  • BUILD_USER: the local user at build host
  • BUILD_AT: the directory on build host where to build the release. must exist.

The built release is then copied to your local directory .deliver/releases and can then be delivered to your production servers by using one of the deploy commands.

If compiling and generating the release build was successful, the release is copied from the remote build host to the release store. The default release store is the local .deliver directory but you can configure any destination with the RELEASE_STORE= environment variables, also remote ssh destinations (in your server network) like RELEASE_STORE=user@releases.acme.org:/releases/ or amazon s3 locations like s3://AWS_ACCESS_KEY_ID@AWS_SECRET_ACCESS_KEY:bucket. The release is copied from the remote build host using the RELEASE_DIR= environment variable. If this is not set, the default directory is found by finding the subdirectory that contains the generated RELEASES file and has the $APP name in the path. e.g. if $APP=myApp and the RELEASES file is found at rel/myApp/myApp/releases/RELEASE the rel/myApp/myApp is copied to the release store.

To build releases and upgrades faster, you might adjust the GIT_CLEAN_PATHS variable in your config file e.g. to something like ="_build rel priv/generated" which defaults to .. That value means, that everything from the last build is reset (beam files, release files, deps, generated assets etc.) before the next build is started to ensure that no conflicts with old (e.g. removed or renamed) files might arise. You can also use the command line option --skip-git-clean to skip this step completely and in addition with the --skip-mix-clean option for full incremental builds.

Build Initial Release

mix edeliver build release [--revision=<git-revision>|--tag=<git-tag>] [--branch=<git-branch>]

Builds an initial release that can be deployed to the production hosts. If you want to build a different tag or revision, use the --revision= or the --tag argument. If you want to build a different branch or the tag / revision is in a different branch, use the --branch= argument.

Build an Upgrade Package for Live Updates of Running Nodes

mix edeliver build upgrade --from=<git-tag-or-revision>|--with=<release-version-from-store>
                          [--to=<git-tag-or-revision>] [--branch=<git-branch>]

Builds an upgrade package that can be deployed to production hosts with running nodes without restarting them. To build an upgrade package you need the release or upgrade package (when using exrm) of the running release. If it is available (in the release store), you can build the upgrade to the new version by passing the old version to the --with=<old-version> option. If not, you can build the old release and the live upgrade from it in a single step by using the --from=<git-tag-or-revision> option. If you don’t want to build an upgrade to the current head of the given branch (master is the default), you can use the --to=<git-tag-or-revision> option. If the upgrade package is built, you might want to modify the generated upgrade instructions (relup) as described in the next section or (more advanced) automatically patch the relup file by implementing your own Edeliver.Relup.Modificationbehaviour to automate this step.

Edit the upgrade instructions (relup)

mix edeliver edit relup [--version=<upgrade-version>]

From the auto-generated appup instructions of all included and updated applications, a relup file is generated during the build upgrade command and included in the upgrade package. It contains the upgrade instructions for the new release version. If there are dependencies between modules or applications, it might be necessary to modify this file, e.g. changing the order of the applications or modules that are reloaded. If there are repeating steps to adjust the relup for your application, you can automate this step by implementing your own Edeliver.Relup.Modification behaviour.

Auto-Versioning

edeliver provides a way to automatically increment the current version for the current build and/or to append the git commit count and/or revision as version metadata. Using a different versions for each release is essential especially if you build hot code upgrades, but also makes sense to see wich version is running, e.g. when using mix edeliver version. For more information check the --auto-version= option described e.g in mix edeliver help upgrade or in the wiki.

Build Restrictions

To build upgrades it is required that there is only one release in the release directory (rel) of you project configured in your rebar.config. E.g. if you want to build two different releases project-dir/rel/release_a and project-dir/rel/release_b you need two rebar.config files that refer only to either one of that release directories in the sub_dirs section. You can then pass the config file to use by setting the environment REBAR_CONFIG= at the command line. The reason for that is, that when the upgrade is build with rebar, rebar tries to find the old version in both release directories.

Deploy Commands

mix edeliver deploy release|upgrade [[to] staging|production] [--version=<release-version>] [Options]

Deploy commands deliver the builds (that were created with a build command before) to your staging or prodution hosts and can perform the live code upgrade. The releases or upgrades to deliver are then available in your local directory .deliver/releases. To deploy releases the following configuration variables must be set:

  • APP: the name of your release which should be built
  • PRODUCTION_HOSTS: the production hosts to deploy to, separated by space
  • PRODUCTION_USER: the local users at the production hosts
  • DELIVER_TO: the directory at the production hosts to deploy the release at

  • STAGING_HOSTS: the staging hosts to test the releases at, separated by space
  • STAGING_USER: the local users at the staging hosts
  • TEST_AT: the directory at the staging hosts. if not set, the DELIVER_TO is used as directory

Deploying to staging can be used to test your releases and upgrades before deploying them to the production hosts. If you don’t pass the [to] production argument deploying to staging is the default.

Deploy an Initial / Clean Release

Deploys an initial release at the production hosts. Requires that the build release command was executed before. If there are several releases in the release store, you will be asked which release to deploy or you can pass the version by the --version= argument variable. If the nodes on the remote deploy hosts are up, the running old release is not affected. The new release will be available only after starting or restarting the nodes on the deploy hosts.

Deploy an Upgrade Package for Live Updates at Running Nodes

Deploys an upgrade at the production hosts and upgrades the running nodes to the new version. Requires that the build upgrade command was executed before and that there is already an initial release deployed to the production hosts and that the node is running.

Release archives in your release store that were created by the build release command cannot be used to deploy an upgrade.

This comand requires that your release start script was generate by a recent rebar version that supports the upgrade command in addition to the start|stop|ping|attach commands. Releases generated with mix and exrm always contain the upgrade command.

If using rebar, make sure that the install_upgrade.escript file which was generated by rebar is included in your release. So ensure, that the following line is in your reltool.config:

{overlay, [ ...
       {copy, "files/install_upgrade.escript", "bin/install_upgrade.escript"}
]}.

Maintenance

edeliver has a set of commands to maintain erlang/elixir nodes on staging or production hosts:

mix edeliver ping production # shows which nodes are up and running
    mix edeliver version production # shows the release version running on the nodes 
    mix edeliver show migrations on production # shows pending database migrations
    mix edeliver migrate production # run database migrations
    mix edeliver restart production # or start or stop
    ...

Extended Options

If something goes wrong, retry with the --verbose option. If you want to see (really) everything what happens, try the --debug option.

For detailed information about the edeliver commands and their options, try mix edeliver help <command>.

For advanced usage have a look also at the wiki.

Recommended Project Structure

your-app/                              <- project root dir
  + rebar                              <- rebar binary
  + mix                                <- optional mix binary when compiling with mix
  + relx                               <- optional relx binary if rebar is not used
  + edeliver                           <- edeliver binary linking to deps/deliver/bin/deliver
  + rebar.config                       <- should have "rel/your-app" in the sub_dirs section
  + mix.exs                            <- if present, mix is used for dependencies and compile
  + relx.config                        <- if present, relx is used for releases
  + .deliver                           <- default release store
  |  + releases/*.tar.gz               <- the built releases / upgrade packages
  |  + appup/OldVsn-NewVsn/*.apppup    <- generated appup files
  |  + config                          <- deliver configuration
  + src/                               <- erlang source files
  |  + *.erl
  |  + your-app.app.src
  + lib/                               <- elixir source files
  |  + *.ex
  + priv/
  + deps/
  |  + edeliver/
  + rel/
     + your-app/
         + files/
         |   + your-app                <- binary to start|stop|upgrade your app
         |   + nodetool                <- helper for your-app binary
         |   + install-upgrade.escript <- helper for the upgrade task of your-app binary
         |   + sys.config              <- app configuration for the release build
         |   + vm.args                 <- erlang vm args for the node
         + reltool.config              <- should have the install_upgrade.escript in overlay section

LICENSE

(The MIT license)

Copyright (c) Gerhard Lazu

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.