Promoting applications from test to production

Setting up an application in the test environment

Adding and naming stacks

The purpose of having a test environment is to allow applications to be stood up and tested without disturbing the other applications in the group. I have tried to not add too many additional stacks to the original set, but the author's interests and mine diverge a little bit. Going forward, adding applications to a stack is done by adding a new stack with a number at the end. For example, adding applications to the admin stack is done with a new stack called admin2. An admin3 can be added so that the number of applications in a stack don't become unmanageable.

Incorporating the docker-compose.yml file into the stack

Docker compose is a great tool for managing stacks of individual services. It is a definite improvement from having a shell script with a bunch of docker run commands in it. Further improvement comes from the addition of the dockge and dockhand tools. These two applications are different approaches to managing the stacks. For the sake of discussion, stacks contain containers and containers contain images. An image, while defined in a single line, cans contain a number of images.

A docker compose file will start out looking a lot like this one:

services:
  weaver:
    image: ghcr.io/scryer-media/weaver:latest
    container_name: weaver
    restart: unless-stopped
    environment:
      # The login Weaver creates on its first start. Pick your own password.
      WEAVER_BOOTSTRAP_LOGIN_USERNAME: "admin"
      WEAVER_BOOTSTRAP_LOGIN_PASSWORD: "choose-a-password"
      # Names that may be used to reach Weaver, on top of localhost and IPs.
      WEAVER_HTTP_ALLOWED_HOSTS: "weaver,localhost"
      WEAVER_INTERMEDIATE_DIR: /downloads/intermediate
      WEAVER_COMPLETE_DIR: /downloads/complete
    ports:
      - "9090:9090"
    volumes:
      - weaver-config:/config
      - /path/to/downloads:/downloads

volumes:
  weaver-config:

There are other directives, but this one is typical. The indentation levels are important and it assumed that lines at the same indentation level are linked together. Generally, the order of lines at the same group and level don't matter. The very first line that you see under services defines how the container will be identified on from the outside. The container_name directive is usually have the same name, but not always.

Since the order is a bit of a free for all, I like to regularize the declaration order and there are a few additions that I put in whether I need them or not.

The volumes listed under the volumes directive almost always need to be modified. Proper definition here allows for data used by the application to be persistent. It also makes things easy to find and backup. It is most common for an application two have two volumes defined. The first is to store configuration data. Often some historical data is stored there along with modest databases. The storage pool, called tank, for all of this is 2TB so putting a large set of data there, even if it is only temporary, could create problems.

The morpheus storage pool has tens of TBs of space so large data sets and data collections go there.

Here is the final compose definition ready to test and promote to production. I will explain the changes after can look at it.

  weaver:
    image: ghcr.io/scryer-media/weaver:latest
    container_name: weaver
    restart: unless-stopped
    env_file:
      - ./env
    environment:
      WEAVER_TRUSTED_CIDRS: 192.168.0.0/16
      WEAVER_INTERMEDIATE_DIR: /data/intermediate
      WEAVER_COMPLETE_DIR: /data/downloads
    user: ${PUID}:${PGID}
    ports:
      - 7090:9090
    volumes:
      - /mnt/${APPS_POOL:-tank}/docker/apps/media/weaver:/config
      # use the same host path and container path as Scryer
      - /mnt/${MEDIA_POOL:-morpheus}/media:/data
  

Everything from before is still there, but I have applied an arbitrary order to the directives. This is why the restart definition is now toward the top. A major addition is adding the three lines below restart.

In docker compose, and probably docker below it, has a hierarchy where it looks for environment variables the docker uses itself or to pass through to the application. Docker will process the files defined by the env_file directive and then in the .env file in the directory where the docker-compose.yml file sits and then definitions in the environment section. It is important that the variables be defined before they are used.

Without getting too far into the weeds, all of the common variables are defined in a file that is picked up with the env_file directive. The environment section adds information that is important to this application.

At volumes, it gets a little interesting. The definition is split by a ':' with the left part defining the external location and the right part defining where it will appear in the file layout where the application runs. The first definition is for tank. The ${APPS_POOL} variable points to the tank storage pool while the ${MEDIA_POOL} to the big bucket that holds the bulk of everything. The APPS_POOL is organized by the application and if you examine the directory layout in tank it follows a convention that includes the stack and container in the definition. The MEDIA_POOL really doesn't generally have an application focus. For example, several applications all work with video files.