Stages

Describes the Staging & Deployment system within Form.io

Overview

Stages provide a way to isolate your Project forms and resources to enable Form Management between different environments. Technically a stage is the same thing as a project, but what distinguishes it from a normal project is that it implements certain CORS restrictions that ensure they are only utilized for Form Management workflows and not for live applications.

Every new project created within Form.io comes with a default stage called Live. This stage is typically the main production project for your application. This is also where your production data will reside.

The amount of stages that you can create is controlled by the License that has been granted to your organization. By default, any new Enterprise project will come with a total of 7 stages. This ensures that you can use your stages to accomplish a "standard" configuration of projects and stages.

Stage Configuration

In most Enterprise deployments, you will want to use stages to achieve a "standard" workflow of both authoring forms as well as testing those forms in different environments. In order to achieve this, stages can be utilized to create a typical configuration that suits the needs of many organizations. The typical configuration for stages consists of the following stages.

Create multiple QA or Dev stages if it's necessary to split form building or testing between different teams.

For each one of these stages, they may (or may not) be connected to their own Environment. The difference between an Environment and a Stage is that an Environment is the backend platform that contains the database and servers that host the Form.io docker containers, whereas a Stage is a project that resides in an Environment used to manage forms and resources in a form publishing workflow. Here is a diagram that illustrates the "standard" configuration.

Isolated Environments

Many people see the diagram above and believe that it will not operate within their "isolated" environment structures. Typically most of the Prod, Non-Prod and Dev environments are completely fire-walled from each other, and the above diagram gives the impression that there is "cross-talk" between these environments. However, this is absolutely not the case. One of the most important technical concepts that Form.io employs is an API-first methodology where the portal interfaces are simple "serverless" applications that use ONLY the APIs for each environment to manage them. This allows a portal (which is loaded within the end user's browser) to serve as an elegant bridge between multiple isolated environments. The following diagram illustrates how our Portal operates when communicating with different environments.

This unique feature enables the Portal application to serve as a management bridge between multiple isolated environments. In order to communicate to each of these different environments a special JWT token is created that provides a way for the portal to establish authenticated access to these environments. Each different environment will define their own PORTAL_SECRET environment variable, which will then be used to validate a special x-remote-token from the portal which will then grant it access (based on the team configurations) to that environment.

Stage Management

The first thing to do once you have a new Project will be to create some additional stages that will map to the different environments. As mentioned above, there will be two stages that will be created by default when you create a new project, Live and Development. By default, these two stages will point to the same environment, but it is typical to isolate the Development environment which can be done at a later time. To start, we will create two additional stages, as described above.

Creating Stages

To create our DEV stage, we will click on the Stages dropdown in the top-left corner of your project page, and then click Create Stage

This will then bring up an interface where you can create your new stage called Dev.

When you are done, click Add Stage button to create the new stage.

If you see a message that says "Can't create more stages", then this means that your license does not permit the creation of more stages. You will need to reach out to support to inquire about purchasing more stages.

Now that you have a new stage, you can navigate to any stage at any time by selecting the Stage dropdown, and then select the stage you wish to navigate to. This process will switch the Portal to now communicate the API endpoints for that stage for all page navigation within that stage.

Stage Remote Connection

Once you have created your stages, the next process is to connect each stage to their Remote Environments. Before you do this step, it is important to ensure that your remote environments have been deployed and are ready to go. Please see the Form.io Deployment Guide for more information on deploying the Form.io software into your own environment. If you wish to perform a Remote connection, you will need to ensure that you establish a PORTAL_SECRET as an environment variable for each of the environments that you create. The value of this should be different between the different environments and should be kept secret.

docker run -d \
  -e "LICENSE_KEY=YOURLICENSE" \
  -e "MONGO=mongodb://mongo:27017/formio" \
  -e "PORTAL_SECRET=CHANGEME" \
  -e "DB_SECRET=CHANGEME" \
  -e "JWT_SECRET=CHANGEME" \
  --restart unless-stopped \
  --network formio \
  --name formio-server \
  -p 3000:80 \
  formio/formio-enterprise

Please see the Form.io Deployment Guide for more information on deploying the Form.io software into your own environment.

Once you have your remote environment ready to go, then the first step is to first navigate to the stage you wish to connect, and then go to the Staging section of that stage, and then click on Connect Environment.

This will display a form that should be provided to make a connection to your remote environment. These configurations are as follows.

As soon as you provide the necessary values, you will then click the Continue button, where you will then see the next page for the Remote connection as follows.

This select dropdown allows you to either connect to an existing stage that exists within this environment or to create a New Stage. If this is a new deployment, then the only option will be to create a New Stage, and so this should be selected for new deployments. It is common, however, that you may wish to "reconnect" to an existing environment, where you would then just select that stage in the dropdown. After you have done this, your stage will now be "connected" to the backend server and all changes made to forms and resources within that stage will affect that remote project.

Once a Stage has been connected, the Portal will point all API traffic for that stage UI to that remote environment. This enables you to use the portal as an external interface on top of any stage deployment.

Disconnecting a Stage

Once a remote environment has been connected to a stage, you will be given the option to "Disconnect" from that stage as follows.

This is a very simple and non-intrusive process where you are simply disconnecting the Portal from the Remote environment it is connected to. This does not turn off, or in any way affect your remote project. There is a common misconception that Disconnecting your remote project will affect the run-time performance of an application pointing to the APIs of this remote project, but this is not the case. The ONLY thing this does is disconnect the Developer Portal from this environment so that any changes made to the forms and resources will no longer affect that remote environment.

Disconnecting from a remote stage will NOT affect the run-time performance of the remote stage. It is perfectly safe to Disconnect a portal interface from a Live Environment.

Reconnecting a Stage

After a stage has been disconnected from an environment, it is a very simple process to reconnect a stage back to an environment. To do this, simply follow the instructions above to Connect a stage to an environment, but on the project selection dropdown, simply select your existing project on that environment, and then click Connect Stage button.

Once you do this, the stage will reconnect to the environment and all APIs for that stage will now communicate the project within that environment.

Stage Versions

The main feature of the Form.io form management workflow is the ability to create versions of your stage so that it can be deployed to other environments. This process creates a JSON representation of the project and then allows you to provide a version (usually using Semantic versioning) of that stage so that it can then be deployed to other environments. A project JSON that is versioned contains the following entities within a project.

forms - The forms of that project
resources - The resources of that project.
roles - The roles for that project.
actions - The form and resource actions.

Project settings and Submission data are NOT included in the stage versioning. In a typical form management workflow, all changes to forms will take place within a single stage. This allows a single point of origin for any of the versioning that will take place. Because of this, we recommend that a dedicated stage, usually called Authoring, be used to build and modify existing forms.

Creating Stage Versions

In order to create a new Stage Version, you will first navigate to the stage you wish to base the "version" against. Whatever stage is in view will be the stage that the version is based upon so it is important to first select the correct stage in the dropdown. Then you will click on the Staging section for that stage.

Once you are in the Staging section, you can create a new stage version by first clicking on Manage Versions and then clicking on Create button.

On the next screen, you can now add the version you would like, add a description (optional) and then click Create Version button once you are ready to create a new version.

We highly recommend that you use Semantic Versioning when creating stage versions.

Now that you have created a new version, you can now deploy this to another stage as follows.

Stage Deployments

Now that we have some versions created from our Authoring environment, we can now perform a deployment into a different stage. This process involves the Portal application, which will "pull" the version of the tag into the browser, and then from there, it will send that version to the destination stage. This allows a deployment to occur between environments that do not have access to one another. See Isolated Environment section to illustrate how this process operates.

The API's that are involved during the deploy process are the following.

In order to deploy a version into a new stage, you must first Select the stage you wish to receive the new version, and then click on the Staging section of that stage.

Once you are in this section, you will then click on Manage Versions, and then select the version you wish to deploy to this stage. Then click the Deploy button.

This will then provide a confirmation as follows. When you are ready to perform the deployment, click on the Deploy version to {STAGE} button where {STAGE} is the name of your stage.

Once this is completed, the stage will then show the version it is based against as follows.

This is how you can be assured that the stage is on the correct version as it is pushed into the higher environments. This process can now be repeated for any stage within your project so that you can easily migrate forms and resources from one stage to another.

Per-Stage Form Configurations

When working with stages, it is very common that there may be some form configurations that need to be dependent upon the stage that form is within. A good example of this is if you have external APIs used within your forms and need this API to change depending on whether it is in the Prod vs. Dev environments. The Form.io platform handles this use case through the Public Configuration section for each stage.

Given the example above, let's suppose you need to alter an API endpoint per stage within your forms. You can first navigate to the Dev Stage and then within the Public Configuration, you will add a configuration called apiEndpoint as follows.

Also, make sure you have Add Public Configuration to Forms schema checked.

Now that you have this saved, you can then go to any form you are building, and then use the following token where you want to use this URL.

{{ form.config.apiEndpoint }}

For example, you can add the following token to the Select dropdown URL as follows.

Now that you have done this, you can deploy form changes between stages while not affecting per-stage form configurations.

Import and Exporting

Another great tool that can be used to save and push changes to other environments is using the Import / Export system. This system allows you to save any project or project partials as a JSON configuration which can then be imported into any other project and apply those forms and resources to those other projects. These project JSON configurations are called Templates and are very useful for migrating forms, resources, actions, and roles to other stages or used as a template when creating new projects and stages.

Exporting a Project Template

The first step to start this process is to pick a stage you wish to export from, and then from there, you will click on Staging and then Export Template. This interface provides a way to either export the entire project as a JSON file, or to select specific forms, resources, actions, and roles to export.

We can see all the available ways we can create an export by unchecking the Include All checkbox which will reveal the following interface.

This interface allows you to select all, or deselect all (which is useful if you wish to only include a few forms in your template), and then you can pick the forms and resources you wish to include in the template. Doing this allows you to create a Partial Template which can be imported into any project where it will ONLY modify the forms and resources that are defined with the same names as the exported template.

The other configuration is the Exclude Access Settings. This configuration provides the ability to not include any access configurations when exporting the template. This is useful if you wish to import the template into a project where you do NOT want to modify any access configurations (such as making forms public). If you uncheck this checkbox, then your template will not modify any access settings when that template is imported into another project.

When you are ready to export your project, you can then click the Export Template button. This will then download the template to your local computer.

Project Template Structure

If you were to now open up the template JSON, you would see the following format.

{
    "title": "Sandbox",
    "version": "0.2.0",
    "description": "A sandbox application",
    "name": "sandbox",
    "roles": {
        "administrator": {
            "title": "Administrator",
            "description": "A role for Administrative Users.",
            "admin": true,
            "default": false
        },
        ...
    },
    "forms": {
        "userLogin": {
            ...
        },
        "userRegister": {
            ...
        },
        ...
    },
    "resources": {
        "user": {
            ...
        },
        ...
    },
    "actions": {
        "userLogin:login": {
            ...
        }
    }
}

Each one of these sections provides the JSON schema for each element, while at the same time replacing any _id with the machine name equivalent. This allows the template to be imported into any other project and then be able to map the ids correctly.

Importing a Project Template

Now that you have a project template, the next step is to import this project template into another project. The import process takes all of the elements within the template and then applies those JSON configurations to the values of the elements within the destination project.

The Project Import process is an "additive" process. This means that it will not touch any forms, resources, etc that are not within the template being imported. This means that if you have a project that contains 10 different forms, and import a template that ONLY includes only one of those forms, the import will modify that one form and leave the other 10 alone.

The other thing that happens during the import process, is that all "machineName" properties are matched against their corresponding _id's within the destination project. This makes it so that you can export from projects with different resource ids and those connections are established during the import process.

The following things are imported during an import process.

Forms
Resources
Actions
Roles
Permissions (if checked in the export)

The following things are NOT imported during an import process.

Submission Data
Project Settings

If you need to migrate submission data from one environment to another, then you will need to use the CLI Migrate Command to perform this action.

There are two ways to use a Project template. During project creation, and importing into an existing project.

Using a template during project creation

Every time a project is created, you will see an option to provide a template under Advanced Options.

You can use this to connect the Export file downloaded from the previous section to now serve as the "template" for any new project created. Once you do this, the project that is created will now contain all of the resources, forms, actions, and roles from the template as soon as it is created. This can also be provided during the POST request of the create project as follows.

Importing a template into an existing project

The next method of using templates is to import them into existing projects. As explained above, this is an additive process and any forms that are not within the template will not be touched and will be left as-is. This allows you to use templates as a method for "partial" updates where you may only wish to modify a few forms within your project. To import a template into an existing project, you need to navigate to the Staging section and then click on Import Template.

Here you will click on the button that says Choose File and then you will select your project template file. After this file has been selected, you will then click the Import Project Template to Live button to import that into this project. Once this is done, you will then see that any forms that were included within the template would have modified any existing forms and resources within the project it is being imported into.