Maintenance and Migration

Upgrading Deployments

Once deployed into the customer environment, the Form.io Platform needs to be upgraded to the latest versions of the software at regular intervals. This does not happen automatically and requires a manual procedure. Please become familiar with the process.

Many cloud providers, such as AWS and Azure, offer their own methods for upgrading container deployments. Refer to the cloud provider's documentation for the container upgrade procedure. This section will describe how to upgrade a local deployment of the Form.io Platform.

Please note before upgrading that the Form.io Platform is a "stateless" system. Each of the containers provided by Form.io does not maintain any files, sessions, or any other data that would be lost between upgrades. It is a best practice that all non-development environments contain multiple container instances behind a load balancer. When the Form.io Platform is deployed in multiple instances behind a load balancer, each deployment is an ephemeral system that can be removed without any effect on the stability of the overall deployment. This allows updates to be performed in a "rolling" fashion where each container is removed from the load-balancer, updated, and then added back into the load balancer once the update is complete.

Major Version Upgrades

When performing an upgrade, please be aware of the distinction between "major" and "minor" upgrades.

A major version upgrade is anytime the version number where the leftmost, "whole" number in the version number has been incremented. For example, updating a 6.10.0 server to version 7.1.0 is considered a major upgrade. Upgrading from 6.10.0 to 6.11.0 is considered a minor version upgrade. These semantic versions are positioned in a way to indicate the "risk" associated with the upgrade and should be taken into consideration. Here are the numbers and what they mean.

• 6.10.2 - The 2 here is considered the "patch" version. Increments of this version are considered patch releases and are the least risky versions. They only include minor bug fixes against the current minor release.

• 6.10.2 - The 10 here is considered the "minor" version. Increments of this version are considered minor releases, which include new minor versions of the renderer which could introduce new features, etc.

• 6.10.2 - The 6 here is considered the "major" version. Increments of this number are considered major releases. They only occur rarely (usually once per year) and include major refactoring and improvements to the platform. There may be some reverse compatibility breaks in these versions so migrations to new major versions must be done with great care.

Adherence to Reverse Compatibility

For every release issued, maintaining reverse compatibility is the highest priority. It is VERY rare that any release will break reverse compatibility, especially within the Form JSON schema that is used to render forms. Because of this, it is generally safe to upgrade the software without concern that forms will "break". If any refactoring does occur, they will usually be paired with an update hook that will automatically update the database to the correct Schema version so that the behavior of the server is maintained. Please see Resolving Update Failures for more explanation on the updated schema.

Database Backup

Before upgrading a server, it is important to create a database backup of the deployment so that it can be restored if any data is lost. While this is never expected or anticipated, it is a best practice to minimize downtime in the event of any errors or interruptions during the upgrade process.

The Form.io Platform only depends on MongoDB for any "state" that is stored within the server, so as long as it is backed up, the upgrade can continue. There are two ways to perform a backup:

• mongodump - This should be used when keeping the same MongoDB database between upgrades. This maintains all indexes and internal metadata about the database.

• mongoexport - This should be used when switching databases during the export process. This is more of a JSON export that will store all records within JSON files that can be re-imported into a new database. This is useful for importing the database into a new database where new indexes will be created.

Please refer MongoDB documentation for both of these options. For most upgrade processes, mongodump is preferred. If situations arise in which it cannot be used, then a mongoexport could be used as a backup plan. In either case, it does not hurt to perform both operations before an upgrade process.

Upgrading Enterprise Server

Once the database has been backed up, update the Form.io API server.

To perform an update with a Docker container system, stop the currently running container, remove it, and then re-register the new version utilizing all of the same environment variables that were used when originally deploying. The following commands illustrate how this can be done on a per-instance basis to perform a manual upgrade within a single environment.

If the values that were used when originally deploying the container were not saved, use the following command to determine the values of these environment variables:

docker inspect formio-server

Once the values of these environment variables are recorded, upgrading is easily achieved with the following command:

docker pull formio/formio-enterprise && \
docker rm formio-server-old || true && \
docker stop formio-server && \
docker rename formio-server formio-server-old && \
docker run -d \
  -e "LICENSE_KEY=YOURLICENSE" \
  -e "MONGO=mongodb://mongo:27017/formio" \
  -e "PORTAL_ENABLED=true" \
  -e "[email protected]" \
  -e "ADMIN_PASS=CHANGEME" \
  -e "DB_SECRET=CHANGEME" \
  -e "JWT_SECRET=CHANGEME" \
  -e "PDF_SERVER=http://pdf-server:4005" \
  --restart unless-stopped \
  --network formio \
  --link pdf-server:pdf-server \
  --name formio-server \
  -p 3000:80 \
  formio/formio-enterprise

After running this command, inspect the logs by typing the following:

docker logs formio-server

For updates that require a database update, the output should appear similar the the following:

Ensure that the updates complete as expected. If any update fails, proceed with the following section to remediate the issue.

Resolving Update Failures

If any of the updates fail to execute, a series of steps that can be taken to help resolve the problem. To understand how to resolve the update failures, it is important to understand how the DB schema works and how to get the server operational once again.

When an update is started, the server first compares the code schema version with the version present in the database. The code schema version can be found by looking at the package.json file within the deployment codebase and noting the "schema" property of that file. This indicates the "DB schema" of the codebase. Compare this value with the value that is found within the database by running the following command:

db.schema.find({key:'formio'}).pretty();

Running this command within the database connection should return a result similar to the following:

> db.schema.find({key:'formio'}).pretty();
{
	"_id" : ObjectId("55cd5c1f2c4aaf01001fe799"),
	"key" : "formio",
	"isLocked" : false,
	"version" : "3.3.9"
}
> 

This indicates that the database is currently on the 3.3.9 version and is not locked. If the "code" schema version were a larger version number than this, this would indicate that an update needs to be run.

If an update fails, the "isLocked" property is set to a timestamp, and the "version" is set to the last successful update.

In the event of a problem, first reset the "isLocked" flag to "null", to retry the update by doing the following:

db.schema.update({key:'formio'},{$set:{isLocked:null}})

After this, retry the update by restarting our docker container:

docker restart formio-server

Once it has restarted, inspect the logs to see if it has moved past the "stuck" update. If it has not, then bump the version of the update by one patch version and then retry as follows. The following example command would be used if th4 server was stuck on version 3.3.7:

db.schema.update({key:'formio'},{$set:{version:'3.3.8',isLocked:null}})

Then restart the server as follows:

docker restart formio-server

In most cases, this will not have any ill effects. However, please reach out to Form.io Support so that we can provide you with the "manual" update script you had to skip so that are able to ensure all updates are applied cleanly to your deployment.

Upgrading PDF Server

Upgrading a PDF Server can be done by using the following command.

docker pull formio/pdf-server && \
docker rm pdf-server-old || true && \
docker stop pdf-server && \
docker rename pdf-server pdf-server-old && \
docker run -itd \
  -e "LICENSE_KEY=YOURLICENSE" \
  -e "MONGO=mongodb://mongo:27017/formio" \
  -e "FORMIO_S3_SERVER=minio" \
  -e "FORMIO_S3_PORT=9000" \
  -e "FORMIO_S3_BUCKET=formio" \
  -e "FORMIO_S3_KEY=CHANGEME" \
  -e "FORMIO_S3_SECRET=CHANGEME" \
  --network formio \
  --link formio-mongo:mongo \
  --link formio-minio:minio \
  --restart unless-stopped \
  --name pdf-server \
  -p 4005:4005 \
  formio/pdf-server;

Follow the same process as described in the Upgrading Enterprise Server section to ensure that the update was processed cleanly. Follow the same steps provided in the Resolving Update Failures section to ensure that any errors in the deployment are fixed.

Database Transfer Between Servers

The following steps describe how to clone a database to different servers. This document refers to the existing database as the "source" database/container and the database on the new server as the "destination."

1. In the destination Mongo database, determine the "mount" folder by running the following command:

docker inspect <MONGO_CONTAINER_NAME>

1. Find the "mounts" for the database. They should look like the following:

{
  "Type": "bind",
  "Source": "/Users/travistidwell/data/db",
  "Destination": "/data/db",
  "Mode": "",
  "RW": true,
  "Propagation": "rprivate"
}

1. Note the "Source" and "Destination" folders. Record them for future use.

2. SSH into the source MongoDB container using the following command:

docker exec -it <MONGO_CONTAINER_NAME> /bin/bash

1. Perform a mongodump of the database, and place it in the "Destination" folder noted above. The sample command below assumes the database is named form.io:

mongodump --archive=/data/db/backup.archive --db=formio

1. Exit the container (type "exit", then press enter) and then verify that the "archive" in the "Source" folder looks like the following:

1. Use the "scp" command to move this file from the Source machine to the Destination machine as follows:

scp source:~/data/db/backup.archive destination:~/data/db

1. SSH into the destination machine and ensure the Form.io Docker Containers are stopped.

2. Ensure that the Environment variables for these containers are the same as the source container environment variables.

3. On the destination machine, find the "Source" and "Destination" folders of the running Docker MongoDB container, similar to step #4.

4. Move the archive file into the "Source" folder of the destination Mongo container folder like so:

cp ~/data/db/backup.archive ~/opt/mongodb

1. Using Bash to connect to the destination container:

docker exec -it <CONTAINER_NAME> /bin/bash

1. Finally, restore the backup to the destination database.

mongorestore --archive=<DESTINATION_FOLDER>/backup.archive --db=formio

Upgrading MongoDB

When upgrading the version of MongoDB deployed to a container by pulling the latest version, note the minimum supported version required by the featureCompatibilityVersion parameter. If an attempt to deploy MongoDB returns a Feature Compatibility Version error, it is usually because the currently installed version is incompatible with the latest's specified featureCompatibilityVersion. In these cases, it may be necessary to upgrade in stages to meet the next supported version until reaching the desired version.

Migrating Projects

The following documentation describes how to perform different kinds of migrations within the Form.io platform. This documentation will refer to both Source and Destination projects, where the source is the project where to migrate FROM while the Destination is the project to migrate into.

Migrating from one project to another can easily be achieved with a combination of the Form.io Staging system as well as the Form.io CLI tool. The Staging system is used to migrate the Forms, Resources, and all Project level configurations. It is used to migrate everything EXCEPT Submissions and Settings.

Migrating Forms, Resources, Actions, and Roles

To start, export the project of the Source Project using the staging interface. This is found by clicking on the Source Project in the Developer portal, and then clicking on Settings > Stage Versions.

For a complete migration, export the whole template. Click on the Export Template button. This will download a JSON file onto the local machine, to migrate to a new project.

The export JSON file on the local machine can be used to create a new project, or update an existing project:

To create a new Destination project:

1. Click on the Create Project button from the Developer portal homepage.

2. Click Upload A Project Template, and select the JSON file.

To update an existing Destination project:

1. Open the Destination project from the Developer Portal.

2. Navigate to Settings > Staging.

3. Click on the Import Template section.

4. Click Choose File and then select the template exported from the Source project. Then, click the Import Project Template button to complete the import.

This will clone the Forms, Resources, Roles, and Actions into this Destination Project.

Migrating submissions is accomplished using the Form.io CLI tool.

Migrating Submissions

To migrate submissions, use the Form.io CLI tool, which can be found on the Form.io Github repository https://github.com/formio/formio-cli. Download this tool to the local machine using the following command:

npm install -g formio-cli

With the CLI tool now on the local machine, make sure that an API key has been configured for both the Source and Destination Projects. To do so, from the Developer Portal, go to the Settings > API Settings section and create an API key.

Once both the Source and Destination project have API keys, issue the following command from the local computer:

formio migrate https://SOURCE_PROJECT.form.io \
    project \
    https://forms.mydomain.com/DESTINATION_PROJECT \
    --src-key [SOURCE_API_KEY] \
    --dst-key [DESTINATION_API_KEY]This command will migrate submission data from a hosted project to a deployed project within your own environment

If you are migrating submissions from open source then https://SOURCE_PROJECT.form.io would instead be the domain of your open source server e.g. localhost:3001

This will copy all submissions from the Source project into the Destination project.

Migrating Project Settings (optional)

To migrate the project settings, use the Project API. The API calls that will be used and the general process are as follows:

1. GET source project using Project GET API.

2. GET destination project using Project GET API.

3. Copy the Destination project API Keys. Record them for later use.

4. Copy the Source project settings, and set them as the Destination settings.

5. Copy the Destination project keys back into the Destination settings.

6. Perform a PUT Request to save the settings into the Destination settings.

The following shell command performs all of the steps above, and can be used to migrate project settings from one project into another while maintaining the same API Keys for future migrations:

SOURCE_URL='https://SOURCE_PROJECT.form.io' && \
SOURCE_APIKEY='SOURCE_API_KEY' && \
DEST_URL='https://DESTINATION_PROJECT.form.io' && \
DEST_APIKEY='DESTINATION_API_KEY' && \
SOURCE_PROJECT=$(curl --location \
    --request GET $SOURCE_URL \
    --header "x-token: $SOURCE_APIKEY"\
) && \
DEST_PROJECT=$(curl --location \
    --request GET $DEST_URL \
    --header "x-token: $DEST_APIKEY"\
) && \
UPDATE=$(node -e "\
    const dst=$DEST_PROJECT; \
    const src=$SOURCE_PROJECT; \
    const keys=dst.settings.keys; \
    dst.settings=src.settings; \
    dst.settings.keys = keys;
    console.log(JSON.stringify(dst))"\
) && \
curl --location \
    --request PUT $DEST_URL \
    --header "x-token: $DEST_APIKEY" \
    --header 'Content-Type: application/json' \
    --data-raw "$UPDATE"

Enabling the Developer Portal on Existing environment

The Form.io platform can use the hosted Developer Portal (https://portal.form.io) to connect to remote deployed environments through the On-Premise Environments section within the project. This is useful to allow a remote environment to serve as an API-only interface for an application, while at the same time, managing that deployment through a hosted portal interface. In some cases, though, it may be neccessary to enable the Developer Portal directly within a remote environment. To do so, configured the following environment variables:

PORTAL_ENABLED=true
[email protected]
ADMIN_PASS=CHANGEME
ADMIN_KEY=CHANGEME123!

Once this is done, restart the server so that the initialization process will install the Portal Base project as well as create the initial admin account for this project.

After the Developer Portal has been enabled, login to the Developer Portal by navigating to the root URL of the deployed API. Once logged in, any existing projects will not be visible. Do not worry, they are still available. The "owner" of these projects must be established in order to view them while logged in as the root user account. To do this, first connect to the MongoDB database, and then run the following command on that environment:

var account = db.submissions.find({'data.email': '[email protected]'}).next();
db.projects.updateMany({}, {$set:{owner:account._id}});

This command will set all of the projects to have the "owner" of the account that was created within the Portal Base project. Now the root user will see all of the existing projects within that environment, so that they can be managed accordingly.

Community to Enterprise Migrations

To deploy all of the forms and resources from the Community Edition into the Form.io Hosted (Saas) Platform at https://form.io or your own Self-Hosted platform, use the Form.io CLI command line tool:

npm install -g formio-cli

Once this tool is installed, you follow these steps general steps:

1. Create a new project within Form.io.

2. Create an API Key within this project by going to the Project Settings > Stage Settings > API Settings.

3. Execute the following command to deploy the local project into Hosted Form.io:

formio deploy http://localhost:3001 https://{PROJECTNAME}.form.io --src-key {srcAPIKEY} --dst-key {dstAPIKEY}

Next, all submissions can be migrated by following the Migrating Submissions documentation.

API & PDF Server Migrations

Be sure to review the Form.io Update Guide before migrating major versions of the API and PDF servers. Before commencing the migration process, create a complete backup of any existing environments.

Please visit the Enterprise Change Log for a full list of changes and fixes.

The following methods are listed below as either available or "no-op" in 9.x. This applies only to the server-side evaluation context.

Other updates in 9.0.0

API Server 7.x to 8.x

Formio.js 5x Upgrade

The 5x renderer includes numerous behind-the-scenes enhancements that significantly boost performance and loading times for forms embedded within your application. It now uses the updated Bootstrap 5 library, providing better styling and theming options for your forms. Additionally, the validation engine has been updated for greater efficiency and improved performance while maintaining backward compatibility

Overall, these changes lay the groundwork for exciting updates and new features in the future. The following information details all of the new features, updates, and breaking changes included with the new formio.js 5x renderer.

New Features

The first thing that you will notice with the 5.x renderer is the package name has changed from formiojs to @formio/js. This is to be more consistent with the naming conventions that we are starting to use with all of our externally facing libraries. This change is as follows:

New Package Name

Before 5.x

npm install --save formiojs

After 5.x

npm install --save @formio/js

Before 5.x

import { Formio } from 'formiojs';

After 5.x

import { Formio } from '@formio/js';

All other libraries and modules within this package remain the same. The following still works as expected.

Bootstrap 5 Support

The 5.x renderer now supports and defaults to Bootstrap 5 as the base template. We have also moved the templates from the renderer to their own repository which can be found @ https://github.com/formio/bootstrap. If you wish to use the 5.x renderer with Bootstrap 4, that is also supported but the following code will need to be included within your application.

import { Formio } from '@formio/js';
import bootstrap4 from '@formio/bootstrap/bootstrap4';
Formio.use(bootstrap4);

It should also be noted, that the Bootstrap templates are no longer part of this library, but rather imported in from the external dependency and bundled along with this library. Because of this, if you wish to make a change to the default Bootstrap template, you will need to make that pull request at the Github repo @ https://github.com/formio/bootstrap

Please see the Breaking Changes section below for more information.

New Validation Engine

The 5.x renderer incorporates our new Core validation engine found @ https://github.com/formio/core/tree/master/src/process.

The new Validation Engine simplifies how data is handled and aims to replace the old method of validation with a more efficient system that checks all the data at once, instead of having each part of the form validate itself These concepts replace the renderer's Validator validations in the existing checkData and checkValidity pipelines with top-down calls to processSync, a synchronous version of the core library's process method. Instead of relying on components to validate themselves, the new validation engine is performed in a single pass using the form JSON and its submission data.

These updates will also improve the Iso-morphic behavior for the renderer validation and significantly improve memory and processor consumption for server-side form validations.

Please see the Breaking Changes section below for more information.

Core SDK

With the 5.x renderer, the Formio SDK is now part of our Core library found @ https://github.com/formio/core/tree/master/src/sdk. It is now imported directly into the @formio/js library and is re-exported to support reverse compatibility. Because of this, there should not be any code upgrade involved.

For example, the following code still works.

import { Formio } from '@formio/js/sdk';
const formio = new Formio('https://examples.form.io');
formio.loadForm().then((form) => {
  console.log(form);
});

Core Utilities

Now, instead of the Form Utilities being defined within the renderer code, they are now maintained within our Core library found @ https://github.com/formio/core/tree/master/src/utils. However, these libraries are re-exported from the renderer to maintain backward compatibility.

For example, the following code still works.

import { eachComponent } from '@formio/js/utils';
import { Formio } from '@formio/js/sdk';
const formio = new Formio('https://examples.form.io/example');
formio.loadForm().then((form) => {
  eachComponent(form.components, (component) => {
    // Called for every component in this form.
    console.log(component);
  });
});

Improved type definitions with JSDoc

5.x officially deprecates our previous "manual" type definitions found in the "types" folder. These type definitions are now automatically generated using a concerted effort to introduce JSDocs into all of our javascript classes and interfaces. The types can be imported using the following:

import { Component } from '@formio/js/types';

Improved renderer build sizes

In the 5.x version of the renderer/builder, significant efforts have been made to reduce the build size of the renderer. While the full renderer is still "large" and over the 1mb goal, we still have been able to trim a lot of size from the renderer/builder as follows.

- formio.form.min.js ~ 1.2mb => ~30% size reduction
- formio.form.min.js ~ 1.4mb => ~30% size reduction
- formio.min.js (SDK) ~ 235k => ~50% size reduction

New lazy-loading Embedding method.

One of the more exciting new additions to the 5.x renderer is the new lazy-loading process for adding the renderer to your application. There is a new file that is included with the 5.x renderer called "formio.embed.js". This file is tiny coming in at ~10kb. What this file does, however, is make it so that you can bundle a lazy-loading renderer within your application without increasing the build sizes of your application.

To use the new lazy-loading features, you will need to change your imports from the following to the new embed source as the following illustrates.

Before 5.x

import { Formio } from 'formiojs';

After 5.x using Embed code.

import { Formio } from '@formio/js/embed';

Whenever you use the Formio.createForm or Formio.builder methods, this functionality will add a simple "loader" where the form is rendered, while the full code needed for rendering is being loaded. All of the code that you are used to still functions as you would expect. For example, the following code still works.

import { Formio } from '@formio/js/embed';

// Before, this would require a +1mb renderer to be bundled with your application to embed a form into a page. Now with the "embed" code, this will only incur a 10kb code penalty and show a loader symbol while the full renderer is lazy-loaded into the application. This improves usability so your application can quickly load and the form will show a loader while the user waits for the renderer to download and form to render.
Formio.createForm(document.getElementById('formio'), 'https://examples.form.io/example').then(function(form) {
  form.submission = {
    data: {
      firstName: 'Joe',
      lastName: 'Smith'
    }
  };
});

The following methods are able to be used within this embedded renderer.

• Formio.createForm

• Formio.builder

• Formio.use

• Formio.setBaseUrl

• Formio.setProjectUrl

If you wish to have code that is executed after the library has been lazy loaded, you can use the following code.

import { Formio as FormioEmbed } from '@formio/js/embed';

FormioEmbed.formioReady.then(function(Formio) {
  // This will be the TextField components.
  console.log(Formio.Components.components.textfield);
});

"childComponents" and improved performance with getComponent method.

The 5.x renderer introduces a new "flat-map" of all component instances and their paths with the "childComponents" map. It can be used as follows.

import { Formio } from '@formio/js';

Formio.createForm(document.getElementById('formio'), 'https://examples.form.io/example').then(function(form) {
  // Prints out all child components and their data paths.
  console.log(form.childComponents);
});

This map is also important to dramatically speed up the heavily used getComponent method.

Formio.createForm(document.getElementById('formio'), 'https://examples.form.io/example').then(function(form) {
  // Now returns the component from map lookup (quick search) vs O^n searching for the component.
  const firstName = form.getComponent('firstName');
});

Breaking Changes

• Bootstrap 5 Default Template - With the 5.x version of the renderer, the default template is now Bootstrap 5 and is found @ https://github.com/formio/bootstrap repo.

• Bootstrap Icons - Now, instead of Font Awesome being the default icon set for our renderer, we are using Bootstrap Icons as the default icon set which is compatible with Bootstrap 5. Of course, you can always change out icon sets, but this is now the default.

• Day component - when left empty, the component data now returns an empty string instead of data '00/00/0000'

• This version implements a new validation system. Within this, there are some changes that you should be aware of.

  • In version 4.x, each component would have an error associated with that component as component.error. This has changed to component.errors which is an array of errors that are associated with the component. For example, if your custom logic references the component.error.messages array, it will need to be changed as follows.

    4.x Renderer

    Copy

    const textField = form.getComponent('textField');
    console.log(textField.error.messages[0]);  // Legacy way of getting the text field error for the first error.

    5.x Renderer

    Copy

    const textField = form.getComponent('textField');
    console.log(textField.errors[0].message);  // 5.x way of getting the error message for the first error.

• EditGrid validateRows method now returns an array of errors instead of a boolean "true" or "false". If you wish to do the same thing as before 5.x, then you can use the following code.

  Copy

  const isValid = form.getComponent('editgrid').validateRow().length === 0;

• In the 5.x renderer, the errors array will always be populated if there are errors in the form. They may not be displayed depending on the "pristine" state of the rendered form, but the error is always populated if there are form errors. This is different in 4.x where the error property would only contain an error if an error is VISIBLE on the form. This means that it is difficult to determine if a form has errors without executing the checkValidity() method with the dirty flag set to "true". You no longer need to do this in the 5.x renderer.

  4.x Renderer

  Copy

  const textField = form.getComponent('requiredField');
  console.log(textField.error);  // This would be null even if there was an error, but it was not visible.

  5.x Renderer

  Copy

  const textField = form.getComponent('requiredField');
  console.log(textField.errors);  // This will be populated with the errors of the textfield even if they are not displayed on the form.

• If you wish to show the "visibleErrors", then there is a new property on each component called "visibleComponents".

  4.x Renderer

  Copy

  const textField = form.getComponent('requiredField');
  console.log(textField.errors);

  5.x Renderer

  Copy

  const textField = form.getComponent('textField');
  console.log(textField.visibleErrors); // This is the equivalent of the 4.x renderer "errors" array.

• With the 5.x renderer, all templates are now stored within a separate repo, and are included as a dependency for this renderer.

Compatible Versions with the 5x Renderer

Minimum version requirements for using the 5x renderer with various Form.io libraries and servers

Change Log

For a full list of changes found within the 5x renderer, please visit the change log below:

5x Renderer Change Log