Audit Logging

Produce audit logs of all user activity on the system. This will generate an auditable trail of user session events and access modification of all entities in the system. Audit logs will output to the standard out for the docker container and can be routed into a log aggregation system if needed.

This feature is included in the Security and Compliance package. Follow the link or contact sales@form.io for more details.

The Audit Log data is presented in the following format:

date EVENT uuid projectId sessionId userId [additional information]

You can turn off audit logging by setting the NOAUDITLOG flag to true in the .env file or in docker secrets.

Audit Logging Events

There are a number of events that are tracked when audit logging is enabled. The following outlines these events that are logged.

Successful/Unsuccessful account login events

When a user is trying to authenticate into the portal, four different types of events can occur based on whether the login was successful, unsuccessful (username), unsuccessful (password), and a log in count. These four events are...

EAUTH_NOUSER (Unsuccessful username login)

The EAUTH_NOUSER event occurs when a user tries to log in with a username/email that does not exist.

Example Log

date
event
uuid
projectId
sessionId
userId
Attempted username/email

2024-04-25T14:50:03.172Z

EAUTH_NOUSER

27759aa2-085b-4719-9193-5bad3fad74a2

659d83632824949f20a882e1

NoSession

NoUser

hello@example.com

EAUTH_PASSWORD (Unsuccessful password login)

The EAUTH_PASSWORD event occurs when a user tries to log in with a username/email that does exist but the password is incorrect.

Example Log

date
event
uuid
projectId
sessionId
userId
submissionId of the resource
Attempted username/email

2024-04-25T17:10:21.472Z

EAUTH_PASSWORD

a3245767-dcc5-4ed2-b998-cd36cdbeb9c6

659d83632824949f20a882e1

NoSession

NoUser

new ObjectId("659d83632824949f20a88389")

admin@example.com

AUTH_LOGIN (Successful login)

The AUTH_LOGIN event occurs when a user logs in successfully.

Example Log

date
event
uuid
projectId
sessionId
userId

2024-04-25T17:10:21.472Z

AUTH_LOGIN

ac245617-fd22-479d-9f8e-4846b0ac8177

659d83632824949f20a882e1

new ObjectId("662a9137cf86fe3fba034396")

new ObjectId("659d83632824949f20a88389")

EAUTH_LOGINCOUNT (Unsuccessful login count)

The EAUTH_LOGINCOUNT event occurs when a user tries to log in with a username/email that does exist but the password is incorrect. A log count is created for the number of times a user unsuccessfully tries to log in.

Example Log

date
event
uuid
projectId
sessionId
userId
Attempted username/email

2024-04-25T14:50:03.172Z

EAUTH_LOGINCOUNT

27759aa2-085b-4719-9193-5bad3fad74a2

659d83632824949f20a882e1

NoSession

NoUser

admin@example.com

Process tracking

These events occur to track the process of other events. They can be used to group processes together.

REQUEST_START (Start of any API request)

The REQUEST_START event occurs when any API request is made to the API server. It denotes the start of the request. Note: All request proceeding the REQUEST_START will have the same uuid until the REQUEST_END. This allows for many events to be grouped by the API request that was made.

Example Log

date
event
uuid
projectId
sessionId
userId
method
path
parameters

2024-04-25T17:43:17.654Z

REQUEST_START

5f46bed7-2a55-4ad7-b21f-cfccd724912b

659d83632824949f20a882e1

NoSession

No User

POST

/project/659d83632824949f20a882e1/user/login/submission

{"live":"1"}

REQUEST_END (End of REQUEST_START event)

The REQUEST_END event occurs when the API request has finished. It denotes the end of the request made by a REQUEST_START event. Note: All request preceding the REQUEST_END will have the same uuid until the REQUEST_START. This allows for many events to be grouped by the API request that was made.

Example Log

date
event
uuid
projectId
sessionId
userId
status code
rtt

2024-04-25T17:43:17.654Z

REQUEST_END

5f46bed7-2a55-4ad7-b21f-cfccd724912b

659d83632824949f20a882e1

NoSession

No User

200

167ms

Authentication checks

These events occur for requests that are related to authentication

AUTH_ANONYMOUS (Request without auth token)

The AUTH_ANONYMOUS event occurs when a request with no authorization token (x-jwt-token) is made.

Example Log

date
event
uuid
projectId
sessionId
userId

2024-04-25T17:43:17.654Z

AUTH_ANONYMOUS

5f46bed7-2a55-4ad7-b21f-cfccd724912b

659d83632824949f20a882e1

NoSession

No User

AUTH_LOGOUT (Logout of account)

The AUTH_LOGOUT event occurs when a user logs out.

Example Log

date
event
uuid
projectId
sessionId
userId

2024-04-25T14:50:03.172Z

AUTH_LOGOUT

27759aa2-085b-4719-9193-5bad3fad74a2

NoProject

new ObjectId("662fab333d908589b8931543")

new ObjectId("659d83632824949f20a88389")

Data Access

When a resource is accessed/viewed in the portal, a data access event will occur. These events have a naming scheme of RESOURCE_ACCESS

SUBMISSION_READ (Submission is viewed)

The SUBMISSION_READ event occurs when a submission is viewed. This can occur multiple times in a request if a list of submissions are viewed.

Example Log

date
event
uuid
projectId
sessionId
userId
submissionId
formId

2024-04-25T17:43:17.654Z

SUBMISSION_READ

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

new ObjectId("662aa339cf86fe3fba0343fe")

new ObjectId("659d83632824949f20a88389")

662aa865cf86fe3fba034d96

662aa7bccf86fe3fba034afa

FORM_READ (Form is viewed)

The FORM_READ event occurs when a form is viewed. This can occur multiple times in a request if a list of forms are viewed.

Example Log

date
event
uuid
projectId
sessionId
userId
formId

2024-04-25T17:43:17.654Z

FORM_READ

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

NoSession

No User

662aa7bccf86fe3fba034afa

ACTION_READ (Action is viewed)

The ACTION_READ event occurs when an action is viewed. This can occur multiple times in a request if a list of actions are viewed.

Example Log

date
event
uuid
projectId
sessionId
userId
actionId
formId

2024-04-25T17:43:17.654Z

ACTION_READ

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

new ObjectId("662aa339cf86fe3fba0343fe")

new ObjectId("659d83632824949f20a88389")

662aa85bcf86fe3fba034d08

662aa7bccf86fe3fba034afa

PROJECT_READ (Project is viewed)

The PROJECT_READ event occurs when a project is viewed. This can occur multiple times in a request if a list of projects are viewed. If you see a log with a projectId of NoProject then the ending Id represents stages within that project.

Example Log

date
event
uuid
projectId
sessionId
userId
projectId / stageId

2024-04-26T16:27:58.279Z

PROJECT_READ

23b2c367-6fb6-41f2-a36a-0ba9a42f8bc9

6627c5f740a5d95c7edc30ab / NoProject

new ObjectId("662aa339cf86fe3fba0343fe")

new ObjectId("659d83632824949f20a88389")

6627c5f740a5d95c7edc30ab

ROLE_READ (Role is viewed)

The ROLE_READ event occurs when a role is viewed. This can occur multiple times in a request if a list of roles are viewed.

Example Log

date
event
uuid
projectId
sessionId
userId
roleId

2024-04-26T16:27:58.279Z

ROLE_READ

a780acd0-2691-4cf4-9934-44f32e5415a3

6627c5f740a5d95c7edc30ab

new ObjectId("662aa339cf86fe3fba0343fe")

new ObjectId("659d83632824949f20a88389")

6627c5f840a5d95c7edc30b2

TEAM_READ (Team is viewed)

The TEAM_READ event occurs when a team is viewed. This can occur multiple times in a request if a list of roles are viewed.

Example Log

date
event
uuid
projectId
sessionId
userId
teamId
teamResourceId

2024-04-26T16:27:58.279Z

TEAM_READ

a780acd0-2691-4cf4-9934-44f32e5415a3

NoProject

new ObjectId("662aa339cf86fe3fba0343fe")

new ObjectId("659d83632824949f20a88389")

662967ad49e0b4f5b46904e8

659d83632824949f20a88316

V_READ (Form revision is viewed)

The V_READ event occurs when a form revision is viewed. This can occur multiple times in a request if a list of revisions are viewed.

Example Log

date
event
uuid
projectId
sessionId
userId
formRevisionId

2024-04-29T14:39:44.708Z

V_READ

e1627c41-1d39-4326-961b-681e5821bc99

6627c5f740a5d95c7edc30ab

new ObjectId("662faecc3d908589b893161a")

new ObjectId("659d83632824949f20a88389")

662c17f09f39954d132d4525

PROJECT_SETTINGS (Project settings is viewed)

The PROJECT_SETTINGS event occurs when project settings is viewed. This event is usually preceded by a GET /project/:projectId request.

Example Log

date
event
uuid
projectId
sessionId
userId

2024-04-26T16:27:58.279Z

PROJECT_SETTINGS

a780acd0-2691-4cf4-9934-44f32e5415a3

6627c5f740a5d95c7edc30ab

new ObjectId("662aa339cf86fe3fba0343fe")

new ObjectId("659d83632824949f20a88389")

Data changes

When data is created or updated, a data change event will occur. These events have a naming scheme of RESOURCE_ACCESS

SUBMISSION_CREATE (Submission is created)

The SUBMISSION_CREATE event occurs when a new submission is created.

Example Log

date
event
uuid
projectId
sessionId
userId
submissionId
formId

2024-04-25T17:43:17.654Z

SUBMISSION_CREATE

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

NoSession

No User

662aa865cf86fe3fba034d96

662aa7bccf86fe3fba034afa

SUBMISSION_UPDATE (Submission is updated)

The SUBMISSION_UPDATE event occurs when a submission is updated.

Example Log

date
event
uuid
projectId
sessionId
userId
submissionId
formId

2024-04-25T17:43:17.654Z

SUBMISSION_UPDATE

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

NoSession

No User

662aa865cf86fe3fba034d96

662aa7bccf86fe3fba034afa

FORM_CREATE (Form is created)

The FORM_CREATE event occurs when a new form is created.

Example Log

date
event
uuid
projectId
sessionId
userId
formId

2024-04-25T17:43:17.654Z

FORM_CREATE

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

new ObjectId("662faecc3d908589b893161a")

new ObjectId("659d83632824949f20a88389")

662faf703d908589b89316c7

FORM_UPDATE (Form is updated)

The FORM_UPDATE event occurs when a form is updated.

Example Log

date
event
uuid
projectId
sessionId
userId
formId

2024-04-25T17:43:17.654Z

FORM_UPDATE

4d7649c7-d24c-4b61-bc7e-6e672186eca6

6627c5f740a5d95c7edc30ab

NoSession

No User

662aa7bccf86fe3fba034afa

PROJECT_CREATE (Project is created)

The PROJECT_CREATE event occurs when a project is created.

Example Log

date
event
uuid
projectId
sessionId
userId
projectId

2024-04-26T18:12:46.382Z

PROJECT_CREATE

23f11fd3-a36e-4ce7-afae-58ab18bc988a

NoProject

new ObjectId("662bcba30dd222f70702f99e")

new ObjectId("659d83632824949f20a88389")

662bf0840dd222f707030342

PROJECT_UPDATE (Project is updated)

The PROJECT_UPDATE event occurs when a project is updated.

Example Log

date
event
uuid
projectId
sessionId
userId
projectId

2024-04-26T18:12:46.382Z

PROJECT_UPDATE

b3672267-494f-4d3d-838d-05e7637c23ef

6627c5f740a5d95c7edc30ab

new ObjectId("662bcba30dd222f70702f99e")

new ObjectId("659d83632824949f20a88389")

6627c5f740a5d95c7edc30ab

ROLE_CREATE (Role is created)

The ROLE_CREATE event occurs when a role is created.

Example Log

date
event
uuid
projectId
sessionId
userId
roleId

2024-04-26T18:12:46.382Z

ROLE_CREATE

4d489fcc-e6a5-4a26-b747-50bd06f46405

6627c5f740a5d95c7edc30ab

new ObjectId("662bcba30dd222f70702f99e")

new ObjectId("659d83632824949f20a88389")

662bf2c10dd222f7070305e0

ROLE_UPDATE (Role is updated)

The ROLE_UPDATE event occurs when a role is updated.

Example Log

date
event
uuid
projectId
sessionId
userId
roleId

2024-04-26T18:12:46.382Z

ROLE_UPDATE

4d489fcc-e6a5-4a26-b747-50bd06f46405

6627c5f740a5d95c7edc30ab

new ObjectId("662bcba30dd222f70702f99e")

new ObjectId("659d83632824949f20a88389")

662bf2c10dd222f7070305e0

TEAM_CREATE (Team is created)

The TEAM_CREATE event occurs when a team is created.

Example Log

date
event
uuid
projectId
sessionId
userId
teamId
teamResourceId

2024-04-26T18:12:46.382Z

TEAM_CREATE

4d489fcc-e6a5-4a26-b747-50bd06f46405

NoProject

new ObjectId("662bcba30dd222f70702f99e")

new ObjectId("659d83632824949f20a88389")

662bf84c0dd222f7070306c1

659d83632824949f20a88316

TEAM_UPDATE (Team is updated)

The TEAM_UPDATE event occurs when a team is updated.

Example Log

date
event
uuid
projectId
sessionId
userId
teamId
teamResourceId

2024-04-26T18:12:46.382Z

TEAM_UPDATE

4d489fcc-e6a5-4a26-b747-50bd06f46405

NoProject

new ObjectId("662bcba30dd222f70702f99e")

new ObjectId("659d83632824949f20a88389")

662bf84c0dd222f7070306c1

659d83632824949f20a88316

Aggregation System Deployment

There are special aggregation systems that help to aggregate and manage your Docker logs. In this article, we will consider connection using the open-source Elastic Stack log analysis software as an example.

The Elastic Stack consists of two different tools: Elasticsearch and Kibana. Elasticsearch is needed to store log data and Kibana to visualize it. To transfer the logs into ElasticSearch we will use the open source data shipper FileBeat.

There are many ways to set up the Elastic Stack, the easiest is using docker-compose. We will use the official docker images and there will be a single ElasticSearch node.

Step 1: Add the following rows to the existing docker-compose.yml:

services:
  elasticsearch:
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.2.0"
    environment:
        - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
        - "discovery.type=single-node"
    ports:
        - "9200:9200"
    volumes:
        - elasticsearch_data:/usr/share/elasticsearch/data

  kibana:
      image: "docker.elastic.co/kibana/kibana:7.2.0"
      ports:
          - "5601:5601"

  filebeat:
      image: "docker.elastic.co/beats/filebeat:7.2.0"
      user: root
      volumes:
          - /MY_WORKDIR/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
          - /var/lib/docker:/var/lib/docker:ro
          - /var/run/docker.sock:/var/run/docker.sock
volumes:
  mdb-data:
  elasticsearch_data:

Note that you need to replace MY_WORKDIR with the path of your working directory.

For ElasticSearch and Kibana basic docker configuration is enough. They are respectively available on ports 9200 and 5601. ElasticSearch has a volume to keep its data. Kibana doesn't need a volume as it uses ElasticSearch to persist its configuration. FileBeat, on the other hand, needs to be configured. To share this config file with the container, we need a read-only volume /usr/share/filebeat/filebeat.yml:ro. FileBeat also needs to have access to docker log files. They can usually be found in /var/lib/docker/containers but that might depend on your docker installation. The docker socket /var/run/docker.sock is also shared with the container. That allows FileBeat to use the docker daemon to retrieve information and enrich the logs with things that are not directly in the log files, such as the name of the image or the name of the container.

Step 2: Create the filebeat configuration file in /MY_WORKDIR/filebeat.yml

filebeat.inputs: 
  - 
    paths:
      - /var/lib/docker/containers/*/*.log
    type: container
logging.json: true
logging.metrics.enabled: false
output.elasticsearch: 
  hosts: 
    - "elasticsearch:9200"
  indices: 
    - 
      index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
processors: 
  - 
    add_docker_metadata: 
      host: "unix:///var/run/docker.sock"
  - 
    decode_json_fields: 
      fields: 
        - message
      overwrite_keys: true
      target: json
  • The FileBeat input type container is needed to import logs from docker. /var/lib/docker/containers/*/*.log is the location of the log files inside the FileBeat container.

  • The output elasticsearch setting allows you to configure the ElasticSearch address as well as the indexes where the logs are imported. The template index filebeat-%{[agent.version]}-%{+yyyy.MM.dd} includes a date. This means that docker logs are imported into the index corresponding to the date they appeared.

  • add_docker_metadata is needed in order to add useful information to the logs, such as the name of the image or the name of the container. Only IDs are displayed by default.

  • decode_json_fields lets to parse logs encoded in JSON. The logs in FileBeat, ElasticSearch, and Kibana consist of multiple fields. The message field is what the application (running inside a docker container) writes to the standard output.

Step 3: Launch docker containers with docker-compose

Kibana is now available on http://localhost:5601. Click on the left on Discover and on Create Index Pattern. Create a pattern using filebeat-* (to include all logs from FileBeat) and @timestamp.

Done! Click on Discover again and you will see the logs. Note that logs of all containers are displayed with the default configuration. The formio logs described at the beginning of this article come from the formio container.

Last updated