Collections

Collections provides a way to define a data structure. A collection is based on a JSON Schema schema which defines the structure of the data. With this schema, you can then insert, update and list the collection entries. These collections are stored inside a MongoDB instance.

Capabilities

Capability	Support	Comment
Access a collection from the front-end		Widgets Collection Table and Form can visualize and modify entries from a collection
Access a collection from scripts		A Collection can be accessed and edited from any JS and Lua scripts
Access a collection from values		You can listen to changes on a collection to set a value with the lasted updated entry each time the collection is updated
Limit the edition access of a collection entries to an user		See Limit the access of a collection entries to an user
Limit the view of a collection form to an user		See Limit the view of a collection form to an user
Define the collection data structure with a schema		See Defining data structure with JSON Schema
Reference other schema from a schema		See Use references to another schema
Define automatically generated fields		See Define automatically generated fields
Declare an auto-increment field		See Declare an auto-increment field
Keep a history of changes to a collection entry		See Track changes to a collection entry (history)
Delete behavior for collection entries		See Delete behavior for collection entries
Create a TTL index to remove inactive collection entries		See Create a TTL index to remove inactive collection entries
Interact with a collection		See Interact with a collection
Track the changes of a collection entry with a value		See Track the changes of a collection entry with a value
Manage conflict (concurrent modifications)		Conflict are managed by the form widget
Improve request times by settings index on field		As the collection grows, response times on large databases can decrease. Setting an index on the queried fields helps the database optimize response times. See Create collections indexes
Ensure unique occurrence.		The unique index is available to do so. See Create collections indexes
Updating order		If multiples source modules (ex: scripts/web/) trigger changes of the same entry of a collection the state will be updated depending of the order of request (sequential).

Examples

Defining data structure with JSON Schema

Concept

Schemas define the structure of a collection and validate an entry when inserting or updating a collection. They require to follow the JSON Schema specification to be considered as a valid collection schema. Therefore, you should start by getting familiar with JSON Schema before attempting to define a collection.

Note

Some of the JSON Schema annotation keywords are used to add a specific behaviour.

Use title to define a string value that will be used whenever we need to print a schema property anywhere on the front-end. If title is not present, we display the name instead. This is mostly meaningful for the CollectionsRights form component.

Use $comment as $comment: "date" to specify that a number property is meant to represent a datetime in a milliseconds/nanoseconds format. For the CollectionsRights form component, it tells it to render this number property as a datetime picker rather than a number input.

Examples

Define a collection

Use references to another schema

Concept

You can use a $ref inside a schema to reference another schema in order to copy the properties of the referenced schema into another one.

Warning

This is only used to simplify the definition of schemas, as it only copy the properties from one to another. The potential elements of the collections defined by these schema have no relation at all between them.

Usage

From a schema report with:

{
  "schema": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "device": {
        "$ref": "root.device"
      }
    }
  }
}

Referencing a schema device like:

{
  "schema": {
    "type": "object",
    "properties": {
      "serial": {"type": "string"},
      "ip": {"type": "string"}
    }
  }
}

We will end up with the same schema as if we would have defined report directly like:

{
  "schema": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "device": {
        "type": "object",
        "properties": {
          "serial": {"type": "string"},
          "ip": {"type": "string"}
        }
      }
    }
  }
}

Define automatically generated fields

Concept

Define automatically generated fields that are computed at each creation and update of an entry of this collection. These are defined by a map where:

key: name of the generated field
value: computed field definition that determines how the field is generated

The result of the computed field is stored within every collection entry. Each field is calculated both at the creation of a new entry and at every update. Theses fields can then be accessed like any normal field in a view or a form for example.

There are multiple type of computed fields. Theses types determine the operation that will be effected.

Referencing a schema property

Every computed field is based on one or more properties define in the collection schema.

To reference which property to pick from, you can use a valid JsonPath expression. When an entry is created or updated, the JsonPath expression will be used with the given payload to retrieve the content you want to feed to the computed field.

This expression must target a property defined in the schema.

CONCAT

Concatenate the values of the properties into a new property. The result is defined by a format that must be given. Always result in a new string property.

format

The format allow the use of placeholders that will be replaced with the value of the property described by the placeholder. These must start by ${{ and end by }}.

Like every other computed fields, the properties can be referenced by giving a valid JsonPath expression.

As examples:

"${{name}}, ${{firstname}}"
"${{$.name}}, ${{$.firstname}}"
"${{person.name}}, ${{person.firstname}}"

In addition to specify directly the property, you can define a list of mappings and use the mapping name instead.

mapping

A mapping allow to map a string key with a schema property path. This can be useful if you have complex path to help clarify what a property represent and it can also make the format more readable and easy to understand.

The name of the mapping is the one you need to use in the format:

"mappings": [
  {"name": "add", "field": "address"},
  {"name": "web", "field": "website"}
],
"format": "Address: ${{add}}, Website: ${{web}}"

Compatible properties

CONCAT operation is available on any type of properties other than array and object.

COALESCE

Take the given properties and retrieve the first one that is not null. The order of properties is kept and defines the order the values are tested.

Compatible properties

COALESCE operation is available on any type of properties, but every property picked for this operation must be the same type.

MERGE_ARRAY

Take two or more arrays and attempt to merge them together into a new array.

Compatible properties

MERGE_ARRAY operation is available on array properties if they share the same internal definition.

SPLIT

Take a string property and separate it into an array by using a given regular expression. The result is either the array containing every element or only the one defined by the given position.

Compatible properties

SPLIT operation is available on string properties.

Declare an auto-increment field

Concept

The predefined collections_counter collection associates a counter to another collection. Each time a new entry is created within a collection that have an auto-increment field defined in its schema, we lookup in this collection to get the current counter and increment it for the next insert. Then, this value is used to set all the auto-increment fields before actually creating the new entry.

Warning

Auto-increment fields are given a value only on creation, meaning they can’t be updated after, making them readonly values.

A collections_counter entry contains :

schemaId: id (osp path) of the schema of the collection
counter: current value of the counter

You can declare which fields should use this counter by editing the collection schema configuration.

Track changes to a collection entry (history)

Concept

By default, each modification done to a collection is saved into a specific predefined collection called collections_history. This can be disabled for each collection by changing the collection schema.

Be aware that if you disable the historic of a collection which was historized previously, the historic will be completely cleared. On the other hand, when enabling historic, an history entry will be created for each current entry of the collection to serve as a base.

The collections_history collection keeps track of every operations performed on any collection. Each time an insert, update or delete request is done, a new document is added to the historic with the following values :

collectionId: id of the document inserted/updated/deleted in the other collection
schemaId: id (osp path) of the schema of the collection
modified_by: name of the user that made the request
modified_at: time when the request was made
operation: is an insert or update (delete is a soft-delete operation, therefore is marked as an update)
diffs: list of every data differences

Delete behavior for collection entries

Deleting an entry from a collection doesn’t completely delete it. It performs a soft delete by setting the __is_active flag to false. This will prevent this entry to be accessible.

This can guarantee that you will never lose important data, as you also have a way to restore inactive elements.

Warning

Since these entries are never fully deleted, the collection will keep increasing in size. All cleaning operation must be manually done directly on your MongoDB instance.

Nevertheless, each collection can be configured to so that inactive element are cleaned after a configured amount of time.

Create a TTL index to remove inactive collection entries

Concept

In the collection schema configuration, specify a TTL higher than 0 seconds in order to automatically create a MongoDB TTL index, allowing you to fully remove an entry that was previously soft deleted.

By default, the TTL is set to 0, which implies that no entry of the collection is truly deleted without a manual operation directly on your MongoDB instance.

Interact with a collection

Concept

You can interact with a collection by :

insert: Inserts a new entry into a collection.
update: Updates an existing entry.
delete: Soft deletes of an entry. The entry is not deleted but is rather set as inactive. By default, inactive entries are not retrieved by list/get requests.
restore: Restore a previously deleted entry.
list: Get the content of a collection.
get: Retrieves one element from a collection.
listen to changes: Listens to this collection updates to set a value with the lasted updated entry of the collection each time the collection is updated.

Examples

Insertion

When attempting to create a new entry, the data must be validated with the schema in order to ensure that it respects the structure of the collection.

When creating a new entry, these following attributes are automatically added :

__created_at: new entry creation time
__created_by: user that created this entry
__modified_at: time of last operation on this entry (same as __created_at)
__modified_by: last user that modified this entry (same as __created_by)
__is_active: flag to define if this entry is active or not. If not, the entry is not accessible (default true)

Once the entry is created, an entry is added to the history collection to keep track of this operation.

The following modules can create an entry:

Module	Can insert	How
Front-end		Collection Table, Form
Scripts		Scripts
Values

Updates

When attempting to update an existing entry, the data must be validated with the schema in order to ensure that it respects the structure of the collection.

There is two different approaches to update a collection:

With full data: you can update an entry by giving the full updated entry. The entry will be match with the _id that must be present in the updated entry. If this id is not present, a new entry will be created instead.
With list of updates: you can update by passing a list of updates to be applied on the given entry. Updates must follow this structure :
- field: name of the updated field
- operation: type of operations: SET, UNSET, PUSH or PULL (PUSH and PULL for arrays only).
- content: updated value

Note

When attempting to pull an object from an array, it’s recommended to have an unique id associated with each object and use this id alone to pull form the array, instead of passing the full object as the update content. Mongo pull update operation may have some issues with matching the element when using the full object as the content, mainly issues that could occur if the order of the fields doesn’t match between the database data and the update content.

Therefore, it’s safer to define an unique id and use it on the pull update. MongoDB should be able to match the correct element with only the id. Your content should look like {<id-key>: '<unique-id-value>'}.

When updating an entry, these following attributes are automatically updated:

__modified_at: time of last operation on this entry
__modified_by: last user that modified this entry

Once the entry is updated, an entry is added to the history collection to keep track of this operation.

The following modules can update an entry:

Module	Can update	How
Front-end		Collection Table, Form
Scripts		Scripts
Values

Delete

When deleting an entry, these following attributes are automatically updated :

__modified_at: time of last operation on this entry
__modified_by: last user that modified this entry
__is_active: flag to define if this entry is active or not. Set to false

Once the entry is deleted, an entry is added to the history collection to keep track of this operation.

The following modules can delete an entry:

Module	Can delete	How
Front-end		Collection Table
Scripts		Scripts
Values

Restore

When restoring an entry previously deleted, these following attributes are automatically updated :

__modified_at: time of last operation on this entry
__modified_by: last user that modified this entry
__is_active: flag to define if this entry is active or not. Set to true

Once the entry is restore, an entry is added to the history collection to keep track of this operation.

The following modules can delete an entry:

Module	Can restore	How
Front-end		Collection Table
Scripts		Scripts
Values

List

You can retrieve all the entries that compose a collection, with or without applying a filter.

Warning

In order to prevent requests with a large amount of results, you will need to define a pageSize limiting the number of result and a pageNumber that you can increase to get the following results. This creates a request where we limit the number of results by pageSize and where we skip the first pageNumber * pageSize, which implies that the pageNumber starts at 0.

In addition to the entries, an additional totalCount is passed into the result. This allows to determine the number of pages by simply dividing totalCount by pageSize. This is here to help you request the rest of the entries.

The following modules offers different ways to retrieve theses entries:

Module	Can list	How
Front-end		Collection Table
Scripts		Scripts
Values

Some modules are capable of maintaining a stream of data when requesting entries of a collection. Therefore, each time an update occurs, the module receives the updated entries of the collection without having to make another request.

The following modules works with streams:

Module	Can be updated	How
Front-end		Collection Table
Scripts
Values

Get

You can retrieve a specific entry by using its id or a filter. With a filter, the request will return the first entry that matches the filter.

The following modules offers different ways to retrieve an entry:

Module	Can retrieve	How
Front-end		Collection Table, Form
Scripts		Scripts
Values		Collection owner

Some modules are capable of maintaining a stream of data when requesting an entry. Therefore, each time an update occurs, the module receives the updated version of this entry without having to make another request.

The following modules works with streams:

Module	Can be updated	How
Front-end		Collection Table, Form
Scripts
Values		Collection owner

Listen to changes

Some modules are capable of maintaining a stream of data when interacting. Therefore, each time an update occurs, the module receives the updated view of the collection.

The following modules can work with streams:

Module	Can be updated	How
Front-end		Collection Table, Form
Scripts
Values		Collection owner

Track the changes of a collection entry with a value

Concept

Collections provide a way to interact with values by using an owner. The owner will update the content of the value each time a collection entry changes, if that entry match the owner definition.

In his basic form, an owner is defined by:

schemaId: the reference of the collection we want to based this owner off
filterId: the id of one of the filters off the specified collection. The value is only updated if the collection entry match the filter

In addition to these basic and mandatory properties, you can set other parameters to change the behaviour:

watchList: List of the fields on which to trigger a change of the value when the collection changes. When an entry of the collections changes, the value is updated only if it matches the filter and if the updated or removed fields are present in watchList. If empty or not defined, every change will trigger an update of the value.
scope: Scope of the document transmitted to the value. Three options are available:
- FULL_DOCUMENT: Set the content of the value directly with the full updated document. Meaning you can access a property (i.e. property name of the collection) like content.name. Note that you have no way to know which properties changed with the last update, which is the reason behind the next scope option.
- CHANGES_ONLY: Set the content of the value with a map that contains two entries:
  - updatedFields: a map with keys that represents the properties changed by the update with their updated values.
  - removedFields: list of properties that were deleted.
- BOTH: Mixed of the two previous options. Set the content of the value with a map that contains three entries:
  - fullDocument: complete document of the collection.
  - updatedFields: a map with keys that represents the properties changed by the update with their updated values.
  - removedFields: list of properties that were deleted.
targetProperty: Target property meant to be set as the value content. Instead of setting the value content with the full document, the value is set with the content of the target property only. Target is represented by a JsonPath, letting you precisely choose which field to target. If the target is not found within the full document, an error is set to the value instead. The content is checked in order to guarantee that it matches the value type. If not, an error is thrown as well. Only works when using scope FULL_DOCUMENT, as we need it to access the target.

Note

targetProperty and watchList are not dependent on each other. You can freely watch only changes on a given property (i.e. date) and retrieve another property (i.e. status).

Examples

Limit the access of a collection entries to an user

Concept

In addition to limit access to a collection schema using the resources access rights management, which will prevent any request on a collection, collections rights can be set for an user, limiting the accessible entries of the collection for the specified user. Therefore, these two features have very different purposes. Access rights are meant to give access to any OnSphere Item, while collections rights are meant to limit the access to the entries of a collection.

For each user and collection, you can:

give the right to create, read, update and delete entries
limit access to specific entries by providing a list of identifiers (MongoDB document ids)
limit access to specific properties by providing a list of properties name (must be from collection schema). Updates are only accepted if they modify one or many of the properties defined in this list
define filters on collection properties. Only entries that match the given filter are available

Collection rights are handled with multiple levels.

users rights: First level which define rights for a specific user.
profiles rights: Second level which define rights for profiles. Profiles can be given rights like users and then be referenced in users rights. When rights are present both at the profile level and the user level, we take into account the rights defines in the user level. Therefore, you can override rights set in a profile for a specific user.
services rights: Third and last level which defines services. Services can also be given rights and are linked to a list of profiles. Only users that have one of theses profiles can enter the service. A service contains a list of active users and is meant to only override the rights of the user when he is active in the service.

Take note that theses three different levels are strongly linked to one another. You cannot use services rights without using profiles rights too. Simply put, you cannot enable a level without enabling every level below.

Collections rights must be enabled in collections module configuration. Furthermore, for each level enabled, you need to provide a valid collection schema. Having a schema defined in the configuration is mandatory to be able to interact with these collections from scripts or from front-end widgets, which gives you a way to create and modify entries of these collections. A default configuration for these collections is available in a specific configuration branch called origin/osp-collections-rights-configuration. Theses schemas are validated to insure that they have the required properties. The schemas validator can be found under the schemas section.

Be aware too that since each one of these three levels correspond to a collection that must be define in your configuration, you are free to add additional properties to the schema and, therefore, to the entries of the collections. For example, we verify that the users rights provided schema contains at least a username property, but you could freely add an email property. Keep in mind that all of theses extras properties won’t be utilized when verifying the rights of an user.

Examples

Limit the view of a collection form to an user

Concept

Using the collections rights, you can set to an user a list of forms rights in addition to the others collection specific rights discussed in the previous chapter. This implies that you need to provide the necessary schemas to enable this feature (see previous chapter).

This list of form rights consist in a basic list of names, each name representing one right. These rights are retrieved when requesting a form and used to determine which part of the form the user has access to. This implies that this feature is only usable with a form.

To see how to use them in a form, look at the form widget document.

Create collections indexes

Concept

In MongoDB, indexes are used to improve query performance by allowing the database to quickly locate documents in a collection. Instead of scanning every document, an index creates a data structure (similar to an index in a book) that points directly to the relevant documents.

The description of the index are available on the official web site

Usage

Faster Queries: Significantly reduce query execution time.
Efficient Sorting: Optimize operations with sort queries.
Support for Uniqueness: Enforce unique values on a field.

The indexes can be defined for history collection in the :ref::module.collections file. And in the schema.collections file for the other collections.

Kind of indexes

The full list of properties is described on the official documentation The possibility can be for example

Assure uniqueness

"index" : "{'ip_address': 1} , {unique: true}"

Partial indexes

From the official documentation

Partial indexes only index the documents in a collection that meet a specified filter expression. By indexing a subset of the documents in a collection, partial indexes have lower storage requirements and reduced performance costs for index creation and maintenance.

"index": "{ cuisine: 1, name: 1 }, { partialFilterExpression: { rating: { $gt: 5 } } }"

Examples

Define history index in a simple collection example