Creating Domain Specific Data Model by Selecting and Extending Concepts from Core Data Model¶

Prerequisite:

• Basic understanding of Data Modeling in CDF
• Basic understanding of Core Data Model
• Access to a CDF Project.
• Know how to install and setup Python.
• Launch a Python notebook.

In this tutorial, we will show you how to extend a core data model making domain specific data model. We will demonstrate this process by building a wind farm data model

Creating the Base Model¶

Cognite's Core Data Model (or short CDM) is an immutable model maintained by Cognite. To extend this model we first create a copy of it in our own space, such that we can start to do modifications to it.

In [1]:

from cognite.neat import NeatSession, get_cognite_client

from cognite.neat import NeatSession, get_cognite_client

In [2]:

client = get_cognite_client(".env")

client = get_cognite_client(".env")

Found .env file in repository root. Loaded variables from .env file.

In [3]:

neat = NeatSession(client)

neat = NeatSession(client)

Neat Engine 2.0.4 loaded.

In [4]:

neat

neat

Out[4]:

Empty session. Get started by reading something with the .read attribute.

Let's:

1. Load CDM in NeatSession , fastest way is through examples short cut neat.read.examples.core_data_mode()
2. Inspect content of NeatSession afterwards by calling neat
3. and visualize data model by calling neat.show.data_model() and neat.show.data_model.implements()

Do not get confused with potential warnings you get when reading the core data model into NeatSession, the warnings just point to users that filters are set on the core data model. We prefer warning users about usage of filters, as filters if used by inexperienced users can lead to unexpected results.

In [5]:

neat.read.examples.core_data_model()

neat.read.examples.core_data_model()

Out[5]:

Succeeded with warnings: Read NEAT(verified,physical,cdf_cdm,CogniteCore,v1)

	count
NeatIssue
NotNeatSupportedFilterWarning	7

Hint: Use the .inspect.issues() for more details.

In [6]:

neat

neat

Out[6]:

Data Model

aspect	physical
intended for	DMS Architect
name	Cognite core data model
space	cdf_cdm
external_id	CogniteCore
version	v1
views	33
containers	29
properties	141

In [7]:

neat.show.data_model()

neat.show.data_model()

http_purl.org_cognite_neat_data-model_verified_physical_cdf_cdm_CogniteCore_v1.html

Out[7]:

In [8]:

neat.show.data_model.implements()

neat.show.data_model.implements()

http_purl.org_cognite_neat_data-model_verified_physical_cdf_cdm_CogniteCore_v1_implements.html

Out[8]:

CDM consists of 33 concepts (divided to 28 so called Core Concepts and 5 so called Core Features), majority of which are related to 3D (15+ views/concepts).

Since we do not need all these concepts, we will select only a subset of them to create a wind farm data model. To do this, we will select the following core concepts:

• CogniteAsset
• CogniteEquipment
• CogniteTimeSeries
• CogniteActivity

Using the above subset of core concepts, by extending them and adjusting them we will create a wind farm data model which will contain the following wind energy specific concepts:

• WindFarm
• WindTurbine
• Substation
• MetMast

Let's start first by selecting the core concepts:

In [9]:

neat.subset.data_model(["CogniteAsset", "CogniteEquipment", "CogniteTimeSeries", "CogniteActivity"])

neat.subset.data_model(["CogniteAsset", "CogniteEquipment", "CogniteTimeSeries", "CogniteActivity"])

Subset to 8 concepts.

[WARNING] Alpha feature 'data_model_subsetting' is subject to change without notice

Out[9]:

Success: NEAT(verified,physical,cdf_cdm,CogniteCore,v1) → NEAT(verified,physical,cdf_cdm,CogniteCore,v1)

Do not mind warning, if something is new feature in neat, it will be marked as Alpha feature.

Let's now inspect content of the subsetted core data model by calling neat , and then visualizing it via neat.show.data_model() ...

In [10]:

neat.show.data_model()

neat.show.data_model()

http_purl.org_cognite_neat_data-model_verified_physical_cdf_cdm_CogniteCore_v1.html

Out[10]:

You should notice that in addition to the core concepts we selected (CogniteAsset, CogniteEquipment, CogniteTimeSeries, CogniteActivity) there are additional concepts (aka core features):

• CogniteVisualizable
• CogntieSourceable
• CogniteDescribable
• CogniteSchedulabel

They are added by NEAT because our the ceoncepts we selected from CDM implement them, and without them the data model would not be complete.

You can inspect this by calling neat.show.data_model.implements()

In [11]:

neat.show.data_model.implements()

neat.show.data_model.implements()

http_purl.org_cognite_neat_data-model_verified_physical_cdf_cdm_CogniteCore_v1_implements.html

Out[11]:

Next we will turn this reduce CDM into enterprise data model that we will further edit in Excel to build up our wind farm data model. We achieve this by calling method ...enterprise_model where we pass data model if for new model we are creating as well org name (org_name).

In [12]:

neat.create.enterprise_model(data_model_id=("wind_space", "WindFarm", "v1"), org_name="NeatOrg")

neat.create.enterprise_model(data_model_id=("wind_space", "WindFarm", "v1"), org_name="NeatOrg")

Out[12]:

Success: NEAT(verified,physical,cdf_cdm,CogniteCore,v1) → NEAT(verified,physical,wind_space,WindFarm,v1)

Let's export data model to Excel:

In [13]:

neat.to.excel("wind-farm-data-model.xlsx")

neat.to.excel("wind-farm-data-model.xlsx")

If we export created enetriprise data model to Excel via neat.to.excel("wind-farm-dm-template.xlsx") we will see in more depth results of ...create.enterprise_model method, which did the following:

1. Create an editable vesions of concepts we selected from CDM, names of which will be prefixed by the `org_name

   If we set org_name to be NeatOrg, neat will create NeatOrgAsset, NeatOrgEquipment, etc., and it will make sure that NeatOrgAsset implements CogniteAsset, NeatOrgEqupiment implements CogniteEquipment, etc.

2. It will adjust connection between the editable versions of concepts

   In CogniteEqupiment, property asset points to CogniteAsset, neat updates this connection in case of NeatOrgEqupimnt, such that it points to NeatOrgAsset instead.

3. It will add dummy property, which if not specified will be in form of <nameOfConcept>GUID

   There are a few purposes of this property. First, to show users how they can add new properties to the editable version of concepts, second by adding specific property to editable version of concepts once can skip adding filters to ensure consumption of data

4. It will add new containers to store additional properties that are not part of the original concepts that editable concepts are implementing

Now we can further edit and tune exported data model to produce desired wind farm data model.

We will do the following:

1. Since we do not intend to futher extend Schedulable, Visualizable, and Sourcable we will simply remove:

• NeatOrgSchedulable
• NeatOrgSourceable
• NeatOrgVisualizable

from Views, Containers and Properties sheets in Excel

2. Since we would like to have location information, which containers name, description, latitude, longitude and height properties, we will rename NeatOrgDescribable to Location and add properties which do not exist in CogniteDescribable, these being latitude, longitude and height.

3. We will set also units to latitude, longitude and height to degree and meter. This is done by specifying Value Type with unit, e.g. float(unit=angle:deg) (list of units and their external ids can be found here)

4. We will rename property neatOrgAssetGUID to location and set it to be the connection of type direct where value type will be set to the above created Location

5. We will create WindFarm, WindTurbine, Substation, MetMast concepts by implementing NeatOrgAsset and adding following specific properties repespectively capacityFactor, activePower, voltageLevel, iecCompliant

6. Since we would like to have explicit connection between WindFarm and its underlaying asset WindTurbine, Substation and MetMast, we will create direct connection from WindTurbine to WindFarm via property windFarm, and similarly for Substation to WindFarm and MetMast to WindFarm via properties of the same name. In addition we will create the reverse connection based of these properties from WindFarm to WindTurbine, Substation and MetMast via corresponding properties windTurbine, substation and metMast

We will read in manual edited Excel file into NeatSession using ...read.excel(filename, enable_manual_edit=True). Beware that we are setting argument enable_manual_edit to True which signals to neat to try to read in manually edited data model and join it into the provenance trail.

You can download wind-farm-data-model-manual-edited.xlsx

In [14]:

neat.read.excel("wind-farm-data-model-manual-edited.xlsx", enable_manual_edit=True)

neat.read.excel("wind-farm-data-model-manual-edited.xlsx", enable_manual_edit=True)

[WARNING] Alpha feature 'enable_manual_edit' is subject to change without notice

Out[14]:

Success: Read NEAT(verified,physical,wind_space,WindFarm,v1)

Let visualize a full provenance from the begining til now:

In [15]:

neat.show.data_model.provenance()

neat.show.data_model.provenance()

data_model_provenance_995928ea.html

Out[15]:

Finally let's push data model to CDF:

In [16]:

neat.to.cdf.data_model()

neat.to.cdf.data_model()

You can inspect the details with the .inspect.outcome.data_model(...) method.

Out[16]:

	name	created
0	spaces	1
1	containers	9
2	views	9
3	data_models	1
4	nodes	0