Asset Hierarchy Source¶

Prerequisite:

• Access to a CDF Project.
• Know how to install and setup Python.
• Launch a Python notebook.

In this tutorial, we will show you how to migrate an asset hierarchy to a data model representing the same hierarchy in CDF.

This tutorial is also a good demonstration of the width of capabilities for neat, going from extracting data, creating a data model, export the data model, and load the data.

Extract Data from Asset Hierarchy¶

We will start by extracting the data from an existing asset hierarchy.

The example we will use in this tutorial is an asset hierarchy for pumps, shown below in CDF's classic Data Exploration

In [1]:

from cognite.neat import NeatSession, get_cognite_client

from cognite.neat import NeatSession, get_cognite_client

We start by instansiating a new NeatSession

In [2]:

client = get_cognite_client(".env")

client = get_cognite_client(".env")

Found .env file in repository root. Loaded variables from .env file.

In [3]:

neat = NeatSession(client, storage="memory")
neat

neat = NeatSession(client, storage="memory") neat

Neat Engine 2.0.3 loaded.

Out[3]:

Empty session. Get started by reading something with the .read attribute.

In [5]:

neat.read.cdf.classic.graph(root_asset_external_id="lift_pump_stations:root")

neat.read.cdf.classic.graph(root_asset_external_id="lift_pump_stations:root")

Extracting Asset relationships: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  8.41it/s]
Extracting data sets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 16.30it/s]

Out[5]:

Success: Read Classic Graph

In [6]:

neat

neat

Out[6]:

Verified Data Model

type	Logical Data Model
intended for	Information Architect
name	Inferred Model
external_id	ClassicDataModel
version	v1
classes	9
properties	71

Instances

Overview:

• 2 types
• 246 instances

	Type	Occurrence
0	ClassicAsset	245
1	ClassicDataSet	1

Provenance:

• Initialize graph store as Memory
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Added rules to graph store as InformationRules
• Upsert prefixes to graph store
• Lookup relationships source and target externalId
• ConvertLiteral is a transformer that improve data typing of a literal value.
• Converts a literal value to new entity

Exporting Data Model¶

Lets export our newly created data model to CDF. First, we need to convert it to an phsycial format.

In [7]:

neat.convert("dms")

neat.convert("dms")

Rules converted to dms

Out[7]:

Success: VerifiedInformationModel → VerifiedDMSModel

In [8]:

neat

neat

Out[8]:

Verified Data Model

aspect	physical
intended for	DMS Architect
name	Inferred Model
space	neat_space
external_id	ClassicDataModel
version	v1
views	9
containers	9
properties	69

Instances

Overview:

• 2 types
• 246 instances

	Type	Occurrence
0	ClassicAsset	245
1	ClassicDataSet	1

Provenance:

• Initialize graph store as Memory
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Extracted triples to graph store using ClassicGraphExtractor
• Added rules to graph store as InformationRules
• Upsert prefixes to graph store
• Lookup relationships source and target externalId
• ConvertLiteral is a transformer that improve data typing of a literal value.
• Converts a literal value to new entity

In [10]:

neat.to.cdf.data_model(existing="recreate")

neat.to.cdf.data_model(existing="recreate")

You can inspect the details with the .inspect.outcome.data_model(...) method.

Out[10]:

	name	unchanged	created	deleted	issues
0	spaces	1	0	0	0
1	containers	0	9	9	0
2	views	0	9	9	0
3	data_models	0	1	1	1
4	nodes	0	0	0	0

We see the data model was succesfully created.

Populating Data Model¶

As the data model is ready, we can move the instances to CDF

In [11]:

neat.to.cdf.instances()

neat.to.cdf.instances()

You can inspect the details with the .inspect.outcome.instances(...) method.

Out[11]:

	name	created	changed
0	ClassicAsset	245	0
1	ClassicDataSet	0	1

As we see from the result above, Neat has created 245 Assets in the new data model.

Results¶

We can now go into CDF and inspect the results. Looking at the data model we created, we can see the schema for the inferred Asset

Furthermore, we can inspect the populated nodes in this Asset schema

Final Remarks¶

• In this tutorial, we used the in-memory version of the Neat store. This works well for small examples, like the toy example here, but for larger asset hierarchies we likely need to use a faster triple store such as GraphDB or Oxigraph. These are also available in Neat, but require extra dependencies.
• This can be considered the first step of a full migration. At least two related problems may remain
  1. First, we might want to infer a more specific type than Asset, for example, Pump and LiftStation. This means adding information that is not explicitly set in the existing Asset Hierarchy. The type might be implicitly defined from the level in the hierarchy, or for example, the external ID of the asset. See part 2 for an example of how to add type in the migration process.
  2. We might want to map the inferred model onto an existing data model. It this case the existing model would be an EnterpriseModel and the inferred model we obtained here would be a Source model.

In [ ]: