Onboard DEXPI data to CDF¶
Prerequisite:
- Installed Neat, see Installation
- Launched a notebook environment.
- Familiar with the
NeatSessionobject, see introduction - Access to
NeatEngine.
In this tutorial, we will:
- extract and transform a knowledge graph from a
dexpiP&ID file for further usage in CDF - infer a data model based on the transformed knowledge graph
- and finally push both the data model and instances from knowledge graph to CDF
Read more about dexpi here: https://www.dexpi.org/
Extract knowledge graph from dexpi P&ID file¶
We will start by instantiating a NeatSession and read the data from a local folder.
Make sure you have a dexpi P&ID file in the folder. In case of this tutorial we are using a DEXPI file provided by DISC initiative. You can download file from this link (if download does not start right click and save link as).
In [1]:
Copied!
from cognite.neat import NeatSession, get_cognite_client
from cognite.neat import NeatSession, get_cognite_client
In [2]:
Copied!
# Note that we use Oxigraph in this example, this will not work in a CDF notebook
# to make it work in a CDF notebook replace set storage="memory"
neat = NeatSession(get_cognite_client(".env"), storage="oxigraph")
# Note that we use Oxigraph in this example, this will not work in a CDF notebook
# to make it work in a CDF notebook replace set storage="memory"
neat = NeatSession(get_cognite_client(".env"), storage="oxigraph")
Found .env file in repository root. Loaded variables from .env file. Neat Engine 2.0.3 loaded.
Process of extracting graph from a dexpi file can take a bit of time, so be patient.
In [3]:
Copied!
neat.read.xml.dexpi("./data.xml")
neat.read.xml.dexpi("./data.xml")
If we now check content of the graph object, we will see that it contains a knowledge graph extracted from the dexpi file:
In [5]:
Copied!
neat
neat
Out[5]:
Instances
Overview:
- 1 named graphs
- Total of 41 unique types
- 206 instances in default graph
default graph:
| Type | Occurrence | |
|---|---|---|
| 0 | PipingNetworkSegment | 29 |
| 1 | Nozzle | 23 |
| 2 | PipingNetworkSystem | 20 |
| 3 | Flange | 19 |
| 4 | ProcessInstrumentationFunction | 19 |
| ... | ... | ... |
| 40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to named graph urn:x-rdflib:default using DexpiExtractor
- Prunes the graph of triples where the object is a node that is not found in the graph.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Pruning the graph of triples where object is a node that is not found in graph.
If we run notebook locally we can preview a part of knowledge graph which should have ProcessPlant in its center:
In [6]:
Copied!
neat.show.instances()
neat.show.instances()
instances.html
Out[6]:
Infer Data Model¶
We can infer a data model from data in the NeatSession by calling .infer().
In [7]:
Copied!
neat.infer()
neat.infer()
Out[7]:
No issues found
In [9]:
Copied!
neat
neat
Out[9]:
Data Model
| type | Logical Data Model |
|---|---|
| intended for | Information Architect |
| name | Inferred Model |
| external_id | NeatInferredDataModel |
| version | v1 |
| classes | 41 |
| properties | 283 |
Instances
Overview:
- 1 named graphs
- Total of 41 unique types
- 206 instances in default graph
default graph:
| Type | Occurrence | |
|---|---|---|
| 0 | PipingNetworkSegment | 29 |
| 1 | Nozzle | 23 |
| 2 | PipingNetworkSystem | 20 |
| 3 | Flange | 19 |
| 4 | ProcessInstrumentationFunction | 19 |
| ... | ... | ... |
| 40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to named graph urn:x-rdflib:default using DexpiExtractor
- Prunes the graph of triples where the object is a node that is not found in the graph.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Pruning the graph of triples where object is a node that is not found in graph.
We now convert verified data model to be DMS compliant
In [10]:
Copied!
neat.convert()
neat.convert()
Rules converted to dms.
Out[10]:
Success: NEAT(verified,logical,neat_space,NeatInferredDataModel,v1) → NEAT(verified,physical,neat_space,NeatInferredDataModel,v1)
Inspect Data Model¶
First we will set data model id to ("dexpi_playground", "DEXPI", "v1.3.1"), where tuple represents (space, external_id, version) of data model
We use the .show to inspect the data model and see how data model objects are connected, the resulting interactive data model visualization should be similar to this one:
In [11]:
Copied!
neat.show.data_model()
neat.show.data_model()
http_purl.org_cognite_neat_data-model_verified_physical_neat_space_NeatInferredDataModel_v1.html
Out[11]:
In [12]:
Copied!
neat.set.data_model_id(("dexpi_playground_2", "DEXPI", "v1.3.1"))
neat.set.data_model_id(("dexpi_playground_2", "DEXPI", "v1.3.1"))
Out[12]:
Success: NEAT(verified,physical,neat_space,NeatInferredDataModel,v1) → NEAT(verified,physical,dexpi_playground_2,DEXPI,v1.3.1)
In [13]:
Copied!
neat
neat
Out[13]:
Data Model
| aspect | physical |
|---|---|
| intended for | DMS Architect |
| name | Inferred Model |
| space | dexpi_playground_2 |
| external_id | DEXPI |
| version | v1.3.1 |
| views | 41 |
| containers | 41 |
| properties | 283 |
Instances
Overview:
- 1 named graphs
- Total of 41 unique types
- 206 instances in default graph
default graph:
| Type | Occurrence | |
|---|---|---|
| 0 | PipingNetworkSegment | 29 |
| 1 | Nozzle | 23 |
| 2 | PipingNetworkSystem | 20 |
| 3 | Flange | 19 |
| 4 | ProcessInstrumentationFunction | 19 |
| ... | ... | ... |
| 40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to named graph urn:x-rdflib:default using DexpiExtractor
- Prunes the graph of triples where the object is a node that is not found in the graph.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Pruning the graph of triples where object is a node that is not found in graph.
- Added dict to urn:x-rdflib:default named graph
- Upsert prefixes to the name graph {named_graph}
- Added dict to urn:x-rdflib:default named graph
- Upsert prefixes to the name graph {named_graph}
Publishing Data Model to CDF¶
Now we are ready to publish this to CDF.
In [26]:
Copied!
neat.to.cdf.data_model()
neat.to.cdf.data_model()
You can inspect the details with the .inspect.outcome.data_model(...) method.
Out[26]:
| name | created | |
|---|---|---|
| 0 | spaces | 1 |
| 1 | containers | 41 |
| 2 | views | 41 |
| 3 | data_models | 1 |
| 4 | nodes | 0 |
Populating Data Model¶
Neat keeps track of the data, so we can immidiately populate this data model with the original data
In [27]:
Copied!
neat.to.cdf.instances()
neat.to.cdf.instances()
INFO | 2025-01-22 15:11:34,936 | Staring DMSLoader and will process 41 views. INFO | 2025-01-22 15:11:34,937 | Starting ViewId(space='dexpi_playground_2', external_id='ActuatingElectricalFunction', version='v1.3.1') 1/41. INFO | 2025-01-22 15:11:34,943 | Finished ViewId(space='dexpi_playground_2', external_id='ActuatingElectricalFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:35,048 | Starting ViewId(space='dexpi_playground_2', external_id='ActuatingFunction', version='v1.3.1') 2/41. INFO | 2025-01-22 15:11:35,054 | Finished ViewId(space='dexpi_playground_2', external_id='ActuatingFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:35,178 | Starting ViewId(space='dexpi_playground_2', external_id='ActuatingSystem', version='v1.3.1') 3/41. INFO | 2025-01-22 15:11:35,187 | Finished ViewId(space='dexpi_playground_2', external_id='ActuatingSystem', version='v1.3.1'). INFO | 2025-01-22 15:11:35,282 | Starting ViewId(space='dexpi_playground_2', external_id='BallValve', version='v1.3.1') 4/41. INFO | 2025-01-22 15:11:35,289 | Finished ViewId(space='dexpi_playground_2', external_id='BallValve', version='v1.3.1'). INFO | 2025-01-22 15:11:35,383 | Starting ViewId(space='dexpi_playground_2', external_id='BlindFlange', version='v1.3.1') 5/41. INFO | 2025-01-22 15:11:35,394 | Finished ViewId(space='dexpi_playground_2', external_id='BlindFlange', version='v1.3.1'). INFO | 2025-01-22 15:11:35,495 | Starting ViewId(space='dexpi_playground_2', external_id='ButterflyValve', version='v1.3.1') 6/41. INFO | 2025-01-22 15:11:35,503 | Finished ViewId(space='dexpi_playground_2', external_id='ButterflyValve', version='v1.3.1'). INFO | 2025-01-22 15:11:35,594 | Starting ViewId(space='dexpi_playground_2', external_id='CentrifugalPump', version='v1.3.1') 7/41. INFO | 2025-01-22 15:11:35,601 | Finished ViewId(space='dexpi_playground_2', external_id='CentrifugalPump', version='v1.3.1'). INFO | 2025-01-22 15:11:35,699 | Starting ViewId(space='dexpi_playground_2', external_id='CheckValve', version='v1.3.1') 8/41. INFO | 2025-01-22 15:11:35,708 | Finished ViewId(space='dexpi_playground_2', external_id='CheckValve', version='v1.3.1'). INFO | 2025-01-22 15:11:35,821 | Starting ViewId(space='dexpi_playground_2', external_id='ControlledActuator', version='v1.3.1') 9/41. INFO | 2025-01-22 15:11:35,833 | Finished ViewId(space='dexpi_playground_2', external_id='ControlledActuator', version='v1.3.1'). INFO | 2025-01-22 15:11:35,935 | Starting ViewId(space='dexpi_playground_2', external_id='CustomHeatExchanger', version='v1.3.1') 10/41. INFO | 2025-01-22 15:11:35,947 | Finished ViewId(space='dexpi_playground_2', external_id='CustomHeatExchanger', version='v1.3.1'). INFO | 2025-01-22 15:11:36,046 | Starting ViewId(space='dexpi_playground_2', external_id='CustomInlinePrimaryElement', version='v1.3.1') 11/41. INFO | 2025-01-22 15:11:36,057 | Finished ViewId(space='dexpi_playground_2', external_id='CustomInlinePrimaryElement', version='v1.3.1'). INFO | 2025-01-22 15:11:36,159 | Starting ViewId(space='dexpi_playground_2', external_id='CustomOperatedValve', version='v1.3.1') 12/41. INFO | 2025-01-22 15:11:36,172 | Finished ViewId(space='dexpi_playground_2', external_id='CustomOperatedValve', version='v1.3.1'). INFO | 2025-01-22 15:11:36,272 | Starting ViewId(space='dexpi_playground_2', external_id='CustomPipingComponent', version='v1.3.1') 13/41. INFO | 2025-01-22 15:11:36,285 | Finished ViewId(space='dexpi_playground_2', external_id='CustomPipingComponent', version='v1.3.1'). INFO | 2025-01-22 15:11:36,379 | Starting ViewId(space='dexpi_playground_2', external_id='ElectricHeater', version='v1.3.1') 14/41. INFO | 2025-01-22 15:11:36,391 | Finished ViewId(space='dexpi_playground_2', external_id='ElectricHeater', version='v1.3.1'). INFO | 2025-01-22 15:11:36,482 | Starting ViewId(space='dexpi_playground_2', external_id='Flange', version='v1.3.1') 15/41. INFO | 2025-01-22 15:11:36,496 | Finished ViewId(space='dexpi_playground_2', external_id='Flange', version='v1.3.1'). INFO | 2025-01-22 15:11:36,637 | Starting ViewId(space='dexpi_playground_2', external_id='FlowInPipeOffPageConnector', version='v1.3.1') 16/41. INFO | 2025-01-22 15:11:36,650 | Finished ViewId(space='dexpi_playground_2', external_id='FlowInPipeOffPageConnector', version='v1.3.1'). INFO | 2025-01-22 15:11:36,747 | Starting ViewId(space='dexpi_playground_2', external_id='FlowOutPipeOffPageConnector', version='v1.3.1') 17/41. INFO | 2025-01-22 15:11:36,758 | Finished ViewId(space='dexpi_playground_2', external_id='FlowOutPipeOffPageConnector', version='v1.3.1'). INFO | 2025-01-22 15:11:36,853 | Starting ViewId(space='dexpi_playground_2', external_id='FlowOutSignalOffPageConnector', version='v1.3.1') 18/41. INFO | 2025-01-22 15:11:36,865 | Finished ViewId(space='dexpi_playground_2', external_id='FlowOutSignalOffPageConnector', version='v1.3.1'). INFO | 2025-01-22 15:11:36,960 | Starting ViewId(space='dexpi_playground_2', external_id='GateValve', version='v1.3.1') 19/41. INFO | 2025-01-22 15:11:36,972 | Finished ViewId(space='dexpi_playground_2', external_id='GateValve', version='v1.3.1'). INFO | 2025-01-22 15:11:37,075 | Starting ViewId(space='dexpi_playground_2', external_id='InstrumentationLoopFunction', version='v1.3.1') 20/41. INFO | 2025-01-22 15:11:37,092 | Finished ViewId(space='dexpi_playground_2', external_id='InstrumentationLoopFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:37,223 | Starting ViewId(space='dexpi_playground_2', external_id='MeasuringLineFunction', version='v1.3.1') 21/41. INFO | 2025-01-22 15:11:37,233 | Finished ViewId(space='dexpi_playground_2', external_id='MeasuringLineFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:37,350 | Starting ViewId(space='dexpi_playground_2', external_id='MetaData', version='v1.3.1') 22/41. INFO | 2025-01-22 15:11:37,360 | Finished ViewId(space='dexpi_playground_2', external_id='MetaData', version='v1.3.1'). INFO | 2025-01-22 15:11:37,500 | Starting ViewId(space='dexpi_playground_2', external_id='Note', version='v1.3.1') 23/41. INFO | 2025-01-22 15:11:37,508 | Finished ViewId(space='dexpi_playground_2', external_id='Note', version='v1.3.1'). INFO | 2025-01-22 15:11:37,629 | Starting ViewId(space='dexpi_playground_2', external_id='Nozzle', version='v1.3.1') 24/41. INFO | 2025-01-22 15:11:37,654 | Finished ViewId(space='dexpi_playground_2', external_id='Nozzle', version='v1.3.1'). INFO | 2025-01-22 15:11:37,767 | Starting ViewId(space='dexpi_playground_2', external_id='OperatedValveReference', version='v1.3.1') 25/41. INFO | 2025-01-22 15:11:37,772 | Finished ViewId(space='dexpi_playground_2', external_id='OperatedValveReference', version='v1.3.1'). INFO | 2025-01-22 15:11:37,869 | Starting ViewId(space='dexpi_playground_2', external_id='PipeOffPageConnectorReferenceByNumber', version='v1.3.1') 26/41. INFO | 2025-01-22 15:11:37,874 | Finished ViewId(space='dexpi_playground_2', external_id='PipeOffPageConnectorReferenceByNumber', version='v1.3.1'). INFO | 2025-01-22 15:11:37,963 | Starting ViewId(space='dexpi_playground_2', external_id='PipeReducer', version='v1.3.1') 27/41. INFO | 2025-01-22 15:11:37,970 | Finished ViewId(space='dexpi_playground_2', external_id='PipeReducer', version='v1.3.1'). INFO | 2025-01-22 15:11:38,066 | Starting ViewId(space='dexpi_playground_2', external_id='PipeTee', version='v1.3.1') 28/41. INFO | 2025-01-22 15:11:38,081 | Finished ViewId(space='dexpi_playground_2', external_id='PipeTee', version='v1.3.1'). INFO | 2025-01-22 15:11:38,252 | Starting ViewId(space='dexpi_playground_2', external_id='PipingNetworkSegment', version='v1.3.1') 29/41. INFO | 2025-01-22 15:11:38,270 | Finished ViewId(space='dexpi_playground_2', external_id='PipingNetworkSegment', version='v1.3.1'). INFO | 2025-01-22 15:11:38,401 | Starting ViewId(space='dexpi_playground_2', external_id='PipingNetworkSystem', version='v1.3.1') 30/41. INFO | 2025-01-22 15:11:38,416 | Finished ViewId(space='dexpi_playground_2', external_id='PipingNetworkSystem', version='v1.3.1'). INFO | 2025-01-22 15:11:38,534 | Starting ViewId(space='dexpi_playground_2', external_id='PlantSystem', version='v1.3.1') 31/41. INFO | 2025-01-22 15:11:38,545 | Finished ViewId(space='dexpi_playground_2', external_id='PlantSystem', version='v1.3.1'). INFO | 2025-01-22 15:11:38,760 | Starting ViewId(space='dexpi_playground_2', external_id='PlateHeatExchanger', version='v1.3.1') 32/41. INFO | 2025-01-22 15:11:38,772 | Finished ViewId(space='dexpi_playground_2', external_id='PlateHeatExchanger', version='v1.3.1'). INFO | 2025-01-22 15:11:38,879 | Starting ViewId(space='dexpi_playground_2', external_id='PressureVessel', version='v1.3.1') 33/41. INFO | 2025-01-22 15:11:38,887 | Finished ViewId(space='dexpi_playground_2', external_id='PressureVessel', version='v1.3.1'). INFO | 2025-01-22 15:11:39,098 | Starting ViewId(space='dexpi_playground_2', external_id='ProcessInstrumentationFunction', version='v1.3.1') 34/41. INFO | 2025-01-22 15:11:39,126 | Finished ViewId(space='dexpi_playground_2', external_id='ProcessInstrumentationFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:39,315 | Starting ViewId(space='dexpi_playground_2', external_id='ProcessPlant', version='v1.3.1') 35/41. INFO | 2025-01-22 15:11:39,322 | Finished ViewId(space='dexpi_playground_2', external_id='ProcessPlant', version='v1.3.1'). INFO | 2025-01-22 15:11:39,434 | Starting ViewId(space='dexpi_playground_2', external_id='ProcessSignalGeneratingFunction', version='v1.3.1') 36/41. INFO | 2025-01-22 15:11:39,447 | Finished ViewId(space='dexpi_playground_2', external_id='ProcessSignalGeneratingFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:39,547 | Starting ViewId(space='dexpi_playground_2', external_id='PropertyBreak', version='v1.3.1') 37/41. INFO | 2025-01-22 15:11:39,556 | Finished ViewId(space='dexpi_playground_2', external_id='PropertyBreak', version='v1.3.1'). INFO | 2025-01-22 15:11:39,651 | Starting ViewId(space='dexpi_playground_2', external_id='RotaryPump', version='v1.3.1') 38/41. INFO | 2025-01-22 15:11:39,661 | Finished ViewId(space='dexpi_playground_2', external_id='RotaryPump', version='v1.3.1'). INFO | 2025-01-22 15:11:39,754 | Starting ViewId(space='dexpi_playground_2', external_id='SafetyValveOrFitting', version='v1.3.1') 39/41. INFO | 2025-01-22 15:11:39,766 | Finished ViewId(space='dexpi_playground_2', external_id='SafetyValveOrFitting', version='v1.3.1'). INFO | 2025-01-22 15:11:39,859 | Starting ViewId(space='dexpi_playground_2', external_id='SignalConveyingFunction', version='v1.3.1') 40/41. INFO | 2025-01-22 15:11:39,874 | Finished ViewId(space='dexpi_playground_2', external_id='SignalConveyingFunction', version='v1.3.1'). INFO | 2025-01-22 15:11:39,986 | Starting ViewId(space='dexpi_playground_2', external_id='SignalOffPageConnectorReferenceByNumber', version='v1.3.1') 41/41. INFO | 2025-01-22 15:11:39,999 | Finished ViewId(space='dexpi_playground_2', external_id='SignalOffPageConnectorReferenceByNumber', version='v1.3.1').
You can inspect the details with the .inspect.outcome.instances(...) method.
Out[27]:
| name | created | changed | |
|---|---|---|---|
| 0 | ActuatingElectricalFunction | 1 | 0 |
| 1 | ActuatingFunction | 4 | 0 |
| 2 | edge | 40 | 0 |
| 3 | ActuatingSystem | 0 | 4 |
| 4 | BallValve | 3 | 0 |
| 5 | BlindFlange | 1 | 0 |
| 6 | ButterflyValve | 1 | 0 |
| 7 | CentrifugalPump | 1 | 0 |
| 8 | CheckValve | 1 | 0 |
| 9 | ControlledActuator | 4 | 0 |
| 10 | CustomHeatExchanger | 1 | 0 |
| 11 | CustomInlinePrimaryElement | 2 | 0 |
| 12 | CustomOperatedValve | 1 | 0 |
| 13 | CustomPipingComponent | 2 | 0 |
| 14 | ElectricHeater | 1 | 0 |
| 15 | Flange | 19 | 0 |
| 16 | FlowInPipeOffPageConnector | 1 | 0 |
| 17 | FlowOutPipeOffPageConnector | 1 | 0 |
| 18 | FlowOutSignalOffPageConnector | 1 | 0 |
| 19 | GateValve | 5 | 0 |
| 20 | InstrumentationLoopFunction | 8 | 0 |
| 21 | MeasuringLineFunction | 5 | 0 |
| 22 | MetaData | 1 | 0 |
| 23 | Note | 1 | 0 |
| 24 | Nozzle | 22 | 1 |
| 25 | OperatedValveReference | 0 | 4 |
| 26 | PipeOffPageConnectorReferenceByNumber | 2 | 0 |
| 27 | PipeReducer | 1 | 0 |
| 28 | PipeTee | 4 | 2 |
| 29 | PipingNetworkSegment | 29 | 0 |
| 30 | PipingNetworkSystem | 20 | 0 |
| 31 | PlantSystem | 0 | 1 |
| 32 | PlateHeatExchanger | 1 | 0 |
| 33 | PressureVessel | 0 | 1 |
| 34 | ProcessInstrumentationFunction | 3 | 16 |
| 35 | ProcessPlant | 0 | 1 |
| 36 | ProcessSignalGeneratingFunction | 0 | 5 |
| 37 | PropertyBreak | 1 | 1 |
| 38 | RotaryPump | 0 | 1 |
| 39 | SafetyValveOrFitting | 1 | 0 |
| 40 | SignalConveyingFunction | 0 | 19 |
| 41 | SignalOffPageConnectorReferenceByNumber | 1 | 0 |
In [ ]:
Copied!