Onboard DEXPI data to CDF¶
Prerequisite:
- Installed Neat, see Installation
- Launched a notebook environment.
- Familiar with the
NeatSession
object, see introduction - Access to
NeatEngine
.
In this tutorial, we will:
- extract a knowledge graph from a
dexpi
P&ID file - transform knowledge graph for further usage in CDF
- infer a data model based on the transformed knowledge graph
- and finally push both the data model and instances from knowledge graph to CDF
Read more about dexpi here: https://www.dexpi.org/
Extract knowledge graph from dexpi P&ID file¶
We will start by instantiating a NeatSession
and read the data from a local folder.
Make sure you have a dexpi P&ID file in the folder. In case of this tutorial we are using a DEXPI file provided by DISC initiative. You can download file from this link (if download does not start right click and save link as).
from cognite.neat import NeatSession, get_cognite_client
# Note that we use Oxigraph in this example, this will not work in a CDF notebook
# to make it work in a CDF notebook replace set storage="memory"
neat = NeatSession(get_cognite_client(".env"), storage="oxigraph")
Found .env file in repository root. Loaded variables from .env file. Neat Engine 2.0.3 loaded.
Process of extracting graph from a dexpi file can take a bit of time, so be patient.
neat.read.xml.dexpi("./data.xml")
If we now check content of the graph
object, we will see that it contains a knowledge graph extracted from the dexpi file:
neat
Instances
Overview:
- 43 types
- 1306 instances
Type | Occurrence | |
---|---|---|
0 | GenericAttribute | 770 |
1 | Association | 330 |
2 | PipingNetworkSegment | 29 |
3 | Nozzle | 23 |
4 | PipingNetworkSystem | 20 |
... | ... | ... |
42 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to graph store using DexpiExtractor
Now we will tranform this graph by calling dexpi specific bundle of transformers, which will:
- attach values of generic attributes to nodes
- create associations between nodes
- remove unused generic attributes
- remove associations between nodes that do not exist in the extracted graph
- remove edges to nodes that do not exist in the extracted graph
neat.prepare.instances.dexpi()
If we run notebook locally we can preview a part of knowledge graph which should have ProcessPlant
in its center:
neat.show.instances()
instances.html
If we now inspect content of neat session we can notice that we went from 1306 to 206 instances, since we perfomed all the above transformations in order to produce rich and well connected graph.
Observ that number of types went only from 43 to 41, and that is because we move information from GenericAttribute
and Association
to correct nodes, making new properties and edges.
These two types where then consecutively removed as all their information has been used.
neat
Instances
Overview:
- 41 types
- 206 instances
Type | Occurrence | |
---|---|---|
0 | PipingNetworkSegment | 29 |
1 | Nozzle | 23 |
2 | PipingNetworkSystem | 20 |
3 | Flange | 19 |
4 | ProcessInstrumentationFunction | 19 |
... | ... | ... |
40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to graph store using DexpiExtractor
- Prunes the graph of triples where the object is a node that is not found in the graph.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Pruning the graph of triples where object is a node that is not found in graph.
Studying the output above, we see that we succesfully read the assets and activities.
Infer Data Model¶
We can infer a data model from data in the NeatSession
by calling .infer()
.
neat.infer()
Inferring classes: 0%| | 0/41 [00:00<?, ?it/s]
Success: Inferred UnverifiedInformationModel
neat
Unverified Data Model
type | Logical Data Model |
---|---|
intended for | Information Architect |
name | Inferred Model |
external_id | NeatInferredDataModel |
space | neat_space |
version | v1 |
classes | 41 |
properties | 283 |
Instances
Overview:
- 41 types
- 206 instances
Type | Occurrence | |
---|---|---|
0 | PipingNetworkSegment | 29 |
1 | Nozzle | 23 |
2 | PipingNetworkSystem | 20 |
3 | Flange | 19 |
4 | ProcessInstrumentationFunction | 19 |
... | ... | ... |
40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to graph store using DexpiExtractor
- Prunes the graph of specified rdf types that do not have connections to other nodes.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Prunes the graph of specified rdf types that do not have connections to other nodes.
This gives us an unverified data model, which we can then verify
Verify Data Model¶
neat.verify()
Success: UnverifiedInformationModel → VerifiedInformationModel
neat
Verified Data Model
type | Logical Data Model |
---|---|
intended for | Information Architect |
name | Inferred Model |
external_id | NeatInferredDataModel |
version | v1 |
classes | 41 |
properties | 283 |
Instances
Overview:
- 41 types
- 206 instances
Type | Occurrence | |
---|---|---|
0 | PipingNetworkSegment | 29 |
1 | Nozzle | 23 |
2 | PipingNetworkSystem | 20 |
3 | Flange | 19 |
4 | ProcessInstrumentationFunction | 19 |
... | ... | ... |
40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to graph store using DexpiExtractor
- Prunes the graph of specified rdf types that do not have connections to other nodes.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Prunes the graph of specified rdf types that do not have connections to other nodes.
- Added rules to graph store as InformationRules
- Upsert prefixes to graph store
We now convert verified data model to be DMS compliant
neat.convert("dms")
Rules converted to dms
Success: VerifiedInformationModel → VerifiedDMSModel
Inspect Data Model¶
First we will set data model id to ("dexpi_playground", "DEXPI", "v1.3.1"), where tuple represents (space, external_id, version) of data model
We use the .show
to inspect the data model and see how data model objects are connected, the resulting interactive data model visualization should be similar to this one:
neat.show.data_model()
http_purl.org_cognite_neat_data-model_verified_physical_neat_space_NeatInferredDataModel_v1.html
neat.set.data_model_id(("dexpi_playground", "DEXPI", "v1.3.1"))
Success: VerifiedDMSModel → VerifiedDMSModel
neat
Verified Data Model
aspect | physical |
---|---|
intended for | DMS Architect |
name | Inferred Model |
space | dexpi_playground |
external_id | DEXPI |
version | v1.3.1 |
views | 41 |
containers | 41 |
properties | 283 |
Instances
Overview:
- 41 types
- 206 instances
Type | Occurrence | |
---|---|---|
0 | PipingNetworkSegment | 29 |
1 | Nozzle | 23 |
2 | PipingNetworkSystem | 20 |
3 | Flange | 19 |
4 | ProcessInstrumentationFunction | 19 |
... | ... | ... |
40 | PipeReducer | 1 |
Provenance:
- Initialize graph store as OxigraphStore
- Extracted triples to graph store using DexpiExtractor
- Prunes the graph of specified rdf types that do not have connections to other nodes.
- Attaches a target property from a target node that is connected to a source node.
- Attaches a target property from a target node that is connected to a source node.
- Prunes nodes of specific rdf types
- Prunes the graph of specified rdf types that do not have connections to other nodes.
- Added rules to graph store as InformationRules
- Upsert prefixes to graph store
Publishing Data Model to CDF¶
Now we are ready to publish this to CDF.
neat.to.cdf.data_model()
You can inspect the details with the .inspect.outcome.data_model(...) method.
name | created | |
---|---|---|
0 | schema | 0 |
1 | spaces | 1 |
2 | containers | 41 |
3 | views | 41 |
4 | data_models | 1 |
5 | nodes | 0 |
Populating Data Model¶
Neat keeps track of the data, so we can immidiately populate this data model with the original data
neat.to.cdf.instances()
INFO | 2024-12-09 12:04:54,228 | Staring DMSLoader and will process 41 views. INFO | 2024-12-09 12:04:54,230 | Starting ViewId(space='dexpi_playground', external_id='ActuatingElectricalFunction', version='v1.3.1') 1/41. INFO | 2024-12-09 12:04:54,248 | Finished ViewId(space='dexpi_playground', external_id='ActuatingElectricalFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,249 | Starting ViewId(space='dexpi_playground', external_id='ActuatingFunction', version='v1.3.1') 2/41. INFO | 2024-12-09 12:04:54,257 | Finished ViewId(space='dexpi_playground', external_id='ActuatingFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,258 | Starting ViewId(space='dexpi_playground', external_id='ActuatingSystem', version='v1.3.1') 3/41. INFO | 2024-12-09 12:04:54,265 | Finished ViewId(space='dexpi_playground', external_id='ActuatingSystem', version='v1.3.1'). INFO | 2024-12-09 12:04:54,265 | Starting ViewId(space='dexpi_playground', external_id='BallValve', version='v1.3.1') 4/41. INFO | 2024-12-09 12:04:54,273 | Finished ViewId(space='dexpi_playground', external_id='BallValve', version='v1.3.1'). INFO | 2024-12-09 12:04:54,273 | Starting ViewId(space='dexpi_playground', external_id='BlindFlange', version='v1.3.1') 5/41. INFO | 2024-12-09 12:04:54,279 | Finished ViewId(space='dexpi_playground', external_id='BlindFlange', version='v1.3.1'). INFO | 2024-12-09 12:04:54,279 | Starting ViewId(space='dexpi_playground', external_id='ButterflyValve', version='v1.3.1') 6/41. INFO | 2024-12-09 12:04:54,285 | Finished ViewId(space='dexpi_playground', external_id='ButterflyValve', version='v1.3.1'). INFO | 2024-12-09 12:04:54,285 | Starting ViewId(space='dexpi_playground', external_id='CentrifugalPump', version='v1.3.1') 7/41. INFO | 2024-12-09 12:04:54,292 | Finished ViewId(space='dexpi_playground', external_id='CentrifugalPump', version='v1.3.1'). INFO | 2024-12-09 12:04:54,292 | Starting ViewId(space='dexpi_playground', external_id='CheckValve', version='v1.3.1') 8/41. INFO | 2024-12-09 12:04:54,298 | Finished ViewId(space='dexpi_playground', external_id='CheckValve', version='v1.3.1'). INFO | 2024-12-09 12:04:54,298 | Starting ViewId(space='dexpi_playground', external_id='ControlledActuator', version='v1.3.1') 9/41. INFO | 2024-12-09 12:04:54,304 | Finished ViewId(space='dexpi_playground', external_id='ControlledActuator', version='v1.3.1'). INFO | 2024-12-09 12:04:54,304 | Starting ViewId(space='dexpi_playground', external_id='CustomHeatExchanger', version='v1.3.1') 10/41. INFO | 2024-12-09 12:04:54,310 | Finished ViewId(space='dexpi_playground', external_id='CustomHeatExchanger', version='v1.3.1'). INFO | 2024-12-09 12:04:54,310 | Starting ViewId(space='dexpi_playground', external_id='CustomInlinePrimaryElement', version='v1.3.1') 11/41. INFO | 2024-12-09 12:04:54,316 | Finished ViewId(space='dexpi_playground', external_id='CustomInlinePrimaryElement', version='v1.3.1'). INFO | 2024-12-09 12:04:54,317 | Starting ViewId(space='dexpi_playground', external_id='CustomOperatedValve', version='v1.3.1') 12/41. INFO | 2024-12-09 12:04:54,322 | Finished ViewId(space='dexpi_playground', external_id='CustomOperatedValve', version='v1.3.1'). INFO | 2024-12-09 12:04:54,322 | Starting ViewId(space='dexpi_playground', external_id='CustomPipingComponent', version='v1.3.1') 13/41. INFO | 2024-12-09 12:04:54,426 | Finished ViewId(space='dexpi_playground', external_id='CustomPipingComponent', version='v1.3.1'). INFO | 2024-12-09 12:04:54,426 | Starting ViewId(space='dexpi_playground', external_id='ElectricHeater', version='v1.3.1') 14/41. INFO | 2024-12-09 12:04:54,431 | Finished ViewId(space='dexpi_playground', external_id='ElectricHeater', version='v1.3.1'). INFO | 2024-12-09 12:04:54,431 | Starting ViewId(space='dexpi_playground', external_id='Flange', version='v1.3.1') 15/41. INFO | 2024-12-09 12:04:54,438 | Finished ViewId(space='dexpi_playground', external_id='Flange', version='v1.3.1'). INFO | 2024-12-09 12:04:54,438 | Starting ViewId(space='dexpi_playground', external_id='FlowInPipeOffPageConnector', version='v1.3.1') 16/41. INFO | 2024-12-09 12:04:54,442 | Finished ViewId(space='dexpi_playground', external_id='FlowInPipeOffPageConnector', version='v1.3.1'). INFO | 2024-12-09 12:04:54,442 | Starting ViewId(space='dexpi_playground', external_id='FlowOutPipeOffPageConnector', version='v1.3.1') 17/41. INFO | 2024-12-09 12:04:54,446 | Finished ViewId(space='dexpi_playground', external_id='FlowOutPipeOffPageConnector', version='v1.3.1'). INFO | 2024-12-09 12:04:54,446 | Starting ViewId(space='dexpi_playground', external_id='FlowOutSignalOffPageConnector', version='v1.3.1') 18/41. INFO | 2024-12-09 12:04:54,450 | Finished ViewId(space='dexpi_playground', external_id='FlowOutSignalOffPageConnector', version='v1.3.1'). INFO | 2024-12-09 12:04:54,450 | Starting ViewId(space='dexpi_playground', external_id='GateValve', version='v1.3.1') 19/41. INFO | 2024-12-09 12:04:54,456 | Finished ViewId(space='dexpi_playground', external_id='GateValve', version='v1.3.1'). INFO | 2024-12-09 12:04:54,456 | Starting ViewId(space='dexpi_playground', external_id='InstrumentationLoopFunction', version='v1.3.1') 20/41. INFO | 2024-12-09 12:04:54,462 | Finished ViewId(space='dexpi_playground', external_id='InstrumentationLoopFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,462 | Starting ViewId(space='dexpi_playground', external_id='MeasuringLineFunction', version='v1.3.1') 21/41. INFO | 2024-12-09 12:04:54,467 | Finished ViewId(space='dexpi_playground', external_id='MeasuringLineFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,467 | Starting ViewId(space='dexpi_playground', external_id='MetaData', version='v1.3.1') 22/41. INFO | 2024-12-09 12:04:54,473 | Finished ViewId(space='dexpi_playground', external_id='MetaData', version='v1.3.1'). INFO | 2024-12-09 12:04:54,473 | Starting ViewId(space='dexpi_playground', external_id='Note', version='v1.3.1') 23/41. INFO | 2024-12-09 12:04:54,478 | Finished ViewId(space='dexpi_playground', external_id='Note', version='v1.3.1'). INFO | 2024-12-09 12:04:54,478 | Starting ViewId(space='dexpi_playground', external_id='Nozzle', version='v1.3.1') 24/41. INFO | 2024-12-09 12:04:54,488 | Finished ViewId(space='dexpi_playground', external_id='Nozzle', version='v1.3.1'). INFO | 2024-12-09 12:04:54,489 | Starting ViewId(space='dexpi_playground', external_id='OperatedValveReference', version='v1.3.1') 25/41. INFO | 2024-12-09 12:04:54,493 | Finished ViewId(space='dexpi_playground', external_id='OperatedValveReference', version='v1.3.1'). INFO | 2024-12-09 12:04:54,494 | Starting ViewId(space='dexpi_playground', external_id='PipeOffPageConnectorReferenceByNumber', version='v1.3.1') 26/41. INFO | 2024-12-09 12:04:54,498 | Finished ViewId(space='dexpi_playground', external_id='PipeOffPageConnectorReferenceByNumber', version='v1.3.1'). INFO | 2024-12-09 12:04:54,498 | Starting ViewId(space='dexpi_playground', external_id='PipeReducer', version='v1.3.1') 27/41. INFO | 2024-12-09 12:04:54,516 | Finished ViewId(space='dexpi_playground', external_id='PipeReducer', version='v1.3.1'). INFO | 2024-12-09 12:04:54,516 | Starting ViewId(space='dexpi_playground', external_id='PipeTee', version='v1.3.1') 28/41. INFO | 2024-12-09 12:04:54,526 | Finished ViewId(space='dexpi_playground', external_id='PipeTee', version='v1.3.1'). INFO | 2024-12-09 12:04:54,526 | Starting ViewId(space='dexpi_playground', external_id='PipingNetworkSegment', version='v1.3.1') 29/41. INFO | 2024-12-09 12:04:54,536 | Finished ViewId(space='dexpi_playground', external_id='PipingNetworkSegment', version='v1.3.1'). INFO | 2024-12-09 12:04:54,536 | Starting ViewId(space='dexpi_playground', external_id='PipingNetworkSystem', version='v1.3.1') 30/41. INFO | 2024-12-09 12:04:54,545 | Finished ViewId(space='dexpi_playground', external_id='PipingNetworkSystem', version='v1.3.1'). INFO | 2024-12-09 12:04:54,546 | Starting ViewId(space='dexpi_playground', external_id='PlantSystem', version='v1.3.1') 31/41. INFO | 2024-12-09 12:04:54,550 | Finished ViewId(space='dexpi_playground', external_id='PlantSystem', version='v1.3.1'). INFO | 2024-12-09 12:04:54,551 | Starting ViewId(space='dexpi_playground', external_id='PlateHeatExchanger', version='v1.3.1') 32/41. INFO | 2024-12-09 12:04:54,556 | Finished ViewId(space='dexpi_playground', external_id='PlateHeatExchanger', version='v1.3.1'). INFO | 2024-12-09 12:04:54,556 | Starting ViewId(space='dexpi_playground', external_id='PressureVessel', version='v1.3.1') 33/41. INFO | 2024-12-09 12:04:54,562 | Finished ViewId(space='dexpi_playground', external_id='PressureVessel', version='v1.3.1'). INFO | 2024-12-09 12:04:54,562 | Starting ViewId(space='dexpi_playground', external_id='ProcessInstrumentationFunction', version='v1.3.1') 34/41. INFO | 2024-12-09 12:04:54,573 | Finished ViewId(space='dexpi_playground', external_id='ProcessInstrumentationFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,573 | Starting ViewId(space='dexpi_playground', external_id='ProcessPlant', version='v1.3.1') 35/41. INFO | 2024-12-09 12:04:54,578 | Finished ViewId(space='dexpi_playground', external_id='ProcessPlant', version='v1.3.1'). INFO | 2024-12-09 12:04:54,578 | Starting ViewId(space='dexpi_playground', external_id='ProcessSignalGeneratingFunction', version='v1.3.1') 36/41. INFO | 2024-12-09 12:04:54,583 | Finished ViewId(space='dexpi_playground', external_id='ProcessSignalGeneratingFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,583 | Starting ViewId(space='dexpi_playground', external_id='PropertyBreak', version='v1.3.1') 37/41. INFO | 2024-12-09 12:04:54,589 | Finished ViewId(space='dexpi_playground', external_id='PropertyBreak', version='v1.3.1'). INFO | 2024-12-09 12:04:54,589 | Starting ViewId(space='dexpi_playground', external_id='RotaryPump', version='v1.3.1') 38/41. INFO | 2024-12-09 12:04:54,594 | Finished ViewId(space='dexpi_playground', external_id='RotaryPump', version='v1.3.1'). INFO | 2024-12-09 12:04:54,594 | Starting ViewId(space='dexpi_playground', external_id='SafetyValveOrFitting', version='v1.3.1') 39/41. INFO | 2024-12-09 12:04:54,599 | Finished ViewId(space='dexpi_playground', external_id='SafetyValveOrFitting', version='v1.3.1'). INFO | 2024-12-09 12:04:54,599 | Starting ViewId(space='dexpi_playground', external_id='SignalConveyingFunction', version='v1.3.1') 40/41. INFO | 2024-12-09 12:04:54,606 | Finished ViewId(space='dexpi_playground', external_id='SignalConveyingFunction', version='v1.3.1'). INFO | 2024-12-09 12:04:54,607 | Starting ViewId(space='dexpi_playground', external_id='SignalOffPageConnectorReferenceByNumber', version='v1.3.1') 41/41. INFO | 2024-12-09 12:04:54,611 | Finished ViewId(space='dexpi_playground', external_id='SignalOffPageConnectorReferenceByNumber', version='v1.3.1').
You can inspect the details with the .inspect.outcome.instances(...) method.
name | created | |
---|---|---|
0 | ActuatingElectricalFunction | 1 |
1 | ActuatingFunction | 4 |
2 | ActuatingSystem | 4 |
3 | BallValve | 3 |
4 | BlindFlange | 1 |
5 | ButterflyValve | 1 |
6 | CentrifugalPump | 1 |
7 | CheckValve | 1 |
8 | ControlledActuator | 4 |
9 | CustomHeatExchanger | 1 |
10 | CustomInlinePrimaryElement | 2 |
11 | CustomOperatedValve | 1 |
12 | CustomPipingComponent | 2 |
13 | ElectricHeater | 1 |
14 | Flange | 19 |
15 | FlowInPipeOffPageConnector | 1 |
16 | FlowOutPipeOffPageConnector | 1 |
17 | FlowOutSignalOffPageConnector | 1 |
18 | GateValve | 5 |
19 | InstrumentationLoopFunction | 8 |
20 | MeasuringLineFunction | 5 |
21 | MetaData | 1 |
22 | Note | 1 |
23 | Nozzle | 23 |
24 | OperatedValveReference | 4 |
25 | PipeOffPageConnectorReferenceByNumber | 2 |
26 | PipeReducer | 1 |
27 | PipeTee | 6 |
28 | PipingNetworkSegment | 29 |
29 | PipingNetworkSystem | 20 |
30 | PlantSystem | 1 |
31 | PlateHeatExchanger | 1 |
32 | PressureVessel | 1 |
33 | ProcessInstrumentationFunction | 19 |
34 | ProcessPlant | 1 |
35 | ProcessSignalGeneratingFunction | 5 |
36 | PropertyBreak | 2 |
37 | RotaryPump | 1 |
38 | SafetyValveOrFitting | 1 |
39 | SignalConveyingFunction | 19 |
40 | SignalOffPageConnectorReferenceByNumber | 1 |
41 | edge | 40 |