Onboard AutomationML (AML) data to CDF¶

Prerequisite:

• Installed Neat, see Installation
• Launched a notebook environment.
• Familiar with the NeatSession object, see introduction
• Access to NeatEngine.

In this tutorial, we will:

• extract a knowledge graph from a aml topology file
• transform knowledge graph for further usage in CDF
• infer a data model based on the transformed knowledge graph
• and finally push both the data model and instances from knowledge graph to CDF

Read more about aml here: https://www.automationml.org/

Extract knowledge graph from AML topology file¶

We will start by instantiating a NeatSession and read the data from a local folder. Make sure you have an AML file in the folder. In case of this tutorial we are using a AML file provided by DISC initiative.

In [1]:

from cognite.neat import NeatSession, get_cognite_client

from cognite.neat import NeatSession, get_cognite_client

In [2]:

# Note that we use Oxigraph in this example, this will not work in a CDF notebook
# to make it work in a CDF notebook replace set storage="memory"

neat = NeatSession(get_cognite_client(".env"), storage="oxigraph")

# Note that we use Oxigraph in this example, this will not work in a CDF notebook # to make it work in a CDF notebook replace set storage="memory" neat = NeatSession(get_cognite_client(".env"), storage="oxigraph")

Found .env file in repository root. Loaded variables from .env file.
Neat Engine 2.0.3 loaded.

Process of extracting graph from an aml file can take a bit of time, so be patient.

In [3]:

neat.read.xml.aml("data.aml")

neat.read.xml.aml("data.aml")

If we now check content of the graph object, we will see that it contains a knowledge graph extracted from the aml file:

In [5]:

neat

neat

Out[5]:

Instances

Overview:

• 4 types
• 5662 instances

	Type	Occurrence
0	Attribute	4689
1	ExternalInterface	858
2	InternalElement	67
3	InternalLink	48

Provenance:

• Initialize graph store as OxigraphStore
• Extracted triples to graph store using AmlExtractor

Now we will tranform this graph by calling aml specific bundle of transformers, which will:

• attach values of attributes to nodes
• remove unused attributes
• remove edges to nodes that do not exist in the extracted graph

In [6]:

neat.prepare.instances.aml()

neat.prepare.instances.aml()

Prunes nodes of specific rdf types:   0%|          | 0/1631 [00:00<?, ?it/s]

Pruning the graph of triples where object is a node that is not found in graph.:   0%|          | 0/1631 [00:0…

If we run notebook locally we can preview a part of knowledge graph. The entire knowledge graph for this specific file we used look like this (while you will see only fragment of it in notebook):

In [7]:

neat.show.instances();

neat.show.instances();

instances.html

If we now inspect content of neat session we can notice that we went from 5662 to 973 instances, since we perfomed all the above transformations in order to produce rich and well connected graph. Observ that number of types went only from 4 to 3, and that is because we moved information from Attributes to correct nodes, making new properties and edges. This type was then consecutively removed as all their information has been used.

In [7]:

neat

neat

Out[7]:

Instances

Overview:

• 3 types
• 973 instances

	Type	Occurrence
0	ExternalInterface	858
1	InternalElement	67
2	InternalLink	48

Provenance:

• Initialize graph store as OxigraphStore
• Extracted triples to graph store using AmlExtractor
• Prunes the graph of triples where the object is a node that is not found in the graph.
• Attaches a target property from a target node that is connected to a source node.
• Prunes nodes of specific rdf types
• Pruning the graph of triples where object is a node that is not found in graph.

Studying the output above, we see that we succesfully read the assets and activities.

Infer Data Model¶

We can infer a data model from data in the NeatSession by calling .infer().

In [8]:

neat.infer()

neat.infer()

Inferring classes:   0%|          | 0/3 [00:00<?, ?it/s]

Out[8]:

Success: Inferred UnverifiedInformationModel

In [10]:

neat

neat

Out[10]:

Unverified Data Model

type	Logical Data Model
intended for	Information Architect
name	Inferred Model
external_id	NeatInferredDataModel
space	neat_space
version	v1
classes	3
properties	45

Instances

Overview:

• 3 types
• 973 instances

	Type	Occurrence
0	ExternalInterface	858
1	InternalElement	67
2	InternalLink	48

Provenance:

• Initialize graph store as OxigraphStore
• Extracted triples to graph store using AmlExtractor
• Prunes the graph of specified rdf types that do not have connections to other nodes.
• Attaches a target property from a target node that is connected to a source node.
• Prunes nodes of specific rdf types
• Prunes the graph of specified rdf types that do not have connections to other nodes.

This gives us an unverified data model, which we can then verify

Verify Data Model¶

In [11]:

neat.verify()

neat.verify()

Out[11]:

Success: UnverifiedInformationModel → VerifiedInformationModel

In [12]:

neat

neat

Out[12]:

Verified Data Model

type	Logical Data Model
intended for	Information Architect
name	Inferred Model
external_id	NeatInferredDataModel
version	v1
classes	3
properties	45

Instances

Overview:

• 3 types
• 973 instances

	Type	Occurrence
0	ExternalInterface	858
1	InternalElement	67
2	InternalLink	48

Provenance:

• Initialize graph store as OxigraphStore
• Extracted triples to graph store using AmlExtractor
• Prunes the graph of specified rdf types that do not have connections to other nodes.
• Attaches a target property from a target node that is connected to a source node.
• Prunes nodes of specific rdf types
• Prunes the graph of specified rdf types that do not have connections to other nodes.
• Added rules to graph store as InformationRules
• Upsert prefixes to graph store

We now convert verified data model to be DMS compliant

In [13]:

neat.convert("dms")

neat.convert("dms")

Rules converted to dms

Out[13]:

Success: VerifiedInformationModel → VerifiedDMSModel

Inspect Data Model¶

First we will set data model id to ("aml_playground", "AML", "topology_v3.0"), where tuple represents (space, external_id, version) of data model

We use the .show to visualize the data model and see how data model objects are connected, the resulting interactive data model visualization should be similar to this one:

In [14]:

neat.show.data_model()

neat.show.data_model()

http_purl.org_cognite_neat_data-model_verified_physical_neat_space_NeatInferredDataModel_v1.html

Out[14]:

In [15]:

neat.set.data_model_id(("aml_playground", "AML", "topology_v3.0"))

neat.set.data_model_id(("aml_playground", "AML", "topology_v3.0"))

Out[15]:

Success: VerifiedDMSModel → VerifiedDMSModel

In [16]:

neat

neat

Out[16]:

Verified Data Model

aspect	physical
intended for	DMS Architect
name	Inferred Model
space	aml_playground
external_id	AML
version	topology_v3.0
views	3
containers	3
properties	45

Instances

Overview:

• 3 types
• 973 instances

	Type	Occurrence
0	ExternalInterface	858
1	InternalElement	67
2	InternalLink	48

Provenance:

• Initialize graph store as OxigraphStore
• Extracted triples to graph store using AmlExtractor
• Prunes the graph of specified rdf types that do not have connections to other nodes.
• Attaches a target property from a target node that is connected to a source node.
• Prunes nodes of specific rdf types
• Prunes the graph of specified rdf types that do not have connections to other nodes.
• Added rules to graph store as InformationRules
• Upsert prefixes to graph store

Publishing Data Model to CDF¶

Now we are ready to publish this to CDF.

In [23]:

neat.to.cdf.data_model()

neat.to.cdf.data_model()

You can inspect the details with the .inspect.outcome.data_model(...) method.

Out[23]:

	name	created
0	schema	0
1	spaces	1
2	containers	3
3	views	3
4	data_models	1
5	nodes	0

Populating Data Model¶

Neat keeps track of the data, so we can immidiately populate this data model with the original data

In [24]:

neat.to.cdf.instances()

neat.to.cdf.instances()

INFO | 2024-12-09 13:11:31,439 | Staring DMSLoader and will process 3 views.
INFO | 2024-12-09 13:11:31,441 | Starting ViewId(space='aml_playground', external_id='ExternalInterface', version='topology_v3.0') 1/3.
INFO | 2024-12-09 13:11:31,618 | Finished ViewId(space='aml_playground', external_id='ExternalInterface', version='topology_v3.0').
INFO | 2024-12-09 13:11:31,619 | Starting ViewId(space='aml_playground', external_id='InternalElement', version='topology_v3.0') 2/3.
INFO | 2024-12-09 13:11:33,151 | Finished ViewId(space='aml_playground', external_id='InternalElement', version='topology_v3.0').
INFO | 2024-12-09 13:11:33,151 | Starting ViewId(space='aml_playground', external_id='InternalLink', version='topology_v3.0') 3/3.
INFO | 2024-12-09 13:11:33,163 | Finished ViewId(space='aml_playground', external_id='InternalLink', version='topology_v3.0').

You can inspect the details with the .inspect.outcome.instances(...) method.

Out[24]:

	name	created	changed
0	ExternalInterface	858	0
1	InternalElement	62	5
2	edge	972	0
3	InternalLink	0	48

In [ ]: