New pysmartdatamodels Version 0.8.2.0: Semantic Identification for Data Spaces

  By , , , , , , , , , , , ,

We are happy to announce a new release of our Python package, pysmartdatamodels, version 0.8.2.0. The headline feature is aimed squarely at data spaces: a new function that can give semantic meaning to a payload even when the party who sent it never provided one.

What’s New in This Version?

Semantic identification for data spaces: identify_or_draft_datamodel()

In a data space, you routinely receive payloads from other participants with no guarantee they follow any particular standard. identify_or_draft_datamodel() takes any such payload and does one of two things:

  • If it recognizes the payload as an existing Smart Data Model, it returns that model’s real schema, together with a full validation of the payload against it.
  • If it doesn’t, it drafts a brand new schema.json on the spot — following the same structural conventions as a real Smart Data Model — so you have a ready-to-review starting point instead of nothing at all.

It accepts the payload in any of the three formats actually used in practice — plain key-values, NGSI-v2 normalized, or NGSI-LD normalized — detecting and handling the right one automatically:

from pysmartdatamodels import pysmartdatamodels as sdm

payload = {
    "id": "urn:ngsi-ld:WeatherObserved:station-042",
    "type": "WeatherObserved",
    "temperature": 21.5,
    "relativeHumidity": 0.6
}

result = sdm.identify_or_draft_datamodel(payload)
print(result["schema"]["source"])   # "existing" -- it recognized WeatherObserved
print(result["schema"]["validation"]["result"])  # True

When the payload’s own “type” doesn’t match anything, you can optionally ask it to try a softer, attribute-based match against the whole catalog before giving up and drafting a new schema:

result = sdm.identify_or_draft_datamodel(payload, fuzzy=True, fuzzy_threshold=0.3)

And when nothing matches at all, the drafted schema isn’t just a bare shape guess — any attribute whose name is already established elsewhere in the catalog (over 160,000 attribute definitions) gets its real description, model reference, and units reused automatically, so only genuinely new attributes are left with a plain TODO placeholder.

A real validate_payload()

validate_payload(datamodel, subject, payload) existed before but didn’t actually validate anything. It now runs full JSON Schema validation against the live schema of the data model you name, confirms the payload’s “type” matches, and separately flags – without failing – any attribute that isn’t part of the official definition.

Reliability and performance fixes

  • generate_sql_schema() no longer crashes on attributes that can hold more than one type (e.g. a Property that can also be a Relationship), and no longer generates colliding enum type names when two unrelated models happen to share an attribute name.
  • validate_data_model_schema() no longer terminates your entire Python process on a bad input — it returns an error result like every other function in the package.
  • The ~160,000-entry attribute database is now cached in memory instead of being re-read from disk on every single function call — noticeably faster for any code calling the per-attribute lookup functions in a loop.

Get the Latest Version

Update your installation with:

pip install --upgrade pysmartdatamodels

➔️ Get the package on PyPI

Bookmark the permalink.

Comments are closed.