Connecting Open Data Soft open data with Smart Data Models for drafting new data models

Many of the datasets published in open data portals are extensively used elsewhere. Well-maintained portals have managers that document the data structure and provide definitions of the types and contents of every field in these datasets. These are some of the requirements for the successful publication of a new Smart Data Model.

With this simple python script (and others to come) at the utils folder it can be drafted a JSON schema compliant with the Smart Data Models Program contribution manual. It is an early version (not everything is updated) but you can check it out.

 Parameters:
– base url of the ODS portal
– dataset_id of the dataset
Returns:
A draft json schema compliant with Smart Data Models Program,

some limitations: it does not translate descriptions (required in English)
some data types
It prints the schema and also returns (if possible a file named schema.json)

test it with this command
SDM_OpenDataSoft_schema_converter.py https://data.ameli.fr/ effectifs

 

New customization option for context

The Smart Data Models Program does not define canonically and uniquely the terms used in the data models. There are many ontologies and vocabularies providing solutions to this issue.

For those users of linked data solutions, every subject includes a context.jsonld file (see example) with long IRI for the terms used in the data models. Besides this, the IRI provided are in fact URL to pages with additional information about the term (see example).

not only this but also two services are available on the tools menu on the front page.

  1. Merging several contexts and detecting conflicts. (for merging several context from different subjects)
  2. Mapping a context with external ontologies. (mapping a local context from SDM to any external ontology). These are the ones available but more could be easily created on demand.

But when mapping an existing open and adopted standard now it is possible to customize the context generated by using a new file notes_context.jsonld (see this empty example) at the root of the subject. It will replace the automatic IRI for a term with the customized one.

Extended data model of RoadSegment

Usually whenever there is an extension of a data model it is a minor update so it is reported in the widget Announcements of the frontpage

However, the update of the RoadSegment data model is so important (more than 30 new attributes) that it is worthy to be announced this way. It is also noticeable the emphasis on the cycling information. Thanks to IUDX for their contribution. It is available in the Transportation subject.

2021, typical "cycling lane" in Tokyo, ie just some white paint on a regular car lane

Export of the full database of attributes

At home -> Search -> json export of attributes database of smart data models  is the full database of attributes (more than 18000), see the statistics page as an array of JSON objects.

Fields for each attribute

_id: identifier of the item

property: the name of the attribute

dataModel: the data model this attribute is present

repoName: the subject this data model belongs to

description: the description of the attribute

typeNGSI: Whether it is a property, Geoproperty, or relationship

modelTags: inherited from the data model tags

license: link to the license for the data model

schemaVersion: version of the data model

type: data type

model: when available the reference model for the attribute

units: when available the recommended units for the attribute

format: either date, or time, or date-time, or URI, etc the format of the attribute

Noise pollution data model and AirQualityForecast published at Environment subject

The Noise Pollution data model and the AirQualityForecast have been published on the Environment subject. The first one merges specific and punctual noise measurements (coming, e.g. from NoiseLevelObservation entities) into average parameters referred to city areas, providing more city-related data about noise pollution status and evolution. The second one helps to store the forecast about the quality of air for a specific period.

Qantas b747 over houses arp

Help to early contributors

If have approached the Smart Data Models Program (SDM) for the first time and you want to become a contributor there are some technical concepts that you need to know about the elements compiled at SDM.

Once checked this presentation maybe you want to review the contribution manual