Validating RDF data

Chapter 7  Comparing ShEx and SHACL

In this chapter we present a comparison between ShEx and SHACL. The technologies have similar goals and similar features. In fact at the start of the Data Shapes Working Group in 2014, convergence on a unified approach was considered possible. However, this did not happen and as of July 2017 both technologies are maintained as separate solutions.

We start by describing some of the common features that they share, followed by a review of the main differences.

7.1  Common Features

ShEx and SHACL share the same goal, to have a mechanism for describing and validating RDF data using a high-level language, so there are a lot of common features that both share.

• Shapes. Both define the notion of a shape, as something that contains constraints on the topology of RDF nodes. SHACL shapes are similar to ShEx shape expressions, with the difference that links to data nodes are expressed in SHACL by target declarations and in ShEx by shape maps. In most of the common cases, it is possible to translate between ShEx and SHACL.
  Example 160 (Similarities between ShEx and SHACL)  

  Consider the following SHACL shapes graph:

  :User a sh:NodeShape ; sh:nodeKind sh:IRI ; sh:property [ sh:path schema:name ; sh:minCount 1; sh:maxCount 1; sh:datatype xsd:string ; ] ; sh:property [ sh:path schema:gender ; sh:minCount 1; sh:maxCount 1; sh:or ( [ sh:in (schema:Male schema:Female) ] [ sh:datatype xsd:string] ) ] ; sh:property [ sh:path schema:birthDate ; sh:maxCount 1; sh:datatype xsd:date ; ] .

  This can be expressed in a ShEx schema:

  :User IRI { schema:name xsd:string ; schema:gender [schema:Male schema:Female] OR xsd:string ; schema:birthDate xsd:date ? }

• Node constraints. Both languages have the notion of node constraints and share similar expressiveness: node kinds, datatypes, datatype facets, value sets, etc. Example 160 shows two declarations which are equivalent in ShEx and SHACL: a node must be an IRI, have exactly one value for the property schema:name that has datatype xsd:string, have exactly one value for the property schema:gender which must be one of ( schema:Male schema:Female) or a xsd:string, and optionally have a value for the property schema:birthDate that has datatype xsd:date.
• Property Constraints. Both languages enable the declaration of constraints on the outgoing and incoming properties of a node.
  Example 161 (Constraints on incoming/outgoing properties in ShEx/SHACL)  

  The following SHACL shapes graph describes that nodes that conform to :User have one outgoing property schema:name and one incoming property schema:member from an organization.

  :User a sh:NodeShape ; sh:property [ sh:path schema:name ; sh:minCount 1; sh:maxCount 1; sh:datatype xsd:string ; ] ; sh:property [ sh:path [ sh:inversePath schema:member ] ; sh:minCount 1; sh:maxCount 1; sh:node :Organization; ] . :Organization a sh:NodeShape ; sh:property [ sh:path rdf:type ; sh:minCount 1; sh:maxCount 1; sh:hasValue :Organization ; ] .

  can be expressed in ShEx as:

  :User { schema:name xsd:string ; ^schema:member @:Organization } :Organization { a [:Organization] }

  Given the following data:

  :alice a :User ; # Passes as a :User schema:name "Alice" . :bob a :User ; schema:name "Robert" . # Fails as a :User :myCompany a :Organization ; schema:member :alice .

  Both ShEx and SHACL check that :alice conforms to the :User shape and raise an error for :bob because there is no arc schema:member from a node with shape :Organization pointing to :bob.

• Cardinalities. Both languages can constraint the number of values for a property in a specific range, or leave the maximum number of value unbound.
• RDF syntax. Both ShEx and SHACL can use RDF concrete syntaxes though with different vocabularies.
• Logical operators. Both ShEx and SHACL have the logical operators And, Or and Not. ShEx has the operators | to represent “oneOf” while SHACL has xone to represent exactly one.
  Example 162 (Example with logical operators)  

  Imagine that in some domain, a :Product must have a schema:productID with a value that either starts by P (matches regular expression "^P") or ends by a digit (regular expression "\\[0-9]$") and is not "P23".

  It can be expressed in ShEx as:

  :Product ({ schema:productID /^P/i ; } OR { schema:productID /[0-9]$/ ; }) AND NOT { schema:productID [ "P23" ] }

  and in SHACL as:

  :ProductShape a sh:NodeShape ; sh:targetClass :Product ; sh:or ( [ sh:path schema:productID ; sh:minCount 1; sh:maxCount 1; sh:pattern "^P" ; sh:flags "i" ] [ sh:path schema:productID ; sh:minCount 1; sh:maxCount 1; sh:pattern "[0-9]$" ; ] ); sh:not [ sh:path schema:productID ; sh:hasValue "P23" ] .

  Given the following data:

  :p45 a :Product ; # Passes as a :Product schema:productID "P45" . :x23 a :Product ; # Passes as a :Product schema:productID "X23" . :p23 a :Product ; # Fails as a :Product schema:productID "P23" . :xx a :Product ; # Fails as a :Product schema:productID "xx" .

• Extension mechanism. Both ShEx and SHACL have extension mechanisms that support the declaration of more advanced constraints. ShEx has semantic actions (see Section 4.10) and SHACL has SHACL-SPARQL (see Section 5.16). In Section 7.18, we compare the ShEx and SHACL extension mechanisms in more detail.

7.2  Syntactic Differences

The design of ShEx emphasized human readability, with a compact grammar that follows traditional language design principles and a compact syntax evolved from Turtle. The specification defines an abstract syntax. The compact syntax (ShExC), a concrete JSON syntax (ShExJ), or any of the concrete syntaxes for RDF may be used to express a ShEx schema.

SHACL uses the RDF abstract syntax and concrete syntaxes directly. The SHACL specification enumerates circa 120 rules that define what constitutes a well-formed SHACL shapes graph.¹ SHACL processors can simply omit ill-formed shapes graphs.

A compact syntax inspired by ShEx has been proposed for a subset of SHACL as a WG Note (see Section 5.18) but it is not mandatory, and compliant SHACL processors are only required to handle the RDF syntax.

As the SHACL compact syntax was inspired by ShExC, they look similar, but there are several semantic differences.

The RDF vocabulary of ShEx is also different from SHACL.

7.3  Foundation: Schema vs. Constraints

Although both languages share a common goal, their designs are based on different approaches.

The designers of ShEx intended the language to be like a grammar or schema for RDF graphs. This design was inspired by languages such as Yacc, RelaxNG, and XML Schema. The main goal was to describe RDF graph structures so they could be validated against those descriptions.

In contrast, the designers of SHACL aimed at providing a constraint language for RDF. The main goal of SHACL is to verify that a given RDF graph satisfies a collection of constraints. In this sense, SHACL follows the Schematron approach, applied to RDF: it declares constraints that RDF graphs must fulfill. Just as Schematron relies strongly on XPath, SHACL relies strongly on SPARQL.

This difference is reflected in how validation results fit in. ShEx implementations usually construct a data structure representing the RDF graph that were validated, containing the nodes and shapes that were matched. After ShEx validation, the result shape map contains a structure which can be considered as an annotated graph that can be traversed or used for further actions, such as transforming RDF graphs into other data structures. This structure is analogous to the Post Schema Validation Infoset from XML Schema (see Section 3.1.3).

In contrast, SHACL describes in detail the errors returned when constraints are not satisfied. A SHACL validation report (see Section 5.5) can be very useful for detecting and repairing errors in RDF graphs. When there are no errors, SHACL processors usually report a single value, sh:conformance true. With SHACL, it can be difficult for users to distinguish the case in which a node is valid because it was checked against some shape, versus the case in which a node is not valid but was ignored by the SHACL processor because it was not reached during the validation process.

The SHACL recommendation prescribes a basic structure for each violation result but does not prescribe what information is to be returned when a node is validated. Nevertheless, SHACL processors can enrich their results. Shaclex, for example, returns information about the nodes validated.

7.4  Invoking Validation

SHACL shapes can include target declarations that associate each shape with a set of RDF nodes and tell SHACL processors how to trigger the validation process (see Section 5.7).

While including the target declarations in the schema is a convenient way to trigger validation, it can be considered an anti-pattern because the shape can’t be reused for other data. Even though this could work in some closed systems, it is impractical for data in open environments. In the interest of keeping schemas reusable, it is a good practice for SHACL to place target declarations in a separate file and link this file to the schema with owl:imports.

A ShEx schema declares a constellation of shape expressions that function as a grammar against which RDF nodes can be tested. The schema itself provides no mechanism for associating a shape expression with the nodes to which the schema applies. In the interest of making schemas reusable, ShEx requires that definitions of shapes be decoupled from their application to particular RDF graphs. ShEx separates the language of schemas, on the one hand, from the association of shapes with nodes to be validated, on the other, by introducing the notion of shape maps (see Section 4.9 for more details). This separation of concerns encourages the community to innovate on node-shape association mechanisms independently from the validation semantics. For example, though the shape map specification currently only supports RDF nodes by direct reference or by triple pattern, Wikidata versions of ShEx include support for SPARQL queries over remote endpoints. As such conventions evolve they can be rolled into future versions of the shape map specification.

Another notable difference between SHACL target node declarations and ShEx shape maps is the following: when a declared target node in SHACL does not exist in the data graph and there are no required values for this node in the shape, the node passes the validation. In ShEx if the node does not exit it always results in a failure, no matter of the shape definition.

7.5  Modularization and Reusability

SHACL leverages the property owl:imports to enable a shapes graph to import other shapes graphs. This mechanism, which can be used to provide the basis of a modular design, is described in Section 5.4.

ShEx has the concept of shapeExternal to declare that the contents of a shape can be obtained from an external source (see Section 4.7.3).

ShEx has a basic import mechanism which allows a schema to derefentiate another schema (see section 4.12) while SHACL has also the possibility to import other shapes graphs using owl:imports (see section 5.4). One difference between ShEx and SHACL import mechanisms is that ShEx dereferentiates the schema while SHACL is a graph merge, so in SHACL the system expects to have already fetched all of the relevant shapes graphs.

Both languages support the reuse of shapes through extending a shape with an AND operator, as described in Section 4.8.1 (ShEx) and Section 127 (SHACL).

The reusability of both languages could be improved. For example, there is no notion of a module, where one might declare internal or hidden shapes, or of public shapes that could be imported by other modules. Also, there is no notion of a shape extending other shape, inheriting some properties and redefining others. Such features could potentially be developed for both languages.

7.6  Shapes, Classes, and Inference

ShEx is only concerned with RDF graphs as they are presented to the validator. There is no interaction between the ShEx processor and any inference mechanism. In this way, ShEx can be used before or after inference. It can even be used to validate the behavior of an inference engine if one defines the shapes that an RDF graph must have before and after inference (see an example in Section 4.11).

In contrast, SHACL has some mechanisms that may interact with inference. For example, the implicit class target (see Section 5.7.3), which associates a shape with a class, triggers validation on all nodes that are SHACL instances. The notion of SHACL instance is different to the RDF Schema notion of instance because it encompasses instances of a class plus its sub-classes (as determined by following rdfs:subClassOf links in the data), but does not take into account all RDFS elements.

The results of applying a SHACL validator may be different if applied to RDF graphs before or after RDFS inference. As SHACL processors are not required to support full RDFS inference, they may ignore other RDFS predicates, such as rdfs:domain, rdfs:range, and sub-properties of rdfs:subClassOf.

For example, consider the following SHACL shape:

and the following RDF data:

If SHACL is applied after RDFS inference, the system checks whether :frank and :grace conform to :UserShape. This is because the domain declaration of :teaches allows RDFS to infer that they are instances of :Teacher and, hence, instances of :User, with the following results:

• :grace has a value for schema:name that is not an xsd:string.
• :oscar has a value for schema:name that is not an xsd:string.

In contrast, if SHACL is applied without RDFS inference, the system returns only one error:

• :oscar has a value for schema:name that is not an xsd:string.

The system does not check :frank or :grace against shape :User because it only follows rdf:type and rdfs:subClassOf declarations. In the absence of RDFS inference, the system only checks that :oscar has shape :User. If SHACL is applied after RDFS inference, the system checks the additional nodes.

This interference between SHACL and RDFS semantics hampers the use of SHACL to validate an inference system as the use case described for ShEx in Example 21.

The property sh:entailment can be used to declare that the SHACL processors should add inferred triples during validation to the data graph following the inference rules declared by a given entailment regime (see Section 5.17). Nevertheless, SHACL processors are not required to support entailment regimes. If a shapes graph declares an entailment and the processor does not support it, a failure must be signalled.

7.7  Violation Reporting and Severities

As pointed out above, SHACL puts more emphasis on validation and provides a dedicated RDF vocabulary for describing conformance and reporting detailed violation results.

For every focus node that does not conform to a shape, an instance of sh:ViolationResult is created in the SHACL results graph. Each violation result links back to the focus node along with metadata, which includes the shape IRI, human readable messages, the failed constraint, the path, and (when available) the value node. The severity level of a SHACL shape, if declared with ( sh:Info, sh:Warning, or sh:Violation), can be included in the violation result (see Section 5.6.5).

ShEx does not have rich violation reporting, but it can provide related functionality. The result of the validation process is a shape map which contains information about the nodes that conform to a shape or not. Every violation can be viewed as an entry showing the focus node and the shape that failed. ShEx processors usually enrich these entries with further information. As shapes in ShEx can contain arbitrary annotations (see Section 4.7.5), these annotations can be included in the results.

In simple and top-level shape definitions, SHACL provides richer and granular violation reporting for each individual constraint that failed. However, violations on nested constraints as formed using sh:node, sh:and, sh:or, sh:xone, or sh:qualifiedValueShape, report only which nested constraint failed (“ sh:node failed”) without detailing why. Implementations could report that information by means of the sh:detail property, but that would be an implementation dependent feature. Also, as a result of validation ShEx produces a Result Map associating nodes with shapes (either validated or non-validated) while SHACL has no comparable feature.

7.8  Default Cardinalities

If no cardinality is declared, ShEx assumes the cardinality to be {1,1} while SHACL assumes {0,*}.

The difference in results is based on the difference between the ShEx and SHACL points of view. In ShEx, a triple expression makes explicit which triples involving the focus node should be found in the graph, and specifying a cardinality may require several such triples. The absence of cardinality means one triple. In SHACL, a shape is a conjunction of constraints. A cardinality constraint is used to constrain the number of allowed triples of a given kind, and the absence of cardinality means no constraint on the number of triples allowed.

7.9  Property Paths

SHACL property shapes can use a subset of SPARQL 1.1 property paths as values for
sh:path. In this way, SHACL leverages on the expressiveness of SPARQL property paths to define constraints.

ShEx does not support arbitrary property paths—only direct and inverse predicates. However, it is easy to emulate this SHACL behavior using nested shapes or recursion.

7.10  Recursion

ShEx supports the definition of cyclic data models with recursive shapes (see Section 4.7.2) while the processing of recursive shapes is undefined in SHACL (see Section 5.12.1). However, some recursion cases can be handled in SHACL through SHACL property paths.

7.11  Property Pair Constraints and Uniqueness

Property pair constraints in SHACL can be used to compare current values with values from another path, checking if they are equal, different or less than them (see Section 5.14).

ShEx 2.0 does not have the concept of property pair constraints, though this possibility is being studied to be included in future versions.

One constraint often required is the ability to declare unique keys. Unique keys are combinations of values that must be unique in a given scope. The scope can be the entire graph or a focus node. One example of a unique constraint for an entire graph is to require that there be no pair of identical values for the properties schema:givenName and schema:lastName. One example of a unique constraint with a focus node scope would be to require that each node not have two values of rdfs:label with the same language tag.

Neither SHACL nor ShEx 2.0 support unique keys in general, although they are supported by OWL 2. SHACL Core offers the sh:uniqueLang constraint to say that there can be no more than one literal for each language tag (see Section 124). Other constraints can be defined using SHACL-SPARQL. In the case of ShEx, there is a proposal to add a UNIQUE keyword to the language, with the scope and the list of predicates that must be unique as parameters.

7.12  Repeated Properties

ShEx allows multiple constraints on triples involving the focus nodes with the same property to be defined. This feature is called repeated properties as explained in Section 4.6.7. In SHACL, repeated properties behave conjunctively, which means that all constraints applied to properties with the same sh:path must be satisfied. The typical SHACL pattern of:

must be changed if we want :p1 and :p2 to be the same property, only with different values. A direct translation of that pattern to:

means that all constraints apply to the path :p conjunctively.