# Introduction

See these publications:

- RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation. Vladimir Alexiev. In Semantic Web in Libraries 2016 (SWIB 16), Bonn, Germany, November 2016: Presentation, HTML, PDF, Video
- Generation of Declarative Transformations from Semantic Models. Vladimir Alexiev. In European Data Conference on Reference Data and Semantics (ENDORSE 2023), Mar 2023: paper, presentation, video

RDF is a graph data model, so the best way to understand RDF data schemas (ontologies, application profiles, RDF shapes) is with a diagram. Many RDF visualization tools exist, but they either focus on large graphs (where the details are not easily visible), or the visualization results are not satisfactory, or manual tweaking of the diagrams is required.

- rdfpuml makes true diagrams directly from Turtle examples using PlantUML and GraphViz. Diagram readability is of prime concern, and rdfpuml introduces various diagram control mechanisms using triples in the `puml:` namespace. Special attention is paid to inlining and visualizing various Reification mechanisms (described with PRV). We give examples from Getty CONA, Getty Museum, AAC (mappings of museum data to CIDOC CRM), Multisensor (NIF and FrameNet), EHRI (Holocaust Research into Jewish social networks), Duraspace (Portland Common Data Model for holding metadata in institutional repositories), Video annotation.

If the example instances include embedded source field names, they can describe a mapping precisely. I've implemented a few more tools to generate transformations:

- rdf2rml generates R2RML transformations for RDBMS tables or SQL queries. Compared to R2RML, this saves about 15x in complexity and is competitive with the dedicated DSL YARRML
- rdf2sparql generates OntoRefine or TARQL transformations from CSV/TSV that take the form of SPARQL UPDATE (for direct GraphDB loading) or CONSTRUCT (for conversion to RDF). (Subsumes two deprecated tools `rdf2tarql` and `rdf2ontorefine`)

See http://twitter.com/hashtag/rdfpuml for news, diagrams and announcements.

## License and Citation

This work is covered by the Artistic-2.0 license.

If you use this software, please cite it as shown above.

- Github shows a link "About> Cite this repository" (see about-citation-files)
- CITATION.cff describes both the software and the above publications. It's a YAML CFF file, see https://citation-file-format.github.io/
- CITATION.bib describes only the above publications. It's a bibtex file

## Documentation

- rdfpuml.md or rdfpuml.html
- rdf2rml.md or rdf2rml.html
- rdf2sparql.md or rdf2sparql.html (subsumes `rdf2tarql` and `rdf2ontorefine`)

### rdfpuml shots

Everyone likes screenshots!

- There are some shots that illustrate each feature in the documentation (rdfpuml.md or rdfpuml.html).
- There are more shots in individual tests in test.
- I'm planning to create a Gallery of all shots in tests

But until I do it, here are a couple of shots.

#### Shot: Company and Person Info

Information about Sirma Group and Atanas Kiryakov from Refinitiv PermID:

test/permid/sample-SirmaGroup-AtanasKiryakov.png

#### Shot: Complex Class Expressions

Intersections and restrictions, represented using blank nodes and `rdf:List`:

test/complex-types/example2.png

#### Shot: CIDOC CRM and Reification

The tool can depict reification using `rdf:Statement` and various other constructs, in this case `crm:E17_Type_Assignment`:

doc/img/CONA-types-reification.png

#### Shot: Arrow Styles

doc/img/arrowStyles.png

#### Shot: Unicode and Links

Unicode (actual) and semantic URL links for each node and property (idea). "Zmrzlina" is Czech for "icecream" and may have the highest concentration of consonants in the world:

test/unicode/svg-links.png

### rdf2sparql shots

Declarative model (left) vs generated SPARQL Update for OntoRefine (right two):

doc/img/rdf2sparql-organizations.png

## Related Work

Software:

- Matjaz Rihtar (2018). https://github.com/mrihtar/rdfgraph: inspired by `rdfpuml`, written in Python 2.7, uses Redland's `librdf` library. I worked with Matjaz in the euBusinessGraph project.

The following papers use or mention this software:

- Ademar Crotti (2019). A Jigsaw Puzzle Metaphor for Representing Linked Data Mappings. PhD Thesis, Knowledge and Data Engineering Group (KDEG), Trinity College, Dublin, Ireland
- C. Debruyne, D. Lewis, D. O’Sullivan (October 2018). Generating Executable Mappings from RDF Data Cube Data Structure Definitions. On the Move to Meaningful Internet Systems" (OTM 2018). LNCS vol 11230. DOI 10.1007/978-3-030-02671-4_21
- L. Zhuhadar, M. Ciampa (2017). Leveraging learning innovations in cognitive computing with massive data sets: Using the offshore Panama papers leak to discover patterns. Computers in Human Behavior. doi:10.1016/j.chb.2017.12.013
- V. Alexiev (2018). Museum Linked Open Data: Ontologies, Datasets, Projects (invited report). In Digital Presentation and Preservation of Cultural and Scientific Heritage (DIPP 2018). Volume 8, pages 19-50. Burgas, Bulgaria, September 2018
- V. Alexiev, A. Kiryakov, P. Tarkalanov (2017) euBusinessGraph: Company and economic data for innovative products and services. 13th International Conference on Semantic Systems (Semantics 2017)
- V. Alexiev, P. Tarkalanov, N. Georgiev, L. Pavlova (2020). Bulgarian Icons in Wikidata and EDM. Digital Presentation and Preservation of Cultural and Scientific Heritage (DIPP 2020).

# Installation

Checkout this repo and add `rdf2rml/bin` to your path. Install the following prerequisites:

- both tools: Perl. Tested with version 5.22 on Windows (cygwin and Strawberry).
- rdfpuml:
  - GraphViz
  - PlantUML. You need a recent version for new features like arrow length and color. I'm currently running 1.2018.10beta7. See in particular plantuml class diagrams.
  - Perl modules: use `cpan` or `cpanm` to install them: `RDF::Trine RDF::Query Encode FindBin Carp::Always Slurp`
  - `RDF::Prefixes::Curie`. This is my own module located in ./lib, and rdfpuml needs `FindBin` to locate it.
- rdf2rml:
  - Apache Jena: `riot`, `update`. Tested with version 3.1.0 of 2016-05-10.
  - cat, grep, rm

## Docker Image

If you prefer to work with Docker so you don't need to install software manually, you can use this rdf2rml image from the public Nexus (Docker Registry) of Ontotext. To run it, use:

    docker run -v <directory>:/files --rm docker-registry.ontotext.com/rdf2rml:latest`

Where `<directory>` is the local directory holding your `.ttl` files. It was made on 31 May 2023 and uses the following versions:

- rdf2rml: 31 May 2023, with fixed issue 22
- PlantUML: 1.2023.7
- Jena: 4.8.0

Note: pull request 7 of 17 Sep 2019 by Jem Rayfield (`@jazzyray`) dockerizes the installation, and makes extra changes related to input/output and configuration. However, it has not been merged yet

# Debian Repo

Jonas Smedegaard (`@jonassmedegaard`, dr at jones fullstop dk) has volunteered for some of the tasks below. His development is at https://salsa.debian.org/debian/rdf2rml/branches. To adopt changes, do something like this.

- To merge all commits in the `salsa/develop` branch:

      cd rdf2rml    # i.e. your local clone of your Github project
      git remote add salsa https://salsa.debian.org/debian/rdf2rml.git
      git fetch salsa
      git merge salsa/develop

- To adopt only single commits from the `salsa/develop` branch, issue `remote` and `fetch` as above, then issue:

      git cherry-pick $commit1 $commit2 $commit3

# ToDo Tasks

See CHANGELOG.org for implemented features and fixes.

Help needed for the following tasks. Post bugs and enhancement requests to this repo!

## Near-term

### Modularize and Package Better

### Regression Tests

- `sort` is added at various places to make the tool more deterministic, i.e. independent of order of RDF statements in the input file. However, this will interfere with the ability to control the layout, especially of disconnected components (see layout_new_line)
- Some regression tests are added.

### rdf2rml: disentangle inverse edge

In the case `Y-P-X` described above:

- Also need to record `?y puml:property ?p` so this prop name can be added to ?y's subject map
- When making ?map, take `puml:property` into account
- But ?map is made many times, and copy-paste is no good…
- Also, this should be done in some cases but not others…
- So it's better to record `?y puml:map ?map` …

### Release on CPAN

### Add Unicode tests

Add ttl with non-ASCII chars: Accented, Cyrillic, French, etc.

- Accented: `"Rudolf Mössbauer"` in ./test/TRR/societyMember.ttl

### Prefixes

#### Allow specifying the prefixes file

See https://github.com/VladimirAlexiev/rdf2rml/pull/7

#### Eliminate Curie.pm

./lib/RDF/Prefixes/Curie.pm remembers `@base` and uses that for URL shortening. Once perlrdf#131 is fixed, eliminate this dependency (local module)

#### Remember prefixes from input file

`rdfpuml` shortens URLs using prefixes only from `prefixes.ttl`, but should also use prefixes defined in the individual input file.

### Support more RDF Formats

Now it only supports Turtle, because it concatenates `prefixes.ttl` to the main file. If it can collect all prefixes from RDF files, such concatenation won't be needed

### Batch Processing

Issue #1: plantuml is slow to start up, so we'd like to process a bunch of `puml` files at once. The best way is to have a smarter script or `Makefile` that uses the following http://plantuml.com/command-line features:

- Keep the intermediate `puml` files (the current `Makefile` doesn't preserve them)
- Run `plantuml` on a whole folder (with `-r[ecurse]` it can even recurse through subfolders)
- Use `-checkmetadata` to skip `png` files that don't need to be regenerated. (The whole `puml` text is stored in the `png`, so `plantuml` can quickly check that there are no changes)
- The `Makefile` should start `plantuml` only once, if some of the `puml` files is newer than its respective `png` file

#### "Manual" Batching

Before I discovered the `-checkmetadata` option, I had the idea that `rdfpuml` could put several diagrams in one `puml` file:

    @startuml file1.png
      # made from file1.ttl
    @enduml
    @startuml file2.png
      # made from file2.ttl
    @enduml

However, this interferes with `make` processing that regenerates only `png` for changed `ttl` files, and makes things less modular overall.

## Mid-Term

### Upgrade to use Attean

Trine (Perl RDF) is end of life. Attean is the new generation

### Integrate in Emacs `org-mode`

Write Turtle, see diagram (easy to do)

### Node colors, icons, tooltips

See ./ideas

### More arrow types and styles

- See `arrows arrows-2` from https://github.com/anoff/blog/tree/master/static/assets/plantuml/diagrams:

- Arrow styles and colors (bold, dashed etc): https://mrhaki.blogspot.com/2016/12/plantuml-pleasantness-get-plantuml.html

- `plantuml -pattern` regexes:

    dotted|dashed|plain|bold|hidden|norank|single|thickness

### Extra Layout Options

Local layout options are described in Help on Layout:

- "hidden" makes a constraint between two nodes, but does not draw the link (`rdfpuml` already implements this)
- norank ignores a link for layout purposes (same as graphviz `constraint=false`)
- "together" groups classes as if they were in the same package (i.e. puts them in a graphviz cluster)

Global options include (eg see this diagram):

skinparam Linetype ortho skinparam NodeSep 80 skinparam RankSep 80 skinparam Padding 5 skinparam MinClassWidth 40 skinparam SameClassWidth true

And there are a lot more undocumented features: https://forum.plantuml.net/7095

### Custom Reification

Ability to describe custom reification situations using the Property Reification Vocabulary (PRV)

### Use MindMap/WBS for Hierarchies

Plantuml now has MindMap and WBS (or OBS) diagrams that use a simple bulleted syntax to draw hierarchies.

It would be nice to use this to draw hierarchies of individuals, in particular taxonomies.

Here are examples of the two styles:

- Mindmap
- WBS

## Long-Term

### rdf2soml to Generate Semantic Object Models

A new tool `rdf2soml` to generate Ontotext Platform SOML from RDF examples.

What's missing? Most importantly: property cardinality and virtual inverses.

PlantUML can show arrow cardinalities, and this simple and natural PlantUML code:

    X "0:1" -left-> "1:m" Y : prop/\ninvProp

Is depicted as follows:

We have two options how to express this in triples:

#### Cardinality With RDF*

    ##### model triples
    :X :prop :Y.
    ##### puml triples
    << :X :prop :Y >>
      puml:arrow puml:left; # direction
      puml:min 1; puml:max puml:inf; # cardinality
      puml:inverseAlias [puml:min 0; puml:max 1; puml:name "invProp"]. # virtual inverse

- Pros: very natural
- Cons:
  - Perl RDF doesn't support RDF*, and few editors support it either.
  - Annotating a triple does not assert it, so we need to assert it as well

#### Cardinality With Blank Node

    ##### model triples
    :X :prop :Y.
    ##### puml triples
    :X puml:left :Y. # direction
    :X :prop [ # a puml:Cardinality; # may need this marker class to skip the node from the diagram
      puml:min 1; puml:max puml:inf; # cardinality
      puml:object :Y; # only needed if X has several relations "prop" and they need different annotations
      puml:inverseAlias [puml:min 0; puml:max 1; puml:name "invProp"] # virtual inverse
    ].

### rdf2shape to Describe & Generate RDF Shapes

### Visualize RDF Shapes (SHACL and ShEx)

Issue #8: discussion with Thomas Francart of Sparna

I developed this SHACL to PlantUML converter, in Java, based on TopQuadrant SHACL lib, and the result is at https://shacl-play.sparna.fr/play/draw and code at https://github.com/sparna-git/shacl-play/tree/master/shacl-diagram

I don't have a strong opinion on the example you provide, an alternative idea that comes to my mind is

    :node1 :link [
      rdf:value :node2;
      puml:min 1 ;
      puml:max 2 ;
    ]

But this changes the structure of the example graph itself, which might not be convenient

### Generate transformations for other than relational sources

R2RML works great for RDBMS, but how about other sources? Extend rdf2rml to generate:

- RML: extends R2RML to handle RDB, XML, JSON, CSV
- XSPARQL: extends XQuery with SPARQL construct and JSON input
- DONE tarql: handles TSV/CSV with SPARQL construct
- DONE OntoRefine: transformation of TSV/CSV and direct loading to GraphDB with SPARQL Update