JSON Cache

No-SQL Interface

Databases of the modules provid a No-SQL interface where the data of the main table and the depending tables are merged as JSON.

Table JsonCache

classDiagram
    class JsonCache {
        ID int
        URI varchar 500
        DisplayText nvarchar 500
        LogUpdatedWhen datetime
        Data JSON
    }

This table is filled via the stored procedure procFillJsonCache.

procFillJsonCache is started by a update trigger trgUpd… of the main table in the database

Interface in clients

All modules provide data via a cache table. In the header of the clients you can inspect the content of the JsonCache with a click on the button.

Example

The JsonCache of the current dataset will be shown as in the example below:

[
    {
        "Type": "Taxon",
        "ID": 4403792,
        "Attributes": {
            "FullName": "Arachnida Cuvier, 1812",
            "Rank": "cl.",
            "GenusOrSupragenericName": "Arachnida",
            "Authors": "Cuvier",
            "Publication": ". 1812",
            "Project": [
                {
                    "Project": "SNSBnames"
                }
            ],
            "Checklist": {
                "Analysis": [
                    {
                        "Checklist": "TaxRef_SNSB_NHC-Pests",
                        "Analysis": "IPM | Group",
                        "Value": "70"
                    },
                    {
                        "Checklist": "TaxRef_SNSB_NHC-Beneficials",
                        "Analysis": "IPM | Group",
                        "Value": "70"
                    },
                    {
                        "Checklist": "TaxRef_SNSB_NHC-Bycatchs",
                        "Analysis": "IPM | Group",
                        "Value": "70"
                    }
                ]
            },
            "Synonymy": [
                {
                    "ID": 4403792,
                    "FullName": "Arachnida Cuvier, 1812",
                    "Name": "Arachnida",
                    "Type": "accepted name"
                }
            ],
            "HierarchyListCache": [
                {
                    "HierarchyListCache": "Animalia | Arthropoda | Arachnida"
                }
            ],
            "Hierarchy": [
                {
                    "ID": 4403792,
                    "FullName": "Arachnida Cuvier, 1812",
                    "Name": "Arachnida",
                    "Rank": "cl."
                },
                {
                    "ID": 4403934,
                    "FullName": "Arthropoda Gravenhorst, 1843",
                    "Name": "Arthropoda",
                    "Rank": "phyl./div."
                },
                {
                    "ID": 4403805,
                    "FullName": "Animalia Linnaeus, 1758",
                    "Name": "Animalia",
                    "Rank": "reg."
                }
            ],
            "CommonNames": [
                {
                    "CommonName": "spider",
                    "LanguageCode": "en",
                    "CountryCode": "GB",
                    "Context": "IPM"
                },
                {
                    "CommonName": "Spinnentier",
                    "LanguageCode": "de",
                    "CountryCode": "DE",
                    "Context": "IPM"
                }
            ],
            "Resource": [
                {
                    "URI": "https://museumsschaedlinge.de/gaeste/",
                    "Title": "diverse animals that migrate or are introduced into buildings seasonally, generally don't casue any damage to objects",
                    "Type": "information"
                }
            ]
        }
    }
]

Update

Apart of the update via the Trigger (see below) you can update the JsonCache via the update button underneath the button.

To update the JsonCache for the whole database select Administration - JsonCache… from the menu. a window as shown below will open where you can update the JsonCache for single datasets or the whole database.

Summary

graph TD;
    TaxonName[main table in database] -->|Update in table| trgUpdTaxonName(trgUpd.. of main table in database)
    Mainform[Buttons in main form] -->ButtonUpdateSingle[Update single dataset] --> procFillJsonCache
    Mainform -->ButtonShowSingle[Show single dataset] --> JsonCache
    trgUpdTaxonName --> procFillJsonCache(procFillJsonCache)
    procFillJsonCache -->JsonCache[JsonCache]
    Adminform[Administration form] -->AdminUpdateSingle[Update single dataset] --> procFillJsonCache
    Adminform --> AdminUpdateDB[Update for whole database] --> procFillJsonCache
    Adminform --> JsonCache

OpenAPI

… hier kommt mal swagger hin sobald verfügbar

Configuration

To configure your Database, choose Administration -> Database -> Rename database to change the name of the database according to your requirements. During this renaming all processes in the database will be terminated (you will get a warning if processes from other host are active).

Afterwards you should adapt the address that is published by the database for access by other modules. Choose Administration -> Database -> Set published address from the menu. This will change the published address to the name of the server where your database is located and an identifier for you database, e.g. http://xy.diversityworkbench.de/Collection/. 

Creation

To create or add a database, you need a windows login on the server and sysadmin permissions.

New installation

If you start with a new installation of DiversityAgents connect with the database with a click on the button. Set server and port and as described in the chapter Database access. After connecting to the server, you will get a message, that no database is available and the button will appear. Click on the button to create a new database on the server. A window as shown below will open where you enter the name or the IP-address of the database server. 

127.0.0.1 is the IP-address of your local machine, so in case of a local installation you can keep this value. Next you have to enter the port of the database server. See the chapter Installation of the database for details. 1433 is the default port for SQL-Server. We do recommend to change this port for security reasons.

After server and port are set, the next step is the definition of the database as described below.

Adding database

If you are already connected to a database and want to add another database, select Connection - Create database... from the menu. A window will open as shown below where you have to enter the name of the database, starting with DiversityCollection.

After creating the database a window with the script for the creation of the objects in the database (tables, views, functions, ...) as shown below will open.

Click on the button to start the script. After the application of the script you get a message as shown below.

Close the window and you will be connected to the new database. In case there are additional updates available, please start the update of the database to ensure the latest version.

Documentation

Tools for the database documentation

These are the tools to describe the parts of the database and create documentations of the structure. To use these tools, choose Administation - Database - Documentation… from the menu. A window will open as shown below.

Click on the List objects button to list the objects of the database. With the  button resp. button you can select resp. deselect the types in the type selection and the object in the list.

Select the objects that should be listed all button resp. none button you can select resp. deselect the types in the type selection and the object in the list.

Select the objects that should be included in the documentation:

• Tables
  • Trigger
• Views
• Roles
• Functions and procedures
• Context

… and exclude the objects that should not be included in the documentation:

• Logging tables
• Enumeration tables
• Old versions of objects
• System objects
• Deprecated objects

The button Set default seletion will select all items in the list without:

• System objects
• Older version of an object indicated by the number at the last position
• Logging tables
• Enumeration tables
• Objects with a description starting with e.g. outdated, deprecated, obsolete etc.
• HTML options:
  • include index for objects
  • include NULL / NOT NULL
  • include relations and dependencies
  • include Description
  • exclude standard trigger
  • exclude definition
  • include permissions for *_Enum etc.
  • exclude obsolete columns
  • exclude columns starting or ending with the given strings
  • include list of tables that are depending on a table

The buttons Add to seletion and  Remove from seletion  will use the given strings with * as wildcard to add resp. remove items from the selection.

With the Context  option you can show or hide the context area for the html and media wiki tab as shown above.

HTML, MediaWiki, JSP-Wiki

To create a documentation choose among the provided options and click on the button Create ... documentation to create the documentation in one of the available formats (HTML, MediaWiki, JSP-Wiki). 

Installation

Diversity Workbench modules use Microsoft SQL-Server 2014 or above as database engine. If you do not have a database server with DiversityAgents already available, you have to install the database engine first. Download the free version of Microsoft SQL Server Express 2016 or above from http://www.microsoft.com/downloads/.. Start the program and follow the instructions for the installation.

Server configuration

To configure your server for remote access, launch the SQL Server Configuration Manager (see image below).

Then click on the "Protocols for SQLEXPRESS" node. Right click on "TCP/IP" in the list of Protocols and choose "enable" for TCP/IP.

Right click on the TCP/IP node and select, "Properties" to open a window as shown below.

In the part IPALL clear out the value for "TCP Dynamic Ports". Give a TCP-Port number to use when making remote connections, e.g. "4321" as shown above. You have to restart the SQL Server Express service before you can connect to your database. 

If you use a database on a server, make sure that the firewall of the server allows access via the port you set for the connections (see below). 

Start the Microsoft SQL Server Managment Studio and attach the database as shown below. Choose the node "databases" and right-click on it to open the context menu (see below). Then choose "attach" from the context menu. A window will open where you can choose the file DiversityAgents_Data.MDF from your database directory and attach it to the database engine.

After the installation make shure to get the latest updates from http://windowsupdate.microsoft.com/.

Database configuration

To configure your Database, use the Client as described in

Database configuration.

Linked server

Databases not available on the local server may be accessible via a linked server. Provided the option for loading the connections is set, the program will automatically try to connect to these databases. Otherwise you can connect to these databases as described here. To administrate the linked servers, choose Administration - Linked servers ... from the menu. A form (see below) will open where you can add linked servers and inspect the content of the available databases.

Use the  resp. buttons to add or remove a linked server. To add a linked server, you need the name of the server and the port, e.g. tnt.diversityworkbench.de, 5432, the login associated with the connection of the linked server e.g. TNT and the password for this login. The available databases will be listed as shown above. To inspect the content, select among the tables or views listed in the right part as shown above. Linked servers have certain restrictions for the availability of data, e.g. XML and geography data are not available via a linked server. For a table or view containing incompatible content you may encounter a corresponding error mentioning the reason for the incompatibility.

An expample for a linked server as provided for central databases is shown below (using the management studio for SQL-Server)

Login administration

To administrate the logins on the database server, their permissions and roles respectively as well as access to projects choose Administration - Database - Logins ... from the menu. A window will open as shown below.

Statistics

To see the activity of a login click on the button. A window will open as shown below listing all databases and tables together with the timespan (From - To) and the number of data sets where any activity of the current login has been found.

To see the current activity on the server click on the button. A window as shown below will open listing all user related processes on the server.

Creation of login

To create a new login click on the button. A window will open as shown below.

Here you can enter the name of the new login, the password and the information about the user which will be stored in a DiversityAgents database. You may either create a new entry in this database or select an existing one: Click on the button to search for a name in the database (see below).

Copy a login

To copy a login including all permissions etc. into a new login, select the original login in the list and click on the button.

Edit a login

To edit the access for a login on the server select the login in the list. If a login should be disabled , uncheck the enabled checkbox (see below).

All databases on the server will be listed with the current database showing a yellow background. The databases where the login has no access will be listed in gray while the databases accessible for a login are black.

Access of a login to a database

To allow the access to a database select the database from the list and choose database as shown below.

Roles of a login in a database

Use the > and < buttons to add or remove roles for the login in the database (see below).

Projects for a login in a database

Depending on the database you can edit the list of projects accessible for a login (see below).

There are 4 states of accessibility for projects

•     Full access: The user can edit the data
•      Read only access: The user can only read the data
•      Locked: The project is locked. Nobody can change the data
•     No access: The user has no access via a project

Projects are related to the module DiversityProjects. To get additional information about a project select it in the the list and click on the button. 

To load additional projects click on the Load projects button. A window will open as shown below. Projects already in the database will be listed in [green]{style=“color: #008000”}, [missing projects in red]{style=“color: #FF0000”} (see below). Check all projects you need in your database and click the Start download button.

Overview for a login

If you see an overview of all permissions and project for a login, click on the button. A window a shown below will open. It lists all modules and their databases, the roles, accessible projects and read only projects for a login. 

To copy the permissions and projects of the current login to another login, select the login where the settings should be copied to from the list at the base of the window and click on the button to copy the settings for all databases or the button to copy the settings of the selected database into this login. 

Overview for a database

If you see an overview of all user and roles in a database, click on the button. A window a shown below will open. It lists all user and roles in the database. 

To remove a user, select it in the list and click on the button. 

Correction of logins

If you select one of the databases, at the base a button may appear. This indicates, that there are windows logins listed where the name of the login does not match the logins of the server. This may happen if e.g. a database was moved from one server to another. To correct this, click on the button. A list of deviating logins will be shown, that can be corrected automatically.

If logins with the same name but different server are found, one of them has to be deleted to make the correction possible. You will get a list where you can select those that should be removed.

Select the duplicate logins that should be removed and click OK.

Tools

These are the tools to handle the basic parts of the database. These tools are only available for the owner of the database and should be [handled with care]{.style2} as any changes in the database may disable the connection of your client to the database. Before changing any parts of the database it is recommended to backup the current state of the database. To use these tools, choose Administation -> Database -> Database tools ... from the menu. A window will open as shown below.

Description

The Description section (see above) shows the basic definitions of the objects in the database and enables you to enter a description for these objects including tables and their columns, functions and their parameter etc. With the buttons SQL for adding, update and both you can generate SQL statements for the creation of the descriptions in your database. Use the button both if you are not sure if a description is already present as it will generate a SQL statement working with existing and missing descriptions (see below).

The button  Fill Cache  fills the table CacheDescription where all descriptions are collected for easy access.

Log table and trigger

In the Log table and trigger section (see below) click on the List tables button to see all tables within the database. The Table section shows the basic definitions of a selected table. If columns for logging the date and responsible user for inserting and updating the data are missing, you can use the Attach ... button to attach these columns to the table. Furthermore you may add a RowGUID to the table as e.g. a preparation for a replication.

In the Log table section (see below) you can create a logging table for the selected table in a format as used within the Diversity Workbench. Click on the Show SQL ... button to show the SQL-statement that will create the logging table. If an old logging table should be kept, choose the Keep old log table option. If your table should support the version setting from a main table, choose the Add the column LogVersion option. To finally create the logging table click on the Create LogTable ... button.

The triggers for insert, update and delete are created in the according sections (see below). If an old trigger exists, its definition will be shown in the upper part of the window. Click on the Show SQL button to see the definition of the trigger according to the current definition of the table in a format as used in the Diversity Workbench. If a trigger should set the version in a main table, which the current table is related to, choose the Add version setting to trigger option. To enable this option you must select the version table first. To finally create the trigger click on the Create trigger button. The update and delete triggers will transfer the original version of the data into the logging tables as defined above, where you can inspect the history of the data sets.

If so far no procedure for setting the version in a main table is defined, you can create this procedure in the last section. Click on the Show SQL button to see the definition and on the Create Procedure button to create the procedure (see below).

Preparation for replication

If you want to use replication within you module, the tables need certain columns and a log table. These preparations can be performed by a script, generated in the section Replication (see below). Select the tables you want to include in the process and create the script. This script can than be included in an update of the database. Please ensure that these changes are only be done by expert staff.

Clear logtables

If for any reason you want to clear the log tables of the database, this can be done in the Clear log tab as shown below. Click on the List tables button to list the log tables. Then select those that should be cleared and click on the Clear log of selected tables button (see below). Please keep in mind that any restoration of data from the log is only possible as long as the data can be retrieved from the log.

Data protection

To implement the General Data Protection Regulation of the European Union several steps have to be performed in a database:

• Generate a skript using this tool (see below) to convert all tables and insert objects according to the requirements:
  • Add columns ID, PrivacyConsent and PrivacyConsentDate in table UserProxy
  • Grant update to PrivacyConsent and PrivacyConsentDate in table UserProxy
  • Create update trigger for UserProxy setting the PrivacyConsentDate
  • Create the function providing the ID of the user from UserProxy
  • Create the function providing the name of the user from UserProxy
  • Create the function PrivacyConsentInfo providing common information
  • For every table:
    • Insert users from a table into UserProxy (if missing so far)
    • Insert users from the corresponding log table into UserProxy (if missing so far)
    • Change the constraints for the logging columns (User_Name() → UserID())
    • Replace user name with ID in logging columns
    • Replace user name with ID in logging columns of the log table
    • Adapt description of the logging columns
• Include the skript in an update of the database
• Check the database for update triggers, functions using e.g. CURRENT_USER, USER_NAME, SUSER_SNAME etc. where user names must be replaced with their IDs. Create a script performing these tasks and include it into an update for the database
• Adapt the client to the now changed informations (e.g. query for responsible etc.)

After these changes the only place where the name of a user is stored is the table UserProxy together with the ID. Removing the name (see below) will remove any information about the user leaving only a number linked to the information within depending data.

To generate a script for the objects and changes needed to implement the General Data Protection Regulation use the Data protection tab as shown below. The generated script will handle the standard objects (logging columns) but not any additional circumstances within the database. For these you need to inspect the database in detail and create a script to handle them on your own.

To set the website where detailed information about the handling of the General Data Protection Regulation within the DiversityWorkbench resp. the current database is provided, click on the button on the Info site tab. If unchanged, the default site for the DiversityWorkbench is set (see below).

If for any reason a user wants his name to be removed from the database, select the users name from the list as shown below and click on the  button (see below).

History

To inspect the history of a dataset click on the button. A form will open, showing all former states of the data in the tables with the current dataset at the top. The version is shown in the header of the main.

The version will be set automatically. If a dataset is changed the version will be increased if the last changes where done by a different user or the last change is more than 24 hours ago (for further details see topic Logging ).

For analysis of the succession of changes the log tables contain additional columns:

• Kind of change: This column is set by the trigger inserting data into the log table
  • current version: This is the current state of the data in the table
  • UPDATE: This is the state of the data before an update happened
  • DELETE: This is the state of the data when the data have been deleted
• Date of change: The date and time of the changes. This column has the default value getdate() that means the current date an time is set when any data are inserted into the log table
• Responsible: The user reponsible for the changes. This column has the default value suser_sname() that means the current user is set when any data are inserted into the log table
• LogID: A unique ID of the logtable. This column is an identity that means it is set by the database when any data are inserted into the log table

Logging

Changes within the database will be documented for each dataset with the time and the responsible user in the columns shown in the image below.

All main tables have a corresponding logging table. If you change or delete a dataset the orignial dataset will be stored in this logging table together with informations about who has done the changes and when it happend. To see the data stored in the logging tables, click on the button to open the history of a dataset.