XDS repository

XDS demo repository

Preliminary content This section is work in progress.

This tutorial is a guide to the XDS demo repository, a simplified implementation of an XDS registry and repository to store documents, folders, submission sets and associations. The demo is useful for everyone who wants to use the IPF XDS components. It shows how to:

• Use the XDS.b components to offer a registry and a repository service
• Transform and process registration, retrieval and query requests
• Configure and use ATNA logging
• Enable secure transport using HTTPS

The demo repository is non-persistent. A restart will therefore always start out with a blank respository/registry.

Overview

The XDS demo repository is implemented in Groovy. Most of the code deals with the query functionality. This is of course used for the ITI-18 transaction (registry stored query), but is also used for the other transactions to perform checks of the input data. Within the project you can find the following source files, tests and configuration files.

Source files:

Test files:

Configuration files:

The repository can be started within Eclipse or from command line using the startup.bat after building an assembly. For this guide it is assumed that you have installed the Groovy Eclipse plugin as described in our development setup. To start the server within Eclipse, right click on Server.groovy and choose Run as/Groovy. The repository can be configured to use HTTPS by specifying the command line argument secure.

Running XDSToolKit tests against the demo repository

You can either implement your own XDS source and consumer or use the XDSToolKit to run tests against the repository. To use the toolkit you should first ensure that it runs fine against the public NIST repository that it is pre-configured with. Then you can change xdstest/actors.xml in the XDSToolKit installation to make it run against the demo repository. Add a new site to the existing <sites>:

  <site name="ipf">
    <transaction name="pr.b">http://localhost:9091/xds-iti41</transaction>
    <transaction name="r.b">http://localhost:9091/xds-iti42</transaction>
    <transaction name="sq.b">http://localhost:9091/xds-iti18</transaction>

    <transaction name="pr.b" secure="1">https://localhost:9091/xds-iti41</transaction>
    <transaction name="r.b" secure="1">https://localhost:9091/xds-iti42</transaction>
    <transaction name="sq.b" secure="1">https://localhost:9091/xds-iti18</transaction>

    <transaction name="pr.a">http://localhost:9091/xds-iti15</transaction>
    <transaction name="r.a">http://localhost:9091/xds-iti14</transaction>
    <transaction name="sq.a">http://localhost:9091/xds-iti18</transaction>

    <transaction name="pr.a" secure="1">https://localhost:9091/xds-iti15</transaction>
    <transaction name="r.a" secure="1">https://localhost:9091/xds-iti14</transaction>
    <transaction name="sq.a" secure="1">https://localhost:9091/xds-iti18</transaction>

    <repository uid="1.19.6.24.109.42.1">http://localhost:9091/xds-iti43</repository>
    <repository uid="1.19.6.24.109.42.1" secure="1">https://localhost:9091/xds-iti43</repository>
  </site>

The demo supports most of the XDS.b tests that are required for registry/repository implementations at the connectathon. Note that some of the XDS tests are checking the forwarding of registration requests to the public repository. At the moment the demo repository does not forward these requests. Instead the entries will be registered within its own registry. The following is the list of tests that we verified:

Provide and Register Document Set:
11966, 11979, 11983, 11981, 11986

Register Document Set:
11990, 11991, 11992, 12022, 11993, 11994, 11995, 11997, 11999, 12000, 12001, 12326, 12323, 12327, 12084, 12002, 12004, 11996

Retrieve Document Set Transaction:
12029, 12021, 12360

Stored Query Transaction:
11897, 11898, 11899, 11901, 11902, 11903, 11904, 11905, 11906, 11907, 11908, 11909

To run the stored query transaction tests you have to run test 12346 to load data into the repository that is used by these tests. Also, the stored query transaction tests assume that no other data has been loaded, i.e. there will be test failures if you run one of the other tests before them. Because the demo repository is not persistent you can simply restart the server to get an empty store.

Running one of these tests is very simple:

xdstest -err --s ipf -t 11966

You can also run tests via HTTPS if you have started the demo repository with the secure command line argument:

xdstest --secure -err --s ipf -t 11966

IPF XDS related code snippets

The main purpose of the demo repository is to demonstrate the features that the IPF XDS components offer. This section takes a closer look at such code pieces.

Basic configuration

Configuration of an XDS application is pretty similar to the standard configuration of an IPF application. The main difference is that we have to configure CXF. Because we run the application within a Tomcat environment, we must ensure that CXF does not start its own Jetty instance. Here is the commented context.xml:

<!-- We can add more CXF related namespaces here if we need to add something to
     the default CXF configuration. -->
<beans xmlns="http://www.springframework.org/schema/beans" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:lang="http://www.springframework.org/schema/lang" 
    xmlns:camel="http://activemq.apache.org/camel/schema/spring"
    xmlns:util="http://www.springframework.org/schema/util"
    xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/lang
http://www.springframework.org/schema/lang/spring-lang-2.5.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util-2.5.xsd
http://activemq.apache.org/camel/schema/spring 
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd
">

    <!-- The following imports are required to configure CXF. cxf-servlet
         is imported to configure CXF to run with servlet support. This 
         allows us to use Tomcat with the CXFServlet instead of using CXF 
         with a standalone Jetty server. -->
    <import resource="classpath:META-INF/cxf/cxf.xml" />
    <import resource="classpath:META-INF/cxf/cxf-extension-soap.xml" />
    <import resource="classpath:META-INF/cxf/cxf-servlet.xml" />

    <!-- Camel context and producer -->
    <camel:camelContext id="camelContext">
        <camel:jmxAgent id="agent" disabled="true" />
    </camel:camelContext>
    
    <bean id="producerTemplate" factory-bean="camelContext" factory-method="createProducerTemplate"/>

    <!-- Our route builders for the ITI transactions -->
    <bean id="iti4142RouteBuilder" depends-on="routeModelExtender"
        class="org.openehealth.ipf.tutorials.xds.Iti4142RouteBuilder">
    </bean>

    <bean id="iti43RouteBuilder" depends-on="routeModelExtender"
        class="org.openehealth.ipf.tutorials.xds.Iti43RouteBuilder">
    </bean>

    <bean id="iti18RouteBuilder" depends-on="routeModelExtender"
        class="org.openehealth.ipf.tutorials.xds.Iti18RouteBuilder">
    </bean>

    <!-- The DSL extensions. We use the core extension of the IPF and the ones 
         that come with the XDS components. In addition, we also define our own. -->
    <bean id="coreModelExtension"
        class="org.openehealth.ipf.platform.camel.core.extend.CoreModelExtension">
    </bean>

    <bean id="xdsModelExtension"
        class="org.openehealth.ipf.platform.camel.ihe.xds.commons.extend.XDSModelExtension">
    </bean>
    
    <bean id="regRepModelExtension" class="org.openehealth.ipf.tutorials.xds.RegRepModelExtension">
        <property name="dataStore" ref="dataStore"/>
    </bean>

    <bean id="routeModelExtender" 
        class="org.openehealth.ipf.platform.camel.core.extend.DefaultModelExtender">
        <property name="routeModelExtensions">
            <list>
                <ref bean="coreModelExtension" />
                <ref bean="xdsModelExtension" />
                <ref bean="regRepModelExtension" />
            </list>
        </property>
    </bean>

    <!-- The store that contains all the in-memory documents and their meta data -->    
    <bean id="dataStore" class="org.openehealth.ipf.tutorials.xds.DataStore" />
</beans>

Exposing the endpoints

To allow clients to communicate with our registry/repository, we need to define routes that automatically expose the SOAP-based endpoints. We do not define all the SOAP related details directly. A from(...) is all it takes. You can find those in the route builders. Take a look at the following snippets:

    public void configure() throws Exception {
        ...
        // Entry point for Stored Query
        from('xds-iti18:xds-iti18')

    public void configure() throws Exception {
        ...
        // Entry point for Provide and Register Document Set
        from('xds-iti41:xds-iti41')
        ...
        // Entry point for Register Document Set
        from('xds-iti42:xds-iti42')

    public void configure() throws Exception {
        ...
        // Entry point for Retrieve Document Set
        from('xds-iti43:xds-iti43')

Validating incoming messages

Once the endpoints have been exposed, clients can send in messages. These messages might or might not conform to the IHE specification. It is usually a good idea to validate incoming messages before processing them. The XDS component offer a simple validation. This is not meant to be complete, e.g. it cannot validate that a patient ID is actually known to the registry, but it performs a variety of checks that will simplify our route implementations. All routes of the demo repository perform this basic validation step right after logging the incoming request, e.g. in ITI-43:

        from('xds-iti43:xds-iti43')
            .log(log) { 'received iti43: ' + it.in.getBody(RetrieveDocumentSet.class) }
            .validate().iti43Request()

A validation failure will throw an XDSMetaDataException. We do not need to put any onException handling in the route for this exception. The XDS components convert such validation failures into an equivalent XDS response with the correct error code. Therefore, we can also throw this exception anywhere else in our routing. E.g. the demo repository contains code to check a specific patient ID used by the XDSToolKit tests to see if we identify patient IDs that should be unknown to the registry. Because the demo does not track patients yet, we simply throw an exception if this specific patient ID is found in a request:

        from('direct:checkPatientIds')
            .choice().when { it.in.body.req.submissionSet.patientId.id == '1111111' }
                .fail(UNKNOWN_PATIENT_ID)
            ...

A failure is reported by throwing an XDSMetaDataException via the fail DSL extension that is implemented in RegRepModelExtension.groovy. The code snippet below shows that this is simply a shortcut to throw the exception.

        ProcessorType.metaClass.fail = { message ->
            delegate.process { 
                throw new XDSMetaDataException(message)
            }
        }

If any other exception is thrown in the route, the XDS components will report a general error in the failure response (either XDSRepositoryError or XDSRegistryError). Of course you can use standard exception handling from Camel to handle such cases.

No matter what exception is thrown in our route, Camel - at least in versions 1.x - will try to redeliver the message by default. We do not expect that the failures will magically go away and therefore, we should disable this redelivery behavior. In all route builders you will find the following snippet at the beginning of the configure method. You can also configure the redelivery policy of the deadLetterChannel, but this way is usually sufficient:

        errorHandler(noErrorHandler())

Using the meta data classes

Once validated, we want to start processing an incoming message. The format of the data structures that we receive in the message body is very important. By default we receive instances of the raw ebXML classes. While these might be interesting for some use cases, we often want to use classes that are closer to the XDS meta classes defined by the IHE specification. These meta classes serve two purposes: They ensure conformance with the XDS specification and they are much easier to use than the more generic ebXML classes. The route builders of the demo repository all convert the ebXML bodies to the meta classes after validation. There are different ways to do this. One way is to simply use convertBodyTo which results in our body to be converted from the ebXML class to the meta data class. This is done in the ITI-43 route:

        // Entry point for Retrieve Document Set
        from('xds-iti43:xds-iti43')
            .log(log) { 'received iti43: ' + it.in.getBody(RetrieveDocumentSet.class) }
            // Validate and convert the request
            .validate().iti43Request()
            .convertBodyTo(RetrieveDocumentSet.class)

Another way is to retrieve the meta class instance via getBody. E.g. in the ITI-41 route builder, we transform the input body into a map that contains the actual request object. This allows us to access the request at any stage in our route, no matter what we were currently doing. To create the map we use a transform processing in which we fill the map:

        from('xds-iti41:xds-iti41')
            .log(log) { 'received iti41: ' + it.in.getBody(ProvideAndRegisterDocumentSet.class) }
            // Validate and convert the request
            .validate().iti41Request()
            .transform { 
                [ 'req': it.in.getBody(ProvideAndRegisterDocumentSet.class), 'uuidMap': [:] ] 
            }

In contrast to convertBodyTo, getBody does not replace the body of the message automatically. Check out the log step at the beginning of the route. It uses getBody to retrieve the meta class. The good thing about these classes is that they have meaningful equals, hashCode and toString implementations. The logging step converts the ebXML class on-the-fly and uses its toString method to get a nice textual representation. If we use convertBodyTo instead, the validation step will fail, because it expects an ebXML class in the message body.

Lets look at some typical use cases that require access to the meta classes.

Evaluating the query type

The next code snippet shows the dispatching of an ITI-18 message based on the stored query type. We use content based routing via choice to call sub routes that perform the corresponding query logic. The queryType method is a simple shortcut to get the query type property from the request message. If non of the supported query types is found we throw an exception via the fail processor. All query types that are defined by the IHE specification are listed in the enum org.openehealth.ipf.platform.camel.ihe.xds.commons.requests.query.QueryType.

    public void configure() throws Exception {
        ...

        from('xds-iti18:xds-iti18')
            ...
            // Dispatch to the correct query implementation
            .choice()
                .when { queryType(it) == FIND_DOCUMENTS }.to('direct:findDocs')
                .when { queryType(it) == FIND_SUBMISSION_SETS }.to('direct:findSets')
                .when { queryType(it) == FIND_FOLDERS }.to('direct:findFolders')
                .when { queryType(it) == GET_SUBMISSION_SET_AND_CONTENTS }.to('direct:getSetAndContents')
                .when { queryType(it) == GET_DOCUMENTS }.to('direct:getDocs')
                .when { queryType(it) == GET_FOLDER_AND_CONTENTS }.to('direct:getFolderAndContents')
                .when { queryType(it) == GET_FOLDERS }.to('direct:getFolders')
                .when { queryType(it) == GET_SUBMISSION_SETS }.to('direct:getSets')
                .when { queryType(it) == GET_ASSOCIATIONS }.to('direct:getAssocs')                
                .when { queryType(it) == GET_DOCUMENTS_AND_ASSOCIATIONS }.to('direct:getDocsAndAssocs')
                .when { queryType(it) == GET_FOLDERS_FOR_DOCUMENT }.to('direct:getFoldersForDoc')
                .when { queryType(it) == GET_RELATED_DOCUMENTS }.to('direct:getRelatedDocs')                
                .otherwise().fail(ErrorCode.UNKNOWN_STORED_QUERY)
            .end()

         ...
    }

    def queryType(exchange) { exchange.in.body.req.query.type }

Splitting for individual entry processing

Many XDS transactions work with sets of entries, e.g. we upload or download multiple documents instead of just one. Using the splitter we can break down the request message into its individual entries and process them individually. In the demo repository this is done in many cases. The next snippet of the ITI-43 route shows how to retrieve a document set by retrieving each document from the store one at a time. Using split we perform the actual splitting of the message by taking the list of documents contained in the meta class. We tell the splitter to aggregate a result list using the retrieved documents. This list is going to be in the message body after the splitting functionality has finished processing (indicated by end()). The entries of the list are the result of the processing of retrieve, which is a custom DSL element that calls DataStore.get() to get the contents of the document. Finally we can simply transform the message using the aggregated list of documents and putting it into the meta class for the response.

            // Retrieve each requested document and aggregate them in a list
            .split { it.in.body.documents }
            .aggregate { target, next -> target.out.body.addAll(next.out.body) }
            .retrieve()            
            .end()
            // Create success response
            .transform { new RetrievedDocumentSet(Status.SUCCESS, it.in.body) }

Secure transport

Using HTTPS instead of HTTP requires very little work. In fact, for a registry/repository it does not require anything related to IPF. We simply configure Tomcat to use secure transport for our webservices. With the embedded Tomcat class of the XDS test package, this is only a few lines of code:

        servletServer.secure = args.length == 1 && args[0].equals('secure')
        servletServer.keystoreFile = 'keystore'
        servletServer.keystorePass = 'changeit'
        servletServer.truststoreFile = 'keystore'
        servletServer.truststorePass = 'changeit'

The keystore provided with the test application is the one that is used in the XDSToolKit. Therefore, you can use the demo repository with secure transport enabled tests from the XDSToolKit. Note that those keystores are changed from time to time. You can replace the keystore of the demo repository by simply overwriting the keystore file.

The configuration of a complete Tomcat installation might be a little bit more but it is well documented here. Of course your mileage may vary if you are using a different container. In any case, you will not require additional configuration steps within a registry/repository implementation. Note that clients (source and consumer) will need to configure the endpoint to use HTTPS. This is not part of this guide. You can find more details in the standard documentation.

Auditing

By default auditing is turned on by all endpoints. To configure the syslog server that receives auditing messages have a look at Server.groovy:

    private static final int SYSLOG_PORT = 514
    ...    
    public static void main(String[] args) {
        ...
        AuditorModuleContext.context.config.auditRepositoryHost = 'localhost'
        AuditorModuleContext.context.config.auditRepositoryPort = SYSLOG_PORT
        ...
    }

Auditing messages will always be send. Because they are send unreliably via the UDP protocol (this is the default), the XDS components "do not care" if there is actually a syslog server running at the specified host and port. If you want to see the audit messages that the demo repository logs, you can install a syslog server at localhost using the standard syslog port 514 or you can change the settings in Server.groovy to match your setup.

If you want to disable auditing you can do so by changing the endpoint configurations, e.g. for ITI-18:

        from('xds-iti18:xds-iti18?audit=false')