Developers' Notes: Importing Data and Models (FinRep and CoRep)

Data Designer is based on an extensible architecture, both through the use of the Eclipse Modeling Framework (EMF), enabling new data models to be rapidly incorporated, and OSGI to allow one to extend application functionality through plug-ins. Here we examine the former, importing metadata from the European Banking Authority's the CRD IV / CRR framework, also known as FinRep and CoRep.

Though EMF uses the XMI (XML Metadata Interchange) format for data and model persistence and specifically ECore for data modeling, numerous adapters exist to manage other formats. For example, EMF has native capabilities to import and work with XML and XSD files.

FinRep and CoRep metadata is provided in Excel and Access database formats, with Word supporting documentation. For our exercise, we exported the Access database tables in XML format (link to an example of domain.xml) with the accompanying schema in XSD (link to an example of domain.xsd). As mentioned, one could also use an adapter incorporating a JDBC / ODBC driver to directly query the database.

Converting XSD to ECore

EMF provides a UI through which one can convert XML and XSD files into XMI and ECore. However, for a large number of files this is very time consuming. A scaleable approach is to do it programmatically.

Figure 1 shows a method used to convert a XSD file into a ECore file. The key method is EMF's XSDEcoreBuilder that takes the XSD file and converts it to a valid ECore. The last line uses a method to register these new Ecore packages as resources in the Eclipse environment so they can be used to convert the XML data files into valid XMI files that conform to their appropriate data models.

public void initModelFromXsd(List<string> xsdFiles) throws IOException {
    
    final Collection<epackage eCorePackages = new LinkedList<epackage>();

    //This method itereates through the list using EMF's 'URI.createFileURI'
    //method to create URI's for each file.
    Collection<uri> uriList = getSchemaUris(xsdFiles);

    for (URI uri : uriList) {
     //Use EMF's XSDEcoreBuilder to create Ecore's from the XSD files.
  Collection<eobject> loadedPkgs = new XSDEcoreBuilder().generate(uri);
    
        for (EObject loadedObject : loadedPkgs) {
            if (loadedObject instanceof EPackage) {
                eCorePackages.add((EPackage) loadedObject);
            }
        }
    }

    //Register the collection of eCore packages in the Eclipse environment
    this.loadedModelResources = registerDynamicPackages(eCorePackages);
}

Figure 1: Creating ECore packages from the XSD files

In the registerDynamicPackages method we carry out several steps (see figure 2). The first is to register that any URI's ending in xml, xmi or ecore extensions will use the corresponding EMF resource factory. We will need this for saving the XMI and ECore files.

Next, we register each EPackage we created in the package registry, create a resource for each and then save each as a '.ecore' file.

private ResourceSet registerDynamicPackages (final Collection<epackage> eCorePackages)
    throws IOException {

    //Create a new resource set that we can add resources to.
    final ResourceSet resourceSet = new ResourceSetImpl();

    // Necessary when running standalone for no factories have been registered yet:
    resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put( "xml",
        new XMLResourceFactoryImpl());
    resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put( "xmi",
 new XMIResourceFactoryImpl());
    resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put( "ecore",
 new EcoreResourceFactoryImpl());

    for (EPackage ePackage: eCorePackages) {
        //Add the ePackages to the package registry so they can be used later.
 resourceSet.getPackageRegistry().put(ePackage.getNsURI(), ePackage);

 //Put ecore on the file name so we have the right extension when saving.
 URI xsdUri = URI.createURI(ePackage.getNsURI());
 URI uri = xsdUri.trimFileExtension().appendFileExtension("ecore");

 //Create a resource from the ePackage and then save it as an eCore file.
 Resource eResourceFromPackage = resourceSet.createResource(uri);
   eResourceFromPackage.getContents().add(ePackage);
   eResourceFromPackage.save(null);       
    }
 
    return resourceSet;
}

Figure 2: Registering resources and saving ECore files

Converting XML to XMI

The code shown in figure 3 loads a XML file and serialises it as a XMI file. With the data in XMI format and the data model in ECore we could load each table into Data Designer if we wish. However, if we translate this data to conform to the same data model used for our FDSF work it gives us the ability to easily make references between the two regulatory frameworks.

public void loadFromXML(String xmlFile) throws IOException {

    // Create the URI of the model file.
    URI fileURI = URI.createFileURI(new File(xmlFile).getAbsolutePath());

    //These are some EMF options we can set when loading a resource.
    Map<String, Object> options = new HashMap<String, Object>();
    options.put(XMLResource.OPTION_EXTENDED_META_DATA, Boolean.TRUE);

    //Load the XML file into a resource and get its ePackage.
    //loadedModelResources is a global resource set we created when converting
    //the XSD files into ECore.
    final Resource resourceXML = loadedModelResources.createResource(fileURI);
    resourceXML.load(options);
    EPackage packageFromResource = resourceXML
        .getContents().get(0).eClass().getEPackage();

    Resource resourceXMI = resourceXML;

    //Some formatting to get the correct namespace URI we want.
    URI uri =
        resourceXML.getURI().trimFileExtension().appendFileExtension("ecore");
    String nsURI = uri.isPlatform() ? uri.toPlatformString(true) : uri.toString();
    String prefix = uri.trimSegments(3).toString();
    nsURI = nsURI.replace(prefix, "");
 
    //Set the namespace and URI for the XMI and save it to a XMI file
    resourceXMI.getContents().get(0).eClass().getEPackage().setNsURI(nsURI);
    resourceXMI.setURI(resourceXML.getURI()
        .trimFileExtension().appendFileExtension("xmi"));
    options.put(XMIResource.OPTION_SCHEMA_LOCATION, Boolean.TRUE);
    resourceXMI.save(options);
}

Figure 3: Load the XML file and serialise it as a XMI file

Transforming EBA's Data Point Model to FDSF's Data Model using ATL

There are numerous translation tools one can use such as the ATL Transformation Language (ATL) or the Epsilon Transformation Language (ETL). We use ATL for our example as shown in figure 4. This code snippet shows a transformation of EBA's Dimensions into FDSF's Explicit and Typed Dimensions.  The first three lines show that it is using the ECore files we generated for Dimensions and Domains from the Access database tables, along with FDSF's meta-model.

The two 'rules' take a Dimension and check whether it is typed or not (i.e., explicitly enumerated).  If it is not typed, the first rule casts it to a FDSF Explicit Dimension, mapping its label, code and description attributes to the corresponding FDSF attributes of name, code and definition.

The last line of each rule uses a 'helper' function called 'getDomain'.    This function uses the domainID attribute of a Dimension and searches through the list of Domains until it finds the match.  This Domain is then referenced by the FDSF reference called 'usesExcplicitDomain'.

For full details on ATL please refer to its documentation.

Once the full ATL transformation is run, all dimensions, domains, hierarchies, metrics, etc. of FinRep / CoRep have been mapped to the FDSF data model with the results stored in a XMI file.  This XMI file is examined in Data Designer: Integrating FDSF and FinRep / CoRep.

-- @path FDSF=/com.jcchapman.fdsf.model/model/FDSFMetamodel.ecore
-- @path Dimension=/com.jcchapman.eba.dpm.import/models/Dimension.ecore
-- @path Dimension=/com.jcchapman.eba.dpm.import/models/Domain.ecore


module eba2fdsf;
create OUT : FDSF from IN1 : Dimension, IN2:Domain;

helper def : getDomain(dID:Integer):Domain!DomainType=
 Domain!DomainType.allInstances()->any(domain | domain.domainID = dID);

rule Dimension2ExplicitDimension {
    from
 e : Dimension!DimensionType (not e.isTyped)
    to 
 f : FDSF!ExplicitDimension (
  name <- e.dimensionLabel,
  code <- e.dimensionCode,
  definition <- e.dimensionDescription,
  usesExcplicitDomain <- thisModule.getDomain(e.domainID)
 )
}

rule Dimension2TypedDimension {
    from
 e : Dimension!DimensionType (e.isTyped)
    to 
 f : FDSF!TypedDimension (
  name <- e.dimensionLabel,
  code <- e.dimensionCode,
  definition <- e.dimensionDescription,
  usesTypedDomain <- thisModule.getDomain(e.domainID)
 )
}

Figure 4: ATL transformation of EBA Dimensions to FDSF Explicit and Typed Dimensions