Transformation Manager is our proven, off-the-shelf data integration and migration tool for developers. The software makes use of some well known data concepts and has some of its own as well. These are the foundation of our approach to transforming data, so read on for a thorough explanation.
Transformation Manager: Data store
The term data store, a general concept, is used in Transformation Manager to describe any structure that holds data, unless a particular type of data store such as a database or an XML document is being referred to.
Transformation Manager: Data model
A data model holds information about the elements, attributes and relationships within a data store. So a data model of a relational database consists of the definitions of its tables, columns and relationships. A data model for an XML Schema or DTD describes both valid elements and attributes for a set of XML instances. An XML data model may also be defined by an XML instance.
Transformation Manager: Relationship
A relationship defines how elements are linked. Relationships can be created, modified and navigated using TM Designer. They are used in projects to access one or more related elements in the source or target.
Transformation Manager: Repository
Transformation Manager uses a repository, which is a central place in which an aggregation of data is kept and maintained in an organized way to hold all aspects of a transformation task. The image below shows the content that a Transformation Manager repository will contain.
Transformation Manager: Project
A project describes the complete transformation from the source model to the target model. A project has references to the source and target models, plus one or more transforms and may include global declarations and Java functions.
There are two principal types of project:
- Primary projects – a project that transforms from a source to target instance.
- Error (or audit) handling projects – a project that processes an error (or audit) output and writes to a target.
Transformation Manager: Project and model versions
Projects and models have both a name and a version within Transformation Manager. The version number is an integer, and the first version number is 1. The name and version number together identify a particular project or model.
Only the version number is relevant to Transformation Manager and this is generally decided by Transformation Manager. Similarly if a model has a name, such as FpML, it need not be known by that name within Transformation Manager; the user can choose a different one.
The version number should be used to differentiate between major versions of projects and models. For example, a project called MYPROJECT Version 1 uses a source model, SRCMODEL Version 1. A major new version of SRCMODEL is to be supported but has limited backward compatibility between the new and old versions of the model. It is likely that the project will also need sizeable changes and in this case a new version of SRCMODEL should be loaded, and a new version of the project created.
The choice of when version numbers should be incremented is a user decision. A user can increment the model and project version each time a minor change is made or never increment the version numbers, using configuration management to maintain previous versions.
Transformation Manager: Model
If a model to be imported into Transformation Manager has an external version number, such as FpML V4.0, then the name can be chosen but the version will be decided by Transformation Manager. It will be 1 the first time the model is loaded.
By default, Transformation Manager will perform incremental loads of Java and relational models. When such a model is imported and a model of that name has already been imported then an incremental load is used. An incremental load is useful when the model changes, for example some new tables are added, and you want them to be available within Transformation Manager. The incremental load will add any unknown tables and columns to the existing model. Any enhancements and changes made within TM Designer to the original model, such as relationships added or modified, will still exist.
Transformation Manager does not support incremental loads for XML models. Whether third party generic loaders support incremental loads depends on the implementation. With incremental loads the version number does not change – it is a modification of an existing model.
With non-incremental loads, if the name already exists in the repository the version number is increased. The model is entirely separate from the original model, and no modifications to the original model will be made to the new model. However, note that this can be achieved using Transformation Manager’s Change Model feature.
Incremental import can be turned off via a user preference. In this case, Java and relational model imports will also increase the version number.
When a user creates a project all versions of the models in the repository can be selected as the source and target for the project.
Transformation Manager: Transform
A transform contains assignments and rules expressed using Simple Mapping Language (SML) in the context of a given source and target model element. Transforms are added to projects and defined using the Editor pane of TM Designer. The Editor pane provides users with a code editor where they can type directly into the transform or where the user can drag and drop into and between the source and target data element to create transform assignments.
Transformation Manager: Independent and dependent transforms
When you execute a project a starting point is required. A project contains one or more independent transforms, and these are executed in turn, processing the relevant source instances or records and applying the rules defined. A project may contain many transforms. They can either be independent or dependent.
- Independent transforms are run one after the other and in no particular order.
- Dependent transforms are only run if they are called from another transform and therefore do have an explicit order. That calling transform can either be an independent or a dependent transform.
An independent transform may call other transforms using the cascade mechanism. A called transform is called a dependent transform. A dependent transform is only executed via a cascade.
Transformation Manager: Identification
Identification specifies what is unique about an element. As such, it can be used to ensure that multiple instances of the same attribute, for instance name, in the source data will not result in multiple instances of the same name in the target. So, the first instance of a particular name in the source will result in the creation of an instance in the target if one does not already exist, but subsequent occurrences of the same name will not. Thus, several source items will be merged into one target item. If no identification is specified, no merging occurs and a new element is created each time.
Click on this link to read more about the software: Transformation Manager. And don’t hesitate to contact us if you have any questions.
