Flow Genesis Terms II – defining Diagnostic Reports, Templates and Deployment Packages

Introduction

Ah, the much-anticipated sequel! Will it exceed the beloved original, like “The Godfather Part II”, or will it be a massive disappointment like “Hannibal”?!

This is a follow-up article to “Flow Genesis Terms I – defining Models and Snapshots“, so please read that article first to get the full perspective.

What is a Diagnostic Report?

The Diagnose tab in Flow Genesis allows you to automatically generate a list of common issues associated with any snapshot you have uploaded. This is a useful check to perform before deployment or UAT, as it can often catch errors that are difficult to detect with the human eye.

You can then save the list into a Diagnostic Report to keep a permanent record of issues found. The file created is referred to as a Diagnostic Report, and contains everything you can see on the screen when you diagnose a snapshot.

Once saved, you can also add Sticky Notes to explain or discuss errors with members of the team. This provides a useful method of collaboration to make sure things are not missed during any particular deployment.

It also provides a way for non-technical team leaders or project managers to verify that no issues exist that have not been sufficiently explained by the developer team.

Note the in the future, we will be adding a feature to export a Diagnostic Report into a more easily-readable issue-list document, so it’s worth saving reports now to take advantage of that additional feature when it becomes available.

What is a Template?

If you’ve ever worked on a multi-country or multi-department roll-out, chances are you’ve already thought about templating without realizing it.

Often when similar models are to be deployed in multiple places, the approach taken is to create one generic template model, then use that as a basis for deployment to the other areas. The generic model can be tested and verified by the customer first, before any area-specific customizations are carried out.

In Genesis, a Template refers to an abstracted Snapshot file which allows users to provide specific customizations for designated objects before deployment.

This is useful in POC development or when starting a project, as it gives you a pre-tested and risk-free starting point for any custom development without locking you into a particular naming standard.

The template file contains all the same data included in the Snapshot, along with instructions on how the template will behave when it is applied.

The output of applying a Template is a new Snapshot containing the customizations specified by the user. This Snapshot can be used to deploy a new server, or to upgrade an existing one with the new names.

We have big plans for the Templating feature of Genesis, including a wizard-based interface for many common customizations beyond naming, so it’s well worth investigating how it can help your business processes now.

What is a Deployment Package?

So you have your model completed, your Templates applied and your Snapshot files created… what next?

That’s where Deployment Packages come in. A Genesis Deployment Package is basically a list of instructions to carry out on a model to convert it from one state of development to another.

More simply put, it stores the differences between two Snapshots so that one can be automatically changed into the other.

The first step toward creating a Deployment Package is to perform a “diff”, or comparison between two Snapshots. Diff is a computing term for an operation that identifies differences between two files, in our case the two Snapshot files.

It is important to note that in almost all cases, the first Snapshot used as the basis for the Deployment Package should be a Snapshot of the server your are deploying to. This is to ensure all the generated instructions are valid for the target server. It is sometimes valid to violate this rule, but make sure you understand exactly what you are doing.

The second Snapshot should be the one that represents the new state of the target server.

So, for instance, if you have completed UAT on a staging server, and would like to deploy changes to production, the first Snapshot should be an up-to-date version of the production server, and the second snapshot should be an up-to-date version of the production server. This would result in a series of instructions that would implement all the recent development on the staging server into the production server.

When exported, Deployment Packages have the file extension .xdpd, which stands for “XML Deployment Package Document”. To perform the deployment, you can load the generated file into the Flow Model Packager.

Conclusion

I know sequels are often more convoluted and confusing that the original, so I guess there is a little “sequelitis” happening here!

Hopefully this gives you a better understanding of the terms and features used in Genesis and how you might apply them to your business. Trust me, once you’ve mastered Diagnosis, Templating and Deployment the Genesis way, you’ll wonder how you ever did without it!

Any questions, feel free to use the comments in this article.

Flow Genesis Terms I – defining Models and Snapshots.

What is a Snapshot?

A snapshot is an XML file that contains information about the structure (metadata) of an OLAP model.

Although it sounds like a simple idea, snapshots can have some very powerful applications. They allow the Flow applications to read and display information about a model without being connected to the TM1 Server via the TM1 API. This means tasks can be carried out “offline”, without the risk of disrupting day-to-day TM1 Server functions.

A snapshot file is also a version. That is to say, it is the model at a specific point in time. This makes tasks such as backup, restore, archiving and version tracking possible, and much easier than they would be with a typical TM1 implementation.

Snapshots have been designed to be OLAP database agnostic, meaning they could potentially also describe an OLAP structure in a product other than TM1. However, Flow’s focus is exclusively on TM1 at present.

When exported to a file, Snapshots have the file extension “.xssd”, which stands for “XML Snapshot Document”.

What is a Model?

It’s tempting to assume the terms Model and Snapshot are interchangeable, but they actually have very specific meanings in Genesis.

A Model is simply a container for one or more Snapshots. In its most common usage, you can think of a Model like a project, with the snapshots it contains being the many versions that the project goes through during the development and production life cycle.

Sometimes you might utilize the Model concept in your own way, and that’s ok. Genesis was designed as a tool that lets you work the way you want without prescribing a rigid methodology.

However, generally, we would recommend using the Category field to collect Models into wider groups, such as customer, department or purpose.

What’s inside a Snapshot?

A snapshot contains XML markup to describe the structure of an OLAP Model. From this information, it is possible to perform various analysis of the server structures, automatically recreate the server from scratch, or update an older server with newer structures and code.

To understand snapshots better, you can open one in a standard text editor and review the markup.

You can see standard XML structures within the file, with each “node” having a simple begin and end tag, with optional attributes. Each node can contain other nodes inside it, as in the following example:

<Node att1="val1" att2="val2">
  <Node att1="val1" att2="val2"></Node>
</Node>

Once you understand the basic XML structure, it becomes possible to decipher the contents of the snapshot file. It can be a little difficult at first, because the markup is designed to be as compact as possible, but let’s review the core elements.

The root node is always Database. Hence, the most simple Snapshot possible would be the following:

<Database ID="1" NM="TM1_Server_Name"></Database>

This simply declares the database with an ID and a Name (NM).

Within a Database, a Snapshot can also include Stats, which are pieces of information about the database, and then any Cubes, Dimensions and ETL Processes it contains.

So equally, the following is also a valid representation of an empty server:

<Database ID="1" NM="TM1_Server_Name">
  <Cubes></Cubes>
  <Dims></Dims>
  <Procs></Procs>
</Database>

Each of the “Cubes”, “Dims” and “Procs” nodes can contain multiple markup descriptions of Cubes, Dimensions and ETL Processes respectively. From this simple structure, we can build up a more complete description of a TM1 server, like so:

<Database ID="1" NM="TM1_Server_Name">
  <Cubes>
    <Cube ID="1" NM="Cube1" DimIDs="1 2 3">...</Cube>
    <Cube ID="2" NM="Cube2" DimIDs="1 2 3">...</Cube>
    <Cube ID="3" NM="Cube3" DimIDs="1 2 3">...</Cube>
    ...
  </Cubes>
  <Dims>
    <Dim ID="1" NM="Dim1">...</Dim>
    <Dim ID="2" NM="Dim2">...</Dim>
    <Dim ID="3" NM="Dim3">...</Dim>
    ...
  </Dims>
  <Procs>
    <Proc ID="1" NM="Proc1">...</Proc>
    <Proc ID="2" NM="Proc2">...</Proc>
    <Proc ID="3" NM="Proc3">...</Proc>
    ...
  </Procs>
</Database>

Note that each object has an ID assigned. This is to make the markup more efficient, so when one object needs to refer to another, it can use it’s ID instead of the full name of the object.

A good example of this is where a Cube defines the Dimensions it contains. This is denoted by a list of IDs of the Dimensions, as opposed to a list of names, which could become quite verbose.

It is beyond the scope of this article to discuss the detailed structures of each object in a Snapshot file, but hopefully this gives you enough of an overview to get a basic understanding of how a Snapshot file works, and what the various terms associated with them mean.

We are currently working on a detailed whitepaper detailing the complete Snapshot format, in the hope it may form the basis of an open standard within the OLAP modelling community.

What does a Snapshot NOT contain?

Snapshot files are data-agnostic, so do not contain any data. The reason for this is both to ensure data security and separation of purpose — it makes sense that the design of a cube should be separate from the data contained within it.

There are already plenty of other formats that can handle data well, such as CSV or Tab Delimited text files. We recommend using those standards to import and export data to and from your TM1 servers.

Hidden control objects (such as TM1 “}ElementAttrbutes” cubes and dimensions) are also excluded, but are represented by more generic XML structures so that the information is preserved in a database-agnostic way.

How can I use a Snapshot?

Once you have developed a TM1 model, you can create a snapshot file using the Flow Model Packager.

Snapshot files can then be uploaded to Flow Genesis to be used as the basis for Exploring, Diagnosing, Templating and Deploying TM1 Models.

You can also save the snapshot files for later use, or archive them. If you require version control, Snapshot files are ideal for adding into a Version Control repository, such as SVN, CVS or Microsoft Visual SourceSafe.

Because Snapshot files are implemented with an open standard, you can also build software to read, write and manipulate Snapshot files.

This opens up a whole wealth of application possibilities, which we hope will lead to some innovative software from the TM1 and wider OLAP developer community.

Conclusion

Hopefully this article will give you an insight into Model and Snapshots, how they are different and how they work within the context of Flow Genesis.

Part II of this article will cover the other file types and standards in Flow Genesis.