TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Friday, September 1, 2017

TDI Getting Started and Ensuring Basic Understanding

The Basics

  • Install TDI (https://ibm.biz/BdjsLw) locally to your laptop. This lets you customize your TDI environment easier. If you can't get a licensed version, start with the free version (https://ibm.biz/BdjiMg). It's fully functionally, will give you time needed to get started solving with TDI, and the license file can be replaced when you get yours.
    • Do NOT install under Program Files on Windows, or any other auto-secured folder. Install to C:\TDI\V7.1.1 (and V7.2). 
    • Note that you can have multiple versions of TDI installed, but each TDI Server needs to have its own Solution Directory (https://ibm.biz/BdXX8h) if they will be running simultaneously. If not then they can share a single Solution Directory. This is how I organize my TDI.
    • If you are installing on Linux there are pre-installation steps as follows:
      • For Redhat/CentOS:
      1. yum upgrade libstdc++
      2. yum install libstdc++.i686
      3. yum upgrade zlib 
      4. yum install zlib.i686
      • For Ubuntu
      1. sudo apt-get update
      2. sudo apt-get install libstdc++5 libstdc++5:i386
      3. touch /etc/inittab
      4. remove /etc/inittab after installation
  • Install the latest TDI Fixpack, linked to from the TDI portal page (http://www.tdi-users.org)
    • Quick Guide to installing a Fixpack: 
      • Unzip the Fixpack zip and grab the file named TDI-7.1.1-FP000*.zip (or SDI-*) and copy it to the TDI installation folder. 
      • Grab the UpdateInstaller.jar file from the zip as well and copy it to the maintenance sub-folder of the TDI installation directory.
      • Make sure TDI is NOT running and then open a commandline window to this folder. Execute the following:
                             (example for installing FP6 on TDI 7.1.1)
                             Windows: bin\applyUpdates -update TDI-7.1.1-FP00006.zip
                             Linux:        bin/applyUpdates -update TDI-7.1.1-FP00006.zip 
  • Make sure you understand the basics:
    • You need to know what an AssemblyLine is (https://ibm.biz/BdjsQL). In addition to being a com.ibm.di.server.AssemblyLine (https://ibm.biz/BdjiMu), every AL is also a Java Task. TDI is pure Java, and you can mix and match standard and custom libraries, and use techniques you know from Java development.
    • Know how to watch data flowing with the AssemblyLine Debugger (https://ibm.biz/Bdjiy3).
    • Grab this PDF detailing the workflows in TDI (https://ibm.biz/BdjeQm) and keep it close while building solutions and exploring options.
    • Understand that AssemblyLine Components are simply the puzzle pieces you can click together to form AssemblyLines. AL Components can be of different types, like Connectors (https://ibm.biz/BdjiVw), Scripts (https://ibm.biz/BdjiVt) or Attribute Maps (https://ibm.biz/BdjiVU).
    • See how Connector Interfaces (https://ibm.biz/BdjsQF) are attached to AL Components in order to connect to data. Connector Interfaces are often referred to just as specific 'Connectors', like LDAP, Database or MQ.
    • And get to know the AssemblyLine Debugger (https://ibm.biz/Bdjiy3). With it you can answer a lot of questions that arise when you're working to get data where it needs to be.
    • You need to know how to use the various Entry objects TDI gives you (https://ibm.biz/Bdjs9f). And you can create your own if you want to, for example, to accumulate data between iterations.
    • As well as understand how Attributes store data (https://ibm.biz/Bdjs9g). Hint: the Debugger shows you.
    • Of course, understand how the built-in flows of the AssemblyLine and all Connector Modes (like Iterator, AddOnly and Update) let you customize behavior through scripting in Hooks (https://ibm.biz/Bdjs9a). Error Hooks and Reconnect configuration are keys to a robust dataflow.
    • You'll want to get familiar with the 'system' object (https://ibm.biz/BdjiyW). It gives you handy functions for all sorts of things like date formatting/parsing, skipping or repeating the current Work Entry, and invoking commandlines. Note, full JavaDocs are installed with your TDI: <InstallDir>/docs/api/index.html
    • Loops, in particular Connector Loops (https://ibm.biz/BdjiM9 and https://ibm.biz/BdjiMC), can be game changers.
    • Last but far from least, you should be comfortable with Javascript (https://ibm.biz/BdjsQV).
  • Follow these links to learn more, and note that even content labelled '7.0' or '7.1' applies for all versions of TDI: 

Solution Guidelines - The Basics

  • Create a project folder under the Solution Directory for each solution. Put all support files (Config, properties, etc.) in this folder and always use relative paths to reference these files. This makes your solution easy to package and portable (https://ibm.biz/BdXX8h).
  • Use the Resources library to define AL components (Connectors, Scripts, Attribute Maps etc) so that they can be re-used in multiple AssemblyLines. Or even Pooled if you want to limit connections.
  • Configure Connectors for Connection Errors and Auto Reconnect (https://ibm.biz/BdjiMP). Usually you only want AutoReconnect for Connection Lost situations. You can even define which Exceptions a Connector should consider as 'Connection Lost', initiating Auto Reconnect behavior (https://ibm.biz/BdjiMY).
  • Implement an Error Handling strategy for your solution (https://ibm.biz/BdjiMS). It could be to simply to log all errors and continue (when possible). If so, try to include as much relevant information as possible, including the original Error you are respondng to.
    • For an Error Hook in a Connector, it could look like this:
var data = "\n##DATA\nwork: "  + work.toJSON(); // handy format for later parsing
if (typeof conn != "undefined") { data += "\nconn: " + conn.toString(); }
if (typeof current != "undefined") { data += "\ncurrent: " + current.toString(); }
var where = thisConnector.getName()
                     + " (" + thisConnector.connector.getClass() + ")"; // Connector Interface
throw "Error for " + where + "\nerror: " + error.toJSON() + data;    

    • Whereas a try-catch block would look like this:
try {
    // doing something that might exception out
} catch (ex) {
    var data = "\n##DATA\nwork: "  + work.toJSON();
    if (typeof conn != "undefined") { data += "\nconn: " + conn.toString();
    if (typeof current != "undefined") { data += "\ncurrent: " + current.toString(); }
    throw "Function foo(bar) failed, bar = '" + bar + "'\n" + ex + data;
}


  • Watch this bullet for future content...

7 comments:

Unknown said...

This is a great help at picking apart the enigma of TDI. I really wish IBM had put together some example documentation like this.

Eddie Hartman said...

@Jared - Thanks :) And please let me know what else you'd like to see.

Unknown said...

Thank you eddie for creating great resource to learn TDI.
It would be great to have a repository with sample AL's to learn from.

I see lot of questions on forum regarding SSL, wouldn't it be easier if people can import a SSL sample AL in their TDI and get past the common mistakes they make.

I am asking for something similar to Angular tour of Heroes Tutorial that introduces new folks to angular.

Eddie Hartman said...

So true. I do have plans for some github resources, but also a bad conscience for not having done anything about it.

Keep pushing me please :D

/Eddie

Unknown said...

Haha !

Well, my mentor always told me not to reinvent the wheel, maybe this git repo will help you : https://github.com/Pradhan10/IBM-TDI-AL

May the force be with you !
https://github.com/Pradhan10/IBM-TDI-Documentation

DeepThought said...

I had an issue installing SDI 72 on CentOS 7. Kept getting the CTGDKG023E error when starting the default server. (java.lang.UnsatisfiedLinkError in i4clntjni).

The SLES8 fix also worked for me on CentOS 7: see: https://www.ibm.com/support/pages/sdi-72-how-install-red-hat-enterprise-linux%C2%AE-8

The relevant commands are:
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/compat-libstdc++-33-3.2.3-72.el7.x86_64.rpm
yum localinstall compat-libstdc++-33-3.2.3-72.el7.x86_64.rpm

Hope this saves some people some time
Cheers!

Eddie Hartman said...

Thanks for sharing, @DeepThought.