TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Monday, June 17, 2019

How To Ask Smart Questions

Although this article is particularly about asking hackers there are some universally relevant pearls here that are ring particularly true today:

http://catb.org/~esr/faqs/smart-questions.html

It's easiest to go for the quick fix when it comes to information gathering. It's apparent in the alarming number of people who's primary news source is social media, and it's highlighted nicely in this article about how Google Maps is making us dumber:

https://www.vocativ.com/418342/google-maps-gps-navigation/index.html

In particular, it's about exercising the gray cells to uncover the desired knowledge. This list of steps from the first article above is by no means exhaustive:

--- excerpt

Before You Ask

Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following:
  1. Try to find an answer by searching the archives of the forum or mailing list you plan to post to.
  2. Try to find an answer by searching the Web.
  3. Try to find an answer by reading the manual.
  4. Try to find an answer by reading a FAQ.
  5. Try to find an answer by inspection or experimentation.
  6. Try to find an answer by asking a skilled friend.
  7. If you're a programmer, try to find an answer by reading the source code.
---

Of course, sometimes speed is critical. But a little forethought and investigation will help make your question easier and faster to answer.

And since I see the world through TDI-colored specs, I going to add that


  • You include 'java' to any searches involving TDI research and troubleshooting (step 2)
  • The AL Debugger is a great tool for step 5
  • The TDI user forum is where many of your skilled friends can be found* :)


*Link here: www.tdi-users.org

Tuesday, June 4, 2019

SDI/TDI as a Windows Service

NOTE: You must remember to copy the ibmdiservice.exe and ibmdiservice.props files from the <InstallDir>\win32_service folder to your Solution Directory (soldir) before you run the service installer.

It is a common mistake to run the service installer from the wrong folder - i.e. not your soldir. Often this is done from <InstallDir>\win32_service where the .exe and .props files are found.

And if you don't know what the soldir means to an SDI/TDI Server, please have a look here: https://ibm.biz/BdjiM6 for this and much more Integrator Know-how you should know.

Note also that you can fire up the Config Editor (CE), ensure the 'Default' Server document points to the same port, install folder and solution directory, and monitor and operate the running server.


Tuesday, September 25, 2018

Thinking about agility

My favorite agile tool is TDI (and please forgive an old man for having trouble shifting to 'SDI'). Not only can I whip together integration service prototypes faster than anything I can do in Java (Spring), or Python or TurboPascal, but I can also build Ops features this way. Monitoring and remote control (pause, restart, failover/failback) for enabling auto-healing, or at least making the solution cheaper to own. As you know, the cost of building a service can quickly be dwarfed by the cost of owning that service over time.

My recent history as a GBS consultant has made DevOps a focus of my job and I have been applying modern tools and techniques to TDI. This includes using git for source management, which ties nicely into CI/CD pipelines. Jason, of Adventures in TDI and TDI support fame, helped me to install eGit in my SDI CE both on Windows and Linux. Git was created by Linux Torvald who gave us Linux, so it's really easy, fast and a de facto standard.

Of course, to build a DevOps pipeline I needed a 'TDI compiler' to turn the TDI workspace Project folder structure and files into the one big Config xml file that the TDI Server needs. So I wrote this Python script. You run it from a command line like this:

    python configify.py -p <Project path>

With these optional parameters:

     -h                 Help & usage instructions
     -v                 Display version
     -c <path>     Write the config to the file specified by <path>
     -n <name>   Set the Solution Instance name (web api) to <name>
     -o                 If present, overwrite the resulting config file

You can grab the script by installing the git commandline and typing someplace where you want the files downloaded:

       git clone https://github.com/eddiehartman/configify.git

Once this was working I focused on Dockerizing a TDI solution. This was pretty simple to do, so I first made a default TDI (SDI) image and then can roll new Dockerfiles to create images based on specific solutions (configs). Then I pushed this up to a Kubernetes cluster and tested it. Pretty simple and the answer to HA and scalability, if you're willing to design for componentization.

Now my next step is to tie this altogether in a pipeline for continuous integration and delivery. Thinking of using Jenkins since I have used it before and have lots of old Jenkinsfiles to copy/paste from. Once I have everything working I though I might quick video.

Sound like a plan? Interested? I know that TDI'ers are often engaged in quick one-off integrations, like security Adapters for ISIM or IGI or PIM. However, for those delivering services (infrastructure wiring) I would think DevOps is important. I look forward to hearing back from y'all :)

Tuesday, October 24, 2017

Collecting my ramblings on Connector Loops in one place

Monday, September 30, 2013 - Why Loops Instead Of Lookup Mode (https://ibm.biz/BdjiMC)

Thursday, November 7, 2013 - Why Connector Loop (https://ibm.biz/BdjiM9)

Tuesday, June 8, 2010 (https://ibm.biz/BdjiMQ)



Friday, September 1, 2017

TDI Getting Started and Ensuring Basic Understanding

The Basics

  • Install TDI (https://ibm.biz/BdjsLw) locally to your laptop. This lets you customize your TDI environment easier. If you can't get a licensed version, start with the free version (https://ibm.biz/BdjiMg). It's fully functionally, will give you time needed to get started solving with TDI, and the license file can be replaced when you get yours.
    • Do NOT install under Program Files on Windows, or any other auto-secured folder. Install to C:\TDI\V7.1.1 (and V7.2). 
    • Note that you can have multiple versions of TDI installed, but each TDI Server needs to have its own Solution Directory (https://ibm.biz/BdXX8h) if they will be running simultaneously. If not then they can share a single Solution Directory. This is how I organize my TDI.
    • If you are installing on Linux there are pre-installation steps as follows:
      • For Redhat/CentOS:
      1. yum upgrade libstdc++
      2. yum install libstdc++.i686
      3. yum upgrade zlib 
      4. yum install zlib.i686
      • For Ubuntu
      1. sudo apt-get update
      2. sudo apt-get install libstdc++5 libstdc++5:i386
      3. touch /etc/inittab
      4. remove /etc/inittab after installation
  • Install the latest TDI Fixpack, linked to from the TDI portal page (http://www.tdi-users.org)
    • Quick Guide to installing a Fixpack: 
      • Unzip the Fixpack zip and grab the file named TDI-7.1.1-FP000*.zip (or SDI-*) and copy it to the TDI installation folder. 
      • Grab the UpdateInstaller.jar file from the zip as well and copy it to the maintenance sub-folder of the TDI installation directory.
      • Make sure TDI is NOT running and then open a commandline window to this folder. Execute the following:
                             (example for installing FP6 on TDI 7.1.1)
                             Windows: bin\applyUpdates -update TDI-7.1.1-FP00006.zip
                             Linux:        bin/applyUpdates -update TDI-7.1.1-FP00006.zip 
  • Make sure you understand the basics:
    • You need to know what an AssemblyLine is (https://ibm.biz/BdjsQL). In addition to being a com.ibm.di.server.AssemblyLine (https://ibm.biz/BdjiMu), every AL is also a Java Task. TDI is pure Java, and you can mix and match standard and custom libraries, and use techniques you know from Java development.
    • Know how to watch data flowing with the AssemblyLine Debugger (https://ibm.biz/Bdjiy3).
    • Grab this PDF detailing the workflows in TDI (https://ibm.biz/BdjeQm) and keep it close while building solutions and exploring options.
    • Understand that AssemblyLine Components are simply the puzzle pieces you can click together to form AssemblyLines. AL Components can be of different types, like Connectors (https://ibm.biz/BdjiVw), Scripts (https://ibm.biz/BdjiVt) or Attribute Maps (https://ibm.biz/BdjiVU).
    • See how Connector Interfaces (https://ibm.biz/BdjsQF) are attached to AL Components in order to connect to data. Connector Interfaces are often referred to just as specific 'Connectors', like LDAP, Database or MQ.
    • And get to know the AssemblyLine Debugger (https://ibm.biz/Bdjiy3). With it you can answer a lot of questions that arise when you're working to get data where it needs to be.
    • You need to know how to use the various Entry objects TDI gives you (https://ibm.biz/Bdjs9f). And you can create your own if you want to, for example, to accumulate data between iterations.
    • As well as understand how Attributes store data (https://ibm.biz/Bdjs9g). Hint: the Debugger shows you.
    • Of course, understand how the built-in flows of the AssemblyLine and all Connector Modes (like Iterator, AddOnly and Update) let you customize behavior through scripting in Hooks (https://ibm.biz/Bdjs9a). Error Hooks and Reconnect configuration are keys to a robust dataflow.
    • You'll want to get familiar with the 'system' object (https://ibm.biz/BdjiyW). It gives you handy functions for all sorts of things like date formatting/parsing, skipping or repeating the current Work Entry, and invoking commandlines. Note, full JavaDocs are installed with your TDI: <InstallDir>/docs/api/index.html
    • Loops, in particular Connector Loops (https://ibm.biz/BdjiM9 and https://ibm.biz/BdjiMC), can be game changers.
    • Last but far from least, you should be comfortable with Javascript (https://ibm.biz/BdjsQV).
  • Follow these links to learn more, and note that even content labelled '7.0' or '7.1' applies for all versions of TDI: 

Solution Guidelines - The Basics

  • Create a project folder under the Solution Directory for each solution. Put all support files (Config, properties, etc.) in this folder and always use relative paths to reference these files. This makes your solution easy to package and portable (https://ibm.biz/BdXX8h).
  • Use the Resources library to define AL components (Connectors, Scripts, Attribute Maps etc) so that they can be re-used in multiple AssemblyLines. Or even Pooled if you want to limit connections.
  • Configure Connectors for Connection Errors and Auto Reconnect (https://ibm.biz/BdjiMP). Usually you only want AutoReconnect for Connection Lost situations. You can even define which Exceptions a Connector should consider as 'Connection Lost', initiating Auto Reconnect behavior (https://ibm.biz/BdjiMY).
  • Implement an Error Handling strategy for your solution (https://ibm.biz/BdjiMS). It could be to simply to log all errors and continue (when possible). If so, try to include as much relevant information as possible, including the original Error you are respondng to.
    • For an Error Hook in a Connector, it could look like this:
var data = "\n##DATA\nwork: "  + work.toJSON(); // handy format for later parsing
if (typeof conn != "undefined") { data += "\nconn: " + conn.toString(); }
if (typeof current != "undefined") { data += "\ncurrent: " + current.toString(); }
var where = thisConnector.getName()
                     + " (" + thisConnector.connector.getClass() + ")"; // Connector Interface
throw "Error for " + where + "\nerror: " + error.toJSON() + data;    

    • Whereas a try-catch block would look like this:
try {
    // doing something that might exception out
} catch (ex) {
    var data = "\n##DATA\nwork: "  + work.toJSON();
    if (typeof conn != "undefined") { data += "\nconn: " + conn.toString();
    if (typeof current != "undefined") { data += "\ncurrent: " + current.toString(); }
    throw "Function foo(bar) failed, bar = '" + bar + "'\n" + ex + data;
}


  • Watch this bullet for future content...

Tuesday, December 13, 2016

How to make the Search/Reconcile AL for an RMI Adapter against a REST or WS API


Writing RMI Adapter AssemblyLines can be tricky at times, especially when it comes to the Search AL - the one used for Reconcile operations. This is because the Dispatcher calls the Search AL in Cycle mode, which causes it to work differently than you might expect.

The usual behavior of an AL is that an Iterator in the Feed section delivers data - one Entry at a time - into the Data Flow section where it is passed from component to component until the end of the AL is reached. At the end of each cycle, control returns to the Iterator which returns the next available Entry. If the iteration data set has been exhausted then the Iterator signals End-Of-Data and the AL terminates. When an Iterator-based AL is started then it normally runs until completion. If you launched it from another AssemblyLine then you can choose to either wait for completion, or allow it to run in the background while your calling AL continues its own processing. There is also a third option, which is to run the called AL in Cycle mode.

If the AssemblyLine is invoked in Cycle mode, then it only makes a single pass each time it is called and the calling process always waits for control to be returned. On the first call, the AL and its components are initialized and then the first processing cycle is performed, resulting in a Work Entry. Each subsequent call drives the AL for another cycle, returning either another Work Entry, or null once End-Of-Data is reached.

As long as the AL uses an Iterator in the Feed section then the built-in flow of the AssemblyLine will provide the caller - for example, the RMI Dispatcher - with a series of Entries as expected. However, the challenge arises when your AL must do more work to prepare each Entry. A common example is integration with a web service where a series of requests must be made to establish an authenticated session, followed by one or more requests to select relevant data - all before the first Entry can be returned.

One approach is to hide this complexity by scripting your own Iterator mode Connector. If you are comfortable using Javascript and have made friends with the AL Debugger, then rolling your own component could be your quickest option. Another route is to build an AL so that it will perform 'normal' iteration in Cycle mode, and here is an example of how to do this.

Create a new AL named 'CycleModeIteration' and then add the FormEntry Connector to the Feed section. This handy component supports only Iterator mode and works just like a FileConnector, except that instead of reading and parsing data from a file, it reads and parses the data entered in the Raw Data Text parameter of its Connection tab. We won't be using this functionality, instead substituting our own via Hook scripting.

Select the Hooks tab and then put this code into the Prolog - Before Init Hook to prepare for processing:

   // Initialize two flags
   eod = false;
   initializing = true;

Then drop the following script into the Override GetNext Hook:

   // If EOD flag set, end iteration
   if (eod) {
   result.setStatus(0); // EOD
   return;
   }

   // Otherwise return dummy data to continue
   // processing
   result.setStatus(1); // More data available

   work._dummyData = "* NOT DONE YET *";

The Override GetNext Hook code will replace both the read next functionality and the Input Map of the Iterator, so our logic here has to directly add attributes to the Work Entry.

Your AL should look like this.



The Iterator now will continue to return a Work Entry with a single dummy attribute until the eod variable has been set to true, at which time it will stop cycling.

Next we set up the Feed section so that initialization and data selection is only performed on the first cycle. Do this by adding an IF-Branch and naming it 'Initializing'. Click on the Script button to add a scripted Condition and use this snippet so that it uses the initializing flag set in the Iterator's Hook:

   return initializing;



In an actual Search AL this Branch would contain the components required to establish a connection to the external service or system. For our example you right-click the IF-Branch and add an Empty Script component that you call 'Perform initialization'. It only needs a line of script that sets the initializing flag to false. Now the IF: Initializing Branch will only be active on the first cycle.



Following this Script component you add another one, this time named 'Prepare data set'. This mirrors the typical selection of entries performed by an Iterator. If a real-world Search AL is working with web service then it will request a data set and parse the return. Once again our example AL is extremely simple, using a hardcoded Array of data and an index variable initialized to 0.



Here is the script shown in the screenshot above for easy cut-and-pasting:

   dataset = [
{name: "Joe Blogs",
title: "Manager",
phone: "555-1234"},
{name: "Jane Doe",
title: "Team Lead",
phone: "555-4321"},
{name: "John Anderson",
title: "Hacker",
phone: "555-1111"},
{name: "Bill Buttons",
title: "Peon",
phone: "555-2222"
    ];


   index = 0;

Now comes the logic that delivers data as long as its available, at which time it then signals End-Of-Data and stops iteration. We implement this through two Branch components: First an IF-Branch labeled 'More data' followed by an ELSE-Branch called 'EOD reached'. For the IF-Branch, click on the Script button and add this scripted Condition:

   return index < dataset.length;

These Branches will control how long iteration continues for our AssemblyLine.



Components placed under the IF: More data Branch set up the Work Entry with attribute values coming from the next available item of data. For the example AL you create a Script named 'Get next entry' that contains the following code:

   // get next entry and increment index
   entry = dataset[index++];

   // set up work entry
   work.removeAllAttributes();
   for (prop in entry) {
   work[prop] = entry[prop];

   }

The first line of script above retrieves the currently indexed Javascript object from the Array, incrementing the index variable immediately after. The next script line removes all attributes from the Work Entry, which at this point will only hold the dummy attribute set in the Override GetNext Hook of our Iterator. Finally a for-loop cycles through the names of all properties in the current object (i.e. 'name', 'title' and 'phone') and creates similarly named Attributes in the Work Entry with the value of each property. This Script component takes care of our Return Next Entry functionality. Now we need the logic to stop iteration after the last available data has been read.

Under the ELSE-Branch you add a Script named 'Signal EOD' containing this script:

   eod = true;
   system.skipEntry(); // skip back to Iterator

At which point this example AL is complete.



To test our new AssemblyLine we will create another AL, which we will call 'TestIteration'. Here you add an AssemblyLine Connector in Iterator mode to the Feed section. In the Connection tab of the Connector, enter the name of our example AL: 'CycleModeIteration'. The AssemblyLine Connector will run our example AL in Cycle mode, just like the RMI Dispatcher will. Also remember to add the wildcard (*) mapping rule to the Input Map so that all data is returned.

Finally, add the canned Script component named 'DumpWorkEntry' to the Data Flow section.



It is now time to run the test AssemblyLine and see our example CycleModeIteration AL works correctly. And if it does, then should see four (4) entries returned and displayed in the console output.



The component statistics displayed at the end of the AL run show us that everything worked as expected.

Note that if you don't get this result, then add the DumpWorkEntry directly under the IF: More data Branch (After 'Get next entry') and try running the example AL directly using the AssemblyLine Debugger. This will allow you to interactively walk component-by-component through its execution, including stepping through all script code. If you are not familiar with the AL Debugger, have a look at this video first.

Once the example AL is running correctly then you have a template from which to build your real-world Search/Reconcile AssemblyLine, simply substituting the hard-coded data set with actual information received from a connected service or system.

Monday, December 12, 2016

Importing a client certificate in order to access an https URL

When you go to make an HTTP request to a service that requires an SSL connection, you'll get an exception that your HTTP Client Connector cannot set up the secure link to the service. Unlike your browser, TDI (SDI) does not automatically import the client certificate. You have to do this yourself, and this post is about how to do this.

For this example, we'll try to access https://login.salesforce.com

If you enter this URL into an HTTP Client Connector and attempt a connection either by pressing the Connect and Next buttons in the Schema area of an Attribute Map, or by dropping the Connector into an AL and running it, then you will see this error:

ERROR - [HTTPClientConnector] CTGDIS810E handleException - cannot handle exception , callreply 
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
java.security.cert.CertPathValidatorException: The certificate issued by CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US is not trusted; internal cause is: 
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.j.a(j.java:10)
at com.ibm.jsse2.qc.a(qc.java:359)

This error is telling us that we are missing the required client certificate. An easy fix available directly in the CE is to press the Get Certificate button located to the right of the HTTP URL parameter in the Connector's Connection tab.



This handy button will request the certificate and then store it in the TDI Server's keystore (as specified in the solution.properties file by the property named javax.net.ssl.trustStore).



Once imported, the Java VM needs a restart in order for it to use this newly acquired cert. This means that you must recycle your TDI Server before a successful connection is possible.

This is all well and good, but if you're scripting your own components then it would be better if the missing certificate could be detected and then automatically imported. Fortunately, this is very simple to do with a single line of script in TDI.

msg = com.ibm.di.security.GetSSLCertificate.installCertificateFrom(baseurl, 443)

The baseurl argument is the 'https://...' address you want to access, and 443 is the default port for HTTPS. If the socket you are trying to reach includes a specific port number, then you can use this in the above call. If the request or the keystore update fail then the returned String is an error message describing this failure. If the client cert exists already in your keystore, then the message will tell you that the cert is already trusted. And if the requested certificate is successfully added to your keystore, the message will indicate this.

CTGDIS1957I Added certificate with subject: CN=login.salesforce.com, OU=Applications, O="Salesforce.com, Inc", L=San Francisco, ST=California, C=US.

Once TDI has been restarted then you can make secure connections to this service.