Certificate chain chain chaining error

You're trying to connect to an https server but get an SSL handshake exception. The error message tells you that the certificate you received from the secure server is not trusted. What does this mean, and what do you do about it?

This exception means that the cert you got from the server is not in your TDI truststore - the one referenced by the javax.net.ssl.truststore property - or it is signed by another certificate that is not found there. To fix this issue you have to get necessary public key certificate and import it into your truststore.

So how do you get hold of this cert? If you know the admin(s) for the server then you can request it from them. Another approach is to connect and blindly assume that the certificate recieved is trusted. Some browsers allow you to save the cert to file. You can also use the Java utility provided in this article.

Once you have the certificate file you import it into your truststore either by using the commandline utility, keytool, found in the jvm/jre/bin folder of your TDI installation, or by using the iKeyMan application linked in button bar at the top of the TDI Configuration Editor (CE).

If you haven't changed the default security setup then the truststore will be the testadmin.jks file found in the serverapi folder of your Solution Directory. The default password is 'administrator'.

Once the certificate is imported, the SSL connection will work.

And, as always, thanks to Jens Thomassen for his invaluable insight and guidance!

Early Christmas Stocking Stuffers

Since this past summer, TDI 7.1 Fixpacks now include a number of new components. However, since they are dropped in the examples folder during fixpack updating and not under jars, they won't show up in the Add Component wizard until you make them available to your TDI installation. This can be done in a couple of ways, as I have outlined here.

To whet your appetite, here is a list of powerful new components provided under examples:

• TPAE IF Connectors include a 'Simple' version which is comparable to the 'Generic Maximo Connector', although it has been extended to work with a broader range of MBOs. The other TPAE IF Connector (without 'Simple' in its name) further enhances integration capabilities to Maximo application data.
• TPAE IF Change Detection Connector allows you to hook into change notifications coming from the Maximo integration framework, for catching and propagating new and modified data to any number of targets. And deletes.
• File Management Connector is for reading and modifying file system structures and file system metadata. More specifically, it can create, find and delete files and directories. You still use the 'File System' Connector to work with file contents, but now you could iterate through a directory and load each returned filepath into a File System Connector (for example, in a Connector Loop) to process the file.
• File Transfer Function component allows you to securely copy files between any two systems, either to or from a remote system, or between two remote systems.
• JSON Parser allows you to read serialized JSON into the hierarchical Entry-Attribute-value model of TDI where you can use features like XPath searches and 'DOM tree walking' techniques to access and manipulate parsed objects. And of course to turn an hierarchical Entry into a JSON stream. Makes going from JSON to XML, and vice versa, a snap :)

Of course there are more components available under examples, like the RegExp Parser and Exchange Change Detection Connector, ready for eager hands to explore and put to good work. Each example folder also includes documentation on that component, as well as example Configs that can be imported into TDI and used to jump start their usage and understanding.

Getting TDI installed on Ubuntu

Courtesy of TDI champion and support guru, Jason Williams: Adventures with IBM TDI

Joined Up Thinking: Tivoli Directory Integrator Web Services Revisited

Joined Up Thinking: Tivoli Directory Integrator Web Services Revisited

Stephen Swann again shares the fruits of his adventures with TDI.

work.FullName.toUpperCase() - What's wrong with this picture?

The fact that TDI lets you use '.' to reference attributes (work.FullName) coupled with JavaScript engine's (limited) automatic type conversion are a constant source of confusion. Although I feel a long rant coming on, I will resist and save some content on this subject for future blogs. For now, suffice to say that using dot notation to reference an attribute gets you just that: a com.ibm.di.entry.Attribute.

Quick refresher alert: Attributes are named containers of values. So for example, work.mail is a container named 'mail' that can in turn hold zero or more values. The values themselves can be any kind of Java object: String, Date, java.sql.Blob - even another attribute, as is the case when working with hierarchical data in TDI (for example, parsing xml, rdf, json, ...).

The JS Engine in TDI sometimes converts the attribute being referenced to a string representation of its value, as with this snippet:

"My name is " + work.FullName >> My name is Slim Shady

Since the Attribute is part of a String concatenation, TDI's JSEngine conveniently returns its value as a Java String instead of the attribute itself. But that's about as far as auto-conversion goes. In most cases when you send an Attribute to do a String's job, the Attribute's toString() method is used. This results in a string value that starts with the attribute name, then a colon and its value.

work.FullName >> "FullName": "Slim Shady"

The toString() method for both the Entry and Attribute classes give you this JSON-like result (of course, you use the JSON Parser for real JSON) and this may not be what you want.

Even worse is when you start calling String functions directly from your dot-referenced Attribute:
work.FullName.toUpperCase() >> com.ibm.jscript.InterpretException: Script interpreter error, line=1, col=12: Error calling method 'toUpperCase()' on an object of type 'com.ibm.di.entry.Attribute'

So what's a poor TDI'er to do? As usual, there is more than one way to flay this feline:

work.getString("FullName").toUpperCase() >> SLIM SHADY
work.FullName.getValue().toUpperCase() >> SLIM SHADY
work["FullName"].getValue().toUpperCase() >> SLIM SHADY
(work.FullName + "").toUpperCase() >> SLIM SHADY

The clue here is to reference the value of the attribute, not the attribute itself. Sometimes you'll want a String representation and will use methods like entry.getString(attributeName) or attribute.getValue(). In those situations where you need the actual Java class used to store the value then you use attribute.getValue(index), where index is is an integer value denoting which value you want: from zero to attribute.size() - 1.

For the sake of completeness, note that calling entry.getObject(attributeName) will return the Java object used to store the first value of the named attribute.

And if any of this is still unclear, please drop a note in the forum and I'll will strive to clarify without further pontification :)

Forum: https://groups.google.com/forum/#!forum/ibm.software.network.directory-integrator

Higher Availability

A question oft asked: How do I make my TDI solution highly available? The answer often boils down to what 'available' means to your solution. In many cases it means that one or more AssemblyLines continue to function. This can be 'wired into' a solution with surprisingly little effort.

To start with, don't create long-running ALs; In other words: don't set the timeout parameter for your Iterator to 'never time out'. Instead, let it wait a bit for new input (e.g. new changelog entries or messages on a queue) and then report End-of-Data so the AL stops. And then restart it.

So when doing a directory sync you let the Change Detection Connector 'listen' for changes and stop if none appear in, say, 10 minutes. Then you have some other process that relaunches the Sync AL: like a cronjob or Windows Scheduled task, or even another AL.

This is the simplest form of HA design, and it also gives you an opportunity to check status and send reports/alerts if needed whenever the Sync AL stops - simply by checking the error object in the AL's After Close Epilog Hook. Since you also expect the AL to stop every once in a while, this can be checked for - for example, using the TDI commandline utility, tdisrvctl, or even another AL. If you detect that the AL is hanging, you can stop and restart it. Furthermore, if an unhandled error occurs and the Sync AL stops abnormally, it is also restarted again.

The idea of monitoring an AL and starting another if it stops seems straightforward. But it is not really that easy. Just because an AL appears to have stopped does not mean it's a good idea to launch a backup. Unless you design for this, running multiple copies of the same AL simultaneously (e.g. reading changelog) may not be a good idea. Also, it is very hard to determine where the failure lies: did the AL stop, has the connection to the TDI Server api been lost, did the TDI server or JVM die, was there a network failure or did the server HW crash, ...? It might be that the AL is waiting for a lock situation on some connected system or resource, or working to re-establish a lost connection. Starting a second copy may serve no purpose.

From experience, the most common situation is that either the AL is hanging - which could be an error in script logic, or I/O latency of connected systems - or it has stopped due to an unexpected (and unhandled) exception. If the AL is actually hanging then it can be killed using api calls or via the tdisrvctl utility, and then restarted. If it has stopped abruptly, restarting is the answer again. If you use a cronjob or scheduled task to (re)launch the AL, then you can be sure the TDI Server is restarted anew each time as well.

Of course, you can use a message queue and divide up your solution into 'feeds' and 'handler' AssemblyLines, allowing you to run multiple copies to increase both performance and availability. You can also use Server mode Connectors to drive solution, since Server mode provides features for pooling and reuse/restart of concurrent AssemblyLines.

These and other techniques and reflections on building robust solutions have been captured by TDI architect Johan Varno and can be found here: http://www.redbooks.ibm.com/redpieces/pdfs/redp4672.pdf

Often the simplest answer is the best one: expect ALs to stop and then restart them again.

Adding custom headers lines to CSV output

I've gotten this questions a few times recently: how to put custom content at the top of a CSV file.

One solution would be to code the After Close Hook for the File System Connector, creating a new file that combined the headers and the CSV content. However, there is an easier solution (thanks again to Jens 'Beautiful Mind' Thomassen):

Add the following script to the After Initialize Hook of the Connector:

// At this point, the Parser is also initialized,
// so you can grab the java.io.BufferedWriter it uses.
//
stream = thisConnector.connector.getParser().getWriter()
//
// Then add your custom content
//
stream.writeln("this is the first line")
stream.writeln("this is the second line")

And that's all there is to it. When the Connector does its first write operation, the field names will be written after the custom lines.