TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Saturday, November 22, 2008

Troubleshoot the SyncDBfromSource AL in the Lotus Connections TDI Config: profiles_tdi.xml

Now that I have your attention, let me point you to this forum post where I describe the steps it takes to bring the SyncDBfromSource AssemblyLine -- the AL that forms the entire API into Connections' Profiles registry -- into the TDI Debugger in order to step through it for purposes of troubleshooting and adventure:

Google Group post about how to debug the Connections TDI ALs (near the bottom of the thread)

My wordy response is at the bottom of this thread. If you have an AssemblyLine that you wish to debug, but are still unable to (even after video #6 here) then let me know!

Monday, November 10, 2008

...now I remember

Remember to check out the FormEntry Connector. This is the perfect companion to the AL Debugger, letting you set up an infinite feed while you interactively set up Attribute values in order to exercise every corner of your AssemblyLine logic.

Plus, you can map out to the entryRawData parameter when this component is used in a Connector Loop, letting you use multiple Parsers on the same bytestream.

Very handy indeed!

New Component and Library .jar Files

Generally, adding a new component to TDI's library amounts to either 1) copying the relevant jar files to the [TDI Installation Folder]/jars sub-directory; or 2) leaving these files where they are and instead telling TDI where to find them.

This latter is done by setting the com.ibm.di.loader.userjars property found near the top of the solution.properties file which is located in your Solution Directory.

According to the instructional comments found in the property file, you can set this property to a list of directory paths, or a list of files, or any combination of the two. For example,
com.ibm.di.loader.userjars=C:\Notes;CustomJars;C:\Drivers\jdbcDrv.jar

Using this property lets you keep custom jar  files either in place, or in a common area like the CustomJars folder I use. This can be preferrable to copying them into the TDI product installation folder.

Due to the mystical dance of Java loaders, sometimes these library files may need to be available to other loaders, in which case you end up having to do one or both of the following:

a). Modifying the PATH in the ibmditk and ibmdisrv files to include additional folders*.

b). Modifying the CLASSPATH line in these files for this same purpose.

I have several sub-folders to my own CustomJars directory, each containing the library files associated with a particular data source, like "Maximo", "Lotus" and so forth. TDI's loader checks all sub-folders under root directory specified, making it simple to keep these organized and backed up.

Note: libraries can be in zip files as well.

*Note: For Domino integration you will want to put the Notes path first: SET PATH=c:\Notes;%PATH% - since a little bird told me that only the first 128 chars are used by the Notes loader. At least for older versions.

...and I know there was something else I wanted to mention...

Thursday, November 6, 2008

Exceptional Solutions

There are two types: those that go beyond the call of duty to deliver scalable, available and maintainable integration; and those that are defined by the stack dumps of unhandled exceptions.

How do you keep an AssemblyLine in the air? By catching and dealing with exceptions yourself. This is broadly done in two ways:

1) By putting code in Error Hooks. At the very least, you should log the error. I tend to use a script function to handle this.

function logerror(msg) {
task.logmsg("ERROR", "@@ERROR - " + msg);
task.logmsg("ERROR", "@@ AL[Component]: " + task.getShortName()
+ "[" + error.getString("connectorname") + "]");
task.logmsg("ERROR", "@@ Operation: " + error.getString("operation"));
task.logmsg("ERROR", "@@ Message: " + error.getString("message"));
task.logmsg("ERROR", "@@ Exception: " + error.getString("exception"));
}

Of course, no error report would be complete without a dump of the various Entry objects available -- at least work and conn. Unfortunately, the dumpEntry() call does not let you set the log level. So I made my own:

function attList( e ) {
if (typeof(e) == "undefined" || !e)
return;

var attnames = e.getAttributeNames();
java.util.Arrays.sort(attnames);

var str = "";
for (var name in attnames) {
var att = e.getAttribute(name);
str += "@@ " + name + ": ";

for (var i = 0; i < att.size(); i++)
str += att.getValue(i) + " | ";

str = str.substring(0, str.length-3) + "\n";
}

return str;
}

function dumpEntry( logLevel, e, name ) {
if (typeof(e) == "undefined" || !e)
return;

task.logmsg(logLevel, "\n@@ ******** Entry Dump: " + name + " ********\n"
+ attList( e )
+ "@@ ***************************************\n")
}

And then added it to my error function, making sure not to throw any exceptions whilst calling it:

function logerror( msg ) {
task.logmsg("ERROR", "@@ **** ERROR - " + msg);
task.logmsg("ERROR", "@@ AL[Component]: " + task.getShortName()
+ "[" + thisConnector.getName() + "]");
logval("operation");
logval("message");
logval("exception");
if (typeof(work) != "undefined" && work)
dumpEntry("ERROR", work, "Work");
if (typeof(conn) != "undefined" && conn)
dumpEntry("ERROR", conn, "Conn");
if (typeof(current) != "undefined" && current)
dumpEntry("ERROR", current, "Current");
}

Now it's easy to customize the format of the messages, as well as include other functionality if needed; for example, using a Passive Connector to write out error information, or using java.lang.System.out.println() to print messages to the command window where the TDI Server was started from.

2) Surround any script that can throw exceptions (like library calls) with a try-catch block:

try {
makeSomeCall(); // if this fails, catch below
} catch (exc) {
task.logmsg("@@ Error: " + exc);
}

If you don't handle it, then your AssemblyLine will stop running.

I'm just saying...

Saturday, November 1, 2008

Integrated Service Management (ISM) leverages TDI

TDI is bundled with IBM's ISM offerings to provide integration services for deployment of solutions like TADDM (System/Asset & Relationship Discovery Tool), CCMDB (Change & Configuration Management DB) and TSRM (IBM's Service Desk and Service Catalog product).

Here's a TSRM scenario captured on film:

Wednesday, October 22, 2008

I didn't always like scripting

Not back when Johan first pulled me into Metamerge. Even though I sometimes just wanted to break free of the Rube Goldberg-esque world of AL construction and do something quick and direct, like:
if (error.getString("status") != "ok")
task.shutdown(); // Invokes AL Shutdown Request Hook
My issue with scripting was that JavaScript code is "opaque" in TDI; there is no Hook flow through it, making it harder to troubleshoot and test. Furthermore, hiding code away in Hooks doesn't help AL legibility. More importantly, new TDI users often had no previous scripting experience.

And then we added real AL debugging and everything changed.

As far as scripting skills go, trial and error is how I learn. The Debugger turns an AL into an interactive tutorial where I can try my hand at executing any series of code snippets, entered one-by-one into the JavaScript commandline (or selected from the history drop-down) and get immediate feedback in the Log Output window. I can play around with stuff until it feels comfortable. If I do something that bombs with an exception, the Debugger catches and displays it, but the running AL is unharmed

So I found myself doing more scripting...

To make my ALs easier to read and debug, I started using Script components (SCs) to define functions that I called from my Hooks. Larger chunks of code are split up into separate, clearly labeled SCs. For example, the snippet at the start of this article would have appeared in an SC called something like Stop AL On Error.

Then I fire up the Debugger, step from script block to script block (since you can't step into them...yet) and examine work and conn, inspect script variables, call any functions and just generally ensure the sanity of my AssemblyLine logic and the health of the data it's working with.

The QA of my solutions improves as well. Since the AL Debugger lets me change any value or invoke any function, it's easy to trigger and test particular parts of my code. I confess to even using the system.skipTo() call - the infamous TDI GOTO command - to move the "Execute Me Next" relay baton around in my AL.

Hand on heart: TDI's interactive, try-test-refine approach lets me sculpt solutions quickly and visually by tooling around in the Debugger. This is where I learned most of what I do about JavaScript, as well as TDI in general, and REST/APP, XML, Domino, DB2, TDS, TSRM... The list goes on.

So the first time I had to deal with Domino-specific Java types, I configured a Notes Iterator, fired up the Debugger, stepped to the After GetNext Hook and then told the Attribute value in question to tell me its Java class name:
rev = conn.getObject("$Revisions"); task.logmsg(rev.getClass().getName())
Which I then looked it up in some Domino Java Classes documentation that Ken Lin pointed me at (thanks again, Ken). I also used system.dumpJavaClass(rev.getClass().getName()) to quickly get the list of publicly available methods:
[1:29:46 PM CEDT] E: system.dumpJavaClass(work.getObject("$Revisions".getClass().getName()) -> true
CTGDIS827I Class name: lotus.domino.cso.DateTime.
getDateTimeData ();
setJavaDate (java.util.Date);
setJavaDate (java.util.Calendar);
makeRDate ();
doAdjust (int, int, boolean);
getRDateTime ();
markInvalid ();
getParent ();
toJavaDate ();
...

Bingo! I could create a java.util.Date object by simply calling the Domino DateTime's toJavaDate() method.

Then I asked the JDBC Connector to query the schema of the DB2 table, learning that the target column for this value was handled as java.sql.Timestamp, and I could easily create one of those like this:
new java.sql.Timestamp(rev.toJavaDate().getTime())
Armed with Google in the one hand and my Norwegian Army Knife in the other, I happily whittled away at the solution, learning and correcting its design as I go.

The same goes for figuring out how to implement a REST-based web service, or scraping web pages for information, or provisioning accounts in Domino, Connections or Content Manager.

Scripted components, e.g. Connectors, Functions or Parsers, are trickier to debug since these execute in their own Script engines- but not impossible. Even though the Debugger gives you full access to the Script environment of the AssemblyLine itself, it can't access those of your scripted components. In order to debug their code, you need to bring it into the AL's environment, preferably as a series of SCs.



Here's an example of one such AL.

I make sure my functions get defined first, just in case I have initialization code that makes use of them. Notice how my Test script is empty. That's because I do my testing interactively from the Debugger. I just use Test as a spot to set my breakpoint.





In this screenshot I am exercising the getNextEntry() function of Iterator Mode. Of course, selectEntries() has already been called by the Variables & Init SC. On the same JavaScript commandline I also do a dumpEntry().


e = getNextEntry(); task.dumpEntry(e);
You'd be surprise how much code you can squeeze into a single line :).

For example, I remember one project where an input data source was giving us over 150 Attributes. I used this one-liner to get a sorted list:
atts=work.getAttributeNames(); java.util.Arrays.sort(atts); for (a in atts) task.logmsg(a)
TDI jockey Jon Elwood is currently building an AJAX interace to launch and monitor the progress of his AssemblyLines. You can imagine how handy the Debugger is for examining the incoming HTTP requests and interactively testing out return snippets of XHTML (or XML/JSON) by simply setting the value of the http.body Attribute before End-of-Flow. Jon will be publishing a HowTo write-up on this that I am looking forward to.

So I guess that wraps up what has turned into an Ode to the AL Debugger. Just felt the need to share :)

Thursday, June 12, 2008

Changing the Solution Directory

Normally, TDI is installed to a program area (e.g. Program Files) and the TDI project files are located in the Solution Directory, which under Windows is "My Documents/TDI"; In the same way that you store your text processor binaries separately from the documents you write with them. This making it easier to include TDI solutions in your existing backup plan.

I've seen a lot of people configure TDI to use the installation folder as the solution directory. Understandable. When I first install some new piece of software, I also want to limit it's spread on my disk until I've decided whether that to let it stay or not.

So when that time comes when you want to change this choice of Solution Directory, then you only have to edit a couple of batch/script files:

o ibmdisrv - which starts the TDI Server.
o ibmditk - for the Config Editor dev environment).

Both are either batch-files or scripts and reside in the TDI install folder.

Just change the third line shown in the snippet below for both files (Windows example):

. . .
rem Only set TDI_SOLDIR if it hasn't been set already in caller's shell
if .%TDI_SOLDIR%==. (
set TDI_SOLDIR="C:\Documents and Settings\\My Documents\TDI"
)
call "C:\Program Files\IBM\TDI\V6.1.1_GA\ibmdicwd" %TDI_SOLDIR%
. . .

As you can see, there is also a Windows environment variable (TDI_SOLDIR) that you can use as well to change this setting.

And finally, there is the -s commandline option for specifying the Solution Directory on startup of either the Server or the CE:

ibmdisrv -c ITIL.xml -r Omnibus_2_TSRM, TSRM_2_Omnibus -s d:\ISM\

This is also a handy option if you need to run someone else's Config, but don't want to mix their project files in with yours. Or if you want to have multiple TDI Servers running on the same machine but with different API ports. Multiple SolDirs means multiple solution.properties files, each with its own api.remote.naming.port value.

P.S. I don't have the Unix scripts in front of me right now, but they are also easy to fix.

Wednesday, June 4, 2008

Newsgroup

The TDI NNTP newsgroup has been out-of-order for a while now. There is a Google Group that mirrors all its content, plus has a good deal of its own activity:

http://groups.google.com/group/ibm.software.network.directory-integrator/topics?lnk=li

It also has good spam filtering.

See you in the forums!

Tuesday, April 15, 2008

Sharing Secrets, aka Synchronizing Passwords

There are plenty of good reasons for synchronizing credentials, like migrating a Portal from one identity store to another, or as an alternative to single sign-on - such as making sure that the Windows password gets put into (for example) an LDAP directory. But while transferring user id's is pretty straightforward, synchronizing passwords is not. This is because passwords are encrypted information, and typically encrypted using a one-way algorithm. So even if you can read the password out of a data store, it will be a binary value that cannot be copied as-is to another authentication system. This pretty much prevents you from using standard change detection and propagation techniques.

Unless of course both the source and target systems employ the exact same encryption algorithm. In this case you should be able to read in the password attribute and then write this binary value to the target. Note that the target system must also support doing a direct write to this attributes -- i.e. without encrypting it again. Note also that in order to read in an attribute with a binary value (like userPassword ) using the TDI LDAP Connector, you must first list it in the Binary Attributes parameter.

There are also technologies available for catching passwords in clear text, when they are changed. TDI provides plugins for a number of popular identity stores, like IBM Tivoli Directory Server, Domino (HTTP InternetPassword only), Microsoft Active Directory and SunOne. These plugins securely capture passwords, making them available for processing by TDI AssemblyLines.

But that only deals with passwords as they are changed, so we're back to the original question: how to synchronize (or migrate) existing, already-encrypted passwords?

That's what I love about TDI: if one path is blocked (e.g. decrypting passwords) then pick a protocol or api and try another route. So in this case, instead of trying to pull passwords out of a system, hook TDI into the login/authentication mechanisms and let clear-text passwords be pushed to your AssemblyLines.

One example is with Tivoli Access Manager for e-Business. TAMeb can be configured to securely call out to TDI with id/password during authentication. This not only allows TDI to check these credentials against multiple identity stores thereby extending the configurability of the authentication service, but it lets you write these clear-text passwords to any number of targets, provisioning users on-the-fly with the same set of credentials. The end result is password synchronization on demand as these credentials are applied.

I've also seen similar solutions where callouts were made directly from the Portal login service, again providing the same multi-backend capability plus the opportunity for sync'ing passwords. One creative solution even re-directed the login page to a TDI AssemblyLine using the HTTP Server Connector (mini web server). TDI provided the login page, verifying (and sync'ing) credentials, directing the connection back to Portal once the user was authenticated.

So while you can't pry encrypted passwords out of a store, there are ways of working around this limitation -- given the right tools :)


Addendum:
Borrowed from a TDI newgroup post by Christian Chateauvieux (of Metamerge and IBM fame) outlining another solution to this challenge: The Tivoli Directory Server pass-through authentication plugin, shipped with TDS 6.1.

This is how it works: On an authentication failure, TDS can be configured to call out to the plugin which attempts to authenticate the same user in another directory. If the password is validated, TDS stores it so that it won't have to look it up next time. Cool solution, as long as the cached passwords are invalidated regularly enough. Or you synchronize them when they are changed.


Addendum II:
You could extend this pass-through feature to support multiple backends, or RDBMS's, Notes db's and even flat files. Simply use the TDI LDAP Server Connector to build a simple LDAP simulator AssemblyLine. Just catch the LDAP bind/rebind operations, then check the credentials (and/or migrate them) against whatever backends you want using the appropriate Connectors and Java calls.

Wednesday, February 6, 2008

If you want TDI to return mutliple values for an Attribute then you
just need to return a JavaScript array:

var mValues = Array();
mValues[0] = "first one";
mValues[1] = "second one";
mValues[2] = "you get the picture";

ret.value = mValues;

This would of course be in a scripted Attribute Map.

If it's just literal values you want to return (like objectClass) you can easily do it like this:

ret.value = ["top",
"person",
"organizationalPersion",
"inetOrgPerson",
"dominoPerson"];

This method does not work when you are adding Attributes to an Entry directly from script. Then you need to either add values to an Attribute and then put it in the Entry:

myAtt = system.newAttribute("Selection");
myAtt.addValue("eenie");
myAtt.addValue("meenie");
myAtt.addValue("mynie moe");
work.setAttribute(myAtt); // works with any Entry

...or you add the Attribute and then the values to the Entry:

ent = system.newEntry();
ent.setAttribute("Selection","eenie");
ent.addAttributeValue("Selection","meenie");
ent.addAttributeValue("Selection","mynie moe");

If instead you want to take a multi-valued Attribute and return it as a comma separated string (as some APIs require), then you could do this using an Attribute Loop and a couple of AttMap components. However, the easiest way is with a snippet of script:

valStr = "";
for (i = 0; i < myMultiVarAttribute.size(); i++)
valStr += ";" + myMultiVarAttribute.getValue(i);
ret.value = valStr.substring(1); // get rid of the first ";"

If you need the values quoted then the script would be:

valStr = "";
for (i = 0; i < myMultiVarAttribute.size(); i++)
valStr += ";\"" + myMultiVarAttribute.getValue(i) + "\"";
ret.value = valStr.substring(1); // get rid of the first ";"

Of course, there could be a double quote (") in the value itself, in which case
you'd need to know how the target system wants this encoded. Perhaps simply by
wrapping the value in single quotes(?) I'd ask Google :)