Reference error: 'java' not found

Just to let you know, if you get the above error then it means you've probably spelled the Java class name wrong.

now = new java.util.Calender()

I was getting ready to start pulling out hair when I saw the light: 'Calendar'.

And I've also gotten other error messages that turned out to be caused by spelling. Just wanted to share that :)

To Google, or not to Google? That is the wrong question.

If you've ever hit an error or connection problem or logical challenge using TDI and cursed the lack of helpful content to be found online, you may not be looking in the right places.

Firstly, use the term 'tdi' in your search arguments only if you're interest is motors or diving. Searches that include 'tivoli directory integrator' will be less exciting but more relevant.

Secondly, remember that TDI is pure Java and also that the core of an issue might be some system that you are integrating. So the trick is to examine available clues, and then make a informed search.

For example, let's look at this error returned while trying to update a Domino server:

10:03:21,410 ERROR - [UpdateDomino] CTGDIS810E handleException - cannot handle exception , update
java.lang.Exception: CTGDKC002E Failed to execute the command. NotesException occurred: Invalid object type for method argument
at com.ibm.di.connector.dominoUsers.DominoUsersConnector.executeCommand(DominoUsersConnector.java:647)
at com.ibm.di.connector.dominoUsers.DominoUsersConnector.putEntry(DominoUsersConnector.java:732)
at com.ibm.di.server.AssemblyLineComponent.executeOperation(AssemblyLineComponent.java:3139)
at com.ibm.di.server.AssemblyLineComponent.add1(AssemblyLineComponent.java:1930)
at com.ibm.di.server.AssemblyLineComponent.update(AssemblyLineComponent.java:1681)
at com.ibm.di.server.AssemblyLine.msExecuteNextConnector(AssemblyLine.java:3669)
at com.ibm.di.server.AssemblyLine.executeMainStep(AssemblyLine.java:3294)
at com.ibm.di.server.AssemblyLine.executeMainLoop(AssemblyLine.java:2930)
at com.ibm.di.server.AssemblyLine.executeMainLoop(AssemblyLine.java:2913)
at com.ibm.di.server.AssemblyLine.executeAL(AssemblyLine.java:2882)
at com.ibm.di.server.AssemblyLine.run(AssemblyLine.java:1296)
10:03:21,410 ERROR - CTGDIS266E Error in NextConnectorOperation. Exception occurred: java.lang.Exception: CTGDKC002E Failed to execute the command. NotesException occurred: Invalid object type for method argument
java.lang.Exception: CTGDKC002E Failed to execute the command. NotesException occurred: Invalid object type for method argument
at com.ibm.di.connector.dominoUsers.DominoUsersConnector.executeCommand(DominoUsersConnector.java:647)
at com.ibm.di.connector.dominoUsers.DominoUsersConnector.putEntry(DominoUsersConnector.java:732)
at com.ibm.di.server.AssemblyLineComponent.executeOperation(AssemblyLineComponent.java:3139)
at com.ibm.di.server.AssemblyLineComponent.add1(AssemblyLineComponent.java:1930)
at com.ibm.di.server.AssemblyLineComponent.update(AssemblyLineComponent.java:1681)
at com.ibm.di.server.AssemblyLine.msExecuteNextConnector(AssemblyLine.java:3669)
at com.ibm.di.server.AssemblyLine.executeMainStep(AssemblyLine.java:3294)
at com.ibm.di.server.AssemblyLine.executeMainLoop(AssemblyLine.java:2930)
at com.ibm.di.server.AssemblyLine.executeMainLoop(AssemblyLine.java:2913)
at com.ibm.di.server.AssemblyLine.executeAL(AssemblyLine.java:2882)
at com.ibm.di.server.AssemblyLine.run(AssemblyLine.java:1296)

Step one is to find the topmost (first) stackdump, and then trace it to the top two or three lines:

10:03:21,410 ERROR - [UpdateDomino] CTGDIS810E handleException - cannot handle exception , update
java.lang.Exception: CTGDKC002E Failed to execute the command. NotesException occurred: Invalid object type for method argument
at com.ibm.di.connector.dominoUsers.DominoUsersConnector.executeCommand(DominoUsersConnector.java:647)
at ...

The first line is timestamped and filled with info coming from the AssemblyLine itself. In the above snippet there is the component name in brackets, UpdateDomino, followed by a codified error message - in this case, a very general one that tells us that the AL was unprepared to handle an exception thrown by one of its components. At the very end of the first line is the operation which failed: update. Although this information gives us context, it brings us no closer to solving the problem.

The second line of the snippet is more interesting here. It also has a numbered error message that translates to 'the desired operation failed because Notes flagged an exception'. After the colon is this error: Invalid object type for method argument. Now we have bait for our hook and can go fishing for answers.

In Google I look for: 'Invalid object type for method argument' update notes
Ok, so this wasn't the best example :) Plenty of TDI content here. The third link in the result page takes me to a redpaper entitled 'Domino Integration using TDI', and here on page 26 is the same Notes error message and the cause: the type of an attribute is not recognized by Notes. A typical situation is that a date value is being written, but was not converted to a Domino Date type. Or that a set of values was being written, and one of them was null. On a side note: This can happen if you are using the template example for AD - Domino synchronization and the AD instance you're working against has a different schema for Users than the standard, out-of-the-box one. In this case the 'Location' attribute in the Output Map or the Domino Connector may be in error.

Getting back to my rant, sometimes the search results aren't this promising. That's when you add 'java' to the list of terms, hoping that some Java developer, deployer or application user has seen this before, and an answer lies beckoning in some forum thread, blog post, presentation or page of documentation somewhere.

And finally, if you learn how to read a JavaDoc then you may find answers in TDI's JavaDocs, or those of the libraries that your solution uses. This includes stuff like database drivers and client APIs, as well as standard Java classes.

So the answer is to Google, but it's the question that's key.

CSV Parsing with a twist

So the question I got was this: how can I get the line being parsed by the CSV Parser?

Unfortunately, the CSVParser class does not have any public method for this, so the following is not possible:

lineRead = thisConnector.getParser().getCurrentLine()

Instead, with the help of Jens Thomassen, TDI surgeon, and the indispensible AL Debugger, I created this example TDI 7.1 AL to do just that. You can download the linked Example AssemblyLine and just drop the file onto a TDI Project. It's self-contained thanks to the ever-handy Form Entry Connector.

This AL contains first a Form Entry Iterator in the Feed section that reads a CSV bytestream line-by-line using the LineReader Parser. This returns each line from the CSV loaded into the Connection parameter of the Connector. Then in the Flow section there is another Form Entry Connector that has the CSV Parser, which gets us the actual CSV attributes.

To make this magic work, I did a couple of things:

1. First I set the Flow Section Iterator to initialize 'only when used'. You do this by pressing the More... button out to the right of the Inherit From setting and changing the Initialize drop-down accordingly. This is to prevent the Connector and Parser from initializing until we have data for it.
2. Then I added this code to the After Initialize Hook of this second Iterator:
   

   outStream = new java.io.PipedOutputStream()
   inpStream = new java.io.PipedInputStream(outStream)
   newline = new java.lang.String("\n\r")
   
   formEntry = thisConnector.getConnector()
   formEntry.initParser(inpStream, null)
   
   csvColumns = formEntry.getParser().getParam("csvColumns")
   firstRow = true

   
   The Piped stream allows me to write into one end of the pipe and have my CSV Parser read from the other end.
3. Now in the Before GetNext Hook I need to write the current line into my pipe.
   

   outStream.write(work.getString("line").getBytes())
   outStream.write(newline.getBytes())
   
   if (firstRow && (csvColumns == "")) {
     outStream.write(work.getString("line").getBytes())
     outStream.write(newline.getBytes())
     firstRow = false;
   }

   
   If this is the first row of the file and no Field Names have been specified for the Parser, I am assuming this must be the column title line of the file, so I have to write it twice to the pipe.

And presto! I am getting both the 'line' Attribute and those parsed out of the CSV. Note that this is a slightly simplistic approach, and as a result the first Entry returned by the CSV Parser contains the column names as Attribute values.

So I made a Second Attempt using just a single Iterator in the Feed section, and scripting the setup and calls to the CSV Parser. You can also just drop this .assemblyline file onto a Project and then play with it. I am using the code that I built for the first example in a slightly different way:

1. The After Initialize Hook script is a bit shorter:
   

   outStream = new java.io.PipedOutputStream()
   inpStream = new java.io.PipedInputStream(outStream)
   newline = new java.lang.String("\n\r")
   
   firstRow = true;
   initParser = true;

2. All the work is done in After GetNext, once I have read in the line:
   

   if (initParser) {
     // In this next line you could instead get a pre-configured
     // Parser from your Project Resource library:
     //    csvParser = system.getParser("Parsers/MyCSV")
     //
     // AND to find out what the 'true name' of a Parser is
     // just add it to a Connector and then use the More...
     // Select Inheritance button to see the inheritance link
     // for the Parser (yeah, it's not that elegant ;)
     //
     csvParser = system.getParser("ibmdi.CSV")
     csvParser.setInputStream(inpStream)
     csvParser.initParser()
     csvColumns = csvParser.getParam("csvColumns")
   
     initParser = false;
   }
   
   if (firstRow && (csvColumns == "")) {
     outStream.write(conn.getString("line").getBytes())
     outStream.write(newline.getBytes())
   }
   
   outStream.write(conn.getString("line").getBytes())
   outStream.write(newline.getBytes())
   
   csvEntry = csvParser.readEntry()
   if (firstRow && (csvColumns == "")) {
     firstRow = false
     system.skipEntry() // skip the column names
   }
   
   if (csvEntry != null)
     conn.merge(csvEntry)

This time only the actual data values are returned.

And now for more coffee :)

Portable Solutions

There are two simple tricks that will make your TDI solutions more portable:

1. Use forward slash in pathnames
2. Use relative pathnames
   

A forward slash will work on Unix and Windows, whereas backslash is Windows-only. The reason I use relative pathnames is that my ultimate goal is a single zip file distributable.

The Solution Directory is the root for all relative paths. If you look in the ibmdisrv and ibmditk batch-files/scripts used to start TDI, you can see that before the TDI Java executable is launched the current directory is changed to the solution directory (bin/defaultSolDir.bat or .sh).

The CE workspace on the other hand is an Eclipse construct. Since TDI follows the Eclipse Project paradigm, you get a workspace with folders for the various projects, and with sub-directories below these that reflect the onscreen Navigator hierarchy.

All the TDI Server itself really needs is a single XML file that is 'compiled' based on Project assets, and this file is written to the sub-directory of the Project that starts with 'Runtime-'. This is subsequently where the Default test Server loads it from when you Run or Debug your AssemblyLines. You can change this preference in the Project properties by using the Linked File option.

The Runtime folder is also where TDI puts your default Property store (also named after the Project). If you look at the Connector tab for the Property store and click on the label of the Collection/Path parameter, you'll see that this value is tied to a substitution Expression:

{config.$directory}/ProjectName.properties

The substitution token {config.$directory} translates at run-time to 'wherever the TDI Server loaded the Config from'. Filepaths set this way ensure that supporting files need only reside in the same folder as the Config file, wherever that happens to be. As you'll see below, I like to be a bit more explicit.

Armed with this knowledge, let me share how I start new projects. For the sake of illustration I'm going to call this project TDI4SyncService.

1. Make a sub-folder of my Solution Directory named 'TDI4SyncService'.
2. Create the project called TDI4SyncService and set the properties to write the Config file to the folder created in the step above: TDI4SyncService/TDI4SyncService.xml.
   
3. Edit the TDI4SyncService Property store so that the collection/filepath is 'TDI4SyncService/TDI4SyncService.properties"

Now I have a single folder that I can zip down and share. It contains both the Config xml and properties files, plus anything else my solution accumulates during development. I will even store .jar files here if they are project specific, editing the com.ibm.di.userjars property as needed. Plus I can drop in batchfiles/scripts for launching the solution so that TDI skills won't be required to use it.

TDI keyboard shortcuts

If you're like me then you're lazy. For example, I'd rather spend the 8 hours making an AssemblyLine than the 1 hour of tedious work it saved me from.

I'm also not fond of mousing about. To limit this while TDI'ing, I've opened Window > Preferences > Keys in TDI and bound the following shortcut keys in All Windows and Dialogs:

• Ctrl+R - Run this AL
  
• Ctrl+D - Debug this AL
• Ctrl + Down Arrow - Step Over
  
• Ctrl+Right Arrow - Step Into
  
• Ctrl+Shift+Down Arrow - Continue
• Ctrl+C - Stop this AL run and/or debug session
  
• ...

And I cycle between AL components by typing the first letter of their name.
So my index fingers never stray far from F and J. Then I can squeeze in a few more dev iterations.

And today it's -10C again. Gulf Stream where are you?!

Loop a Connector today

If you've never tried Connector Loops you should. Especially when you find yourself stuffing initialization scripts into your Prolog Hooks - because that's the only place where you can do something before the Feed Connector (e.g. Iterator) kicks in and reads the first entry; Or if you're parsing data entries out of Attribute values (like XML/SOAP, JSON, ...) being returned by your Feed Iterator.

Let's look at pre-Feed handling first. When your solution requires initial processing to happen before the Feed section engages, why not skip using the Feed section altogether? Instead, you implement your own Feed-Flow loop down in your AL Flow section. You do this with a Connector Loop and attach any Iterator or Lookup Connector you want to it. Now add components under the loop, mix well, and violá you have a mini assemblyline baked into your AL. How handy is that. And you can have as many as you need, with the results of one Loop feeding into the next one. You can also nest them:

FOR-EACH FileInDirectory
   FOR-EACH EntryInFile
      ...

As an observant reader you're thinking, sure, but how does the inner Loop know which file to read for every cycle of the outer Loop? I'm glad you asked that.

One of the cool things about Connector Loops is their "Connector Parameters" tab where you leverage your Attribute mapping skills to configure the connection, for example mapping to the filePath parameter of a File Connector. By default the Connector initializes each time the Loop starts up, and this is perfect for the file example above. However, if the inner loop were talking to a secure LDAP or database server, the cost of negotiating the connection each time is wasteful. So you press the "More..." button and tell the Loop to skip (re)initialization and just do the query. Ok, that was a couple of cool things :)

Digressing slightly, there has been some confusion, particularly among new users, in differentiating between Iterator and Loop modes. We all learn after a while that they are conceptually the same: a search query is made and results returned. But while the Iterator gracefully deals with 0, 1 or many entries returned, Loop mode bails, leaving the handling of 0 and many to the user (through the ever-popular mandatory Hooks: On No Match and On Multiple Found). Despair no longer. The Connector Loop handles this for you. You can still code the On No Match or On Multiple Found Hooks, but these are not mandatory for the Loop. So while Lookup mode is great for checking assumptions about data uniqueness and presence, with a Connector Loop you get the best of both worlds.

Moving to the second part of this lecture: Connector Loops are also how you Iterate entries found in the value of an Attribute. The secret ingredient is the Form Entry Connector which uses a Parser to iterate over entries found in the value of its Raw Data Text parameter. So you add a Connector Loop, attach the Form Entry Connector, map the Attribute with the payload to the 'entryRawData' parameter, choose your Parser and away you go. What more could you ask for?

Seriously, write me if you have more to ask for.

What Is The Solution Directory?

The Solution Directory is the current working directory for an  SDI/TDI Server.

Each TDI Server running on the same machine should have it's own Solution Directory (or SolDir for short). If you don't know where your SolDir is pointing, simply open defaultSolDir.sh (or .bat) found in the bin sub-folder of your installation directory (e.g. V7.2). This script sets the TDI_SOLDIR environment variable that controls Server behavior.

     export/set TDI_SOLDIR="some/place/on/your/disk"

Both the ibmditk script to launch the CE, and ibmdisrv that starts a Server call defaultSolDir, along with other scripts in this folder. Like javaHome.

The Solution Directory holds configuration settings, including Java settings, for a Server. These settings are found in a file named solution.properties.

After reading solution.properties file, the Server then reads the global.properties file in the etc sub-folder of the TDI installation directory. Settings in solution.properties override those in global.properties.

If you open solution.properties in a text editor you will find two important properties that are personal to any running TDI Server:

• The port that the Server listens to for RMI API calls: api.remote.naming.port
• The port that the Server listens to web calls: web.server.port (dashboard, FDS, CURI and REST APIs)

This is also where you provide the paths to keystores and truststores, as well as credentials to access these.

By default, the one for the TDI Server's APIs is named testserver.jks and is found in the SolDir. Certificates used for SSL connections in your solutions are under the serverapi sub-folder and are named testadmin.jks.

NOTE: The default password for testserver.jks is server, and for testadmin.jks is administrator. To import your own certificates use keytool (found under the jvm/jre/bin sub-folder of your TDI installation directory) or the Key Manager button in the TDI CE (Configuration Editor).

NOTE II: If you remove all properties in solution.properties that you are not explicitly setting, then a TDI version upgrade might change global.properties and could then affect settings for your Server.

Leveraging the Solution Properties folder to make your solutions more portable

As mentioned above, the SolDir is the current working directory for TDI, both the CE and the Server. Any Config xml files placed in the configs sub-folder of the SolDir will be automatically loaded when the Server starts up. If they have Schedulers, then these are activated as well.

Another approach is to make a Project folder in the SolDir. Solution filepaths should point in this directory using simply the name of the folder. For example, if you project is named "MyProject" then the path to the Property Store file would be:

    MyProject/MyProject.properties

The next step is to instruct the CE where to write the compiled Config xml, which is written each time you save a change. By default this file is written as:

    MyProject/Runtime-MyProject/MyProject.xml

You can change this easily to have the Config written to your Project folder. Simply right-click on the name of your Project in the CE Navigator View and select Properties.




Choose Tivoli Directory Integrator properties at the bottom and set the Linked File parameter to a relative path that points to a config file under this same folder, like MyProjext/MyProject.xml.

NOTE III: If you use forward-slashes for paths instead of backslashes, your solution runs on Windows and all flavors of *nix (including mainframe).

Make sure you also put any other support files in this folder so you can easily move the solution by simply compressing this directory and then uncompressing it in the SolDir of the target TDI installation.

The ReadMe.txt file you make in your project directory could look something like this:

To install this solution:

1) Unzip QRTrigger.zip to your Solution Directory;

2) Edit QRTrigger.properties to configure connections

3) Run the solution with this commandline:

ibmdisrv -c QRTrigger/QRTrigger.xml -r QRListener

Violá, you have something that you can hand to the operations team for deployment, pass on to a colleague, and even publish it online (and sending me the link :)

Pointing to a specific Solution Directory

Stephen Swann pointed out to me that I should mention the scripts used to start up the CE (ibmdisrv) and the one for the Server (ibmditk). Each accepts the -s argument for specifying the SolDir to use:

    ibmdisrv -s /MyOtherSolDir -c MyOtherProject/MyOtherProject.xml -d

NOTE IV: If the SolDir does not exist then the Server will set it up the first time it starts.

Your Eclipse workspace

So the SolDir is a tool for making solutions portable, and it is independent of where you put your TDI workspace. The workspace is where the Eclipse framework housing TDI organizes your source files, and the directory structure there mirrors what you see in the CE Navigator. It is not recommended that you work directly with source files, apart from tying them to SCM like git.

What if..?

But what if you chose the program directory or "current location" when you installed, or want to move the Solution Directory for some other reason? No problem. Simply edit defaultSolDir.

Changing TDI_SOLDIR is only half the job. You also have to instruct the CE where the new SolDir is when it automatically starts up your local TDI Server. This is visible in the Server view at the bottom left of the CE, and is called 'Default'. Can change the Server Launcher settingsby right-clicking on 'Default' in the Server view and selecting Properties and making the change here too.

Final note about the Servers view

The view is not called Servers for nothing. TDI lets you define new Servers, allowing you to start multiple Servers on your PC, or connect to those running on other platforms. You can point a Project at any Server by editing its settings (right-click > Properties). This lets you test and debug the ALs there. Each Server has its own Solution Directory settings, so using relative paths means you can build your ALs on your laptop and then run them elsewhere. You will also want to use forward slashes for filepaths, since backslash only works on Windows and TDI runs a lot of other places as well.

You can also right-click on a Server and select Debug Server. This opens a blank debugger tab which then catches an ALs that are started on this Server.

And while I'm on the soapbox here are some additional tips: Define Logging for where your task.logmsg() messages are written; Code Error Hooks and log enough info for manual intervention; Write status messages to the command line by using main.logmsg(); And set up Connector Auto-Reconnect (Connection Errors).