TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Thursday, November 17, 2016

JSON and XML Tutorial - Part 5, Scripting a Connector

Continuing from my last rambling post, we'll script an Iterator Connector that will return the same parsed entries as our Conditional Loop did. I'll begin with a little theory, which you will find duplicated other places like the How-To page of www.tdi-users.org and this venerable video tutorial on the subject.

To script a component in TDI (SDI) you start with the basic Script version of that component. So in order to script a Connector, you add a Script Connector.

Typing 'script' in the Search box filter the selection list. Since we'll be implementing only Iterator mode, we may as well choose this mode in the wizard.
Our new Connector looks pretty much like any other Connector at this point. All the features provided by the TDI AL framework are visible in the various tabs: Input/Output Map, Hooks, Delta, Connection Errors and Pool. You can also choose the Mode and State for your Connector, as well as when the component will initialize. Only the Connection tab is specific to the technology our Connector is interfacing with. That is why this aspect is called the Connector Interface, or CI for short.

The Connection tab of our Script Connector does not hold the usual set of parameter (yet), but instead gives us access to the interface logic of the component via the Edit Script... button.
Press this button to open the script editor.

Once in the editor, we can see that our new Connector has a preloaded example script. At the top are the two functions necessary for Iterator mode: selectEntries() and getNextEntry().
This is how a scripted component works: if the required functions are present in the script, then that operation or mode is available. Note however that for a scripted Connector, the Mode drop-down will always contain all modes - including those not supported by its script.

Iterator mode selects its data set with selectEntries() and then iterates through it using getNextEntry() to return one Entry at a time and to signal End-of-data when the data set is exhausted. Here is the standard template for these function with a little comment added for clarity:

var counter = 0;

function selectEntries()
{
// initialize the 'data set'
counter = 0;
}

function getNextEntry ()
{
// if the data set is exhausted, setStatus to 0
if (counter > 100) {
result.setStatus (0);
        // This next line is no longer required after TDI 7.1
result.setMessage ("End of input");
return;
}

// otherwise, populate the conn Entry via the 'entry' variable
entry.setAttribute ("counter", counter);
counter++;
// setStatus is 1 by default, which means 'more data available'
}

A scripted Iterator is easy to test in the Schema section of the Connector's Input or Output Map. Pressing the Connect button calls the initialize() function (if one exists), followed by querySchema()and then selectEntries()

The reason information shows up under Schema when you press the Connect button is because there is a template script for querySchema(). Here is a copy of it with clarifying comments:

function querySchema ()
{
// the 'list' variable is a Java Vector for returning 
// one or more schema items. Each one is added to the
// Vector as an Entry with two attributes:
//    name - the name of the attribute
//    syntax - a string value with guidance to the developer
var e = new com.ibm.di.entry.Entry();
e.addAttributeValue("name","counter");
e.addAttributeValue("syntax","Number");
list.add(e);
// once again, setStatus to 1 means info available
result.setStatus (1);
}

If there is no querySchema() function then the Schema grid is first populated when you start stepping through the data set. You will also note that there is no way to signal if a schema item is Required or not. This is schema item option only available if you code your component using Java. However, I have been creative with what I put in the Syntax value and have not missed this feature :)

Once connection and selection are complete then the Connect button is grayed and the Next button becomes available. Each time you press this button it results in a call to getNextEntry() and either returns data or signals EOD.

And now for something cool: Simply open the script editor again and change the If-statement in getNextEntry() to signal EOD after 10 entries instead of 100. Then return to the Input Map tab, press the Close button in the Schema area and then Connect again. This causes the script to be reprocessed and lets you test any changes you made to the code immediately.

Ok, enough theory for now. Let's strip out all the template functions except for selectEntries() and getNextEntry() which we will be rewriting. In addition, we will add the initialize() function as well as some global variables. Here's what the start of the script looks like:

// MyRestConnector
vrs = 0.9; // 20161117 created this component
// main.logmsg() provides info in the Console view in the CE
// You could also write to stdout: java.lang.System.out.println(...)
main.logmsg("MyRestConnector v." + vrs);
// All variables defined outside of function bodies
// are global for the entire script.
// 
// Here is the HTTP Client Connector we'll use to make requests
  // Discovering what the 'internal name' of a Connector is requires a little legwork,
  // and I'll get back to this in a later post.
var http = system.getConnector("ibmdi.HTTPClient");
// Hardcoding the URL for now. Later we'll create a Connection Form
var url = "http://www.ideal-status.nl/static/consumer_notification_advice.json";
// Variables for our NodeList and index
var nodeList = null;
var nodeIndex = 0;
function initialize()
{
// Initialize the HTTP Client CI loaded above
http.initialize(null);
// Reset the data set
nodeList = null;
nodeIndex = 0;
}

We can test the validity of this script by bringing up the Input Map tab again and pressing Connect. If you did not empty out the selectEntries() function then you'll get an error when you try this.

Now we come to the meat of our Connector: selectEntries(). This function does most of the work of component, setting up and making the REST request, parsing the JSON payload received and then preparing the data set to be returned by subsequent calls to getNextEntry(). These three steps are called out in the comments included in the code below:

function selectEntries()
{
// PREPARE REQUEST
// Set up an Entry for passing Output attributes that
// control parameter settings
var requestEntry = system.newEntry();
requestEntry["http.url"] = url;
requestEntry["http.Accept"] = "application/json";
// The HTTP method is GET by default, but just in case...
requestEntry["http.Method"] = "GET";
// Make the request and store the response in a local Entry
// Note that the queryReply() method is what CallReply mode uses
var responseEntry = http.queryReply(requestEntry);
// Check the responseCode returned by the service
var responseCode = responseEntry.getString("http.responseCode");
// If this is not a 2xx return, then throw an exception
if (!responseCode.startsWith("2")) {
var responseMsg = responseEntry.getString("http.responseMsg");
throw "Error making request: " + responseCode 
+ " - " + responseMsg;
}
// PARSE JSON RECEIVED
// Wrapping this in try-catch for better error handling
try {
var json = responseEntry.getString("http.bodyAsString");
// fromJSON() is a static method, so we can call it via the class
var hEntry = com.ibm.di.entry.Entry.fromJSON(json);
} catch (ex) {
throw "Error parsing JSON return: " + ex 
+ "\nJSON: " + json;
}
// PREPARE DATA SET
nodeList = hEntry.getChildNodes();
nodeIndex = 0;

}

Once again we can test that our script so far is correct (at least syntactically) by using the Close and Connect buttons. We are still not seeing any data returned yet because that's the responsibility of getNextEntry() which we code to look like this:

function getNextEntry ()
{
// If the data set is exhausted, setStatus to 0
if (nodeIndex >= nodeList.getLength()) {
result.setStatus (0);
return;
}
// Otherwise, populate the conn Entry via the 'entry' variable
// Borrowing this code from the AL made in the previous blog
// First grab the next node and increment the index
item = nodeList[nodeIndex];
nodeIndex++;
// The 'entry' variable is the conn Entry for this Connector
entry.id = item.getName();
var children = item.getChildNodes(); // get next level
for (var child in children) {
// set conn attribute using child att name and value
entry[child.getName()] = child.getValue();
}
// For completeness sake, I like to set the return status
result.setStatus(1);
}

Once this is in place then we can start testing it using the Close, Connect and Next buttons.

And violá! we have a fully functional Iterator. We can use it in the Feed section of an AL or in a Connector Loop, we can code any of the Iterator mode Hooks, and we can enable Delta if we want to detect changes between each call to the service.

And what's more, if we drop our Connector into an AL then we can step through the script in the AL Debugger.
At this point, we can either double click on ConnectorScript and then set breakpoints in the code, or just start stepping into using the Step into button. Note that pressing Step into while in the script takes debug processing into for- and while-loops, as well as if-statements and try-catch blocks. Step over will do just that: stepping over these code blocks.
As always, the Debugger lets you examine and change variables directly by executing Javascript in the Javascript Commandline located just under the script window.

Each time you press the ENTER key then the script snippet is passed to the running server and run in the context of where the AL processing is at that point. This lets you inspect the state of your component by logging the results of all statements executed, as shown in the screenshot above. Furthermore, you can modify the values of variables and attributes, as well as the contents of Entry objects. You can also define new variables and instantiate new objects, calling scripted functions and invoking Java methods as desired.

I'll keep my love ballad to the AL Debugger short and conclude with the recommendation that you learn to love this time-saving tool.

And I'll wrap up another extended blog here. But I will return to show you how to set up the custom Connection tab for your scripted Connector, so stay tuned!

8 comments:

notesnl said...

Hi Eddie,
If I try this in my TDI I get a connection timed out message.
Any ideas what can be wrong, I have the same when I setup a connection via the default available Http Client Connector.

rgrds,
Tom

Eddie Hartman said...

@Tom Steenbergen - A connection timeout could mean a couple of things. First, that the IP address or hostname is wrong, or the server it points to is not running. Try pinging the server to see if it is up. Then you can try to telnet to the port to see that's it's listening on that port. Of course, if it were simply the wrong port you were trying the you'd get a connection refused.

Another option is that the server is behind a firewall. Some of these let you authenticate if you telnet or dial up the socket with a browser. Give this a shot.

This is what comes to mind, Tom (at least, my mind :). You could also post your query on the TDI forum: https://ibm.biz/Bdi83A

Happy TDIing!

notesnl said...

Thanx Eddie,
I had also posted this on the TDI Forum. And we found the problem to be in the company firewall.
So it is solved for now.
Rgrds Tom

Ram Chauhan said...

Great Blog Thank you so much for sharing this.
Hikvision Full HD & Night Vision Camera Kit

SteveMartin said...

Hi,
All is good until I get to the final function getNextEntry() , save the script, hit Connect, OK, hit Next and get
null
java.lan.NullPointerException at com.ibm.di.parser.HTTPParser.writeEntry(HTTPParser.java.333) ...
Have gone back to the function initialize as well as function selectEntries(), etc and compared the script.
My getNext looks like this:
function getNextEntry()
{
// If the data set is exhausted, setStatus to 0
if (nodeIndex >= nodeList.getLength()) {
result.setStatus (0);
return;
}
// Otherwise, populate the conn Entry via the 'entry' variable
// Borrowing this code from the AL made in the previous blog
// First grab the next node and increment the index
item = nodeList[nodeIndex];
nodeIndex++;
// The 'entry' variable is the conn Entry for this Connector
entry.id = item.getName();
var children = item.getChildNodes(); // get next level
for (var child in children) {
// set conn attribute using child att name and value
entry[child.getName()] = child.getValue();
}
// For completeness sake, I like to set the return status
result.setStatus(1);
}

SteveMartin said...

Getting a null pointer after the last addition of function getNextEntry().
Have it pasted in from the provided script. Hitting Next gives me :
null
java.lang.NullPointerException at com.ibm.di.parser.HTTPParser.writeEntry(HTTPParser.java:333)

Eddie Hartman said...

@SteveMartin - strange, I just copy/pasted the script and run the connector as an iterator in an AL... Could it be you are getting strange data back? Have you tried debugging? If you press the Debugger button (next to Run in Console), and then make sure you get the 'full' debugger by pressing the Debugger button that may be visible at the far right of your screen, you will see the Connector Script under your Scripted Iterator. Of course, you first have put this in the Feed of a test AL :) If you double click Connector Script you can set break points in your scripted code and use the Stepping buttons (top left) to walk the logic. Just above the log Output pane you'll see a one-liner input felt with '// Enter javascript and hit Return' in it. Anything you type here is passed to the running AL and executed as though it were part of the AL. So if you enter a variable name (like 'entry' or 'work' or whatever) and press Return, the result of this statement will appear in the log Output. Note that this will be the toString() method of the object referenced. You can also change variables and attributes directly using this commandline. More on the Debugger here:

https://www.youtube.com/watch?v=6h9Fg-SsToM

Eddie Hartman said...

NOTE if you double-click an open tab you go to full-screen. Double-clicking again returns to normal perspective. If you don't see the Servers pane at the bottom left of the CE then you may be in the wrong Eclipse perspective. Use Windows > Open Perspective to open the TDI (SDI) perspective. If you already have this selected, but are missing one or more panels (i.e. Console, Javascript, Navigator...) then use Windows > Respect Perspective to fix things.

And once you are debugging your scripted connector, just use Step Into to walk through it's execution. Also enable the Show Hooks checkbox at the bottom of the Debug AL panel (top left). Then as you Step Into you can explore the built-in workflows of TDI.