TDIing out loud

Ramblings on the paradigm-shift that is TDI.

Tuesday, October 24, 2017

Collecting my ramblings on Connector Loops in one place

Monday, September 30, 2013 - Why Loops Instead Of Lookup Mode (https://ibm.biz/BdjiMC)

Thursday, November 7, 2013 - Why Connector Loop (https://ibm.biz/BdjiM9)

Tuesday, June 8, 2010 (https://ibm.biz/BdjiMQ)



Friday, September 1, 2017

TDI Getting Started and Ensuring Basic Understanding

The Basics

  • Install TDI (https://ibm.biz/BdjsLw) locally to your laptop. This lets you customize your TDI environment easier. If you can't get a licensed version, start with the free version (https://ibm.biz/BdjiMg). It's fully functionally, will give you time needed to get started solving with TDI, and the license file can be replaced when you get yours.
    • Do NOT install under Program Files on Windows, or any other auto-secured folder. Install to C:\TDI\V7.1.1 (and V7.2). 
    • Note that you can have multiple versions of TDI installed, but each TDI Server needs to have its own Solution Directory (https://ibm.biz/BdXX8h) if they will be running simultaneously. If not then they can share a single Solution Directory. This is how I organize my TDI.
  • Install the latest TDI Fixpack, linked to from the TDI portal page (http://www.tdi-users.org)
    • Quick Guide to installing a Fixpack: 
      • Unzip the Fixpack zip and grab the file named TDI-7.1.1-FP000*.zip (or SDI-*) and copy it to the TDI installation folder. 
      • Grab the UpdateInstaller.jar file from the zip as well and copy it to the maintenance sub-folder of the TDI installation directory.
      • Make sure TDI is NOT running and then open a commandline window to this folder. Execute the following:
                             (example for installing FP6 on TDI 7.1.1)
                             Windows: bin\applyUpdates -update TDI-7.1.1-FP00006.zip
                             Linux:        bin/applyUpdates -update TDI-7.1.1-FP00006.zip 
  • Make sure you understand the basics:
    • You need to know what an AssemblyLine is (https://ibm.biz/BdjsQL). In addition to being a com.ibm.di.server.AssemblyLine (https://ibm.biz/BdjiMu), every AL is also a Java Task. TDI is pure Java, and you can mix and match standard and custom libraries, and use techniques you know from Java development.
    • Know how to watch data flowing with the AssemblyLine Debugger (https://ibm.biz/Bdjiy3).
    • Grab this PDF detailing the workflows in TDI (https://ibm.biz/BdjeQm) and keep it close while building solutions and exploring options.
    • Understand that AssemblyLine Components are simply the puzzle pieces you can click together to form AssemblyLines. AL Components can be of different types, like Connectors (https://ibm.biz/BdjiVw), Scripts (https://ibm.biz/BdjiVt) or Attribute Maps (https://ibm.biz/BdjiVU).
    • See how Connector Interfaces (https://ibm.biz/BdjsQF) are attached to AL Components in order to connect to data. Connector Interfaces are often referred to just as specific 'Connectors', like LDAP, Database or MQ.
    • And get to know the AssemblyLine Debugger (https://ibm.biz/Bdjiy3). With it you can answer a lot of questions that arise when you're working to get data where it needs to be.
    • You need to know how to use the various Entry objects TDI gives you (https://ibm.biz/Bdjs9f). And you can create your own if you want to, for example, to accumulate data between iterations.
    • As well as understand how Attributes store data (https://ibm.biz/Bdjs9g). Hint: the Debugger shows you.
    • Of course, understand how the built-in flows of the AssemblyLine and all Connector Modes (like Iterator, AddOnly and Update) let you customize behavior through scripting in Hooks (https://ibm.biz/Bdjs9a). Error Hooks and Reconnect configuration are keys to a robust dataflow.
    • You'll want to get familiar with the 'system' object (https://ibm.biz/BdjiyW). It gives you handy functions for all sorts of things like date formatting/parsing, skipping or repeating the current Work Entry, and invoking commandlines. Note, full JavaDocs are installed with your TDI: <InstallDir>/docs/api/index.html
    • Loops, in particular Connector Loops (https://ibm.biz/BdjiM9 and https://ibm.biz/BdjiMC), can be game changers.
    • Last but far from least, you should be comfortable with Javascript (https://ibm.biz/BdjsQV).
  • Follow these links to learn more, and note that even content labelled '7.0' or '7.1' applies for all versions of TDI: 

Solution Guidelines - The Basics

  • Create a project folder under the Solution Directory for each solution. Put all support files (Config, properties, etc.) in this folder and always use relative paths to reference these files. This makes your solution easy to package and portable (https://ibm.biz/BdXX8h).
  • Use the Resources library to define AL components (Connectors, Scripts, Attribute Maps etc) so that they can be re-used in multiple AssemblyLines. Or even Pooled if you want to limit connections.
  • Configure Connectors for Connection Errors and Auto Reconnect (https://ibm.biz/BdjiMP). Usually you only want AutoReconnect for Connection Lost situations. You can even define which Exceptions a Connector should consider as 'Connection Lost', initiating Auto Reconnect behavior (https://ibm.biz/BdjiMY).
  • Implement an Error Handling strategy for your solution (https://ibm.biz/BdjiMS). It could be to simply to log all errors and continue (when possible). If so, try to include as much relevant information as possible, including the original Error you are respondng to.
    • For an Error Hook in a Connector, it could look like this:
var data = "\n##DATA\nwork: "  + work.toJSON(); // handy format for later parsing
if (typeof conn != "undefined") { data += "\nconn: " + conn.toString(); }
if (typeof current != "undefined") { data += "\ncurrent: " + current.toString(); }
var where = thisConnector.getName()
                     + " (" + thisConnector.connector.getClass() + ")"; // Connector Interface
throw "Error for " + where + "\nerror: " + error.toJSON() + data;    

    • Whereas a try-catch block would look like this:
try {
    // doing something that might exception out
} catch (ex) {
    var data = "\n##DATA\nwork: "  + work.toJSON();
    if (typeof conn != "undefined") { data += "\nconn: " + conn.toString();
    if (typeof current != "undefined") { data += "\ncurrent: " + current.toString(); }
    throw "Function foo(bar) failed, bar = '" + bar + "'\n" + ex + data;
}


  • Watch this bullet for future content...

Tuesday, December 13, 2016

How to make the Search/Reconcile AL for an RMI Adapter against a REST or WS API


Writing RMI Adapter AssemblyLines can be tricky at times, especially when it comes to the Search AL - the one used for Reconcile operations. This is because the Dispatcher calls the Search AL in Cycle mode, which causes it to work differently than you might expect.

The usual behavior of an AL is that an Iterator in the Feed section delivers data - one Entry at a time - into the Data Flow section where it is passed from component to component until the end of the AL is reached. At the end of each cycle, control returns to the Iterator which returns the next available Entry. If the iteration data set has been exhausted then the Iterator signals End-Of-Data and the AL terminates. When an Iterator-based AL is started then it normally runs until completion. If you launched it from another AssemblyLine then you can choose to either wait for completion, or allow it to run in the background while your calling AL continues its own processing. There is also a third option, which is to run the called AL in Cycle mode.

If the AssemblyLine is invoked in Cycle mode, then it only makes a single pass each time it is called and the calling process always waits for control to be returned. On the first call, the AL and its components are initialized and then the first processing cycle is performed, resulting in a Work Entry. Each subsequent call drives the AL for another cycle, returning either another Work Entry, or null once End-Of-Data is reached.

As long as the AL uses an Iterator in the Feed section then the built-in flow of the AssemblyLine will provide the caller - for example, the RMI Dispatcher - with a series of Entries as expected. However, the challenge arises when your AL must do more work to prepare each Entry. A common example is integration with a web service where a series of requests must be made to establish an authenticated session, followed by one or more requests to select relevant data - all before the first Entry can be returned.

One approach is to hide this complexity by scripting your own Iterator mode Connector. If you are comfortable using Javascript and have made friends with the AL Debugger, then rolling your own component could be your quickest option. Another route is to build an AL so that it will perform 'normal' iteration in Cycle mode, and here is an example of how to do this.

Create a new AL named 'CycleModeIteration' and then add the FormEntry Connector to the Feed section. This handy component supports only Iterator mode and works just like a FileConnector, except that instead of reading and parsing data from a file, it reads and parses the data entered in the Raw Data Text parameter of its Connection tab. We won't be using this functionality, instead substituting our own via Hook scripting.

Select the Hooks tab and then put this code into the Prolog - Before Init Hook to prepare for processing:

   // Initialize two flags
   eod = false;
   initializing = true;

Then drop the following script into the Override GetNext Hook:

   // If EOD flag set, end iteration
   if (eod) {
   result.setStatus(0); // EOD
   return;
   }

   // Otherwise return dummy data to continue
   // processing
   result.setStatus(1); // More data available

   work._dummyData = "* NOT DONE YET *";

The Override GetNext Hook code will replace both the read next functionality and the Input Map of the Iterator, so our logic here has to directly add attributes to the Work Entry.

Your AL should look like this.



The Iterator now will continue to return a Work Entry with a single dummy attribute until the eod variable has been set to true, at which time it will stop cycling.

Next we set up the Feed section so that initialization and data selection is only performed on the first cycle. Do this by adding an IF-Branch and naming it 'Initializing'. Click on the Script button to add a scripted Condition and use this snippet so that it uses the initializing flag set in the Iterator's Hook:

   return initializing;



In an actual Search AL this Branch would contain the components required to establish a connection to the external service or system. For our example you right-click the IF-Branch and add an Empty Script component that you call 'Perform initialization'. It only needs a line of script that sets the initializing flag to false. Now the IF: Initializing Branch will only be active on the first cycle.



Following this Script component you add another one, this time named 'Prepare data set'. This mirrors the typical selection of entries performed by an Iterator. If a real-world Search AL is working with web service then it will request a data set and parse the return. Once again our example AL is extremely simple, using a hardcoded Array of data and an index variable initialized to 0.



Here is the script shown in the screenshot above for easy cut-and-pasting:

   dataset = [
{name: "Joe Blogs",
title: "Manager",
phone: "555-1234"},
{name: "Jane Doe",
title: "Team Lead",
phone: "555-4321"},
{name: "John Anderson",
title: "Hacker",
phone: "555-1111"},
{name: "Bill Buttons",
title: "Peon",
phone: "555-2222"
    ];


   index = 0;

Now comes the logic that delivers data as long as its available, at which time it then signals End-Of-Data and stops iteration. We implement this through two Branch components: First an IF-Branch labeled 'More data' followed by an ELSE-Branch called 'EOD reached'. For the IF-Branch, click on the Script button and add this scripted Condition:

   return index < dataset.length;

These Branches will control how long iteration continues for our AssemblyLine.



Components placed under the IF: More data Branch set up the Work Entry with attribute values coming from the next available item of data. For the example AL you create a Script named 'Get next entry' that contains the following code:

   // get next entry and increment index
   entry = dataset[index++];

   // set up work entry
   work.removeAllAttributes();
   for (prop in entry) {
   work[prop] = entry[prop];

   }

The first line of script above retrieves the currently indexed Javascript object from the Array, incrementing the index variable immediately after. The next script line removes all attributes from the Work Entry, which at this point will only hold the dummy attribute set in the Override GetNext Hook of our Iterator. Finally a for-loop cycles through the names of all properties in the current object (i.e. 'name', 'title' and 'phone') and creates similarly named Attributes in the Work Entry with the value of each property. This Script component takes care of our Return Next Entry functionality. Now we need the logic to stop iteration after the last available data has been read.

Under the ELSE-Branch you add a Script named 'Signal EOD' containing this script:

   eod = true;
   system.skipEntry(); // skip back to Iterator

At which point this example AL is complete.



To test our new AssemblyLine we will create another AL, which we will call 'TestIteration'. Here you add an AssemblyLine Connector in Iterator mode to the Feed section. In the Connection tab of the Connector, enter the name of our example AL: 'CycleModeIteration'. The AssemblyLine Connector will run our example AL in Cycle mode, just like the RMI Dispatcher will. Also remember to add the wildcard (*) mapping rule to the Input Map so that all data is returned.

Finally, add the canned Script component named 'DumpWorkEntry' to the Data Flow section.



It is now time to run the test AssemblyLine and see our example CycleModeIteration AL works correctly. And if it does, then should see four (4) entries returned and displayed in the console output.



The component statistics displayed at the end of the AL run show us that everything worked as expected.

Note that if you don't get this result, then add the DumpWorkEntry directly under the IF: More data Branch (After 'Get next entry') and try running the example AL directly using the AssemblyLine Debugger. This will allow you to interactively walk component-by-component through its execution, including stepping through all script code. If you are not familiar with the AL Debugger, have a look at this video first.

Once the example AL is running correctly then you have a template from which to build your real-world Search/Reconcile AssemblyLine, simply substituting the hard-coded data set with actual information received from a connected service or system.

Monday, December 12, 2016

Importing a client certificate in order to access an https URL

When you go to make an HTTP request to a service that requires an SSL connection, you'll get an exception that your HTTP Client Connector cannot set up the secure link to the service. Unlike your browser, TDI (SDI) does not automatically import the client certificate. You have to do this yourself, and this post is about how to do this.

For this example, we'll try to access https://login.salesforce.com

If you enter this URL into an HTTP Client Connector and attempt a connection either by pressing the Connect and Next buttons in the Schema area of an Attribute Map, or by dropping the Connector into an AL and running it, then you will see this error:

ERROR - [HTTPClientConnector] CTGDIS810E handleException - cannot handle exception , callreply 
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
java.security.cert.CertPathValidatorException: The certificate issued by CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US is not trusted; internal cause is: 
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.j.a(j.java:10)
at com.ibm.jsse2.qc.a(qc.java:359)

This error is telling us that we are missing the required client certificate. An easy fix available directly in the CE is to press the Get Certificate button located to the right of the HTTP URL parameter in the Connector's Connection tab.



This handy button will request the certificate and then store it in the TDI Server's keystore (as specified in the solution.properties file by the property named javax.net.ssl.trustStore).



Once imported, the Java VM needs a restart in order for it to use this newly acquired cert. This means that you must recycle your TDI Server before a successful connection is possible.

This is all well and good, but if you're scripting your own components then it would be better if the missing certificate could be detected and then automatically imported. Fortunately, this is very simple to do with a single line of script in TDI.

msg = com.ibm.di.security.GetSSLCertificate.installCertificateFrom(baseurl, 443)

The baseurl argument is the 'https://...' address you want to access, and 443 is the default port for HTTPS. If the socket you are trying to reach includes a specific port number, then you can use this in the above call. If the request or the keystore update fail then the returned String is an error message describing this failure. If the client cert exists already in your keystore, then the message will tell you that the cert is already trusted. And if the requested certificate is successfully added to your keystore, the message will indicate this.

CTGDIS1957I Added certificate with subject: CN=login.salesforce.com, OU=Applications, O="Salesforce.com, Inc", L=San Francisco, ST=California, C=US.

Once TDI has been restarted then you can make secure connections to this service.

Monday, November 21, 2016

JSON and XML Tutorial - Part 6, Making a custom Form

As promised, this post will walk you through creating a custom Connection Tab Form for the Connector scripted in the previous post. Myself being in true form, I will begin with a promised explanation and then a bit of theory.

Internal name of a component

In my last post I promised to show you how to discover the internal name of a component. Otherwise, how can you know that the HTTP Client Connector is loaded using this script call:

http = system.getConnector("ibmdi.HTTPClient");

There are a couple of routes to this knowledge. For one, you can ask the TDI Server for a report on installed components via the object menu (right-click) on your running Server.



Another approach is to select this component in the Inherit from setting of an AL Connector, Function or Connector Loop. Note that if you select a Library component found under the Resources of your project, then the internal name is not displayed. You must choose a basic component in order to get the internal name shown.


Bit o' theory: Internal names of parameters

Just as with components, every parameter in TDI (SDI) has an internal name which can be used to retrieve or set this value programmatically. To discover the internal name of a parameter, either click on the label for that paramter, or on the pencil button to the right of it.


This opens up the Parameter dialog where you can configure that the value is to be dynamically populated from a Property, a snippet of Javascript or a TDI/SDI Substitution expression. Near the top of this dialog box is the internal name of the parameter.


As noted above, the internal name of the parameter can be used to read and set its value, as in this example snippet bursting with helpful comments:

// 'thisConnector' is available in any Connector Hook and
// refers to the AL Connector owning the Hook.
// The property 'connector' references the Connector Interface
// selected for it via the Inherit From selector.
URLused = thisConnector.connector.getParam("url");

// Furthermore, all AL components are also script variables.
myHTTPClient.connector.setParam("method""POST");

// If the name of a component is not suitable for a variable name,
// you can still access it by requesting a reference from the AL (i.e. task object).
ALconn = task.getConnector("My HTTP Connector");
ALconn.connector.setParam("url", "myrestservice.com:8080/aresource");

Another way to see the internal names of all the Parameters of a Connector is to attach it to a Connector Loop and then look at the Parameter Map tab.


This feature lets you map values to Connector settings as easy as mapping attributes. If Connector Loops are new to you, I suggest you read my other posts (or rather, 'love letters') on this handy feature. But I digress...

Back to creating a component Form

Returning to the topic for this post, we will now change the Connector scripted in the previous post so that the url used to make our REST request is entered as a parameter in the Connection tab of the Connector instead of being hardcoded.

The first thing we'll do is to close the editor for this Connector first. It is important that we do not open multiple editors for the same item, otherwise changes may be overwritten or lost. Once no editor is open for our Connector then we use the right-click option Open With... to edit it in the Forms Editor.


The first time you edit a component with the Forms Editor you are asked if you want to use the default form as a template. If you answer Yes for our example then you will get the default form for a Script Connector. Otherwise, the Forms Editor will start with an empty Form. For this example we'll answer Yes.


The Forms Editor screen is divided into three areas:

  1. The top pane where the name of the component and NLS resources (if any) are located.
  2. The navigator for the Form at the left.
  3. The details pane for any item selected in the navigator area.


In the top pane (1), the Form Title is entered. This value will be displayed at the top of the Connection tab. Unless you understand how translation resources are managed, leave the Translation File parameter unchanged or even empty. There are also two buttons located in this area: Test Form to launch the Form in a dialog for testing and for setting default values for parameters, and Delete Form.

The navigator area (2) provides an Init Script option for writing Javascript to be preloaded when this Form is opened in the CE (Connection tab), as well as an Events Script option for script that provides dynamic functionality to the Form. Neither of these will be discussed in this example, although the CouchDB Connector linked in a previous post uses this scripting option in the Form.

Immediately under these two script items are the Sections of the Form. Most standard components include a General and an Advanced section in the Connection tab. However, you can define as many sections as you need. Each section has a name (which is optional), the flag for whether this section is already expanded when the Connection tab is selected, and finally the list of Fields that are visible for this section. This list of Fields is ordered from top-down in the same way as they will appear in the Connection tab.

Below Sections is the list of Fields defined for the Form. Each field is identified by its internal name.

The details pane (3) provides the editor for whatever item is currently selected in the navigator. If the item selected is one of the two script options, then the details area will provide a script editor. If a Section or Field is selected then its editor appears in the details area.

For our example, we will start by creating the 'url' Field for our new Form by pressing the Add Field button at the bottom of the navigator pane and naming this Field 'url'.


In the Field Editor that appears to the right we can enter the Label for the field. This value appears onscreen to the left of the parameter setting in the Connection tab. We can also specify a ToolTip to be displayed on mouse hover-over. The Field type is chosen, which is String in this case. The Mode Selection option can be used to only display a field for certain modes by specifying these in a comma-separated list. We will leave this setting blank.

At the bottom of the editor are a series of tabs where one or more Buttons  to follow the parameter can be defined, drop-down values specified for Fields of a drop-down type, and where custom components and panel logic can be loaded.

If you want a bit more detail on most options available in the Forms editor, consult the TDI/SDI User Guide. We'll proceed now based on the description provided above. For example, this page for TDI 7.1.1 describes Forms Editor features, and is unchanged since TDI 7.1, as well as for later versions like SDI 7.2.

Now we select the General Section in the navigator and remove the $GLOBAL.script Field already visible here using the Remove Field button.  Then we add our new 'url' parameter by pressing the Add Field button at the bottom of the editor pane.


Now select the Advanced Section and remove the Fields already there. Finally, we add the $GLOBAL.script Field to this section.

Note that the $GLOBAL.* fields are predefined and provided access to standard features like file browsing and, in this case, script editing. Please note also that the $GLOBAL.debug Field represents the standard Detailed log checkbox found in most components. However, if you enable this for a Script component, then the entire script is written to the log. As a result, you will want to implement your own 'debug logging' Field if this functionality is desired.

Now if you press the Test Form button you should get a dialog with the Form you just defined.


Any values you enter for parameters here will become the default values for this component once you close and save the Form.

Now close and save the Forms Editor and use Open With... to open the Connector with the Connector Editor again. Now when you select the Connection tab then you will see your newly created Form.


Access field values in your script code

Our last step will be to edit the Connector script in order to have the 'url' variable set to the value entered for our REST URL parameter (internal name: url). We access the script using the Edit Script... button that we added to the Advanced section of the Form. We now need to decide how our component will behave regarding dynamic changes made to our parameter setting.

When the values entered in the Connection form of a component are actually used to refresh the settings of a Connector or Function differs from component to component, and in many cases, from parameter to parameter in a component. For example, the LDAP URL Parameter of the LDAP Connector is only refreshed when that component is initialized. However, the Search Filter parameter (internal name: ldapSearchFilter) is also refreshed before selectEntries() is performed for Iterator mode.

In our example here we will refresh the 'url' variable in the initialize() function only.

function initialize()
{
// Initialize the HTTP Client CI loaded above
http.initialize(null);
// Reset the data set
nodeList = null;
// Refresh internal settings based on Form values
url = connector.getParam("url");

}

The predefined 'connector' variable provides access to our Connector Interface where all parameter values entered in the Form are kept.

Once we have made these changes to the script then our Connector is configurable via its Connection tab. Now we have a few options for sharing our new component.

Of course, the .connector file itself can be shared and dragged-dropped into new Projects. You can also use copy-paste directly in the CE to do this.

If we want to use our new REST Connector in multiple ALs in the same Project, we do so easily as long as our Connector is stored under Resources > Connectors.

If we want to leverage the component in another Project, then one way would be to define a Reference in this Project to the Config xml where the Connector is defined. Then it will be available for use in this other Project.

Another route would be to right-click on our Connector and choose the Publish... option. This allows us to export a 'package' to the packages folder of the installation directory. Components found in packages are available for use by any TDI Server started from this installation directory.

And until I get feedback warranting an update (as you may have seen me do for other posts) that'll be all for now.

Thursday, November 17, 2016

JSON and XML Tutorial - Part 5, Scripting a Connector

Continuing from my last rambling post, we'll script an Iterator Connector that will return the same parsed entries as our Conditional Loop did. I'll begin with a little theory, which you will find duplicated other places like the How-To page of www.tdi-users.org and this venerable video tutorial on the subject.

To script a component in TDI (SDI) you start with the basic Script version of that component. So in order to script a Connector, you add a Script Connector.

Typing 'script' in the Search box filter the selection list. Since we'll be implementing only Iterator mode, we may as well choose this mode in the wizard.
Our new Connector looks pretty much like any other Connector at this point. All the features provided by the TDI AL framework are visible in the various tabs: Input/Output Map, Hooks, Delta, Connection Errors and Pool. You can also choose the Mode and State for your Connector, as well as when the component will initialize. Only the Connection tab is specific to the technology our Connector is interfacing with. That is why this aspect is called the Connector Interface, or CI for short.

The Connection tab of our Script Connector does not hold the usual set of parameter (yet), but instead gives us access to the interface logic of the component via the Edit Script... button.
Press this button to open the script editor.

Once in the editor, we can see that our new Connector has a preloaded example script. At the top are the two functions necessary for Iterator mode: selectEntries() and getNextEntry().
This is how a scripted component works: if the required functions are present in the script, then that operation or mode is available. Note however that for a scripted Connector, the Mode drop-down will always contain all modes - including those not supported by its script.

Iterator mode selects its data set with selectEntries() and then iterates through it using getNextEntry() to return one Entry at a time and to signal End-of-data when the data set is exhausted. Here is the standard template for these function with a little comment added for clarity:

var counter = 0;

function selectEntries()
{
// initialize the 'data set'
counter = 0;
}

function getNextEntry ()
{
// if the data set is exhausted, setStatus to 0
if (counter > 100) {
result.setStatus (0);
        // This next line is no longer required after TDI 7.1
result.setMessage ("End of input");
return;
}

// otherwise, populate the conn Entry via the 'entry' variable
entry.setAttribute ("counter", counter);
counter++;
// setStatus is 1 by default, which means 'more data available'
}

A scripted Iterator is easy to test in the Schema section of the Connector's Input or Output Map. Pressing the Connect button calls the initialize() function (if one exists), followed by querySchema()and then selectEntries()

The reason information shows up under Schema when you press the Connect button is because there is a template script for querySchema(). Here is a copy of it with clarifying comments:

function querySchema ()
{
// the 'list' variable is a Java Vector for returning 
// one or more schema items. Each one is added to the
// Vector as an Entry with two attributes:
//    name - the name of the attribute
//    syntax - a string value with guidance to the developer
var e = new com.ibm.di.entry.Entry();
e.addAttributeValue("name","counter");
e.addAttributeValue("syntax","Number");
list.add(e);
// once again, setStatus to 1 means info available
result.setStatus (1);
}

If there is no querySchema() function then the Schema grid is first populated when you start stepping through the data set. You will also note that there is no way to signal if a schema item is Required or not. This is schema item option only available if you code your component using Java. However, I have been creative with what I put in the Syntax value and have not missed this feature :)

Once connection and selection are complete then the Connect button is grayed and the Next button becomes available. Each time you press this button it results in a call to getNextEntry() and either returns data or signals EOD.

And now for something cool: Simply open the script editor again and change the If-statement in getNextEntry() to signal EOD after 10 entries instead of 100. Then return to the Input Map tab, press the Close button in the Schema area and then Connect again. This causes the script to be reprocessed and lets you test any changes you made to the code immediately.

Ok, enough theory for now. Let's strip out all the template functions except for selectEntries() and getNextEntry() which we will be rewriting. In addition, we will add the initialize() function as well as some global variables. Here's what the start of the script looks like:

// MyRestConnector
vrs = 0.9; // 20161117 created this component
// main.logmsg() provides info in the Console view in the CE
// You could also write to stdout: java.lang.System.out.println(...)
main.logmsg("MyRestConnector v." + vrs);
// All variables defined outside of function bodies
// are global for the entire script.
// 
// Here is the HTTP Client Connector we'll use to make requests
  // Discovering what the 'internal name' of a Connector is requires a little legwork,
  // and I'll get back to this in a later post.
var http = system.getConnector("ibmdi.HTTPClient");
// Hardcoding the URL for now. Later we'll create a Connection Form
var url = "http://www.ideal-status.nl/static/consumer_notification_advice.json";
// Variables for our NodeList and index
var nodeList = null;
var nodeIndex = 0;
function initialize()
{
// Initialize the HTTP Client CI loaded above
http.initialize(null);
// Reset the data set
nodeList = null;
nodeIndex = 0;
}

We can test the validity of this script by bringing up the Input Map tab again and pressing Connect. If you did not empty out the selectEntries() function then you'll get an error when you try this.

Now we come to the meat of our Connector: selectEntries(). This function does most of the work of component, setting up and making the REST request, parsing the JSON payload received and then preparing the data set to be returned by subsequent calls to getNextEntry(). These three steps are called out in the comments included in the code below:

function selectEntries()
{
// PREPARE REQUEST
// Set up an Entry for passing Output attributes that
// control parameter settings
var requestEntry = system.newEntry();
requestEntry["http.url"] = url;
requestEntry["http.Accept"] = "application/json";
// The HTTP method is GET by default, but just in case...
requestEntry["http.Method"] = "GET";
// Make the request and store the response in a local Entry
// Note that the queryReply() method is what CallReply mode uses
var responseEntry = http.queryReply(requestEntry);
// Check the responseCode returned by the service
var responseCode = responseEntry.getString("http.responseCode");
// If this is not a 2xx return, then throw an exception
if (!responseCode.startsWith("2")) {
var responseMsg = responseEntry.getString("http.responseMsg");
throw "Error making request: " + responseCode 
+ " - " + responseMsg;
}
// PARSE JSON RECEIVED
// Wrapping this in try-catch for better error handling
try {
var json = responseEntry.getString("http.bodyAsString");
// fromJSON() is a static method, so we can call it via the class
var hEntry = com.ibm.di.entry.Entry.fromJSON(json);
} catch (ex) {
throw "Error parsing JSON return: " + ex 
+ "\nJSON: " + json;
}
// PREPARE DATA SET
nodeList = hEntry.getChildNodes();
nodeIndex = 0;

}

Once again we can test that our script so far is correct (at least syntactically) by using the Close and Connect buttons. We are still not seeing any data returned yet because that's the responsibility of getNextEntry() which we code to look like this:

function getNextEntry ()
{
// If the data set is exhausted, setStatus to 0
if (nodeIndex >= nodeList.getLength()) {
result.setStatus (0);
return;
}
// Otherwise, populate the conn Entry via the 'entry' variable
// Borrowing this code from the AL made in the previous blog
// First grab the next node and increment the index
item = nodeList[nodeIndex];
nodeIndex++;
// The 'entry' variable is the conn Entry for this Connector
entry.id = item.getName();
var children = item.getChildNodes(); // get next level
for (var child in children) {
// set conn attribute using child att name and value
entry[child.getName()] = child.getValue();
}
// For completeness sake, I like to set the return status
result.setStatus(1);
}

Once this is in place then we can start testing it using the Close, Connect and Next buttons.

And violá! we have a fully functional Iterator. We can use it in the Feed section of an AL or in a Connector Loop, we can code any of the Iterator mode Hooks, and we can enable Delta if we want to detect changes between each call to the service.

And what's more, if we drop our Connector into an AL then we can step through the script in the AL Debugger.
At this point, we can either double click on ConnectorScript and then set breakpoints in the code, or just start stepping into using the Step into button. Note that pressing Step into while in the script takes debug processing into for- and while-loops, as well as if-statements and try-catch blocks. Step over will do just that: stepping over these code blocks.
As always, the Debugger lets you examine and change variables directly by executing Javascript in the Javascript Commandline located just under the script window.

Each time you press the ENTER key then the script snippet is passed to the running server and run in the context of where the AL processing is at that point. This lets you inspect the state of your component by logging the results of all statements executed, as shown in the screenshot above. Furthermore, you can modify the values of variables and attributes, as well as the contents of Entry objects. You can also define new variables and instantiate new objects, calling scripted functions and invoking Java methods as desired.

I'll keep my love ballad to the AL Debugger short and conclude with the recommendation that you learn to love this time-saving tool.

And I'll wrap up another extended blog here. But I will return to show you how to set up the custom Connection tab for your scripted Connector, so stay tuned!

Wednesday, November 16, 2016

JSON and XML Tutorial - Part 4, JSON example


Practice examples are helpful when mastering new techniques. In this example we'll make a REST request and then handle the return JSON payload. We'll be using the service available at the following URL:

http://www.ideal-status.nl/static/consumer_notification_advice.json

If you dial up this URL you'll get JSON that looks like this:

{"0021":{"issuer_id":"0021","issuer_bic":"RABONL2U","issuer_name":"Rabobank","datetime":"2016-11-21 11:22:49","status":"OK","message":""},"0721":{"issuer_id":"0721","issuer_bic":"INGBNL2A","issuer_name":"ING Bank","datetime":"2016-11-21 11:23:05","status":"OK","message":""},"0031":{"issuer_id":"0031","issuer_bic":"ABNANL2A","issuer_name":"ABN Amro Bank","datetime":"2016-11-21 11:22:51","status":"OK","message":""},"0751":{"issuer_id":"0751","issuer_bic":"SNSBNL2A","issuer_name":"SNS Bank","datetime":"2016-11-21 11:23:07","status":"OK","message":""},"0811":{"issuer_id":"0811","issuer_bic":"BUNQNL2A","issuer_name":"Bunq","datetime":"2016-11-21 11:23:16","status":"OK","message":""},"0771":{"issuer_id":"0771","issuer_bic":"RBRBNL21","issuer_name":"RegioBank","datetime":"2016-11-21 11:23:12","status":"OK","message":""},"0761":{"issuer_id":"0761","issuer_bic":"ASNBNL21","issuer_name":"ASN Bank","datetime":"2016-11-21 11:23:10","status":"OK","message":""},"0091":{"issuer_id":"0091","issuer_bic":"FRBKNL2L","issuer_name":"Friesland Bank","datetime":"2016-11-21 11:22:58","status":"OK","message":""},"0511":{"issuer_id":"0511","issuer_bic":"TRIONL2U","issuer_name":"Triodos Bank","datetime":"2016-11-21 11:23:03","status":"OK","message":""},"0161":{"issuer_id":"0161","issuer_bic":"FVLBNL22","issuer_name":"Van Lanschot Bankiers","datetime":"2016-11-21 11:23:00","status":"OK","message":""},"0801":{"issuer_id":"0801","issuer_bic":"KNABNL2H","issuer_name":"Knab","datetime":"2016-11-21 11:23:14","status":"OK","message":""}}

In JSON terms, this represents a single object with a number of properties, each one with a four digit numeric name (e.g. '0021', '0721', ...). The values of these properties are in turn objects with their own properties like 'issuer_id' and 'issuer_name'. We will look at how to parse this notation into objects that we can leverage in an AssemblyLine.

To do this, create a new AL named jsonxmlTut4 and add the HTTP Client Connector in CallReply mode.

NOTE: CallReply is the most flexible mode for this component, allowing you to dynamically set any and all parameters via specially named attributes in the Output Map. For example, to specify a URL that overrides the parameter setting in the Connection tab, simply map out an attribute named 'http.url'. All parameters can be mapped in this way by prepending the Internal name of the parameter (which you get by pressing the pencil button
 next to a parameter) with the text 'http.'. And now back to the meat of this post...

Paste the URL above into the URL parameter of the Connector and use the 'map everything' wildcard (*) rule in the Input Map. Finally, add the DumpWorkEntry script in order to display what the HTTP Client Connector returns.


When you run the AL then you'll see output similar to this:



The prefix 'http.' is used by the HTTP Parser - which is an important part of the Connector - both for defining the request being sent, as well as for parsing response information. In the Attributes handled by the Parser you can find the standard HTTP headers like Content-Type, Content-Length, responseCode and responseMsg. You will also find special headers like Authorization and cookies (e.g. Set-Cookie). The service called here is also returning the ETag for the queried resource, which is like a fingerprint and/or version number for the resource.

In addition, there are three 'http.' attributes that represent the body of the response:
  • http.body which is the body of the HTTP response, presented according to the Content-Type setting. The above screenshot shows that this value is being returned as a byte array, although for some content types this attribute will have a String value.
  • http.bodyAsBytes always returns the HTTP body as a byte array.
  • http.bodyAsString always returns the HTTP body as a String.
Since we know the payload is a JSON string, we should be able to attach the JSON Parser to our HTTP Client Connector and get back the encoded data.




Now when we hit Run in console the AL, we get both the HTTP content plus a number of additional attributes. And note that even the HTTP content looks different in the log output now.


When reading, Parsers turn byte streams into Entries. Our wildcard (*) Input Map instructs that all attributes created in the conn Entry (the Connector's local cache) are copied into the work Entry. Typically when you see an Attribute displayed in dumpEntry() output it looks something like this:

        "attribute name": "attribute value"

If there are multiple values, then they are shown on separate lines and enclosed with square brackets. In the screenshot above you're seeing quoted attribute names at multiple layers, like "issuer_id" which appears in each of the parsed entities. Even the HTTP attributes appear without the 'http.' prefix and instead look to be contained in an object named 'http'.

This means that the JSON contained hierarchical data, so the Parser parses this into a hierarchical Entry object. Or, as in this case, it takes the flat work Entry holding the HTTP attributes returned by the Input Map of our Connector and adds the tree of parsed data.

Unfortunately, this gives us an Entry holding two types of data which can cause us grief. For example, if we wanted to convert the data to XML format by calling work.toXML() then we'll get an exception due to the mix of hierarchical and non-hierarchical data. For this reason, I never tie a parser to the HTTP Client Connector. Instead, I grab the value of the http.bodyAsString attribute and do the parsing myself:

  json = work.getString("http.bodyAsString");
  hEntry = work.fromJSON(json); // convert JSON to hierarchical Entry

If we replace the contents of the DumpWorkEntry Script with the above snippet then we get a cleaner hierarchical Entry (aka a true hEntry). An hEntry offers a number of handy features, including DOMInterface methods for traversing, searching and editing the tree, like .search(<xpath expressions>).getChildNodes().getElementsByTagName(<tag name>) and .addChildNode(). All these methods are detailed in the TDI JavaDocs for the Entry class. Just look for functions that return Node or NodeList, or that accept these as arguments.

So if we had a payload containing a series of nodes named 'Person' at whatever level in the hierarchy, we could retrieve the list of them like this:

  personList = work.getElementsByTagName("Person");
  for (node in personList) {
      task.logmsg(node.toString());
  }

The NodeList has only two methods: .getLength() and .item(<index>). But as you can see in the above example, TDI's Javascript engine lets you easily for-loop through the Nodes of a NodeList. You can also use square brackets to reference list members: personList[0].

And remember that this works for any hierarchical Entry, like that parsed from JSON or XML, as well as created by custom Parsers and script.

The Nodes returned by methods like .item() are both DOM Nodes and TDI Attributes, so it's wise to study the Javadocs for both. Since there are multiple types of Nodes - like those that represent XML node attributes and those that carry node values - it sometimes takes me a little trial and error to get my script right. And faithful readers will no doubt realize that I do this kind of data exploration either in the AL Debugger or in the Javascript tab at the bottom of the TDI CE.

However, sometimes instead of working with an hEntry, I just stick to Javascript. This means that I convert the JSON payload to a Javascript object and unwrap the data directly.

  json = work.getString("http.bodyAsString");
  jsobj = fromJson(json);
  for (prop in jsobj) { // return the name of each top-level property
      task.logmsg(prop + ": " + jsobj[prop]); // log prop name and value
  }

If I put this code into the DumpWorkEntry Script then the output looks like this:



So we see each of the entities in the JSON as a property with what looks like an identifier followed by '[object object]', which is the TDI JS engine's way of telling us that each value logged was another JS object. If we instead want to display the contents of each item we could either convert each back to JSON representation - toJson(jsobj[prop]) - or we could continue to explore the next level in the data tree. In fact, our logic could check if the value can be logged directly, or should be unwrapped:

  json = work.getString("http.bodyAsString");
  jsobj = fromJson(json);
  for (prop in jsobj) { // return the name of each top-level property
  subobj = jsobj[prop];
  if (typeof subobj == "object") { // explore next level
  task.logmsg(prop + ": ");
  for (subprop in subobj) {
  task.logmsg("  " + subprop + ": " 
+ subobj[subprop]);
  }
       } else {
  task.logmsg(prop + ": " + jsobj[prop]);
  }
  }

This time the output of our AL will look like this:


This is all well and good, we have a handle on the data now and could continue to do all our iterating in script. 

Sometimes however it's easier if we can loop through a data set using AL components. We can do this by scripting an Iterator to use in the built-in loop provided by the Feed section of the AL, or in a Connector Loop component. Or we could use a Conditional Loop and simply step through the entities with script. 

In either case we will want to convert the data into an entry object so that we can use it in Attribute Maps and other AL component features. And I usually leverage a hierarchical Entry when iterating data using components.

Let's start by using a Conditional Loop to do this. First we need to change the code in the DumpWorkEntry Script to this:

  json = work.getString("http.bodyAsString");
  hEntry = work.fromJSON(json);
  items = hEntry.getChildNodes(); // get top level NodeList
  index = 0; // initialize index

Now we can add the Conditional Loop calling it 'more items'. Press the Script button to add this scripted condition:


  return index < items.getLength();


Now we just have to remember to increment our index so we don't loop forever.
Finally, under the Loop we add another Empty Script with the following code:


  item = items[index]; // get next Node
  index++; // increment index

  work.removeAllAttributes(); // empty out work

  // set work attributes from Node values
  work.id = item.getName();
  children = item.getChildNodes(); // get next level
  for (child in children) {
  // set work attribute using child att name and value
       work[child.getName()] = child.getValue();
  }

  task.logmsg(work); // display this Entry

Our AL looks like this now:
And when we run the AL, we see the following output:


Now any components added under the Loop after our Set up work Entry script will have access to the work Entry holding data for a single entity from the parsed JSON payload.

And this post has gotten pretty long already. Look for follow-up post showing how to script a Connector to Iterate through the entities we just handled here.